Method for placing box on outer container truck of gantry crane

Abstract

The invention discloses a method for placing a container on an outer container truck of a gantry crane, which relates to the field of container lifting and comprises the following steps of: 1, mounting a plurality of image acquisition components on a target lifting appliance, adjusting an acquisition area to be a lifting appliance and container working area, and butting a PLC (Programmable Logic Controller) bus; 2, based on the current acquisition area, arranging a plurality of shooting points and corresponding acquisition angle parameters, shooting the calibration board at different positions, obtaining a plurality of groups of images, calculating internal parameters and external parameters of a plurality of image acquisition components, and generating a calibration file after processing; visual information can be obtained from multiple angles, comprehensive cognition on an operation area is improved, information blind areas are remarkably reduced, the environment adaptability is enhanced, higher-precision container and button lock positioning is achieved by combining internal parameters, external parameters and PLC information of an image acquisition component, depth estimation and semantic segmentation technologies are used, and the positioning accuracy is improved. The recognition and positioning precision of the target object is improved, so that operation errors are reduced, and high-level automation is realized.

Description

Technical Field

[0001] The invention relates to the technical field of container lifting, and in particular to a method for placing containers on a container truck outside a gantry crane. Background Art

[0002] In recent years, with the rapid development of global trade, the container throughput of port terminals has continued to increase. Improving terminal loading and unloading efficiency and automation level has become an important trend in the development of the industry. Gantry cranes are important equipment for terminal loading and unloading. They undertake the task of transferring containers from ships to container trucks. In order to achieve real-time guidance and automatic container release for truck drivers outside the terminal, modern gantry crane systems integrate a variety of advanced technologies. Traditional container lifting and lowering operations often rely on manual operation, with limited visual information extraction of the environment and lack of comprehensive information from multiple angles and viewpoints, resulting in reduced accuracy and increased risk of missed or wrong lifting. The positioning of the spreader and container often relies on simple sensors, which have limited accuracy and response speed, and fail to fully utilize visual and depth information, resulting in inaccurate positioning and insufficient automation. In complex environments, operators have difficulty in operation and are prone to operator fatigue; When a fault occurs, there is often a lack of effective real-time monitoring and fault diagnosis mechanisms, resulting in inefficient fault recovery, increased operational risks and downtime. In many vision-based systems, image and depth information are insufficiently utilized, and monocular depth data cannot be effectively combined with other information, limiting the understanding and utilization of the spatial environment. Summary of the invention

[0003] 1. Technical issues to be resolved In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for placing containers on a container truck outside a gantry crane, which can effectively solve the problems of the prior art.

[0004] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: The present invention discloses a method for placing a container on a container truck outside a gantry crane, comprising the following steps: Step 1: Install several image acquisition components on the target spreader, adjust the acquisition area to the spreader and container working area, and connect to the PLC bus; Step 2: Based on the current acquisition area, arrange several shooting points and their corresponding acquisition angle parameters, shoot the calibration plate at different positions, obtain multiple sets of images, calculate the internal and external parameters of several image acquisition components, and generate a calibration file after processing; Step 3: Obtain the coding information of the PLC bus, edit the data acquisition plan based on the calibration file, and record the container release process data of the external container truck in real time; Step 4: Analyze the positions and postures of several image acquisition components based on the box placement process data, and restore the positions and postures of the image acquisition components in absolute scale according to the applied data acquisition scheme, and perform restoration state analysis to obtain a position and posture restoration data set; Step 5: Use the position and posture recovery dataset as training samples, build a monocular depth estimation model through a deep learning algorithm, reconstruct the loss function to train the monocular depth estimation model, output depth information, introduce a semantic segmentation module to detect container knobs and containers, and output the absolute scale data of the corresponding image acquisition components; Step 6: Generate three-dimensional coordinates through depth information and store them as point cloud data structures. Combined with target detection, locate the corner points of the container and the position coordinates of the button lock. Step 7: Based on the positioning data of the container corner points and the button lock position coordinates, perform actual deviation analysis, output correction instructions and feedback to the driver control terminal to execute the container release operation.

[0005] Furthermore, the intrinsic parameter attributes of the two image acquisition components include: focal length, optical center and distortion coefficient, and the extrinsic parameter attributes are: relative positions of several image acquisition components.

[0006] Furthermore, the data attributes of the container release process of the foreign container truck in step 3 include: container position and posture data, spreader operating force and torque setting data, on-site temperature, humidity and wind speed environmental data, time records of each operation link, path records of the spreader moving during the operation process and any event records that trigger safety alarms.

[0007] Furthermore, during the recovery state analysis in step 4, if the camera posture recovery process fails, fault diagnosis and tracking are performed, and the specific process includes: Record error codes, log information and status data, capture and mark error data, algorithm output, running time, image acquisition component position and posture, spreader position and posture and environmental data during recovery; Verify whether the viewing angle obtained by the image acquisition component matches the position information provided by the PLC bus; Verify that the parameters used in the data collection plan match those of the original data collection plan; Statistically analyze the collected error data and match it with historical fault data, count the frequency and time of similar faults, obtain historical related adjustment plans as a reference, identify the current fault mode and related characteristics, and output the final adjustment plan for secondary recovery application.

[0008] Furthermore, the monocular depth estimation model in step 5 is based on the output depth information, including: the frame, the floor and the container, and converts the evaluated depth information into absolute scale data in the actual scene.

[0009] Furthermore, the construction process of the point cloud data in step 6 is as follows: Perform denoising based on the depth information, generate a depth map and perform normalization until the depth value meets the preset range; The internal parameters of the image acquisition component are used to obtain the mapping from the image coordinate space to the acquisition coordinate space. For each pixel and its corresponding depth value, its three-dimensional coordinates in the acquisition coordinate system are calculated.

[0010] Furthermore, the three-dimensional coordinates in the acquisition coordinate system are calculated by converting the two-dimensional pixel coordinates, the three-dimensional coordinates are expressed as (X, Y, Z), the two-dimensional pixel coordinates are expressed as (u, v), and the calculation formula is: ; ; ; In the formula, Respectively represent the position data of the optical center on the x and y axes, and They represent the corresponding x- and y-axis focal lengths respectively, and d represents the depth value.

[0011] Furthermore, the process of actual deviation analysis in step 7 is: obtaining the actual coordinates of the container corner points and knob lock positions, comparing the target positioning coordinates with the actually detected coordinates, calculating the deviations of the container corner points and knob lock positions, analyzing the deviation calculation results, determining the deviation direction and the deviation distance of each axis, and performing an impact analysis. If the deviation is large, the position that has the greatest impact on the spreader and container placement is adjusted first. Based on the calculated deviation, the format of the correction instruction is determined, and the detected viewing angle diagram, deviation data and movement direction are displayed to the driver in a visual interface. Real-time feedback based on the current operating status, environmental information and action suggestions is provided, and the driver chooses to apply manually or adjust automatically by the program.

[0012] (III) Beneficial effects Compared with the known prior art, the technical solution provided by the present invention has the following beneficial effects: By carrying multiple image acquisition components, it is possible to obtain visual information from multiple angles, improve the overall cognition of the operating area, significantly reduce information blind spots, and enhance environmental adaptability. Combining the internal and external parameters of the image acquisition components and PLC information, it can achieve higher-precision container and button lock positioning. The use of depth estimation and semantic segmentation technology improves the recognition and positioning accuracy of target objects, thereby reducing errors in operation, achieving a high level of automation, and reducing the necessity of human intervention. The automation and intelligence of the operation process make the operation more efficient and reduce the dependence on manual operation skills. Through the application of the monocular depth estimation model, the depth information in the environment can be converted into real space coordinates, thereby achieving more accurate spatial perception. This method provides the necessary depth information support for container operations in complex scenarios. The introduction of the semantic segmentation module improves the recognition accuracy of containers and button locks, enabling the system to better understand and process multiple objects in complex environments. Through real-time data acquisition and processing, it provides rapid feedback, reduces the delay between decision-making and execution, and greatly improves the safety and efficiency of operations. By designing a fault tracking and real-time adjustment mechanism, it is possible to quickly diagnose and take corresponding measures when problems occur. This real-time feedback improves the stability and reliability of the system, reduces operation downtime, integrates multi-angle data and in-depth information, can better adapt to complex operating environments, and avoids the problem that traditional methods are not sensitive to environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0015] The present invention will be further described below in conjunction with the embodiments.

[0016] Example 1 A method for placing a container on a container truck outside a gantry crane in this embodiment is as follows: Figure 1 As shown, the following steps are included: Step 1: Install several image acquisition components on the target spreader, adjust the acquisition area to the spreader and container working area, and connect to the PLC bus; Step 2: Based on the current acquisition area, arrange several shooting points and their corresponding acquisition angle parameters, shoot the calibration plate at different positions, obtain multiple sets of images, calculate the internal and external parameters of several image acquisition components, and generate a calibration file after processing; the internal parameter attributes of the image acquisition component include: focal length, optical center and distortion coefficient, and the external parameter attributes are: the relative positions of several image acquisition components; Step 3: Obtain the coding information of the PLC bus, edit the data acquisition plan based on the calibration file, and record the data of the container release process of the external container truck in real time; the data attributes of the container release process of the external container truck include: container position and posture data, spreader operating force and torque setting data, on-site temperature, humidity and wind speed environmental data, time records of each operation link, the path record of the spreader moving during the operation process and any event record that triggers a safety alarm; Step 4: Analyze the positions and postures of several image acquisition components based on the box placement process data, and restore the positions and postures of the image acquisition components in absolute scale according to the applied data acquisition scheme, and perform restoration state analysis to obtain a position and posture restoration data set; Step 5: Using the position and posture recovery dataset as training samples, a monocular depth estimation model is constructed through a deep learning algorithm, and the loss function is reconstructed to train the monocular depth estimation model, output the depth information, introduce the semantic segmentation module to detect the container button lock and container, and output the absolute scale data of the corresponding image acquisition components; the monocular depth estimation model is based on the output depth information, including: frame, floor and container, and converts the evaluated depth information into absolute scale data in the actual scene; Step 6: Generate three-dimensional coordinates through depth information and store them as point cloud data structures. Combined with target detection, locate the corner points of the container and the position coordinates of the button lock. The construction process of point cloud data is as follows: Perform denoising based on the depth information, generate a depth map and perform normalization until the depth value meets the preset range; Use the intrinsic parameters of the image acquisition component to obtain a mapping from the image coordinate space to the acquisition coordinate space, and for each pixel and its corresponding depth value, calculate its three-dimensional coordinates in the acquisition coordinate system; The three-dimensional coordinates in the acquisition coordinate system are calculated by converting the two-dimensional pixel coordinates. The three-dimensional coordinates are expressed as (X, Y, Z), and the two-dimensional pixel coordinates are expressed as (u, v). The calculation formula is: ; ; ; In the formula, Respectively represent the position data of the optical center on the x and y axes, and Represent the corresponding x and y axis focal lengths respectively, d represents the depth value, and a filtering algorithm is used to remove noise points to improve the quality of the point cloud. Normal information is estimated for each point in the point cloud for use in the subsequent 3D reconstruction and surface reconstruction process. Monocular depth information is used to successfully generate high-quality point clouds, laying a good foundation for subsequent 3D reconstruction and analysis. Step 7: Based on the positioning data of the container corner points and the button lock position coordinates, perform actual deviation analysis, output correction instructions and feedback to the driver control terminal to execute the container release operation.

[0017] Compared with existing technologies, by combining advanced image acquisition technology, real-time data recording, deep learning and point cloud processing, it not only improves the accuracy and real-time performance of container unloading operations, but also enhances the safety and intelligence level of the system, making this method show greater application potential and value in modern logistics and lifting operations.

[0018] Example 2 In other aspects, this embodiment also provides a recovery state analysis process. During the recovery state analysis process, if the camera posture recovery process fails, fault diagnosis and tracking are performed. The specific process includes: Record error codes, log information and status data, capture and mark error data, algorithm output, running time, image acquisition component position and posture, spreader position and posture and environmental data during the recovery process; help to fully understand the background and cause of the fault, thereby improving the accuracy of fault diagnosis, providing a rich foundation for subsequent data analysis, and enabling a deeper understanding of the root cause of the fault; Verify whether the viewing angle obtained by the image acquisition component matches the position information provided by the PLC bus; Verify whether the viewing angle of the image acquisition component matches the position information provided by the PLC bus, which can timely identify errors caused by inconsistent data and enhance the reliability of the system; Verify whether the parameters used in the data acquisition plan match the original data acquisition plan; by confirming whether the parameters used in the data acquisition plan are consistent with the original plan, potential failures caused by parameter changes can be avoided and the stability of the system can be ensured; Statistically analyze the collected error data, match historical fault data, count the frequency and time of similar faults, obtain historical related adjustment plans as reference, identify the current fault mode and related features, and output the final adjustment plan for secondary recovery application. By statistically analyzing historical fault data, the current fault mode can be compared with the historical mode, so that past experience can be used to guide the current fault handling method and improve the recovery efficiency. The final adjustment plan is output for secondary recovery application, which promotes the continuous optimization of the system in continuous fault handling, enhances the adaptability and reliability of the system, and timely identifies and handles faults, which can reduce downtime caused by faults and improve overall operating efficiency; Through comprehensive fault diagnosis and corresponding adjustment plans, the safety hazards caused by equipment failure can be reduced and the safety of operators can be guaranteed.

[0019] Example 3 In this embodiment, a process of actual deviation analysis is provided, specifically: obtaining the actual coordinates of the container corner points and knob lock positions, comparing the target positioning coordinates with the actually detected coordinates, calculating the deviations of the container corner points and knob lock positions, analyzing the deviation calculation results, determining the deviation direction and the deviation distance of each axis, and performing an impact analysis. If the deviation is large, the position that has the greatest impact on the spreader and container placement is adjusted first, and the format of the correction instruction is determined based on the calculated deviation. The detected viewing angle diagram, deviation data and moving direction are displayed to the driver in a visual interface, and real-time feedback based on the current operating status, environmental information and action suggestions is provided. The driver chooses to apply manually or the program automatically adjusts, and the detected knob lock and container corner point deviations can be effectively used for secondary guidance and correction instructions to ensure that the spreader posture and the position of the trolley are accurately adjusted during the overall operation process, thereby ensuring the safety and efficient container placement operation of the external container truck.

[0020] In summary, the present invention can achieve high-precision recovery of the position and posture of containers and spreaders through accurate calibration of multiple image acquisition components and their internal and external parameters. This method greatly improves the accuracy of position and posture estimation through deep learning and image processing technology, and has better robustness than traditional methods. With the help of the PLC system, various data in the process of external container truck release are recorded in real time, including environmental data and operating parameters, which can effectively monitor and optimize the workflow, improve the safety and controllability of operations, and provide data support for subsequent operations; The introduction of a monocular depth estimation model and a semantic segmentation algorithm for target detection provides a more flexible and intelligent solution compared to traditional depth measurement methods. Through the reconstruction of depth information, not only can a three-dimensional model be obtained, but also the geometric features of the container can be accurately estimated, thereby improving the overall operation efficiency. Through the denoising of depth information and the calculation of three-dimensional coordinates, the point cloud data generated by this method has higher accuracy. Compared with existing technologies, the point cloud data has higher quality and richer information, and can support more complex analysis requirements. By analyzing the actual deviations of the container corners and button lock positions, corrective instructions can be output instantly, improving the adaptability and intelligence level of the operation, and more effectively reducing the risk of human errors and equipment damage.

[0021] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for placing containers on a container truck outside a gantry crane, characterized in that: The following steps are involved: Step 1: Install several image acquisition components on the target spreader, adjust the acquisition area to the spreader and container working area, and connect to the PLC bus; Step 2: Based on the current acquisition area, arrange several shooting points and their corresponding acquisition angle parameters, shoot the calibration plate at different positions, obtain multiple sets of images, calculate the internal and external parameters of several image acquisition components, and generate a calibration file after processing; Step 3: Obtain the coding information of the PLC bus, edit the data acquisition plan based on the calibration file, and record the container release process data of the external container truck in real time; Step 4: Analyze the positions and postures of several image acquisition components based on the box placement process data, and restore the positions and postures of the image acquisition components in absolute scale according to the applied data acquisition scheme, and perform restoration state analysis to obtain a position and posture restoration data set; Step 5: Use the position and posture recovery dataset as training samples, build a monocular depth estimation model through a deep learning algorithm, reconstruct the loss function to train the monocular depth estimation model, output depth information, introduce a semantic segmentation module to detect container knobs and containers, and output the absolute scale data of the corresponding image acquisition components; Step 6: Generate three-dimensional coordinates through depth information and store them as point cloud data structures. Combined with target detection, locate the corner points of the container and the position coordinates of the button lock. Step 7: Based on the positioning data of the container corner points and the button lock position coordinates, perform actual deviation analysis, output correction instructions and feedback to the driver's control terminal to execute the container release operation.

2. A method for placing containers on a container truck outside a gantry crane according to claim 1, characterized in that: The internal parameter attributes of the image acquisition components in the second embodiment include: focal length, optical center and distortion coefficient, and the external parameter attributes include: relative positions of several image acquisition components.

3. The method for placing containers on a container truck outside a gantry crane according to claim 1, characterized in that: The data attributes of the container release process of the foreign container truck in step 3 include: container position and posture data, spreader operating force and torque setting data, on-site temperature, humidity and wind speed environmental data, time records of each operation link, the path record of the spreader moving during the operation and any event record that triggers a safety alarm.

4. The method for placing containers on a container truck outside a gantry crane according to claim 1, characterized in that: During the recovery state analysis in step 4, if the camera posture recovery process fails, fault diagnosis and tracking are performed. The specific process includes: Record error codes, log information and status data, capture and mark error data, algorithm output, running time, image acquisition component position and posture, spreader position and posture and environmental data during recovery; Verify whether the viewing angle obtained by the image acquisition component matches the position information provided by the PLC bus; Verify that the parameters used in the data collection plan match those of the original data collection plan; Statistically analyze the collected error data and match it with historical fault data, count the frequency and time of similar faults, obtain historical related adjustment plans as a reference, identify the current fault mode and related characteristics, and output the final adjustment plan for secondary recovery application.

5. The method for placing containers on a container truck outside a gantry crane according to claim 1, characterized in that: The monocular depth estimation model in step 5 is based on the output depth information, including: the frame, the floor and the container, and converts the estimated depth information into absolute scale data in the actual scene.

6. The method for placing containers on a container truck outside a gantry crane according to claim 1, characterized in that: The construction process of the point cloud data in step 6 is as follows: Perform denoising based on the depth information, generate a depth map and perform normalization until the depth value meets the preset range; The internal parameters of the image acquisition component are used to obtain the mapping from the image coordinate space to the acquisition coordinate space. For each pixel and its corresponding depth value, its three-dimensional coordinates in the acquisition coordinate system are calculated.

7. A method for placing containers on a container truck outside a gantry crane according to claim 6, characterized in that: The three-dimensional coordinates in the acquisition coordinate system are calculated by converting the two-dimensional pixel coordinates. The three-dimensional coordinates are expressed as (X, Y, Z), and the two-dimensional pixel coordinates are expressed as (u, v). The calculation formula is: ； ； ； In the formula, Respectively represent the position data of the optical center on the x and y axes, and They represent the corresponding x- and y-axis focal lengths respectively, and d represents the depth value.

8. The method for placing containers on a container truck outside a gantry crane according to claim 1, characterized in that: The process of actual deviation analysis in step 7 is as follows: obtaining the actual coordinates of the container corner points and knob lock positions, comparing the target positioning coordinates with the actually detected coordinates, calculating the deviations of the container corner points and knob lock positions, analyzing the deviation calculation results, determining the deviation direction and the deviation distances of each axis, and performing an impact analysis. If the deviation is large, the position that has the greatest impact on the spreader and container placement is adjusted first. Based on the calculated deviation, the format of the correction instruction is determined, and the detected viewing angle diagram, deviation data and moving direction are displayed to the driver in a visual interface. Real-time feedback based on the current operating status, environmental information and action suggestions is provided, and the driver chooses to apply manually or adjust automatically by the program.

Images