Visual box stacking method for gantry crane

Abstract

The invention discloses a visual box stacking method for a gantry crane, and relates to the field of automatic box stacking, and the method comprises the following steps: step 1, mounting a plurality of cameras on a lifting appliance, and enabling the collection angles of the cameras to cover the range of the lifting appliance; 2, initial lifting appliance working pictures are collected through a camera, a distortion line segment on the collected pictures is marked, and camera distortion parameter calibration is carried out; step 3, performing camera radial distortion removal on the originally collected picture through the calibrated distortion parameter to obtain a camera radial distortion removal picture; through effective camera calibration and distortion correction, the precision and consistency of image acquisition are ensured, the container identification accuracy is further improved, the condition that the image acquired by the camera is closer to the real world is ensured, the discrimination capability of the model on the position and attitude of the container is improved, and the identification accuracy of the container is improved through real-time monitoring and data analysis. Fine adjustment control can be rapidly carried out on the lifting appliance, and it is ensured that all parameters in the box stacking process are kept within a safe and effective range.

Description

Technical Field

[0001] The present invention relates to the technical field of automated box stacking, and in particular to a method for visual box stacking for a gantry crane. Background Art

[0002] In modern port logistics, stacking containers in gantry crane dock crane yards is a complex and critical task. In order to improve operation efficiency and accuracy, advanced visual technology and machine learning methods have been introduced into this field. In automated operations, gantry crane equipment needs to stack boxes in the container area. The traditional method is to use lasers to determine the deviation between containers. Traditional visual recognition systems have a certain error rate, especially in complex environments or special postures, which leads to reduced accuracy in the stacking process. Due to camera distortion and environmental noise, many existing methods fail to effectively correct these deviations, thus affecting object recognition and position calculation. They often fail to cope well with dynamically changing environments, such as lighting changes, diverse box types, and cluttered yards. These will cause the model to be unstable and lack robustness in practical applications. Traditional methods usually rely on manually annotated datasets, but in complex and varied scenarios, a single static dataset is not enough to train an efficient model. Summary of the invention

[0003] 1. Technical issues to be resolved In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for visual box stacking for a gantry crane, which can effectively solve the problems of the prior art.

[0004] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: The invention discloses a method for visual stacking of boxes for a gantry crane, comprising the following steps: Step 1: Install several cameras on the spreader so that the camera acquisition angle covers the spreader range; Step 2: Use the camera to collect the initial lifting equipment working picture, mark a distortion line segment on the collected picture, and calibrate the camera distortion parameters; Step 3: Remove the camera radial distortion of the original collected image by using the calibrated distortion parameters to obtain the camera radial distortion-free image. Analyze and obtain several feature points of the target container. Based on several container feature points, map the pixels on the camera radial distortion-free image in parallel, and solve the camera's intrinsic and extrinsic parameters by using the direct linear method. Step 4: Collect the spreader lowering signal and container stacking operation video through PLC, and collect a preset number of pictures; Step 5: For a preset number of collected images, use the MAE self-supervision method to perform pre-training and label some images; Step 6: Obtain the mask of the locked area through instance segmentation, and use posture estimation to determine the key point A where the box is hung And the key point B of the lock area below , calculate the deviation values ​​of all cameras relative to the container below; Step 7: The deviation values ​​of the obtained trolley direction, trolley direction and rotation angle are micro-controlled until the deviation of the trolley direction is within the threshold of stacking boxes; Step 8: Use model assertions to collect corner cases and corresponding pseudo labels during the bin stacking process for fine-tuning the model.

[0005] Furthermore, the distortion line segment marked in step 2 is a line segment which is a curve after imaging and a straight line in the real scene, and the distortion parameter of the camera is obtained by the plumb line method.

[0006] Furthermore, the camera radial distortion model in step 2 is: ; in is the center point of camera distortion, where r represents the distance from each pixel of the image to the center of distortion. is the pixel coordinate on the distorted image, is the pixel coordinate of the corresponding dedistorted image, and the polynomial division model is used as The equation of the line through the set of points on the marked line is ax+by+c=0. The line parameters a, b, and c are obtained by the least squares method. , obtained through the polynomial division model , minimizing the energy equation To solve the distortion parameters.

[0007] Furthermore, in step 5, the MAE pre-trained weights are used for the labeled part of the picture, the original decoder is used to output the mask annotation, a network head is added after the encoder for posture output, the mask output is trained using focal loss and dice loss, and the posture head uses mean square error to fine-tune the model.

[0008] Furthermore, the deviation in step 6 includes the deviation in the direction of the small vehicle and the deviation in the direction of the large vehicle, and setting the deviation coordinates includes: ( ,in: The calculation formula of the deviation of the trolley direction is:

[0009] The calculation formula for the deviation of the vehicle direction is: ; According to the direction of the trolley, calculate the trolley direction deviation values ​​of the two cameras on the left side of the spreader and the trolley direction deviation values ​​of the two cameras on the right side. Based on the current state of the container, obtain the container feature points and calculate the length of the long side of the container, and calculate the relative deviation angle.

[0010] Furthermore, in the process of calculating the trolley direction deviation values ​​of the two cameras on the left side of the spreader and the trolley direction deviation values ​​of the two cameras on the right side: The calculation formula for the trolley direction deviation value of the two cameras on the left side of the spreader is:

[0011] The calculation formula for the car direction deviation value of the two cameras on the right is: ; The deviation between the two is: , according to the state of the spreader, the type of the current box is obtained, and the length of the long side of the container is obtained. , by calculating arctan( ) to complete the relative deviation angle acquisition.

[0012] Furthermore, the edge cases include: model misclassification, high-complexity scenes, special postures or environmental changes.

[0013] Furthermore, the specific process of fine-tuning the model in step 8 is as follows: Define and identify corner cases that occur during the stacking process, implement model assertion mechanisms, and set standards to monitor the model output, including predicted masks, poses, and classification results; In the actual stacking process, the data obtained from the camera and sensor are monitored in real time, and the output of the model is analyzed. When the model detects the above conditions, the current image and the corresponding model prediction and true label are recorded and stored in real time. The collected images are post-processed, compared with the actual situation, and pseudo-labels are automatically added. The collected corner cases and their pseudo-labels are combined into a new training set. The original data and the newly collected corner case data are integrated to form a standard training data set. The pre-trained model is used for fine-tuning with a preset learning rate.

[0014] Furthermore, the model assertion mechanism defines a trigger condition as follows: when the prediction confidence of the model is lower than a certain threshold and the prediction result is inconsistent with the actual result, it is considered a corner case.

[0015] Furthermore, the pseudo-label is obtained by calculating the container position based on the known geometric relationship, and the corner cases judged to be critical are manually verified to confirm the accuracy of the pseudo-label.

[0016] (III) Beneficial effects Compared with the known prior art, the technical solution provided by the present invention has the following beneficial effects: Through effective camera calibration and distortion correction, the accuracy and consistency of image acquisition are ensured, thereby improving the accuracy of container recognition. The plumb line method and polynomial division model are used to correct camera distortion to ensure that the images obtained by the camera are closer to the real world, thereby improving the model's ability to distinguish the position and posture of the container. Through real-time monitoring and data analysis, the spreader can be quickly fine-tuned to ensure that all parameters in the stacking process remain within a safe and effective range. The model assertion mechanism is used to monitor and collect corner cases in complex environments in real time, and the collected data is pseudo-labeled to continuously improve and fine-tune the model to adapt to the changing on-site environment. Through the collection of corner cases and model fine-tuning, this method shows a strong ability to adapt to environmental changes and improves the robustness of the overall system. By continuously collecting new data and fine-tuning during operation, the long-term use efficiency and stability of the model are improved. By using MAE for self-supervised pre-training and relying on the self-learning ability of image features, good preliminary recognition capabilities can be obtained in the absence of labeled data. Self-supervised learning and data enhancement strategies are adopted to improve the adaptability of the model to new data and effectively solve the problem of insufficient data annotation. By collecting the position of the small car, the position of the large car, PLC signals and camera data, and conducting comprehensive analysis, the operation strategy is continuously adjusted to adapt to the current working status. By integrating various sensors and visual information, the overall understanding of the operation scene is improved, thereby optimizing the scheduling and execution strategy. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] The present invention will be further described below in conjunction with the embodiments.

[0020] Example 1 A method for stacking boxes visually for a gantry crane in this embodiment, such as Figure 1 As shown, including: Step 1: Install several cameras on the spreader so that the camera acquisition angle covers the spreader range; Step 2: Use the camera to collect the initial lifting equipment working pictures, mark a distortion line segment on the collected pictures, and calibrate the camera distortion parameters; the marked distortion line segment is a line segment that is a curve after imaging and a straight line in the real scene. The camera distortion parameters are calculated by the plumb line method; the camera radial distortion model is ; in is the center point of camera distortion, where r represents the distance from each pixel of the image to the center of distortion. is the pixel coordinate on the distorted image, is the pixel coordinate of the corresponding dedistorted image, and the polynomial division model is used as The equation of the line through the set of points on the marked line is ax+by+c=0. The line parameters a, b, and c are obtained by the least squares method. , obtained through the polynomial division model , minimizing the energy equation To solve the distortion parameters; Step 3: Remove the camera radial distortion of the original collected image by using the calibrated distortion parameters to obtain the camera radial distortion-free image. Analyze and obtain several feature points of the target container. Based on several container feature points, map the pixels on the camera radial distortion-free image in parallel, and solve the camera's intrinsic and extrinsic parameters by using the direct linear method. Step 4: Collect the spreader lowering signal and container stacking operation video through PLC, and collect a preset number of pictures; Step 5: For a preset number of collected images, use MAE self-supervision to pre-train and annotate some images; for the annotated images, use the MAE pre-trained weights, the original decoder is used to output the mask annotation, and a network head is added after the encoder for posture output. The output of the mask is trained using focal loss and dice loss, and the posture head uses mean square error to fine-tune the model; Step 6: Obtain the mask of the locked area through instance segmentation, and use posture estimation to determine the key point A where the box is hung And the key point B of the lock area below , calculate the deviation values ​​of all cameras relative to the container below; Step 7: The deviation values ​​of the obtained trolley direction, trolley direction and rotation angle are micro-controlled until the deviation of the trolley direction is within the threshold of stacking boxes; Step 8: Use model assertions to collect corner cases and corresponding pseudo labels in the stacking process for fine-tuning the model. Corner cases include: model misclassification, highly complex scenes, special postures or environmental changes. Pseudo labels are calculated based on container positions with known geometric relationships. Manual verification is performed on critical corner cases to confirm the accuracy of pseudo labels.

[0021] Compared with the existing technology, by calibrating the distortion parameters of the camera and correcting it using the deformation line segments, the radial distortion of the camera lens can be effectively eliminated, thereby improving the accuracy of image processing. The MAE self-supervised pre-training method can effectively use unlabeled data for model training, reducing the demand for a large amount of labeled data. At the same time, through the combined training of posture output and loss function, the model's adaptability to different states can be improved. The mask of the lock area is obtained through instance segmentation, and combined with posture estimation to identify key points A and B. This method improves the ability to accurately locate the container and its lock area and reduces the risk of erroneous operation. In terms of micro-motion control, real-time adjustment is made according to the obtained deviation value, which helps to maintain precise operation in complex stacking situations and reduces the risk of container stacking operations. By collecting corner cases and confirming and verifying pseudo-labels, the robustness of the model can be improved and its adaptability to complex scenarios can be enhanced. This mechanism can continuously optimize the model to adapt to challenges in different environments.

[0022] Example 2 In other aspects, this embodiment also provides a method for fine-tuning model processing, and the specific process of fine-tuning model processing is: Define and identify corner cases that occur during the stacking process, implement the model assertion mechanism, and set standards to monitor the model output, including the predicted mask, posture, and classification results; the model assertion mechanism defines the trigger conditions as follows: when the model's prediction confidence is lower than a certain threshold and the predicted result is inconsistent with the actual result, it is judged as a corner case; In the actual stacking process, the data obtained from the camera and sensor are monitored in real time, and the output of the model is analyzed. When the model detects the above conditions, the current image and the corresponding model prediction and true label are recorded and stored in real time. The collected images are post-processed, compared with the actual situation, and pseudo-labels are automatically added. The collected corner cases and their pseudo-labels are combined into a new training set. The original data and the newly collected corner case data are integrated to form a standard training data set. The pre-trained model is used for fine-tuning with a preset learning rate.

[0023] Compared with existing technologies, by defining the criteria and trigger conditions for edge cases, it is possible to effectively identify special situations encountered by the model in actual applications, such as low confidence or inconsistent predictions. This accurate identification ensures that subsequent modifications and optimizations are based on real problems. During the stacking process, the camera and sensor data are monitored in real time, so that changes in the model output can be captured in a timely manner. Through the real-time feedback mechanism, problems can be quickly discovered and adjusted, thereby improving the safety and efficiency of operations. The collected corner cases are combined with the original data to form a more comprehensive training data set, so that the model can better learn and adapt to different scenarios and enhance its generalization ability. Fine-tuning with a pre-trained model at a lower learning rate can effectively avoid overfitting of the model on new data while maintaining the performance on the original task; The application of this strategy makes the model more stable when processing new data. Through this dynamic fine-tuning and incremental learning method, the training set is continuously updated and improved. The model can gradually overcome various complex situations encountered in the real environment, thereby improving its own robustness and adaptability.

[0024] Example 3 In this embodiment, a method for calculating the direction deviation between the small vehicle and the large vehicle and a method for calculating the relative deviation angle are provided, which specifically includes: setting the deviation coordinates including: ( ,in: The calculation formula of the deviation of the trolley direction is:

[0025] The calculation formula for the deviation of the vehicle direction is: ; According to the direction of the trolley, calculate the trolley direction deviation values ​​of the two cameras on the left side of the spreader and the trolley direction deviation values ​​of the two cameras on the right side. Based on the current state of the container, obtain the container feature points and calculate the length of the long side of the container, and calculate the relative deviation angle; In the process of calculating the trolley direction deviation values ​​of the two cameras on the left side of the spreader and the trolley direction deviation values ​​of the two cameras on the right side: The calculation formula for the trolley direction deviation value of the two cameras on the left side of the spreader is:

[0026] The calculation formula for the car direction deviation value of the two cameras on the right is: ; The deviation between the two is: , according to the state of the spreader, the type of the current box is obtained, and the length of the long side of the container is obtained. , by calculating arctan( ) to complete the relative deviation angle acquisition.

[0027] In summary, the present invention can simultaneously output the posture information and instance segmentation results of the container through the decoder of the posture estimation and instance segmentation multi-task learning model. This multi-task learning model not only improves the overall efficiency of the system, but also can better understand the spatial relationship and shape characteristics of the container. Through the above method, millimeter-level deviation estimation can be achieved, which is crucial in automated container stacking. Millimeter-level accuracy ensures that the containers can be accurately aligned during the stacking process, avoiding operational errors and safety hazards caused by deviations. The present invention not only focuses on image processing and analysis, but also combines real-time signal acquisition, control feedback and model fine-tuning to form a relatively comprehensive system solution. The automatic container placement method based on advanced visual technology and machine learning methods and the system using the method can achieve higher accuracy and more robust container stacking perception at a lower cost, which plays an important role in improving port operation efficiency, improving container stacking accuracy and ensuring operation safety. It not only reduces the need for manual intervention, but also lays a technical foundation for the development of future smart ports.

[0028] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visual stacking of boxes for gantry cranes, characterized in that: The following steps are involved: Step 1: Install several cameras on the spreader so that the camera acquisition angle covers the spreader range; Step 2: Use the camera to collect the initial lifting equipment working picture, mark a distortion line segment on the collected picture, and calibrate the camera distortion parameters; Step 3: Remove the camera radial distortion of the original collected image by using the calibrated distortion parameters to obtain the camera radial distortion-free image. Analyze and obtain several feature points of the target container. Based on several container feature points, map the pixels on the camera radial distortion-free image in parallel, and solve the camera's intrinsic and extrinsic parameters by using the direct linear method. Step 4: Collect the spreader lowering signal and container stacking operation video through PLC, and collect a preset number of pictures; Step 5: For a preset number of collected images, use the MAE self-supervision method to perform pre-training and label some images; Step 6: Obtain the mask of the locked area through instance segmentation, and use posture estimation to determine the key point A where the box is hung And the key point B of the lock area below , calculate the deviation values ​​of all cameras relative to the container below; Step 7: The deviation values ​​of the obtained trolley direction, trolley direction and rotation angle are micro-controlled until the deviation of the trolley direction is within the threshold of stacking boxes; Step 8: Use model assertions to collect corner cases and corresponding pseudo labels during the bin stacking process for fine-tuning the model.

2. A method for visual stacking of boxes for gantry cranes according to claim 1, characterized in that: The distortion line segment marked in step 2 is a line segment which is a curve after imaging and a straight line in the real scene. The distortion parameters of the camera are calculated by the plumb line method.

3. A method for visual stacking of boxes for gantry cranes according to claim 1, characterized in that: The camera radial distortion model in step 2 is: ; in is the center point of camera distortion, where r represents the distance from each pixel of the image to the center of distortion. is the pixel coordinate on the distorted image, is the pixel coordinate of the corresponding dedistorted image, and the polynomial division model is used as The equation of the line through the set of points on the marked line is ax+by+c=0. The line parameters a, b, and c are obtained by the least squares method. , obtained through the polynomial division model , minimizing the energy equation To solve the distortion parameters.

4. A method for visual stacking of boxes for gantry cranes according to claim 1, characterized in that: In step 5, for the image of the annotated part, the weights pre-trained by MAE are used, the original decoder is used to output the mask annotation, a network head is added after the encoder for posture output, the mask output is trained using focal loss and dice loss, and the posture head uses mean square error to fine-tune the model.

5. The method for visual stacking of boxes for gantry crane according to claim 1, characterized in that: The deviation in step 6 includes the deviation in the direction of the small vehicle and the deviation in the direction of the large vehicle, and setting the deviation coordinates includes: ,in: The calculation formula of the deviation of the trolley direction is: The calculation formula for the deviation of the vehicle direction is: ; According to the direction of the trolley, calculate the trolley direction deviation values ​​of the two cameras on the left side of the spreader and the trolley direction deviation values ​​of the two cameras on the right side. Based on the current state of the container, obtain the container feature points and calculate the length of the long side of the container, and calculate the relative deviation angle.

6. A method for visual stacking of boxes for gantry cranes according to claim 5, characterized in that: In the process of calculating the trolley direction deviation values ​​of the two cameras on the left side of the spreader and the trolley direction deviation values ​​of the two cameras on the right side: The calculation formula for the trolley direction deviation value of the two cameras on the left side of the spreader is: The calculation formula for the car direction deviation value of the two cameras on the right is: ; The deviation between the two is: , according to the state of the spreader, the type of the current box is obtained, and the length of the long side of the container is obtained. , by calculating arctan( ) to complete the relative deviation angle acquisition.

7. A method for visual stacking of boxes for gantry cranes according to claim 1, characterized in that: The corner cases include: model misclassification, high complexity scenes, special postures or environmental changes.

8. The method for visual stacking of boxes by a gantry crane according to claim 1, characterized in that: The specific process of fine-tuning the model in step 8 is as follows: Define and identify corner cases that occur during the stacking process, implement model assertion mechanisms, and set standards to monitor the model output, including predicted masks, poses, and classification results; In the actual stacking process, the data obtained from the camera and sensor are monitored in real time, and the output of the model is analyzed. When the model detects the above conditions, the current image and the corresponding model prediction and true label are recorded and stored in real time. The collected images are post-processed, compared with the actual situation, and pseudo-labels are automatically added. The collected corner cases and their pseudo-labels are combined into a new training set. The original data and the newly collected corner case data are integrated to form a standard training data set. The pre-trained model is used for fine-tuning with a preset learning rate.

9. A method for visual stacking of boxes for gantry cranes according to claim 8, characterized in that: The model assertion mechanism defines the triggering condition as follows: when the prediction confidence of the model is lower than a certain threshold and the prediction result is inconsistent with the actual result, it is determined to be a corner case.

10. The method for visual stacking of boxes by a gantry crane according to claim 1, characterized in that: The pseudo-label is obtained by calculating the container position based on the known geometric relationship, and the corner cases judged to be critical are manually verified to confirm the accuracy of the pseudo-label.

Images