Method for collecting, clamping and placing box in gantry crane

Abstract

The invention discloses a method for container placing of a container truck in a gantry crane, and relates to the field of container carrying, and the method comprises the following steps: step 1, arranging a plurality of cameras for a lifting appliance, and obtaining a placing area image of a container in a balanced manner at each visual angle; 2, shooting the calibration plate through a plurality of preset angles and positions, collecting a plurality of calibration images, and calculating internal parameters and relative positions of a plurality of hanger cameras by applying a calibration algorithm; step 3, connecting the programmable logic controller with a container operating system, acquiring a data acquisition authority, and acquiring the size, weight and current position data of the currently operated container; through joint calibration and image processing of multiple cameras, real-time and accurate space information and container positions are obtained, dependence of a traditional method on experienced operators is reduced, key information is extracted from a large amount of data through intelligent algorithms such as an instance segmentation model and semi-supervised learning, the operation process is optimized, and the accuracy of the operation process is improved. And automation of container loading and unloading operation is promoted.

Description

Technical Field

[0001] The invention relates to the technical field of container handling, and in particular to a method for placing containers on a container truck in a gantry crane. Background Art

[0002] In modern port and terminal operations, gantry cranes and dock cranes are key loading and unloading equipment. Especially in the process of placing containers on container trucks in the terminal, how to complete the operation accurately and efficiently has always been a technical difficulty. Traditional terminal operations require a lot of manual intervention, especially when container trucks place containers at designated locations, manual precise guidance and adjustment are required, which not only increases labor costs, but also easily leads to inefficiency and misoperation. The feedback mechanism of traditional engineering equipment in detection and adjustment is usually slow and cannot respond to changes in real time, resulting in inefficient adjustments. In the case of a rapidly changing operating environment, errors and deviations cannot be corrected in a timely manner. Today's logistics industry requires a higher level of automation and intelligence, but traditional technologies often lack this capability, limiting efficient and safe operating modes. In addition, in visual monitoring systems, traditional methods are insufficient in the collection, processing and comprehensive decision-making of spatial information, and often cannot provide clear and complete information on the position and posture of objects in space. Summary of the invention

[0003] 1. Technical issues to be resolved In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for placing containers on a container truck in a gantry crane, which can effectively solve the problems of the prior art.

[0004] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: The present invention discloses a method for placing containers on a container truck in a gantry crane, comprising the following steps: Step 1: Arrange several cameras on the spreader to obtain images of the container placement area in a balanced manner from different viewing angles; Step 2: Photograph the calibration plate at several preset angles and positions, collect multiple calibration images, and apply the calibration algorithm to calculate the intrinsic parameters of several sling cameras and their relative positions; Step 3: Connect the programmable logic controller and the container operating system, obtain data collection permissions, and collect the size, weight, and current location data of the currently operated container; Step 4: Measure the distance data between the front baffle and side guide of the frame and the corner of the container as the reference item data of the relative position during the container placement process; Step 5: Randomly select some collected data and reference data, perform screening and labeling data processing, build initial training sets and validation sets, input labeled data for machine learning, train the initial instance segmentation model, and output the areas of the frame guide and fender; Step 6: After obtaining the initial instance segmentation model, use a semi-supervised method to iteratively filter the collected data until the model training is completed; Step 7: Use the trained instance segmentation model to process the real-time image to identify and extract the areas of the guide plate and baffle on the frame; Step 8: Calculate the deviation data between the frame and the spreader through the measured position and posture information of the front baffle and guide plate of the frame; Step 9: Generate adjustment instructions based on the deviation data and feed them back to the spreader control end. When the deviation data is within the set threshold, the spreader performs adjustment operations.

[0005] Furthermore, in the process of inputting labeled data for machine learning in step 5, the training parameters of the learning rate, batch size, optimizer and loss function are set, the semantic segmentation algorithm is selected to build the model architecture, the model is trained using the training set, the loss value and accuracy of the validation set are monitored during the training process, and the hyperparameters are adjusted. If the validation set loss no longer decreases or shows an increasing trend, the training is stopped.

[0006] Furthermore, the annotated data in step 5 includes a mask image of each instance and a corresponding category label, which are synchronously uploaded to the database for storage after annotating.

[0007] Furthermore, the model training process in step 6 is: using a semi-supervised learning method, expanding the training set by reasoning on unlabeled data, and iterating the initial instance segmentation model, cyclically screening and updating the collected data according to a preset period until the strongly enhanced and weakly enhanced labeled data are consistent in the instance segmentation model results.

[0008] Furthermore, in the process of the cyclic screening, the current model is used to generate pseudo labels for unlabeled samples, the pseudo labels are added to the training data set, and then retrained on the training data set, the pseudo labels are classified, and for each pseudo label, the current instance segmentation model is used to predict it to obtain a confidence score for each pseudo label, and the confidence of the pseudo label prediction is compared with a preset threshold, and only pseudo labels with confidence higher than the threshold are selected for labeling.

[0009] Furthermore, the calculation process of the deviation data between the frame and the sling in step 8 is: set a global coordinate system, define the frame calibration points and the key points of the sling in the coordinate system, perform posture transformation on each key point, obtain the position and attitude vectors of the corresponding frame and sling, calculate the posture difference vector and rotation matrix of the frame and sling, and obtain the deviation data.

[0010] Furthermore, the position and attitude vectors of the frame and the spreader are expressed as a 4x4 transformation matrix, specifically: ; ; The deviation between the frame and the spreader is calculated by the combined matrix: ; In the formula, Represents the pose transformation matrix of the front fender of the frame, Represents the pose transformation matrix of the spreader, represents the rotation matrix of the front fender of the frame, represents the rotation matrix of the spreader, represents the position vector of the front fender of the frame, represents the position vector of the spreader, Represents the inverse matrix of the pose matrix of the spreader.

[0011] Furthermore, the The dimension is 4x4, including the translation deviation d and the rotation deviation , specifically: − ; ; In the formula, The matrix representing the inverse of the spreader's rotation matrix.

[0012] (III) Beneficial effects Compared with the known prior art, the technical solution provided by the present invention has the following beneficial effects: Through the joint calibration and image processing of multiple cameras, real-time and accurate spatial information and container positions are obtained, which greatly improves the accuracy of container loading and unloading, reduces errors caused by human factors, and ensures that containers can be accurately placed in the predetermined position. The automated and intelligent visual inspection and control system reduces the dependence of traditional methods on experienced operators, and can still achieve efficient operations in the case of insufficient personnel or lack of experience, thereby improving the utilization efficiency of human resources. Through intelligent algorithms such as instance segmentation models and semi-supervised learning, key information is extracted from a large amount of data, the operation process is optimized, the automation of container loading and unloading operations is promoted, and the limitations of manual operations are reduced. Based on data-driven models, the system can flexibly adapt to the integration of new equipment or new technologies, improve overall operating efficiency, and reduce operating costs. Camera calibration is performed at multiple angles and multiple positions to establish a robust visual processing algorithm that can adapt to the changing yard environment. Even in highly dynamic and complex operating environments, the system can maintain good performance and ensure the safe and efficient placement of containers. Through the collection and integration of multiple data, information islands are eliminated, and a global information monitoring and decision-making system is established, the real-time and accuracy of data are improved, which helps to optimize the overall operation process, reduce operating costs and time, and manage container resources more efficiently. Through precise posture control and deviation adjustment, the impact and wear of equipment caused by errors in container operation are reduced, the service life of the equipment is increased, the maintenance and replacement costs are reduced, and the safety of operation is improved. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] The present invention will be further described below in conjunction with the embodiments.

[0016] Example 1 A method for placing a container truck in a gantry crane according to this embodiment is as follows: Figure 1 As shown, the following steps are included: Step 1: Several cameras are placed on the spreader to obtain images of the container placement area from different perspectives in a balanced manner; this significantly improves the real-time monitoring of the container status and provides rich visual information for subsequent data processing and model training; Step 2: Photograph the calibration plate at several preset angles and positions, collect multiple calibration images, and apply the calibration algorithm to calculate the internal parameters of several sling cameras and their relative positions; this eliminates image distortion caused by camera position changes and greatly improves the accuracy of subsequent image processing; Step 3: Connect the programmable logic controller and the container operating system, obtain data collection permissions, and collect the size, weight, and current location data of the currently operated container; this information is the basis for accurate placement and subsequent algorithm processing, which improves the operational accuracy of the entire platform; Step 4: Measure the distance data between the front baffle and side guide of the frame and the corner of the container as the reference item data for the relative position during the container placement process; ensure that in actual operation, the container placement accuracy is higher and the error range is reduced; Step 5: Randomly select a number of collected data and reference item data, perform screening and labeling data processing, construct the initial training set and validation set, input the labeled data for machine learning, train the initial instance segmentation model, and output the area of ​​the frame guide and baffle; in the process of inputting the labeled data for machine learning, set the learning rate, batch size, training parameters of the optimizer and loss function, select the semantic segmentation algorithm to build the model architecture, use the training set to train the model, monitor the loss value and accuracy of the validation set during training, and adjust the hyperparameters. If the loss of the validation set no longer decreases or shows an increasing trend, stop training; the labeled data contains the mask image and the corresponding category label of each instance, which are uploaded to the database for storage after labeling; in the process of machine learning, the setting and optimization of various parameters improve the training efficiency and final effect of the instance segmentation model. Monitor the loss value and accuracy of the validation set to make model training more scientific and reasonable, and avoid the occurrence of overfitting; Step 6: After obtaining the initial instance segmentation model, use a semi-supervised method to iteratively filter the collected data until the model training is completed; the model training process is as follows: using a semi-supervised learning method, expand the training set by reasoning on unlabeled data, and iterate the initial instance segmentation model, cyclically filter and update the collected data according to a preset period until the strongly enhanced and weakly enhanced labeled data are consistent in the instance segmentation model results; during the cyclic screening process, use the current model to generate pseudo labels for unlabeled samples, add the pseudo labels to the training data set, and then retrain on the training data set to classify the pseudo labels. For each pseudo label, use the current instance segmentation model to predict it, obtain the confidence score of each pseudo label, compare the confidence of the pseudo label prediction with the preset threshold, and only select the pseudo labels with a confidence higher than the threshold for labeling; using the reasoning of unlabeled data, the scale of the training set is expanded, and the recognition and generalization capabilities of the model are further enhanced. By generating pseudo labels and classifying them, the quality of the training data set is ensured and the accuracy of the final model is improved; Step 7: Use the trained instance segmentation model to process the real-time image, identify and extract the areas of the guide plate and baffle on the frame; the effect of this step directly affects the operating accuracy of the spreader and ensures the safe placement of the container; Step 8: Calculate the deviation data between the frame and the spreader through the measured position and posture information of the front baffle and guide plate of the frame; provide data basis for subsequent adjustments and ensure the accuracy and safety of the operation; Step 9: Generate adjustment instructions based on the deviation data and feed them back to the spreader control end. When the deviation data is within the set threshold, the spreader will perform adjustments. This automated feedback mechanism greatly improves operational efficiency and reduces the impact of human factors on operational accuracy.

[0017] Compared with existing technologies, multi-angle monitoring, precise calibration, effective data collection and processing, as well as efficient machine learning and real-time feedback control have greatly improved the accuracy and safety of container truck placement in the gantry crane. The refined processing of each step ensures the intelligence and automation of the operation process, which has brought innovation to the traditional container handling method. Through continuous iterative optimization and model training, more efficient resource utilization and a safer operating environment can be achieved, providing a solid foundation for future smart logistics.

[0018] Example 2 On other levels, this embodiment also provides a process for calculating the deviation data between the frame and the sling, specifically: setting a global coordinate system, defining the frame calibration points and the key points of the sling in the coordinate system, performing posture transformation on each key point, obtaining the position and attitude vectors corresponding to the frame and sling, calculating the posture difference vector and rotation matrix of the frame and sling, and obtaining the deviation data.

[0019] As a preferred implementation in this embodiment, the position and attitude vectors of the frame and the sling are expressed as a 4x4 transformation matrix, specifically: ; ; The deviation between the frame and the spreader is calculated by the combined matrix: ; The position and posture of the frame Convert to the coordinate system of the spreader to facilitate calculation of the deviation between them; In the formula, Represents the pose transformation matrix of the front fender of the frame, Represents the pose transformation matrix of the spreader, represents the rotation matrix of the front fender of the frame, represents the rotation matrix of the spreader, represents the position vector of the front fender of the frame, represents the position vector of the spreader, Represents the inverse matrix of the pose matrix of the spreader.

[0020] Said The dimension is 4x4, including the translation deviation d and the rotation deviation , specifically: − ; ; By taking the inverse of the rotation matrix, the coordinate system of the spreader can be transformed to the coordinate system of the frame, so that the positions and orientations of the two can be compared in the same coordinate system. When calculating the orientation difference between the frame and the spreader, the orientation of the spreader is transformed from its own coordinate system to the coordinate system of the frame to facilitate the calculation of the relative orientation difference between the two; In the formula, The inverse matrix representing the rotation matrix of the spreader enables the control system to accurately determine the actual relative position and direction between the frame and the spreader, providing an accurate basis for subsequent fine-tuning. In actual work, by continuously measuring and calculating the deviation, a closed-loop feedback system can be formed to evaluate the rotation deviation. , can effectively prevent potential collisions and mistakes, especially in complex and dynamic yard environments. Precise posture control can prevent equipment damage and personal injury.

[0021] In summary, the present invention can accurately locate the guide plate and baffle on the truck frame through the instance segmentation technology, and estimate the posture deviation of the spreader and the frame accordingly, thereby achieving real-time guidance for the truck driver, greatly improving the accuracy and efficiency of the operation; Specifically, instance segmentation technology can accurately identify and segment key components on the truck frame, and then estimate the posture deviation between the spreader and the frame by calculating the spatial position and posture of these components. This information is fed back to the truck driver in real time through the visual guidance system to help him make precise adjustments, thereby ensuring that the container can be accurately placed at the target location. Through the above method, the gantry crane terminal crane can automatically place the container on the container truck in the yard on the lane side, effectively solving the problem of manual intervention in placing containers on the container truck on the lane side, reducing dependence on manual operation, and improving overall operational efficiency and safety. Through this intelligent guidance system, ports and terminals can better cope with the growing demand for cargo throughput while reducing labor costs and operational risks.

[0022] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for placing containers on a container truck in a gantry crane, characterized in that: The following steps are involved: Step 1: Arrange several cameras on the spreader to obtain images of the container placement area in a balanced manner from different viewing angles; Step 2: Photograph the calibration plate at several preset angles and positions, collect multiple calibration images, and apply the calibration algorithm to calculate the intrinsic parameters of several sling cameras and their relative positions; Step 3: Connect the programmable logic controller and the container operating system, obtain data collection permissions, and collect the size, weight, and current location data of the currently operated container; Step 4: Measure the distance data between the front baffle and side guide of the frame and the corner of the container as the reference item data of the relative position during the container placement process; Step 5: Randomly select some collected data and reference data, perform screening and labeling data processing, build initial training sets and validation sets, input labeled data for machine learning, train the initial instance segmentation model, and output the areas of the frame guide and fender; Step 6: After obtaining the initial instance segmentation model, use a semi-supervised method to iteratively filter the collected data until the model training is completed; Step 7: Use the trained instance segmentation model to process the real-time image to identify and extract the areas of the guide plate and baffle on the frame; Step 8: Calculate the deviation data between the frame and the spreader through the measured position and posture information of the front baffle and guide plate of the frame; Step 9: Generate adjustment instructions based on the deviation data and feed them back to the spreader control end. When the deviation data is within the set threshold, the spreader performs adjustment operations.

2. A method for placing containers on a container truck in a gantry crane according to claim 1, characterized in that: In the process of inputting the labeled data for machine learning in step 5, the training parameters of the learning rate, batch size, optimizer and loss function are set, the semantic segmentation algorithm is selected to build the model architecture, the model is trained using the training set, the loss value and accuracy of the validation set are monitored during the training process, and the hyperparameters are adjusted. If the validation set loss no longer decreases or shows an increasing trend, the training is stopped.

3. The method for placing containers on a container truck in a gantry crane according to claim 1, characterized in that: The labeled data in step 5 includes the mask image of each instance and the corresponding category label, which are synchronously uploaded to the database for storage after labeling.

4. The method for placing containers on a container truck in a gantry crane according to claim 1, characterized in that: The model training process in step 6 is: using a semi-supervised learning method, expanding the training set by reasoning on unlabeled data, and iterating the initial instance segmentation model, cyclically screening and updating the collected data according to a preset period until the strongly enhanced and weakly enhanced labeled data are consistent in the instance segmentation model results.

5. The method for placing containers on a container truck in a gantry crane according to claim 4, characterized in that: In the process of the cyclic screening, the current model is used to generate pseudo labels for unlabeled samples, the pseudo labels are added to the training data set, and then retrained on the training data set, the pseudo labels are classified, and for each pseudo label, the current instance segmentation model is used to predict it to obtain a confidence score for each pseudo label, and the confidence of the pseudo label prediction is compared with a preset threshold, and only pseudo labels with confidence higher than the threshold are selected for labeling.

6. The method for placing containers on a container truck in a gantry crane according to claim 1, characterized in that: The calculation process of the deviation data between the frame and the sling in step 8 is: set a global coordinate system, define the frame calibration points and the key points of the sling in the coordinate system, perform posture transformation on each key point, obtain the position and attitude vectors of the corresponding frame and sling, calculate the posture difference vector and rotation matrix of the frame and sling, and obtain the deviation data.

7. A method for placing containers on a container truck in a gantry crane according to claim 6, characterized in that: The position and attitude vectors of the frame and the spreader are expressed as a 4x4 transformation matrix, specifically: ； ； The deviation between the frame and the spreader is calculated by the combined matrix: ； In the formula, Represents the pose transformation matrix of the front fender of the frame, Represents the pose transformation matrix of the spreader, represents the rotation matrix of the front fender of the frame, represents the rotation matrix of the spreader, represents the position vector of the front fender of the frame, represents the position vector of the spreader, Represents the inverse matrix of the pose matrix of the spreader.

8. The method for placing containers on a container truck in a gantry crane according to claim 1, characterized in that: Said The dimension is 4x4, including the translation deviation d and the rotation deviation , specifically: − ； ； In the formula, The matrix representing the inverse of the spreader's rotation matrix.

Images