Container grabbing and positioning methods, devices, industrial control computers, and storage media

By using a deep learning model to detect the coordinates of the detection boxes for containers and keyholes, and combining this with guide block information, the time consumption and occlusion problems of traditional vision processing algorithms are solved, enabling efficient and accurate positioning of container grabbing and improving the efficiency of automated operations.

CN117726682BActive Publication Date: 2026-11-10MATRIXTIME ROBOTICS (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311760094.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-19
Publication Date
2026-11-10
Estimated Expiration
2043-12-19

AI Technical Summary

Technical Problem

Traditional vision processing algorithms are time-consuming when grabbing containers, and the field of vision is obstructed when the spreader is descending, making it difficult to accurately locate the lock hole, which affects the accuracy and efficiency of container grabbing.

Method used

A container detection model and a keyhole detection model based on deep learning are adopted. The center coordinates of the container are determined by detecting the detection frame coordinates of the container and guide block, and the positioning accuracy is optimized by combining the keyhole detection frame coordinates. Distortion correction and filtering conditions are used to optimize the positioning accuracy.

Benefits of technology

It improves the efficiency and accuracy of container grabbing detection, avoids positioning inaccuracies caused by blocked lock holes, and enhances the efficiency of automated operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117726682B_ABST
    Figure CN117726682B_ABST
Patent Text Reader

Abstract

This invention proposes a container gripping and positioning method, device, industrial control computer, and storage medium, relating to the field of logistics technology. The method utilizes a deep learning-based container detection model to obtain the coordinates of the container detection frame and the guide block detection frame. Based on the container detection frame coordinates, a deep learning-based keyhole detection model is used to obtain the keyhole detection frame coordinates, effectively solving the problem of inaccurate detection caused by the small size of the keyhole. Combining the guide block detection frame coordinates and the keyhole detection frame coordinates to determine the center coordinates of the container to be inspected effectively solves the problem of inaccurate positioning due to keyhole obstruction, further improving detection accuracy, facilitating automated container gripping, and increasing the efficiency of automated operations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of logistics technology, and more specifically, to a container grabbing and positioning method, device, industrial control computer, and storage medium. Background Technology

[0002] Automated container handling is an essential part of automated operations at ports, and the key to container handling lies in quickly locating the container. Traditional vision processing algorithms often consume a lot of time, affecting the efficiency of automated operations.

[0003] In addition, because the container grabbing operation is at a high height, the video sensor is often installed at a height of more than ten meters above the ground. When the spreader descends, it often obstructs the container, causing blind spots in the field of view of the visual sensor, making it difficult to accurately locate the container and affecting the accuracy of container grabbing. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a container grabbing and positioning method, device, industrial control computer and storage medium, which can improve the efficiency and accuracy of container grabbing and positioning.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, the present invention provides a container gripping and positioning method, applied to an industrial control computer, the method comprising:

[0007] Acquire a container grabbing detection image; the container grabbing detection image includes the container to be inspected, the guide block to be inspected, and the lock hole to be inspected;

[0008] The captured container detection image is input into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected. The container detection model is obtained by training a deep learning object detection model based on the labeled container and guide block labels, as well as the corresponding label coordinates of the container and guide block labels.

[0009] The container detection image is cropped according to the detection frame coordinates of the container to be inspected to obtain the container image to be inspected;

[0010] The container image to be detected is input into a pre-trained keyhole detection model to obtain the coordinates of the detection box of the keyhole; the keyhole detection model is obtained by training a deep learning object detection model based on the container image with labeled keyhole boxes and the coordinates of the corresponding labeled boxes.

[0011] The center coordinates of the container to be inspected are determined based on the detection frame coordinates of the keyhole to be inspected and the detection frame coordinates of the guide block to be inspected; the center coordinates of the container to be inspected are used to guide the spreader to grab the container.

[0012] In an optional implementation, before the step of inputting the container detection image into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected, the following steps are included:

[0013] The distortion correction is performed on the box detection image according to the preset distortion coefficient.

[0014] In an optional implementation, the container inspection model and the keyhole inspection model are obtained in the following way:

[0015] Acquire training images and corresponding annotation description files; the annotation description files record a one-to-one correspondence between annotation box types and annotation box coordinates; the annotation box types include container type, guide block type, and keyhole type; the training images include labeled container annotation boxes, guide block annotation boxes, and keyhole annotation boxes;

[0016] The training image, the annotation description file, the container type, and the guide block type are input into the deep learning object detection model to train the container detection model.

[0017] The training image is cropped based on the coordinates of the container bounding boxes to obtain the container image;

[0018] The container image, the annotation description file, and the keyhole type are input into the deep learning target detection model to train the keyhole detection model.

[0019] In an optional implementation, the step of determining the center coordinates of the container to be inspected based on the detection frame coordinates of the lock hole to be inspected and the detection frame coordinates of the guide block to be inspected includes:

[0020] The center coordinates of the keyhole are determined based on the detection frame coordinates of the keyhole to be detected and the detection frame coordinates of the guide block to be detected.

[0021] The center coordinates of the container to be inspected are determined based on the center coordinates of the keyhole, the slope of the long side of the container to be inspected, and the pixel width of the container to be inspected.

[0022] In an optional implementation, the step of determining the center coordinates of the keyhole based on the detection frame coordinates of the keyhole to be detected and the detection frame coordinates of the guide block to be detected includes:

[0023] The guide block to be detected and the keyhole to be detected that meet the preset filtering conditions are identified as the target guide block and the target keyhole;

[0024] The center coordinates of the guide blocks are determined based on the detection frame coordinates of all the target guide blocks.

[0025] When there are two target keyholes, the center coordinates of the keyholes are determined based on the detection frame coordinates of all the target keyholes, and the relative coordinate set is updated; the relative coordinate set is used to record the coordinate difference between the center coordinates of the guide block and the center coordinates of the keyhole in each of the box-grabbing detection images;

[0026] When there are no two target keyholes, the center coordinates of the keyholes are determined based on the relative coordinate set and the center coordinates of the guide block.

[0027] In an optional implementation, the step of determining the center coordinates of the guide block based on the detection frame coordinates of all the target guide blocks includes:

[0028] The abscissa of the center coordinate of the guide block is determined based on the average abscissa of the detection frame of all the target guide blocks.

[0029] The ordinate of the center coordinate of the guide block is determined based on the average value of the ordinate of the detection frame of all the target guide blocks.

[0030] In an optional implementation, the step of determining the center coordinates of the keyhole based on the detection frame coordinates of all the target keyholes and updating the relative coordinate set includes:

[0031] The horizontal coordinate of the center of the keyhole is determined based on the average value of the horizontal coordinates of the detection frames of all the target keyholes.

[0032] The ordinate of the center coordinate of the keyhole is determined based on the average value of the ordinate of the detection frame of all the target keyholes;

[0033] The relative coordinates are determined based on the center coordinates of the guide block and the center coordinates of the keyhole, and the relative coordinates are saved to the relative coordinate set.

[0034] In an optional implementation, the step of determining the center coordinates of the keyhole based on the relative coordinate set and the center coordinates of the guide block includes:

[0035] The median coordinate of the set of relative coordinates is determined as the target relative coordinates;

[0036] The abscissa of the center coordinate of the keyhole is determined based on the difference between the abscissa of the center coordinate of the guide block and the abscissa of the relative coordinate of the target.

[0037] The ordinate of the lock hole center is determined based on the difference between the ordinate of the guide block center coordinate and the ordinate of the target relative coordinate.

[0038] Secondly, the present invention provides a container gripping and positioning device for use in an industrial control computer, the device comprising:

[0039] The preprocessing module is used to acquire container grabbing detection images; the container grabbing detection images include the container to be inspected, the guide block to be inspected, and the lock hole to be inspected;

[0040] The detection module is used to input the captured container detection image into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected; the container detection model is obtained by training a deep learning target detection model based on the labeled container and guide block labels, as well as the corresponding label coordinates of the container and guide block labels.

[0041] The detection module is also used to crop the container detection image according to the detection box coordinates of the container to be detected to obtain the container image to be detected; input the container image to be detected into a pre-trained keyhole detection model to obtain the detection box coordinates of the keyhole to be detected; the keyhole detection model is obtained by training a deep learning target detection model based on the container image with labeled keyhole labels and the corresponding label box coordinates of the keyhole labels;

[0042] The determining module is used to determine the center coordinates of the container to be inspected based on the detection frame coordinates of the lock hole to be inspected and the detection frame coordinates of the guide block to be inspected; the center coordinates of the container to be inspected are used to guide the spreader to grab the container.

[0043] Thirdly, the present invention provides an industrial control computer, the industrial control computer including a memory and a processor, the memory being used to store a computer program, and the processor being used to execute the container grabbing and positioning method as described in any of the foregoing embodiments when the computer program is invoked.

[0044] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the container grabbing and positioning method as described in any of the foregoing embodiments.

[0045] Compared to existing technologies, the container grabbing and positioning method, device, industrial control computer, and storage medium provided in this invention utilize a deep learning-based container detection model to detect the coordinates of the container detection frame and the guide block detection frame. Then, based on the container detection frame coordinates, a deep learning-based keyhole detection model is used to detect the keyhole detection frame coordinates, effectively solving the problem of inaccurate detection caused by the small size of the keyhole target. Combining the guide block detection frame coordinates and the keyhole detection frame coordinates to determine the center coordinates of the container to be inspected effectively solves the problem of inaccurate positioning due to keyhole obstruction, further improving detection accuracy, facilitating automated container grabbing, and increasing the efficiency of automated operations.

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic flowchart of a container grabbing and positioning method provided in an embodiment of the present invention is shown.

[0049] Figure 2 A schematic diagram of an image coordinate system provided in an embodiment of the present invention is shown.

[0050] Figure 3 This diagram illustrates another flow chart of the container grabbing and positioning method provided in an embodiment of the present invention.

[0051] Figure 4 A block diagram of a container grabbing and positioning device provided in an embodiment of the present invention is shown.

[0052] Figure 5 A block diagram of an industrial control computer provided in an embodiment of the present invention is shown.

[0053] Icons: 10-Industrial control computer; 110-Memory; 120-Processor; 130-Communication module; 200-Container grabbing and positioning device; 201-Preprocessing module; 202-Detection module; 203-Determination module. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0056] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0057] With the rapid development of port operations, the demand for efficient and automated port operations is becoming increasingly urgent. Among these, automated container handling has become a key link in automated port terminal operations, and the key to container handling is the rapid and accurate location of containers.

[0058] In existing technologies, cameras that capture images of containers are mounted on spreader arms. These cameras photograph the containers from different angles, and traditional visual processing algorithms are used to identify the keyhole information in the images from each angle. This identification process is time-consuming and affects the efficiency of automated operations. Furthermore, as the spreader descends, the keyholes may be obstructed, making it difficult to accurately locate the container and impacting the accuracy of container grabbing.

[0059] Based on this, the container grabbing and positioning method, device, industrial control computer, and storage medium provided in this invention are based on the following core idea: Utilizing a deep learning-based container detection model to detect the coordinates of the container detection frame and the guide block detection frame, and then using a deep learning-based keyhole detection model to detect the keyhole detection frame coordinates based on the container detection frame coordinates, effectively solving the problem of inaccurate detection caused by the keyhole being too small. Combining the guide block detection frame coordinates and the keyhole detection frame coordinates to determine the center coordinates of the container to be inspected effectively solves the problem of inaccurate positioning due to keyhole obstruction, further improving detection accuracy.

[0060] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0061] Please refer to Figure 1 , Figure 1 A schematic flowchart of a container grabbing and positioning method provided by an embodiment of the present invention is shown. This method is applied to an industrial control computer and includes the following steps:

[0062] Step S10: Obtain the box detection image.

[0063] The image captured during container inspection includes the container to be inspected, the guide block to be inspected, and the keyhole to be inspected.

[0064] In this embodiment of the invention, the industrial control computer is connected to the quay crane. Assuming that a vision sensor (e.g., a camera) is installed on the quay crane trolley, the vision sensor moves along the trolley track with the quay crane trolley. After receiving the container grabbing task, the industrial control computer controls the quay crane trolley to move to the position of the vision sensor's field of view, which includes the guide block of the container truck and the container on the container truck. At this time, the vision sensor collects the container grabbing detection image.

[0065] In this embodiment of the invention, the top left corner of the container detection image is taken as the origin of the image coordinate system, the X direction is parallel to the long side of the container, and the Y direction is perpendicular to the long side of the container, as shown below. Figure 2 As shown, the image includes a container truck, a guide block for the container truck, a container inside the container truck, and a lock hole for the container. It should be noted that the image coordinate system can be constructed according to the actual application scenario; however, this invention does not limit this.

[0066] Step S20: Input the container detection image into the pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected.

[0067] The container detection model is obtained by training a deep learning object detection model based on the labeled container bounding boxes and guide block bounding boxes, as well as the coordinates of the corresponding bounding boxes.

[0068] Step S30: Crop the container detection image according to the detection frame coordinates of the container to be inspected to obtain the container image to be inspected.

[0069] Step S40: Input the image of the container to be detected into the pre-trained keyhole detection model to obtain the coordinates of the detection box of the keyhole to be detected.

[0070] The keyhole detection model is obtained by training a deep learning object detection model based on the container image with labeled keyhole bounding boxes and the coordinates of the corresponding bounding boxes.

[0071] In this embodiment of the invention, a container detection model is used to detect containers and guide blocks in a container grabbing detection image, obtaining the detection frame coordinates of the container to be detected and the detection frame coordinates of the guide block to be detected. The container in the corrected image is then cropped to obtain the image of the container to be detected. A keyhole detection model is then used to detect keyholes in the image of the container to be detected, obtaining the detection frame coordinates of the keyholes to be detected.

[0072] It should be noted that the detection frame coordinates of the container to be inspected, the guide block to be inspected, and the lock hole to be inspected are used to describe the detection frames of the container to be inspected, the guide block to be inspected, and the lock hole to be inspected, respectively. Specifically, the detection frame of the container to be inspected represents its position in the container grabbing detection image, the detection frame of the guide block to be inspected represents its position in the container grabbing detection image, and the detection frame of the lock hole to be inspected represents its position in the container grabbing detection image.

[0073] Step S50: Determine the center coordinates of the container to be inspected based on the detection frame coordinates of the lock hole to be inspected and the detection frame coordinates of the guide block to be inspected.

[0074] The center coordinates of the container to be inspected are used to guide the spreader to grab the container.

[0075] In this embodiment of the invention, since the relative position between the guide block on the container truck and the container lock hole remains unchanged during the descent of the spreader, the center coordinates of the container to be inspected can be determined by combining the detection frame coordinates of the lock hole to be inspected and the detection frame coordinates of the guide block to be inspected, so as to guide the spreader to grab the container.

[0076] In summary, the container grabbing and positioning method provided by this invention firstly uses a deep learning-based container detection model to detect the coordinates of the container detection frame and the guide block detection frame. Secondly, based on the container detection frame coordinates, a deep learning-based keyhole detection model is used to detect the keyhole detection frame coordinates, effectively solving the problem of inaccurate detection caused by the keyhole being too small, thus improving detection efficiency and accuracy and avoiding grabbing failure due to excessive detection delay. Finally, the center coordinates of the container to be inspected are determined by combining the guide block detection frame coordinates and the keyhole detection frame coordinates, effectively solving the problem of inaccurate positioning due to keyhole obstruction, further improving detection accuracy, facilitating automated container grabbing, and increasing the efficiency of automated operations.

[0077] Optionally, as another possible implementation, the accuracy of localization detection can be improved by correcting the container grabbing detection image. Before the step of inputting the container grabbing detection image into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected, the following steps are included:

[0078] The distortion of the captured image is corrected according to the preset distortion coefficient.

[0079] In this embodiment of the invention, in addition to directly detecting and recognizing the container grabbing image using a pre-trained container detection model, distortion correction can also be performed on the container grabbing image according to a preset distortion coefficient to obtain a corrected container grabbing image, effectively correcting the distortion at the image edges. The corrected container grabbing image is then input into the pre-trained container detection model to obtain the detection frame coordinates of the container to be detected and the detection frame coordinates of the guide block to be detected. This improves the accuracy of positioning and detection.

[0080] Assuming the camera is used as a visual sensor to acquire images, the camera distortion coefficients, i.e., preset distortion parameters, are obtained based on camera calibration. Distortion correction is then performed on the box-detection images using these preset distortion coefficients. The Taylor formula for calculating the distortion is as follows:

[0081] x dr =x*(1+k1r) 2 +k2r 4 +k3r 6 )

[0082] y dr =y*(1+k1r) 2 +k2r 4 +k3r 6 )

[0083] x dt =2p1xy+p2*(r 2 +2x 2 )+1

[0084] y dt =2p1*(r 2 +2y 2 )+2p2xy+1

[0085] Where (x,y) are the corrected coordinates, (x... dr ,y dr (x) represents the radially distorted coordinates in the box-detection image; dt ,y dt ) represents the coordinates of the captured image after tangential distortion, (k1,k2,k3,p1,p2) represents the preset distortion coefficients, k1, k2 and k3 represent the radial distortion coefficients, p1 and p2 represent the tangential distortion coefficients, and r represents the preset distortion distance.

[0086] Optionally, before performing the container grabbing and positioning process described above, it is necessary to first train a deep learning object detection model to obtain a container detection model and a keyhole detection model. The training process for the container detection model and the keyhole detection model is explained below. Please refer to... Figure 3 , Figure 3 This diagram illustrates another flowchart of the container gripping and positioning method provided in an embodiment of the present invention. The container detection model and the keyhole detection model are obtained in the following manner:

[0087] Step S60: Obtain the training images and their corresponding annotation description files.

[0088] The annotation description file records a one-to-one correspondence between annotation box types and annotation box coordinates. Annotation box types include container type, guide block type, and keyhole type. The training image includes annotated container annotation boxes, guide block annotation boxes, and keyhole annotation boxes.

[0089] In this embodiment of the invention, it is necessary to acquire a large number of images of container trucks loaded with containers, and manually label the containers, guide blocks, and keyholes in the images to obtain corresponding container label boxes, guide block label boxes, and keyhole label boxes in the training images. Labeling refers to drawing a label box at the location containing the container, guide block, or keyhole; the label box can be rectangular or circular. The setting can be determined according to the actual application scenario, and this invention is not limited thereto. This embodiment of the invention uses a rectangular label box as an example for illustration.

[0090] When manually annotating each training image, a corresponding annotation description file is generated. The annotation description file records a one-to-one correspondence between the type of bounding boxes and their coordinates in the corresponding training image.

[0091] Step S70: Input the training image, annotation description file, container type and guide block type into the deep learning object detection model to train the container detection model.

[0092] In this embodiment of the invention, training images, annotation description files, container types, and guide block types are input into a deep learning object detection model so that the deep learning object detection model can detect container bounding boxes and guide block bounding boxes in the training images according to the container type and guide block type. The deep learning object detection model is then iteratively trained based on the detection results and the corresponding bounding box coordinates in the annotation description file to obtain a container detection model.

[0093] Step S80: Crop the training image according to the coordinates of the container annotation box to obtain the container image.

[0094] Step S90: Input the container image, annotation description file, and keyhole type into the deep learning target detection model to train the keyhole detection model.

[0095] In this embodiment of the invention, since the area occupied by the keyhole in the training image is too small, directly using the keyhole bounding boxes marked in the training image as the training target is too small and affects the training effect. Therefore, the training image is cropped into a container image according to the coordinates of the container bounding boxes, and then a deep learning target detection model is trained according to the keyhole bounding boxes marked in the container image to obtain the keyhole detection model.

[0096] It should be noted that the deep learning object detection model can be any of the following: You Only Look Once (YOLO) V3, YOLO V4, or YOLO V5. The deep learning object detection model used to train the container detection model and the deep learning object detection model used to train the keyhole detection model can be the same (e.g., both using YOLO V5) or different; this invention does not limit this.

[0097] Optionally, in practical applications, the container may be tilted in the container truck, and the center coordinates of the container are determined based on the tilt state of the container in the container truck. Figure 1 The sub-steps of step S50 may include:

[0098] The center coordinates of the keyhole are determined based on the detection frame coordinates of the keyhole and the guide block. The center coordinates of the container to be inspected are then determined based on the keyhole center coordinates, the slope of the long side of the container, and the pixel width of the container.

[0099] In this embodiment of the invention, the formula for calculating the center coordinates of the container to be inspected is as follows:

[0100] x′=xh c ×sin(arctanγ)

[0101] y′=y+h c ×cos(arctanγ)

[0102] Where (x',y') are the center coordinates of the container to be inspected; (x,y) are the center coordinates of the lock hole; γ is the slope of the long side of the container to be inspected; h c It is half the pixel width of the container to be inspected, and is a fixed value.

[0103] Optionally, in practical applications, to further reduce the impact of image distortion on the accuracy of keyhole detection, preset filtering conditions are used to screen the guide block and the keyhole to be detected. The step of determining the center coordinates of the keyhole based on the detection frame coordinates of the keyhole and the detection frame coordinates of the guide block can include:

[0104] First, the guide block and keyhole to be tested that meet the preset filtering conditions are identified as the target guide block and target keyhole.

[0105] In this embodiment of the invention, the camera is mounted on the quay crane trolley at a fixed shooting angle, so the distortion in each region of the image captured by the camera is essentially constant. Before the industrial control computer performs the container grabbing and positioning task, the distortion of each region can be analyzed based on the captured image, and preset filtering conditions can be set through a visual interface or a third-party server based on the analysis results.

[0106] Based on the ordinate of the container detection frame in the image, the guide block and keyhole corresponding to the long side of the container are divided into two parts. If the image distortion corresponding to the ordinate of the guide block detection frame being less than that corresponding to the ordinate of the container detection frame is better than that corresponding to the ordinate of the guide block detection frame being greater than that of the container detection frame, the preset filtering condition is set to be less than the ordinate of the container detection frame; otherwise, the preset filtering condition is set to be greater than the ordinate of the container detection frame. The preset filtering condition can be set according to the actual application.

[0107] In this embodiment of the invention, due to errors in the distortion coefficients, the image after distortion correction is often not an ideal image and still has some deformation. A preset filtering condition is used to filter the guide block and keyhole to be detected, resulting in a more ideal corrected guide block and keyhole, which are then used as the target guide block and target keyhole. This optimizes the problem of inaccurate detection caused by distortion correction at long distances in visual sensors, effectively improving detection accuracy.

[0108] Secondly, the center coordinates of the guide blocks are determined based on the detection frame coordinates of all target guide blocks.

[0109] Finally, the target keyhole is determined. When there are two target keyholes, the center coordinates of the keyhole are determined based on the detection frame coordinates of all target keyholes, and the relative coordinate set is updated. The relative coordinate set is used to record the coordinate difference between the center coordinates of the guide block and the center coordinates of the keyhole in each capture box detection image. When there are no two target keyholes, the center coordinates of the keyhole are determined based on the relative coordinate set and the center coordinates of the guide block.

[0110] In this embodiment of the invention, when the keyhole is not obstructed by the spreader and two target keyholes can be detected from the image of the container to be inspected, the center coordinates of the keyhole are directly determined based on the detection frame coordinates of the target keyhole, and the coordinate difference between the center coordinates of the guide block and the center coordinates of the keyhole is updated to the relative coordinate set in sequence. When the keyhole is obstructed by the spreader and two target keyholes cannot be detected from the image of the container to be inspected, there may be one target keyhole or there may be no target keyhole. In this case, it is necessary to combine the center coordinates of the guide block and the relative coordinate set to determine the center coordinates of the keyhole.

[0111] As can be seen, by binding the center of the guide block and the center of the lock hole, the center coordinates of the container can be accurately located even when the lock hole is blocked and the spreader or container truck is moving.

[0112] Optionally, as a possible implementation, the center coordinates of the guide block are determined using the average value of the detection frame coordinates of the target guide blocks. The step of determining the center coordinates of the guide block based on the detection frame coordinates of all target guide blocks may include:

[0113] The abscissa of the center of the guide block is determined by the average abscissa of the detection frame of all target guide blocks; the ordinate of the center of the guide block is determined by the average ordinate of the detection frame of all target guide blocks.

[0114] Optionally, as a possible implementation, the center coordinates of the keyhole are determined using the average of the detection frame coordinates of the target keyholes. The step of determining the center coordinates of the keyholes based on the detection frame coordinates of all target keyholes and updating the relative coordinate set may include:

[0115] The horizontal coordinate of the lock hole center is determined based on the average horizontal coordinate of the detection frames of all target lock holes; the vertical coordinate of the lock hole center is determined based on the average vertical coordinate of the detection frames of all target lock holes; the relative coordinates are determined based on the center coordinates of the guide block and the center coordinates of the lock hole, and the relative coordinates are saved to the relative coordinate set.

[0116] In this embodiment of the invention, relative coordinates are saved into a relative coordinate set according to the order of the horizontal coordinates or the order of the vertical coordinates.

[0117] Optionally, in practical applications, when the lifting device obstructs the keyhole, the center coordinates of the keyhole are determined by utilizing the relatively fixed positional characteristics of the guide block and the keyhole. The steps for determining the center coordinates of the keyhole based on the set of relative coordinates and the center coordinates of the guide block may include:

[0118] The median coordinate of the relative coordinate set is determined as the target relative coordinate; the abscissa of the keyhole center is determined based on the difference between the abscissa of the guide block center coordinate and the abscissa of the target relative coordinate; the ordinate of the keyhole center is determined based on the difference between the ordinate of the guide block center coordinate and the ordinate of the target relative coordinate.

[0119] In this embodiment of the invention, the median coordinate is determined based on the number of relative coordinates in the set of relative coordinates. If the number of relative coordinates is 10, then the 6th relative coordinate is the median coordinate; if the number of relative coordinates is 9, then the 5th relative coordinate is the median coordinate.

[0120] Based on the same inventive concept, this embodiment of the invention also provides a container gripping and positioning device. Its basic principle and the resulting technical effects are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0121] Please refer to Figure 4 , Figure 4 The diagram shows a block illustration of a container gripping and positioning device 200 according to an embodiment of the present invention. The container gripping and positioning device 200 is applied to an industrial control computer and includes a preprocessing module 201, a detection module 202, and a determination module 203.

[0122] Preprocessing module 201 is used to acquire container grabbing detection images; the container grabbing detection images include the container to be inspected, the guide block to be inspected, and the lock hole to be inspected;

[0123] The detection module 202 is used to input the container detection image into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected. The container detection model is obtained by training a deep learning target detection model based on the labeled container and guide block labels, as well as the corresponding label coordinates of the container and guide block labels.

[0124] The detection module 202 is also used to crop the container detection image according to the detection box coordinates of the container to be detected to obtain the container image to be detected; input the container image to be detected into a pre-trained keyhole detection model to obtain the detection box coordinates of the keyhole to be detected; the keyhole detection model is obtained by training a deep learning target detection model based on the container image with the labeled keyhole annotation box and the corresponding annotation box coordinates of the keyhole annotation box;

[0125] The determining module 203 is used to determine the center coordinates of the container to be inspected based on the detection frame coordinates of the lock hole to be inspected and the detection frame coordinates of the guide block to be inspected; the center coordinates of the container to be inspected are used to guide the spreader to grab the container.

[0126] In summary, the container gripping and positioning device provided in this embodiment of the invention firstly uses a deep learning-based container detection model to detect the coordinates of the container detection frame and the guide block detection frame. Secondly, based on the container detection frame coordinates, a deep learning-based keyhole detection model is used to detect the keyhole detection frame coordinates, effectively solving the problem of inaccurate detection caused by the keyhole being too small, thus improving detection efficiency and accuracy and avoiding gripping failure due to excessive detection delay. Finally, the center coordinates of the container to be inspected are determined by combining the guide block detection frame coordinates and the keyhole detection frame coordinates, effectively solving the problem of inaccurate positioning due to keyhole obstruction, further improving detection accuracy, facilitating automated container gripping, and improving the efficiency of automated operations.

[0127] Optionally, the preprocessing module 201 is also used to perform distortion correction on the box-grabbing detection image according to a preset distortion coefficient.

[0128] Optionally, the preprocessing module 201 is further used to acquire training images and corresponding annotation description files; the annotation description files record a one-to-one correspondence between annotation box types and annotation box coordinates; the annotation box types include container type, guide block type, and keyhole type; the training images include labeled container annotation boxes, guide block annotation boxes, and keyhole annotation boxes; the training images, annotation description files, container types, and guide block types are input into a deep learning object detection model to train a container detection model; the training images are cropped according to the container annotation box coordinates to obtain container images; the container images, annotation description files, and keyhole types are input into a deep learning object detection model to train a keyhole detection model.

[0129] Optionally, the determining module 203 is specifically used to determine the center coordinates of the lock hole based on the detection frame coordinates of the lock hole to be detected and the detection frame coordinates of the guide block to be detected; and to determine the center coordinates of the container to be detected based on the center coordinates of the lock hole, the slope of the long side of the container to be detected, and the pixel width of the container to be detected.

[0130] Optionally, the determining module 203 is specifically used to determine the guide block and the keyhole to be detected that meet the preset filtering conditions as the target guide block and the target keyhole; determine the center coordinates of the guide block according to the detection frame coordinates of all target guide blocks; when there are two target keyholes, determine the center coordinates of the keyhole according to the detection frame coordinates of all target keyholes, and update the relative coordinate set; the relative coordinate set is used to record the coordinate difference between the center coordinates of the guide block and the center coordinates of the keyhole in each capture detection image; when there are no two target keyholes, determine the center coordinates of the keyhole according to the relative coordinate set and the center coordinates of the guide block.

[0131] Optionally, the determining module 203 is specifically used to determine the abscissa of the center coordinate of the guide block based on the average abscissa of the detection frame of all target guide blocks; and to determine the ordinate of the center coordinate of the guide block based on the average ordinate of the detection frame of all target guide blocks.

[0132] Optionally, the determining module 203 is specifically used to determine the abscissa of the lock hole center coordinate based on the average abscissa of the detection frame of all target lock holes; determine the ordinate of the lock hole center coordinate based on the average ordinate of the detection frame of all target lock holes; determine the relative coordinate based on the center coordinate of the guide block and the center coordinate of the lock hole, and save the relative coordinate to the relative coordinate set.

[0133] Optionally, the determining module 203 is specifically used to determine the median coordinate of the relative coordinate set as the target relative coordinate; determine the abscissa of the lock hole center coordinate based on the difference between the abscissa of the guide block center coordinate and the abscissa of the target relative coordinate; and determine the ordinate of the lock hole center coordinate based on the difference between the ordinate of the guide block center coordinate and the ordinate of the target relative coordinate.

[0134] Please refer to Figure 5 , Figure 5 This is a block diagram of an industrial control computer 10 provided in an embodiment of the present invention. The industrial control computer 10 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0135] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0136] The processor 120 is used to read / write data or programs stored in the memory 110 and perform corresponding functions. For example, when a computer program stored in the memory 110 is executed by the processor 120, the container grabbing and positioning method disclosed in the above embodiments can be implemented.

[0137] The communication module 130 is used to establish a communication connection between the industrial control computer 10 and other communication terminals via the network, and to send and receive data via the network.

[0138] It should be understood that, Figure 5 The structure shown is only a schematic diagram of the industrial computer 10. The industrial computer 10 may also include components such as... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown. Figure 5 The components shown can be implemented using hardware, software, or a combination thereof.

[0139] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor 120, implements the container grabbing and positioning method disclosed in the above embodiments.

[0140] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0141] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0142] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of positioning a container gripping device, characterized in that, Applied to industrial control computers, the method includes: Acquire a container grabbing detection image; the container grabbing detection image includes the container to be inspected, the guide block to be inspected, and the lock hole to be inspected; The captured container detection image is input into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected. The container detection model is obtained by training a deep learning object detection model based on the labeled container and guide block labels, as well as the corresponding label coordinates of the container and guide block labels. The container detection image is cropped according to the detection frame coordinates of the container to be inspected to obtain the container image to be inspected; The container image to be detected is input into a pre-trained keyhole detection model to obtain the coordinates of the detection box of the keyhole; the keyhole detection model is obtained by training a deep learning object detection model based on the container image with labeled keyhole boxes and the coordinates of the corresponding labeled boxes. The center coordinates of the container to be inspected are determined based on the detection frame coordinates of the keyhole to be inspected and the detection frame coordinates of the guide block to be inspected; the center coordinates of the container to be inspected are used to guide the spreader to grab the container.

2. The method of claim 1, wherein, Before the step of inputting the container detection image into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected, the following steps are included: The distortion correction is performed on the box detection image according to the preset distortion coefficient.

3. The method of claim 1, wherein, The container inspection model and the keyhole inspection model were obtained in the following ways: Acquire training images and corresponding annotation description files; the annotation description files record a one-to-one correspondence between annotation box types and annotation box coordinates; the annotation box types include container type, guide block type, and keyhole type; the training images include labeled container annotation boxes, guide block annotation boxes, and keyhole annotation boxes; The training image, the annotation description file, the container type, and the guide block type are input into the deep learning object detection model to train the container detection model. The training image is cropped based on the coordinates of the container bounding boxes to obtain the container image; The container image, the annotation description file, and the keyhole type are input into the deep learning target detection model to train the keyhole detection model.

4. The method of claim 1, wherein, The step of determining the center coordinates of the container to be inspected based on the detection frame coordinates of the keyhole to be inspected and the detection frame coordinates of the guide block to be inspected includes: The center coordinates of the keyhole are determined based on the detection frame coordinates of the keyhole to be detected and the detection frame coordinates of the guide block to be detected. The center coordinates of the container to be inspected are determined based on the center coordinates of the keyhole, the slope of the long side of the container to be inspected, and the pixel width of the container to be inspected.

5. The method of claim 4, wherein, The step of determining the center coordinates of the keyhole based on the detection frame coordinates of the keyhole to be detected and the detection frame coordinates of the guide block to be detected includes: The guide block to be detected and the keyhole to be detected that meet the preset filtering conditions are identified as the target guide block and the target keyhole; The center coordinates of the guide blocks are determined based on the detection frame coordinates of all the target guide blocks. When there are two target keyholes, the center coordinates of the keyholes are determined based on the detection frame coordinates of all the target keyholes, and the relative coordinate set is updated; the relative coordinate set is used to record the coordinate difference between the center coordinates of the guide block and the center coordinates of the keyhole in each of the box-grabbing detection images; When there are no two target keyholes, the center coordinates of the keyholes are determined based on the relative coordinate set and the center coordinates of the guide block.

6. The method of claim 5, wherein, The step of determining the center coordinates of the guide block based on the detection frame coordinates of all the target guide blocks includes: The abscissa of the center coordinate of the guide block is determined based on the average abscissa of the detection frame of all the target guide blocks. The ordinate of the center coordinate of the guide block is determined based on the average value of the ordinate of the detection frame of all the target guide blocks.

7. The method of claim 5, wherein, The step of determining the center coordinates of the keyhole based on the detection frame coordinates of all the target keyholes and updating the relative coordinate set includes: The horizontal coordinate of the center of the keyhole is determined based on the average value of the horizontal coordinates of the detection frames of all the target keyholes. The ordinate of the center coordinate of the keyhole is determined based on the average ordinate of the detection frames of all the target keyholes. The relative coordinates are determined based on the center coordinates of the guide block and the center coordinates of the keyhole, and the relative coordinates are saved to the relative coordinate set.

8. The method of claim 5, wherein, The step of determining the center coordinates of the keyhole based on the relative coordinate set and the center coordinates of the guide block includes: The median coordinate of the set of relative coordinates is determined as the target relative coordinates; The abscissa of the lock hole center is determined based on the difference between the abscissa of the center coordinate of the guide block and the abscissa of the relative coordinate of the target. The ordinate of the lock hole center is determined based on the difference between the ordinate of the guide block center coordinate and the ordinate of the target relative coordinate.

9. A container gripping and positioning device, characterized in that Applied to industrial control computers, the device includes: The preprocessing module is used to acquire container grabbing detection images; the container grabbing detection images include the container to be inspected, the guide block to be inspected, and the lock hole to be inspected; The detection module is used to input the captured container detection image into a pre-trained container detection model to obtain the detection box coordinates of the container to be detected and the detection box coordinates of the guide block to be detected; the container detection model is obtained by training a deep learning target detection model based on the labeled container and guide block labels, as well as the corresponding label coordinates of the container and guide block labels. The detection module is also used to crop the container detection image according to the detection box coordinates of the container to be detected to obtain the container image to be detected; input the container image to be detected into a pre-trained keyhole detection model to obtain the detection box coordinates of the keyhole to be detected; the keyhole detection model is obtained by training a deep learning target detection model based on the container image with labeled keyhole labels and the corresponding label box coordinates of the keyhole labels; The determining module is used to determine the center coordinates of the container to be inspected based on the detection frame coordinates of the lock hole to be inspected and the detection frame coordinates of the guide block to be inspected; the center coordinates of the container to be inspected are used to guide the spreader to grab the container.

10. An industrial computer, characterized by The industrial control computer includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the container grabbing and positioning method as described in any one of claims 1-8 when the computer programs are invoked.

11. A computer readable storage medium having stored thereon a computer program, characterized in that When the computer program is executed by the processor, it implements the container grabbing and positioning method as described in any one of claims 1-8.