Container lock hole identification and positioning method, system and storage medium

Abstract

The present invention discloses a container lock hole identification and positioning method, system and storage medium, wherein the method comprises the following steps: S1: using a set of diagonal stereo cameras set on the container spreader to continuously collect the top view plane image of the container; S2: using an optimized deep learning target detection model to detect the lock hole position in the top view plane image for real-time identification, tracking and positioning, and according to the lock hole position obtained by identification, controlling the spreader to descend and move toward the center of the container at the same time, until the height of the spreader from the container is less than a set threshold; S3: the stereo camera collects the depth image of the container, pre-processes the depth image, and determines the position of the four corners of the container by edge detection; S4: according to the geometric relationship between the four corners of the container and the center of the lock hole, calculate the center position of the lock hole. The above method infers the lock hole position according to the edge detection of the container depth image, and the inferred lock hole position has higher accuracy and is less affected by complex outdoor scenes and light environments.

Description

Technical Field

[0001] The invention belongs to the technical field of port automation, and in particular relates to a container lock hole identification and positioning method, system and storage medium. Background Art

[0002] Automated ports adopt unmanned management and operation modes, which not only saves labor costs, but also has high efficiency in automated cargo transfer and management, which can greatly increase the cargo throughput of ports, thereby achieving effective growth in economic benefits. Container grabbing is an important part of the cargo transfer process in automated ports. It is difficult to achieve and has high technical complexity. The key is the identification and positioning of the container lock hole. If manual operation is used, it not only requires high proficiency of workers, but also easily causes worker fatigue, posing a greater safety risk.

[0003] At present, automated ports mainly use high-precision radar to identify and locate container locks. Although this method is effective, it is expensive and not conducive to large-scale promotion. The method based on machine vision is cheaper, but it is difficult to accurately identify and locate the locks with high precision in the face of complex outdoor scenes and light environments, which limits the application of automatic container grabbing technology. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to propose a container keyhole identification and positioning method, which can improve the position recognition accuracy of the keyhole and can adapt to complex outdoor scenes and light environments.

[0005] Another object of the present invention is to provide a system that can implement the above container lock hole identification and positioning method, and a storage medium storing a computer program that instantiates the above identification and positioning method.

[0006] Technical solution: The container lock hole identification and positioning method of the present invention comprises the following steps:

[0007] S1: Using a set of diagonal stereo cameras set on the container spreader to continuously collect the top view plane image of the container;

[0008] S2: Use the optimized deep learning target detection model to detect the keyhole position in the top-view plane image for real-time identification, tracking and positioning. According to the keyhole position obtained by identification, the spreader is controlled to descend and move toward the center of the container until the height of the spreader from the container is less than the set threshold.

[0009] S3: The stereo camera collects the depth image of the container, pre-processes the depth image, and determines the positions of the four corners of the container through edge detection;

[0010] S4: Calculate the center position of the keyhole according to the geometric relationship between the four corners of the container and the center of the keyhole.

[0011] Furthermore, the optimization performed by the deep learning target detection model in step S2 includes: constraining the aspect ratio of the anchor frame with the length-to-width ratio of the standard container key hole, and calculating the size of the key hole in the image based on the sling height, the number of layers where the container is located, and the stereo camera parameters, and constraining the size of the anchor frame with this size.

[0012] Furthermore, the optimization performed by the deep learning target detection model in step S2 also includes: performing key point detection on the acquired planar image, determining the location of anchor frame generation according to the key point distribution density cluster center, and generating a corresponding number of anchor frames.

[0013] Furthermore, the deep learning target detection model in step S2 is a YOLO V4 target detection model.

[0014] Furthermore, the step S2 comprises:

[0015] S2.1: Two stereo cameras collect a top-down plane image of the container, and use a deep learning object detection model to identify the keyhole position in the plane image;

[0016] S2.2: Calculate the positional relationship between the center of the spreader and the center of the container based on the spreader height and the position of the lock hole;

[0017] S2.3: Control the spreader to descend and move toward the center of the container according to the positional relationship between the spreader center and the container center, collect the top view plane image of the container again, and calculate the position range of the lock hole on the plane image according to the spreader displacement;

[0018] S2.4: Use the deep learning target detection model to identify the keyhole position within the estimated position range, and return to step S2.2 until the height of the spreader from the container is less than the set threshold.

[0019] The container lock hole identification and positioning system of the present invention comprises a controller and at least two stereo cameras, wherein the two stereo cameras are electrically connected to the controller and are respectively arranged at a set of diagonal corners of the container spreader.

[0020] The storage medium of the present invention stores a computer program, and the computer program is configured to implement the container lock hole identification and positioning method when running.

[0021] Beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. The spreader is guided to move by a deep learning target detection model, and a stereo camera is used to collect depth images and identify the edge of the container, and the position of the keyhole is calculated based on the edge of the container. Since the depth image recognizes the edge of the container with higher accuracy and reliability, and the container specifications are standard, the calculated position of the keyhole is more accurate and less affected by complex outdoor scenes and light environments. 2. The position range of the keyhole in the image is calculated based on the spreader movement, and the size of the target detection model anchor frame is adjusted in real time according to the spreader height. The anchor frame generation position and quantity are determined according to the distribution of key points, which improves the accuracy of target tracking and effectively reduces the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a container lock hole identification and positioning method according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the installation position of a stereo camera according to an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a method for tracking a keyhole position according to an embodiment of the present invention;

[0025] Figure 4 A top view of the container. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0027] Reference Figure 1 According to an embodiment of the present invention, a container lock hole identification and positioning method comprises the following steps:

[0028] S1: Using a set of diagonal stereo cameras set on the container spreader to continuously collect the top view plane image of the container;

[0029] S2: Use the optimized deep learning target detection model to detect the keyhole position in the top-view plane image for real-time identification, tracking and positioning. According to the keyhole position obtained by identification, the spreader is controlled to descend and move toward the center of the container until the height of the spreader from the container is less than the set threshold.

[0030] S3: The stereo camera collects the depth image of the container, pre-processes the depth image, and determines the positions of the four corners of the container through edge detection;

[0031] S4: Calculate the center position of the keyhole according to the geometric relationship between the four corners of the container and the center of the keyhole.

[0032] Since the depth information near the corners of the container will change dramatically, and the structure of the container meets the corresponding standards, the positional relationship between the keyhole and the corners of the container is clear and accurate, such as Figure 4 Therefore, according to the above characteristics, a set of stereo cameras are installed at a set of diagonal angles of the spreader to collect the depth images of a set of diagonal angles of the container, as shown in Figure 2 As shown, by identifying the corner point positions of the depth image of the container and then inferring the position of the keyhole of the container according to the structural geometry of the container, it is possible to overcome the noise influence caused by outdoor scenes and complex light environments, and the detection reliability and accuracy are higher, which can assist the lifting equipment to lift the container through the keyhole more accurately.

[0033] And because the working principle of stereo cameras is usually binocular parallax, synchronous focusing is required, and the anti-interference ability is poor, it is not suitable for tracking the position of the keyhole during the movement of the spreader. Therefore, in the early stage, it is only necessary to let the stereo camera collect the plane image of the container, track the position of the keyhole in the collected plane image through the deep learning target detection model, calculate the relative position of the spreader and the center of the container and feed it back to the controller, control the spreader to descend and move to the center of the container. The deep learning target detection model can use convolutional neural network target detection models such as YOLO V3, YOLO V4 or Faster RCNN.

[0034] In this embodiment, the deep learning target detection model adopts an improved YOLO V4 target detection model, and the training method is as follows:

[0035] Collect a large number of 2D images of containers at different heights, different time periods, and different lighting environments;

[0036] The keyholes in the collected images are manually annotated using rectangular frames, and a container keyhole target detection dataset is established;

[0037] The YOLO V4 target detection model is trained and tested multiple times using the established dataset, and the model with the best test structure is selected as the detection model.

[0038] At the same time, in order to improve the adaptability of the YOLO V4 target detection model to the container keyhole recognition scenario and improve the recognition speed and accuracy, the aspect ratio, size, generation position and number of the anchor boxes generated by the YOLO V4 target detection model are optimized as follows according to the shape and size of the keyhole:

[0039] a) Since the length of the standard keyhole is 178 mm × 162 mm and the aspect ratio is about 1:1.099, the aspect ratio of the anchor frame is constrained to 1:1 to 1:1.2;

[0040] b) When the spreader is at different heights, the size of the keyhole in the acquired plane image is calculated according to the spreader height, the number of layers where the container is located, and the camera parameters. The size of the anchor frame is set according to the size of the keyhole in the image, and a 10% error is allowed;

[0041] c) Use the ORB (Oriented Fast and Rotated Brief) algorithm to detect key points in the images captured by the camera, determine the anchor frame generation position based on the distribution of key points, cluster the key point positions where the key point density reaches a certain threshold, calculate the cluster center, and generate a corresponding number of anchor frames based on the key point density of the area with the cluster center as the center. The higher the key point density, the more anchor frames are generated, and the lower the density, the fewer anchor frames are generated. No anchor frames are generated in areas without key points, which reduces the amount of calculation and improves tracking efficiency.

[0042] At the same time, in order to further improve the accuracy of target tracking and reduce the amount of calculation, such as Figure 3 As shown in FIG. 1 , during the descent of the spreader, the position range of the keyhole in the image after the movement is calculated according to the movement amount of the spreader, as shown in FIG. Figure 3 As shown in the left figure, the YOLO V4 target detection model is then used to identify the keyhole within the inferred position range.

[0043] In practice, due to the poor accuracy and anti-interference ability of the stereo camera, the depth value of some areas in the acquired depth image is obviously missing, so it is necessary to perform image enhancement and edge detection on the depth image to determine the position of the container corner. In this embodiment, based on the color image taken at the same time as the depth image, a joint bilateral filtering method is used to filter and fill holes in the depth image, and the Canny operator is used to detect the edges with high significance in the enhanced depth image, and the intersection of the edge fitting straight line is used as the position of the container corner.

[0044] According to the container lock hole identification system of the embodiment of the present invention, Figure 2 As shown, it includes a controller and at least two stereo cameras, the controller is connected to the two stereo cameras, and collects container images taken by the stereo cameras. The two stereo cameras are respectively set at a set of diagonal angles of the spreader, and the controller can be shared with the spreader controller. In this embodiment, the stereo camera uses Intel's RealSense D435i. The storage medium according to the embodiment of the present invention stores a computer program instantiated with the above container keyhole identification and positioning method.

Claims

1. A container lock hole identification and positioning method, characterized in that: The steps include: S1: Using a set of diagonal stereo cameras set on the container spreader to continuously collect the top view plane image of the container; S2: Use the optimized deep learning target detection model to detect the keyhole position in the top-view plane image for real-time identification, tracking and positioning. According to the keyhole position obtained by identification, the spreader is controlled to descend and move toward the center of the container until the height of the spreader from the container is less than the set threshold. S3: The stereo camera collects the depth image of the container, pre-processes the depth image, and determines the positions of the four corners of the container through edge detection; S4: Calculate the center position of the keyhole according to the geometric relationship between the four corners of the container and the center of the keyhole; The step S3 comprises: S3.1: Two stereo cameras each collect a depth image of the container; S3.2: Based on the color images collected by two stereo cameras at the same time, the depth image is filtered and the holes are filled using the joint bilateral filtering method; S3.3: Use the Canny operator to detect the edges with high significance in the processed depth image, and identify the intersection of the edge fitting straight lines as the location of the container corner; S3.4: Establish a two-dimensional coordinate system based on the positions of one set of diagonals obtained, and deduce the positions of another set of diagonals.

2. The container lock hole identification and positioning method according to claim 1, characterized in that: The optimization performed by the deep learning target detection model in step S2 includes: constraining the aspect ratio of the anchor frame with the length-to-width ratio of the standard container keyhole, and calculating the size of the keyhole in the image based on the sling height, the number of layers where the container is located, and the stereo camera parameters, and constraining the size of the anchor frame with this size.

3. The container lock hole identification and positioning method according to claim 2, characterized in that: The optimization performed by the deep learning target detection model in step S2 also includes: performing key point detection on the acquired planar image, determining the location of anchor frame generation according to the key point distribution density cluster center, and generating a corresponding number of anchor frames.

4. The container lock hole identification and positioning method according to claim 1, characterized in that: The deep learning target detection model in step S2 is a YOLO V4 target detection model.

5. The container lock hole identification and positioning method according to claim 1, characterized in that: The step S2 comprises: S2.1: Two stereo cameras collect a top-down plane image of the container, and use a deep learning object detection model to identify the keyhole position in the plane image; S2.2: Calculate the positional relationship between the center of the spreader and the center of the container based on the spreader height and the position of the lock hole; S2.3: Control the spreader to descend and move toward the center of the container according to the positional relationship between the spreader center and the container center, collect the top view plane image of the container again, and calculate the position range of the lock hole on the plane image according to the spreader displacement; S2.4: Use the deep learning target detection model to identify the keyhole position within the estimated position range, and return to step S2.2 until the height of the spreader from the container is less than the set threshold.

6. A container lock hole identification and positioning system for implementing the container lock hole identification and positioning method according to any one of claims 1 to 5, characterized in that: It comprises a controller and at least two stereo cameras, wherein the two stereo cameras are electrically connected to the controller and are respectively arranged on a group of diagonal corners of the container spreader.

7. A storage medium storing a computer program, characterized in that: The computer program is configured to implement the container lock hole identification and positioning method according to any one of claims 1 to 5 when running.

Images