An automatic container keyhole recognition and positioning system and method

Abstract

The present invention relates to an automatic identification and positioning system and method for container keyholes, including a data acquisition system, a control system and related algorithms. The data acquisition system includes a monocular camera and a laser range finder; the control system and related algorithms mainly have algorithms built into an industrial control computer, and a wire-pulling sensor is used to obtain the actual position of the spreader to assist in position matching. The image information and distance information of the keyhole on one side of the container are respectively collected through the monocular camera and the laser range finder. After algorithm parsing, the keyhole coordinates on this side are obtained, and the keyhole coordinates on the other side of the container body are obtained in combination with the container size. According to the deviation between the keyhole and the lock catch position coordinates, the position of the spreader is adjusted to make the keyhole match the lock catch position of the spreader, and the work of locking and grasping the container is completed. Compared with the prior art, the present invention has the advantages of fast and accurate identification and positioning, and is applicable to reach stackers with a very limited effective installation range, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of container loading and unloading by reach trucks, and in particular to an automatic container keyhole recognition and positioning system and method. Background Art

[0002] Under the background of economic globalization, container logistics transportation has grown rapidly. Traditional container terminals rely on manual operations, with low work efficiency, high error rates, and inability to fully guarantee the personal safety of workers, which are no longer suitable for the current rapid development of social economy. To achieve efficient container terminal loading and unloading operations and minimize the time that ships stay in port, efforts have been made around the world to improve the automation and intelligence level of terminals to enhance the work efficiency of container loading and unloading at terminals. In 1993, the ECT terminal in Rotterdam Port, the Netherlands, was put into operation. Since then, the exploration and practice of automated terminals have been continuously deepened, and many achievements have been made. In recent years, China has also accelerated the transformation and upgrading of ports to be intelligent. Since 2017, the Mawan Smart Port, Xiamen Ocean Shipping Terminal, and Qingdao Port have successively started the planning and construction of automated container terminals.

[0003] Among them, the automation and intelligence transformation of reach trucks is particularly important for the work of loading and unloading containers at terminals. During the process of loading and unloading containers by reach trucks, the grabbing action is completed by quickly and accurately aligning and locking the spreader with two keyholes above the front of the container. In the traditional container grabbing method, the reach truck driver manually adjusts the positions of the reach truck and the spreader based on experience to align the keyholes with the lock catches and lock them. This method not only has low work efficiency but also requires high skills from the driver. Prolonged work is likely to cause fatigue to the driver, thus increasing the probability of accidents.

[0004] Patent application CN114863250A discloses a container keyhole recognition and positioning method, system, and storage medium. It uses a stereo camera to continuously collect container images, detects the keyhole positions in the images through a deep learning object detection model, and real-time recognizes and tracks the positions. The spreader moves towards the center of the container according to the keyhole positions until the distance from the container height is less than a set threshold. Then, the stereo camera collects the depth image of the container and preprocesses it. The positions of the four corners of the container are determined through edge detection. According to the positional relationship between the four corners and the keyhole positions, the center position of the keyhole is calculated. This positioning method is suitable for outdoor complex scenarios and light environments and can track the target in real time. However, only one position matching is performed, and the matching accuracy is not high. Moreover, the installation of the stereo camera and the like cannot be applied to reach trucks with a limited effective installation range.

[0005] Patent application CN113213340A discloses a container unloading method, system, device and storage medium based on keyhole recognition. It uses a camera on the spreader to collect keyhole pictures, establishes a plane coordinate system, obtains the alignment error information between the keyhole center and the preset hoisting positioning point, and then the container is moved forward and backward by the container truck to perform alignment, realizing the coordinated loading and unloading operation of the driverless container truck and the driverless crane, meeting the container truck alignment function. However, it only performs one position matching, cannot achieve the precise matching of the keyhole and the lock position, and is not applicable to a reach stacker with a limited effective installation range. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects existing in the above-mentioned prior art and provide a container keyhole automatic recognition and positioning system and method, which can quickly and accurately identify and position the container keyhole and is applicable to a reach stacker with a limited effective installation range.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] The present invention provides a container keyhole automatic recognition and positioning system, including a data acquisition system and a control system;

[0009] The data acquisition system includes a monocular camera and a ranging sensor. The monocular camera is used to collect the RGB image of the target container, and the ranging sensor is used to collect the horizontal distance information from the ranging sensor to the target container. The ranging sensor is preferably a laser ranging sensor.

[0010] The control system includes a reach stacker keyhole recognition and positioning industrial computer and a wire rope sensor;

[0011] The reach stacker keyhole recognition and positioning industrial computer is built-in with a visual computing algorithm, a sensor data fusion algorithm, and a vector coordinate calculation algorithm. The visual computing algorithm is used to predict the keyhole area in the target container according to the RGB image of the target container collected by the monocular camera and process the predicted keyhole area picture to obtain the image pixel coordinates of the keyhole center of the target container. The vector coordinate calculation algorithm is a general algorithm for calculating and processing coordinate data in the prior art. The vector coordinate calculation algorithm converts the image pixel coordinates and the horizontal distance information collected by the ranging sensor into the world coordinate system. The sensor data fusion algorithm is an algorithm in the prior art for integrating and combining the data of multiple sensors. Since different sensors have different measurement errors, sampling frequencies, and data accuracies, the sensor data fusion algorithm can be used to improve the data accuracy and integrity, so as to better reflect the actual situation. Through the sensor data fusion algorithm, the pixel coordinate information and the horizontal distance information are fused to obtain the three-dimensional coordinate value of the keyhole center of the target container in the world coordinate system;

[0012] The wire-pulling sensor determines the actual positions of the spreader lock and the lock hole by detecting the telescopic degree of the oil cylinder, and secondarily calibrates and matches the positions. Among them, the spreader is installed on the inner gantry track in front of the stacker truck body, and the lifting and left-right telescoping of the spreader can be controlled by controlling the telescopic of the oil cylinder.

[0013] Further, the monocular camera is arranged on the handrail of the stairs on one side of the stacker truck cab. The axial direction of the lens of the monocular camera faces the outside of the stacker truck and forms an angle of 30° with the driving direction of the stacker truck. It is powered by the on-vehicle power supply of the stacker truck. The ranging sensor is arranged on the side of the vehicle lamp on one side of the handrail of the stacker truck stairs and is parallel to the horizontal plane. It is powered by the on-vehicle power supply of the stacker truck;

[0014] Further, the industrial control computer for identifying and positioning the lock hole of the stacker truck is arranged on the side of the stacker truck body on one side of the handrail of the stacker truck stairs, and the wire-pulling sensor is arranged beside the gantry of the stacker truck.

[0015] Further, the control system further includes an electric control cabinet. The industrial control computer for identifying and positioning the lock hole of the stacker truck is arranged inside the electric control cabinet, and the electric control cabinet is arranged on the side of the stacker truck body on one side of the handrail of the stairs.

[0016] Further, the visual computing algorithm is the SSD deep learning object detection model algorithm and the traditional image processing method provided by the OpenCV computer vision library.

[0017] The present invention provides a method for automatically identifying and positioning the lock hole of a container. The steps include:

[0018] S1: The monocular camera collects the container picture, and predicts the container lock hole area through the deep learning object detection model algorithm. The deep learning object detection model algorithm is preferably the SSD deep learning object detection model algorithm;

[0019] S2: Process the predicted container lock hole area image through the traditional image processing method provided by the OpenCV computer vision library to obtain the image pixel coordinates of the center of the unilateral lock hole of the target container;

[0020] S3: Measure the horizontal distance from the laser ranging sensor to the lock hole of the target container through the laser ranging sensor;

[0021] S4: Convert the obtained image pixel coordinates of the center of the unilateral lock hole of the target container and the horizontal distance from the laser ranging sensor to the lock hole of the target container to the world coordinate system through the vector coordinate calculation algorithm, and fuse the image pixel coordinate information and the horizontal distance information through the sensor data fusion algorithm to obtain the three-dimensional coordinate value of the center of the unilateral lock hole of the target container in the world coordinate system;

[0022] S5: Combine the known dimensions of the container fixation to calculate the three-dimensional coordinate values of the center of the lock hole on the other side of the target container in the world coordinate system;

[0023] S6: Combine the three-dimensional coordinate values of the known spreader lock in the world coordinate system to obtain the three-dimensional coordinate difference between the center of the lock hole and the lock, and adjust the telescopic degree of the oil cylinder according to this difference, and adjust the telescopic degrees of the lifting oil cylinder and the swing oil cylinder to preliminarily match the positions of the lock hole and the lock;

[0024] S7: Measure the actual position after the oil cylinder adjustment through the wire-pulling sensor, and perform precise position matching on the positions of the lock hole and the lock again until the positions of the lock hole and the lock are completely matched, and complete the locking and container-grabbing actions.

[0025] Further, the specific steps of predicting the container lock hole area through the deep learning object detection model algorithm include:

[0026] S11: Calibrate the monocular camera through the Zhang's calibration method to obtain the internal parameters, external parameters and distortion parameters of the camera;

[0027] S12: Before recognition, use the monocular camera to collect a large number of container sample pictures, manually mark the lock hole area and non-lock hole area in them as positive and negative samples, and train them through the deep learning network framework;

[0028] S13: Use the trained network model to predict the lock hole area in the container picture taken during the recognition process.

[0029] Further, the specific steps of processing the predicted container lock hole area image through the traditional image processing method provided by the OpenCV computer vision library include:

[0030] S21: Perform image preprocessing of image enhancement and noise reduction on the predicted lock hole area picture;

[0031] S22: Perform binaryzation processing on the preprocessed image to find all contours;

[0032] S23: Fit the contour of the target container lock hole with the minimum bounding rectangle;

[0033] S24: Use the center point coordinates of the minimum bounding rectangle as the pixel coordinates of the target container lock hole position.

[0034] By installing a lock hole automatic recognition and positioning system on the reach stacker, it can automatically recognize the lock holes of the target container to be grabbed, obtain the three-dimensional coordinate information of the lock holes, adjust the positions of the reach stacker and the spreader according to the coordinate information, and complete the lock dropping and container grabbing operations, enabling the reach stacker to quickly and efficiently complete the position matching between the spreader lock and the container lock hole, and complete the lock dropping and container grabbing actions, thus greatly improving the working efficiency of the reach stacker, reducing the working intensity of the reach stacker driver, and reducing the occurrence of accidents.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] (1) Fully automatic lock hole recognition and guiding positioning, greatly improving the working efficiency of loading and unloading containers, reducing the working intensity of the reach stacker driver, and reducing the occurrence of accidents.

[0037] (2) Quickly and accurately recognize and position the container lock holes. It can automatically recognize the lock holes of the target container to be grabbed, obtain the three-dimensional coordinate information of the lock holes, adjust the positions of the reach stacker and the spreader according to the coordinate information, and complete the lock dropping and container grabbing operations, enabling the reach stacker to quickly and efficiently complete the position matching between the spreader lock and the container lock hole, and complete the lock dropping and container grabbing actions.

[0038] (3) On a reach stacker with a very limited effective installation range, it can accurately obtain image and distance information and obtain the lock hole coordinates on both sides. The installation of the monocular camera, the reach stacker lock hole recognition and positioning industrial control computer, and the wire rope displacement sensor is simple and flexible, suitable for different application scenarios, and the monocular camera is simpler to install and easier to integrate than the stereo camera. Description of the Drawings

[0039] Figure 1 It is a schematic structural diagram of the container lock hole automatic recognition and positioning system.

[0040] Figure 2 It is a working flow chart of the container lock hole automatic recognition and positioning system.

[0041] Reference numerals: 1 - monocular camera; 2 - laser ranging sensor; 3 - electric control cabinet; 4 - wire rope displacement sensor. Detailed Embodiments

[0042] The present invention will be described in detail below with reference to the drawings and specific embodiments. In the technical solution of the present invention, features such as component models, material names, connection structures, control methods, algorithms, etc. that are not clearly described are regarded as common technical features disclosed in the prior art.

[0043] Embodiment 1

[0044] This embodiment provides a container lock hole automatic recognition and positioning system, as Figure 1 shown, including a data acquisition system, a control system, and related algorithms.

[0045] Among them, the data acquisition system includes a monocular camera 1 and a laser range finder 2: The monocular camera 1 is installed on the handrail of the right staircase of the reach stacker cab. The lens axis of the monocular camera faces outward of the reach stacker, forming a 30° angle with the driving direction of the reach stacker, and is powered by the on-vehicle power supply of the reach stacker, and is used to collect the RGB image of the target container; The laser range finder 2 is arranged on the left side of the reach stacker head, parallel to the horizontal plane, and is powered by the on-vehicle power supply of the reach stacker, and is used to collect the distance information from the sensor to the target container.

[0046] The control system and related algorithms include the reach stacker lock hole recognition and positioning industrial computer and the wire rope sensor 4. The reach stacker lock hole recognition and positioning industrial computer is installed inside the electric control cabinet 3 located on the left side of the reach stacker body. The reach stacker lock hole recognition and positioning industrial computer is built-in with vision calculation algorithms, sensor data fusion algorithms and vector coordinate calculation algorithms; The wire rope sensor 4 is arranged beside the reach stacker gantry, and determines the actual position of the spreader and the lock hole by detecting the telescopic degree of the oil cylinder, and is used to assist in completing the precise position matching between the container lock hole and the buckle on the spreader.

[0047] This embodiment provides a method for automatically identifying and positioning a container lock hole, as Figure 2 shown, including the following steps:

[0048] S1: Use the monocular camera 1 to collect the RGB picture of the container, and train and predict the container lock hole area through the SSD deep learning algorithm;

[0049] S2: Process the predicted container lock hole area image through the traditional image processing method provided by the OpenCV computer vision library, including: performing image preprocessing of image enhancement and noise reduction on the predicted lock hole area picture; performing binaryzation processing on the preprocessed image to find all contours; fitting the container lock hole contour with the minimum bounding rectangle; using the center point coordinates of the minimum bounding rectangle as the pixel coordinates of the target container lock hole position;

[0050] S3: Use the laser range finder 2 to measure the horizontal distance from the laser range finder 2 to the target container lock hole;

[0051] S4: Convert the image pixel coordinates and the horizontal distance to the world coordinate system through the vector coordinate calculation algorithm, and fuse the pixel coordinates and the horizontal distance through the sensor data fusion algorithm to obtain the three-dimensional coordinate value of the center of the target container single-side lock hole in the world coordinate system;

[0052] S5: Combine the known fixed size of the container to calculate the three-dimensional coordinate value of the center of the lock hole on the other side of the target container in the world coordinate system;

[0053] S6: Combine the three-dimensional coordinate values of the known spreader lock in the world coordinate system to obtain the three-dimensional coordinate difference between the center of the lock hole and the lock, and adjust the telescopic degree of the oil cylinder according to this difference to preliminarily match the positions of the lock hole and the lock.

[0054] S7: Measure the actual position of the oil cylinder after adjustment through the wire-pulling sensor 4, and perform another precise position matching on the positions of the lock hole and the lock until the positions of the lock hole and the lock are completely matched, completing the locking and container-grabbing actions.

[0055] The above description of the embodiments is for the convenience of those of ordinary skill in the art to understand and use the invention. It is obvious that those who are familiar with the technology in this field can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative labor. Therefore, the present invention is not limited to the above embodiments, and the improvements and modifications made by those skilled in the art without departing from the scope of the present invention according to the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. An automatic recognition and positioning system for container keyholes, characterized in that, It includes a data acquisition system and a control system; The data acquisition system includes a monocular camera and a ranging sensor. The monocular camera is used to collect the RGB image of the target container, and the ranging sensor is used to collect the horizontal distance information from the ranging sensor to the target container; The control system includes an industrial computer for identifying and positioning the keyhole of the stacker and a wire rope sensor; The ROM of the industrial computer for identifying and positioning the keyhole of the stacker pre-stores a visual calculation algorithm, a sensor data fusion algorithm, and a vector coordinate calculation algorithm; The industrial computer for identifying and positioning the keyhole of the stacker analyzes the RGB image of the target container collected by the monocular camera through the visual calculation algorithm, thereby predicting the keyhole area in the target container, and processes the predicted keyhole area picture to obtain the image pixel coordinates of the keyhole center of the target container; The industrial computer for identifying and positioning the keyhole of the stacker converts the image pixel coordinates of the keyhole center of the target container and the horizontal distance information collected by the ranging sensor into the world coordinate system through the vector coordinate calculation algorithm. The industrial computer for identifying and positioning the keyhole of the stacker fuses the pixel coordinate information and the horizontal distance information through the sensor data fusion algorithm to obtain the three-dimensional coordinate value of the keyhole center of the target container in the world coordinate system; The wire rope sensor determines the actual positions of the locking buckle and keyhole of the stacker spreader by detecting the telescopic degree of the oil cylinder, and secondarily calibrates and matches the positions.

2. The automatic recognition and positioning system for container keyholes according to claim 1, wherein The ranging sensor is a laser ranging sensor.

3. The automatic recognition and positioning system for container keyholes according to claim 1, characterized in that, The monocular camera is installed on the handrail of the stairs on one side of the stacker cab. The axis of the lens of the monocular camera faces the outside of the stacker, forming a 30° angle with the driving direction of the stacker, and is powered by the on-vehicle power supply of the stacker. The ranging sensor is installed on the side of the vehicle lamp on one side of the stacker stairs handrail, parallel to the horizontal plane, and is powered by the on-vehicle power supply of the stacker.

4. The automatic identification and positioning system for container keyholes according to claim 1, wherein, The industrial computer for identifying and positioning the keyhole of the stacker is installed on the side of the stacker body on one side of the stacker stairs handrail.

5. The automatic recognition and positioning system for container keyholes according to claim 1, characterized in that, The control system further includes an electric control cabinet. The industrial computer for identifying and positioning the keyhole of the stacker is installed inside the electric control cabinet, and the electric control cabinet is installed on the side of the stacker body on one side of the stairs handrail.

6. The automatic identification and positioning system for container keyholes according to claim 1, wherein, The wire rope sensor is installed beside the mast of the stacker.

7. The automatic recognition and positioning system for container keyholes according to claim 1, wherein The visual calculation algorithm is the SSD deep learning object detection model algorithm and the traditional image processing method provided by the OpenCV computer vision library.

8. An automatic recognition and positioning method for container keyholes, characterized in that the steps It includes: S1: Collect the container picture through the monocular camera, and predict the keyhole area in the container picture through the deep learning object detection model algorithm; S2: Process the predicted container keyhole area image through the traditional image processing method provided by the OpenCV computer vision library to obtain the image pixel coordinates of the center of the unilateral keyhole of the target container; S3: Measure the horizontal distance from the laser ranging sensor to the keyhole of the target container through the laser ranging sensor; S4: Convert the image pixel coordinates of the center of the unilateral lock hole of the target container obtained and the horizontal distance from the laser ranging sensor to the lock hole of the target container into the world coordinate system through the vector coordinate calculation algorithm, and fuse the image pixel coordinate information and the horizontal distance information through the sensor data fusion algorithm to obtain the three-dimensional coordinate value of the center of the unilateral lock hole of the target container in the world coordinate system; S5: Combine the known fixed dimensions of the container to deduce the three-dimensional coordinate value of the center of the lock hole on the other side of the target container in the world coordinate system; S6: Combine the known three-dimensional coordinate value of the spreader lock in the world coordinate system to obtain the three-dimensional coordinate difference between the lock hole center and the lock, and adjust the telescopic degree of the oil cylinder according to this difference to preliminarily match the positions of the lock hole and the lock; S7: Measure the actual position after the oil cylinder is adjusted through the wire-pulling sensor, and perform precise position matching on the positions of the lock hole and the lock again until the positions of the lock hole and the lock are completely matched, and complete the locking and container-grabbing actions.

9. A method for automatically identifying and positioning a container keyhole according to claim 8, characterized in that, The specific steps for predicting the container lock hole area through the deep learning object detection model algorithm include: S11: Calibrate the monocular camera through the Zhang's calibration method to obtain the internal parameters, external parameters and distortion parameters of the camera; S12: Before recognition, use a monocular camera to collect a large number of container sample pictures, manually mark the lock hole areas and non-lock hole areas among them as positive and negative samples, and train them through the deep learning network framework; S13: Use the trained network model to predict the lock hole area in the container pictures taken during the recognition process.

10. A method for automatically identifying and positioning a container keyhole according to claim 8, characterized in that, The specific steps for processing the image of the predicted container lock hole area through the traditional image processing method provided by the OpenCV computer vision library include: S21: Perform image preprocessing of image enhancement and noise reduction on the predicted lock hole area picture; S22: Perform binary processing on the preprocessed image to find all the contours; S23: Use the minimum bounding rectangle to approximate the contour of the lock hole of the target container; S24: Use the center point coordinates of the minimum bounding rectangle as the pixel coordinates of the position of the lock hole of the target container.

Images