A hatch detection method and system for cargo holds based on lidar and monocular camera

Abstract

The present invention discloses a hatch detection method and system based on a lidar and a monocular camera. The method is as follows: by means of the image collected by the monocular camera, the hatch of the cargo hold is first identified, greatly reducing the computational amount and improving the recognition accuracy; then, by means of the spatial position conversion between the camera and the lidar, the rapid extraction of the point cloud near the hatch of the cargo hold is realized; and then, by using the mapping into a two-dimensional image and the remapping method, the spatial positions of the four corner points of the hatch of the cargo hold are quickly obtained. And by means of the fusion of the detection results of the two channels of the forward lidar camera and the lateral lidar camera, the reliability of the hatch detection of the cargo hold is ensured. This method can well reduce the computational amount for identifying the hatch of the cargo hold and improve the accuracy and reliability.

Description

Technical Field

[0001] The present invention relates to the field of automation of port gantry cranes, and particularly to a method and system for detecting hatch covers of cargo holds based on lidar and a monocular camera. Background Art

[0002] With the need for social and economic development, bulk cargo transportation at bulk terminals plays an increasingly important role in the national economy. Common bulk cargo terminal commodities in bulk cargo transportation, such as media, iron ore, grains, etc., have always occupied a relatively large scale in terms of freight volume. At bulk terminals, gantry cranes are one of the main equipment for bulk cargo transportation. The traditional operation mode of gantry cranes requires the driver to perform manual operations in the driver's cab. At the same time, due to the obstruction of the driver's line of sight in the driver's cab, it is difficult to observe the specific situation of the hatch cover of the cargo hold below, and an observer needs to be configured on the deck of the cargo ship to cooperate with the driver to complete the operation together. Such an operation mode's efficiency depends on the driver's operation proficiency and cooperation with the observer, with a relatively high operation difficulty and significant potential safety hazards. Moreover, in the actual production operation process, the time for loading cargo onto the ship is often very tight. In order to complete the loading and unloading tasks within the specified time, gantry cranes are often required to operate continuously within 24 hours, resulting in problems such as unstable operation efficiency, high labor intensity of staff, and high operation safety risks.

[0003] In order to reduce safety risks and improve the working environment of staff to meet the requirements of the era of intelligent bulk terminals, it is necessary to carry out automation transformation and upgrading of gantry cranes. Among them, one of the key technologies in the automation transformation process of gantry cranes is to achieve the identification, positioning of the hatch cover of the cargo hold, and measurement of the position distribution of the cargo inside the hold. Real-time, accurate, and reliable detection results of the hatch cover of the cargo hold are an important guarantee for the automated operation of gantry cranes.

[0004] Existing methods for identifying hatch covers of cargo holds, such as patents with publication numbers CN 113340287, CN 112529958, etc., solely rely on lidar to identify the hatch covers of cargo holds. Relying on a single laser sensor has the risk of insufficient safety redundancy, and it is prone to detection failure in extreme cases, bringing potential hazards to production operations. At the same time, since the lidar scans to obtain three-dimensional point cloud information, compared with two-dimensional image information, the computational amount for recognition increases significantly. Only relying on three-dimensional point clouds and using feature recognition methods to achieve the identification of hatch covers of cargo holds, the computational efficiency is too slow, it is difficult to meet the real-time requirements, and the recognition accuracy is difficult to guarantee. Summary of the Invention

[0005] The present invention mainly solves the technical problems existing in the prior art, such as large calculation amount for hatch identification in cargo holds, low calculation efficiency, poor real-time performance, and insufficient accuracy. It provides a hatch detection method and system based on lidar and monocular camera, which can efficiently and accurately identify and detect the position of the cargo hold hatch in real time.

[0006] The present invention mainly solves the above technical problems through the following technical solutions: A hatch detection method based on lidar and monocular camera includes the following steps:

[0007] S1. When the angle sensor on the gantry crane determines that it enters the detection range, start the detection: The forward lidar collects forward point cloud data, and the forward camera collects forward images; The side lidar collects side point cloud data, and the side camera collects side images; The forward point cloud data and the side point cloud data are collectively referred to as point cloud data hereinafter.

[0008] S2. Perform template matching on the forward image and the side image respectively to obtain the areas where the cargo hold hatches are located in the forward image and the side image.

[0009] S3. Perform cargo hold hatch detection on the areas where the cargo hold hatches are located in the forward image and the side image to obtain the pixel positions of the four corner points of the cargo hold hatch in the forward image and the side image.

[0010] S4. Utilize the relevant parameters of the calibrated forward lidar and forward camera, and combine with the pixel positions of the four corner points of the cargo hold hatch obtained in step S3 to remap the pixel positions of the corner points back to the forward point cloud data, so as to obtain the rough spatial position information of the cargo hold hatch in the forward point cloud data; Similarly, obtain the rough spatial position information of the cargo hold hatch in the side point cloud data.

[0011] S5. Segment the collected forward point cloud data according to the rough spatial position information of the cargo hold hatch in the forward point cloud data to obtain the forward point cloud of the cargo hold hatch and its surrounding area; Similarly, obtain the side point cloud of the cargo hold hatch and its surrounding area.

[0012] S6. Map the forward point cloud of the cargo hold hatch and its surrounding area into a forward two-dimensional image and perform dilation processing; Similarly, obtain the side two-dimensional image.

[0013] S7. Perform cargo hold hatch detection on the forward two-dimensional image and the side two-dimensional image respectively to obtain the positions of the four corner points of the cargo hold hatch.

[0014] S8. Remap the forward point cloud of the cargo hold hatch and its surrounding area into a 2D image, record the spatial information of the point cloud mapped to the pixel values of the four corner points of the detected cargo hold hatch, so as to obtain the spatial position information of the four angles of the cargo hold hatch detected by the forward lidar and the forward camera; similarly, obtain the spatial position information of the four angles of the cargo hold hatch detected by the side lidar and the side camera.

[0015] S9. Use the forward point cloud of the cargo hold hatch and its surrounding area and the side point cloud of the cargo hold hatch and its surrounding area for point cloud matching to obtain the spatial pose relationship between the forward lidar and the side lidar.

[0016] S10. Use the spatial pose relationship between the forward lidar and the side lidar, combined with the spatial position information of the four corner points of the cargo hold hatch, to obtain the final position information of the cargo hold hatch.

[0017] Preferably, in step S2, the template matching is specifically: adopt standard square difference matching, traverse the image to be detected, calculate the matching degree between the template image and the overlapping sub-image, so as to obtain the area where the cargo hold hatch is located; the template image is an image of the approximate range of the cargo hold hatch collected by the camera in advance and specified manually.

[0018] Preferably, the detection of the cargo hold hatch in steps S3 and S7 is specifically:

[0019] A1. First, perform filtering to enhance edge detection on the image, and then search for rectangles in each color channel of the image.

[0020] A2. Separate the channels of the image, set a threshold, then use canny to extract edges, perform dilation processing on the image, and then perform contour search.

[0021] A3. Check the found contours, perform polygon fitting using the image contour points, calculate the four vertices of the rectangle after calculating the contour area, and finally obtain the maximum cosine of the angle between the contour edges, filter according to the previously set threshold, and finally combine the information of each channel to obtain the positions of the four corner points of the final cargo hold hatch.

[0022] Preferably, the conversion formula for mapping the pixel position of the corner point to the point cloud data is:

[0023]

[0024] where z c is the preset estimated height value, u and v are the pixel coordinates of the corner point in the picture, is the 3*3 internal parameter matrix K of the camera, is the position coordinate of the point in the point cloud data after the corner points of the image detection are mapped back to the point cloud. R is the rotation matrix of the transformation between the lidar and the camera, and t is the displacement matrix of the transformation between the lidar and the camera. are the three-dimensional coordinates in the camera coordinate system with the camera coordinate system as the reference after transformation.

[0025] Preferably, the specific formula for mapping the point cloud to a two-dimensional image is:

[0026]

[0027] where u cw and v cw are the pixel coordinates of the point cloud mapped to the picture, is the 3*3 internal parameter matrix K of the camera, is the position information of each point of the lidar point cloud after dilation processing.

[0028] Preferably, the step S10 is specifically:

[0029] Convert the spatial position information of the four angles of the hatch detected by the forward lidar and the forward camera using the following formula:

[0030]

[0031] are the spatial position information of the four angles of the hatch detected by the forward lidar and the forward camera, are the spatial position information of the four angles of the hatch detected by the side lidar and the side camera; then calculate the difference between the four angles of the spatial position after conversion, that is, calculate and the difference, compare the difference with a preset threshold. When the differences of all four corner points are less than the threshold, then use the spatial position of the four corner points as the spatial position information of the four corner points of the measured hatch; if any one or several of the differences of the four corner points are greater than or equal to the threshold, re-detect and issue a warning.

[0032] After obtaining the spatial position information of the hatch, the position distribution information of the goods inside the hatch can be further filtered.

[0033] A hatch detection system for cargo holds based on lidar and monocular cameras operates using the hatch detection method for cargo holds based on lidar and monocular cameras as described above. The system includes a forward lidar, a forward camera, a lateral lidar, a lateral camera, a first angle sensor, and a second angle sensor. Both the forward camera and the lateral camera are monocular cameras. The forward lidar and the forward camera are installed on a stabilized pan-tilt head, and the stabilized pan-tilt head is installed at the elephant trunk of a gantry crane. The lateral lidar and the lateral camera are installed above the driver's cab of the gantry crane. And when the elephant trunk of the gantry crane moves above the cargo hold hatch, both the forward lidar and the lateral lidar can collect complete point cloud data of the cargo hold hatch, and both the forward camera and the lateral camera can collect complete images of the cargo hold hatch. The forward lidar and the lateral lidar obtain the pose relationship between the two lidars through off-line calibration. The forward lidar and the forward camera are jointly calibrated. The lateral lidar and the lateral camera are jointly calibrated.

[0034] The substantial effect brought by the present invention is that, by means of the images collected by the monocular camera, the cargo hold hatch is first identified, greatly reducing the calculation amount and improving the recognition accuracy. Then, through the spatial position conversion between the camera and the lidar, the rapid extraction of the point cloud near the cargo hold hatch is realized. By using the mapping to a two-dimensional image and the remapping method, the spatial positions of the four corner points of the cargo hold hatch are quickly obtained.

[0035] At the same time, the present invention overcomes the potential hazard of large recognition errors that may cause production safety accidents due to the shaking during the operation of the crane by means of the dual detection of the cargo hold by the forward lidar and camera and the lateral lidar and camera. The present invention realizes dual detection by using the dual-channel detection method of the forward lidar camera and the lateral lidar camera, improving the reliability and effectiveness of the detection of the cargo hold hatch. Description of the Drawings

[0036] Figure 1 is a flowchart of a detection method of the present invention;

[0037] Figure 2 is a diagram of the installation positions of sensors of the present invention;

[0038] Figure 3 is a registration diagram of lidar point cloud data and images of the present invention;

[0039] Figure 4 is an image collected by the forward camera of the present invention;

[0040] Figure 5 is an image of a roughly defined area of the cargo hold hatch specified by a person of the present invention;

[0041] Figure 6It is an image of the template matching detection effect of the present invention;

[0042] Figure 7 It is an image for detecting the hatch of a cargo ship in an image of the present invention;

[0043] Figure 8 It is the point cloud information collected by the forward lidar of the present invention;

[0044] Figure 9 It is the point cloud of the hatch of the cargo ship and the surrounding area after segmentation of the present invention;

[0045] Figure 10 It is an image of the mapped hatch of the cargo ship and the nearby area of the present invention;

[0046] Figure 11 It is the detection of the hatch of the mapped image cargo ship of the present invention;

[0047] Figure 12 It is a comparison diagram of the point cloud of the actual cargo ship and the point cloud of the position distribution of the cargo in the cabin obtained after filtering of the present invention;

[0048] In the figure: 1. Anti-sway pan-tilt; 2. First angle sensor; 3. Lateral lidar and lateral camera; 4. Second angle sensor. Specific embodiments

[0049] The technical solution of the present invention will be further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0050] Embodiment: A hatch detection system for a cargo ship based on a lidar and a monocular camera in this embodiment includes a forward lidar, a forward camera, a lateral lidar, a lateral camera, a first angle sensor, and a second angle sensor, and both the forward camera and the lateral camera are monocular cameras.

[0051] Before installing the lidar and the camera, it is necessary to jointly calibrate the lidar and the camera of the same group (both forward or both lateral) to achieve the information fusion of the image and the point cloud. Through the Zhang-Zhengyou calibration method, the internal parameters of the camera are calibrated using the pinhole model, where the distortion considers radial distortion and tangential distortion. First, use the camera to take calibration plates at different angles, and through the cameraCalibrator tool in matlab, obtain the 3*3 internal parameter matrix K and 5 distortion correction coefficients k1, k2, p1, p2, k3.

[0052] On the basis of obtaining the internal parameters of the camera, use the calibration plate to jointly calibrate the lidar and the camera to obtain the conversion relationship [Rt] between the camera coordinate system and the lidar coordinate system, where R is the rotation matrix (matrix size 3*3) and t is the displacement vector (vector size 3*1).

[0053] In the case of removing distortion (the distortion is calculated and removed by five distortion correction coefficients), the conversion relationship between the position coordinates of the laser point cloud and the pixel coordinates of the camera shot can be obtained through the above parameters, and the conversion formula is as follows:

[0054]

[0055] where u and v are the pixel coordinates in the picture, is the obtained 3×3 intrinsic parameter matrix K, is the position coordinate obtained by the lidar collecting the point cloud, R is the rotation matrix obtained by the transformation between the lidar and the camera, and t is the displacement matrix obtained by the transformation between the lidar and the camera, is the three-dimensional coordinate in the camera coordinate system with the camera coordinate system as the reference after transformation. After the calibration of the lidar and the camera, the registration of the point cloud collected by the lidar and the image collected by the camera can be realized through the conversion formula, Figure 3 is the registration map of the point cloud and the picture installed on the gantry crane.

[0056] Then install the calibrated lidar and camera on the gantry crane. Install a lidar and a monocular camera at two places, namely at the front of the jib nose (frontward) and above the driver's cab (sideward) of the gantry crane. The installation schematic diagram is as Figure 2 shown. Install the forward radar and camera at position 1, and the field of view is vertically downward. At the same time, to ensure stability, the forward radar and camera are first installed on the anti-sway platform 1, and then the anti-sway platform is fixed to the jib nose. The reason for installing a lidar and a camera forward is that only at this position can the position distribution of the goods inside the cargo hold be observed well, but there is shaking in this position during the actual production process. Install the sideward lidar and sideward camera 3 above the driver's cab. The shaking at the sideward installation position is small, but the inside of the cargo hold cannot be fully observed. After installation, by adjusting the viewing angle, when the jib nose of the gantry crane moves above the cargo ship, a complete image and point cloud of the cargo hold hatch can be collected. At the same time, after installation, through the off-line calibration method, the pose relationship [R 1 t 1 between the two lidars is obtained. Where R 1 is the rotation matrix for the transformation between the two radar coordinate systems, and t 1 is the displacement vector. The transformation between the two radar coordinate systems can be carried out through the following formula:

[0057]

[0058] In the preliminary preparation, first manually operate the gantry crane to move the jib nose of the gantry crane above the cargo hold hatch, and record the rotation angle value of the gantry crane tower and the boom angle value at this time, that is, useFigure 2 The first angle sensor 2 and the second angle sensor 4 shown record angle values. Using the pictures acquired by the forward camera and the side camera, a rough range of a cargo ship hatch is manually specified in advance, and the image of the manually specified rough range of the cargo ship hatch is saved. For example Figure 4 is the complete image acquired by the forward camera, Figure 5 is the image of the rough range of the cargo ship hatch manually specified, thus obtaining the template image for template matching.

[0059] As Figure 1 shown, a method for detecting a cargo ship hatch based on a lidar and a monocular camera in this embodiment is as follows:

[0060] Step 1: At each stage when the cargo ship hatch needs to be detected, when it is judged that it is roughly above the cargo hold according to the angle sensor information on the gantry crane, that is, when running to the positions of the previous angle sensor 1 and angle sensor 2, detection starts. The forward and side cameras and the lidar simultaneously collect image and lidar point cloud data. The point cloud data of the collected lidar is corrected and fused in real time with the help of imu sensor data and the slam map construction method to reduce the influence caused by shaking. After the image data is collected, using the image of the rough range of the cargo ship hatch manually specified, on the acquired image, the template matching method is adopted, specifically the standard square difference matching, to traverse the image to be detected, calculate the matching degree between the template image and the overlapping sub-image, and judge the rough position of the cargo ship hatch in real time, obtaining the rough regional position of the cargo ship hatch. The specific result is as Figure 6 shown.

[0061] Step 2: Through the method of template matching in Step 1, the rough position of the cargo ship hatch is judged, and the image of the matching area of the rough position of the cargo ship hatch is obtained. For this area, the cargo ship hatch is detected to obtain the pixel positions of the four corner points of the cargo ship hatch in the image. Specifically, first, the image is filtered to enhance edge detection, and then rectangles are searched in each color channel of the image. The image is separated into channels, relevant thresholds are set, then canny is used to extract edges, the image is dilated, and then contour search is carried out. The found contours are checked, polygon fitting is carried out using the image contour points, the four vertices of the rectangle are obtained after calculating the contour area, and finally the maximum cosine of the angle between the contour edges is obtained, and filtered according to the previously set threshold. Finally, combining the information of each channel, the positions of the four corner points of the final cargo ship hatch are obtained. The detection result is as Figure 7 shown.

[0062] Step 3: Using the relevant parameters of the pre-calibrated lidar and camera, and combining with the pixel position information of the four corner points of the cargo hatch obtained in Step 5, remap them back to the point cloud data processed after lidar acquisition. Preset an estimated height z c value (generally higher than the distance from the cargo hatch to the lidar), and then perform the conversion through the following formula:

[0063]

[0064] where u and v are the pixel coordinates in the image, is the obtained 3*3 intrinsic matrix K, is the position coordinate of the point in the point cloud data after the corner points detected in the image are mapped back to the point cloud, R is the rotation matrix obtained from the transformation between the lidar and the camera, and t is the displacement matrix, is the three-dimensional coordinate in the camera coordinate system with the camera coordinate system as the reference after transformation.

[0065] Calculate the approximate spatial position information of the cargo hatch in the point cloud collected by the lidar through the above formula. Use the calculated spatial position information to segment the collected data to obtain the point cloud of the cargo hatch and its surrounding area. Specifically Figure 8 is the point cloud information collected by the forward lidar, Figure 9 is the point cloud of the cargo hatch and its surrounding area after segmentation.

[0066] Step 4: Using the point cloud obtained in Step 3, remap the segmented point cloud back to a two-dimensional image. The method of remapping to a two-dimensional image follows the pinhole camera model method shown in Step 1. The specific formula is:

[0067]

[0068] where u and v are the pixel coordinates in the image, is the obtained 3*3 intrinsic matrix K, is the position information of each point of the processed lidar point cloud.

[0069] At the same time, because the collected point cloud has a certain sparsity, during the mapping process, perform dilation processing on the image mapped from the point cloud to ensure that the image can form a complete connected graph. Figure 10 is the image of the cargo hatch and its nearby area after mapping.

[0070] Step 5: In the two-dimensional image obtained in Step 4, perform hatch detection on the cargo hold to obtain the four-angle information of the cargo hold hatch. Perform hatch detection on the mapped image to obtain the pixel positions of the four corner points of the cargo hold hatch in the image. Specifically, first perform filtering to enhance edge detection on the image, and then search for rectangles in each color channel of the image. Separate the channels of the image, set relevant thresholds, then use canny to extract edges, perform dilation processing on the image, and then perform contour search. Check the found contours, use the image contour points for polygon fitting, calculate the contour area to obtain the four vertices of the rectangle, and finally obtain the maximum cosine of the angles between the contour edges, filter according to the previously set threshold, and finally combine the channel information to obtain the positions of the four corner points of the final hatch. The detection results are as Figure 11 shown.

[0071] After detecting the positions of the four corner points of the hatch, remap the point cloud obtained in Step 3. During the remapping process, record the spatial position information of the laser points mapped to the pixels of the four corner points. For the spatial position information of multiple laser points corresponding to each corner point, use the average weighted method to obtain the spatial position information of the four corner points of the cargo hold hatch.

[0072] Step 6: Use the segmented point cloud information obtained by the forward lidar and camera in Step 3 and the segmented point cloud information obtained by the side lidar and camera to perform point cloud matching. Since preliminary matching has been carried out in advance, but there is still a certain amount of shaking in the forward lidar during the actual production process, it is necessary to recalculate the position change relationship between the forward radar and the side radar. Using the previously obtained position transformation relationship between the two lidars as the initial value, use the segmented point clouds of the cargo hold hatch and its nearby area obtained by the forward lidar and the side lidar after Step 3, and perform matching with the ICP matching algorithm to obtain the updated spatial pose relationship [R 1 t 1 of the side lidar camera and the forward lidar camera.

[0073] Step 7: Use the spatial pose relationship [R 1 t 1 of the side lidar camera and the forward lidar camera obtained in Step 6, and combine the spatial position information of the four corner points of the forward lidar camera and the side lidar camera in Step 5 to make a judgment to obtain the final cargo hold hatch position information.

[0074] Since the shaking of the side lidar is very small, the credibility is the best. Therefore, the spatial position information of the four angles of the cargo hold hatch detected by the forward lidar and the camera is converted using the following formula:

[0075]

[0076] After that, calculate the differences in the spatial positions of the four angular spaces obtained after conversion, that is, calculate and the difference. Set a preset threshold. When the differences of all four corner points are less than the threshold, then use the spatial positions of the four corner points as the spatial position information of the four corner points of the hatch of the cargo ship measured. If it is greater than the threshold, measure again and issue a warning.

[0077] Furthermore, using the obtained spatial position information of the four corner points of the cargo ship hatch, filter the obtained point cloud of the cargo ship detection, and the position distribution information of the goods in the cabin can be obtained. Figure 12 This is the comparison diagram between the actual point cloud of the cargo ship and the point cloud of the position distribution of the goods in the cabin obtained after filtering.

[0078] This solution relies on the method of multi-sensor fusion such as lidar and camera for hatch detection, which can effectively improve safety redundancy and ensure operation safety. At the same time, the current object detection method for images is relatively mature and reliable. By means of the method of the cargo ship hatch in the image, it can well reduce the calculation amount of identifying the cargo ship hatch and improve the accuracy and reliability.

[0079] The specific embodiments described in this article are only examples to illustrate the spirit of the present invention. Those skilled in the technical field to which the present invention belongs can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, but will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0080] Although terms such as point cloud data and position information are used more in this article, the possibility of using other terms is not excluded. Using these terms is only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.

Claims

1. A hatch detection method for cargo holds based on lidar and monocular camera, characterized in that, it includes the following steps: S1. When the angle sensor on the gantry crane determines that it enters the detection range, start the detection: the forward lidar collects forward point cloud data, and the forward camera collects forward images; The side lidar collects side point cloud data, and the side camera collects side images; S2. Perform template matching on the forward image and the side image respectively to obtain the area where the cargo hold hatch is located in the forward image and the side image; S3. Perform cargo hold hatch detection on the areas where the cargo hold hatch is located in the forward image and the side image, and obtain the pixel positions of the four corner points of the cargo hold hatch in the forward image and the side image; S4. Utilize the relevant parameters of the calibrated forward lidar and forward camera, and combine the pixel positions of the four corner points of the cargo hold hatch obtained in step S3 to remap the pixel positions of the corner points back to the forward point cloud data to obtain the rough spatial position information of the cargo hold hatch in the forward point cloud data; Similarly, obtain the rough spatial position information of the cargo hold hatch in the side point cloud data; S5. Segment the collected forward point cloud data according to the rough spatial position information of the cargo hold hatch in the forward point cloud data to obtain the forward point cloud of the cargo hold hatch and its surrounding area; Similarly, obtain the side point cloud of the cargo hold hatch and its surrounding area; S6. Map the forward point cloud of the cargo hold hatch and its surrounding area into a forward two-dimensional image and perform dilation processing; Similarly, obtain the side two-dimensional image; S7. Perform cargo hold hatch detection on the forward two-dimensional image and the side two-dimensional image respectively to obtain the positions of the four corner points of the cargo hold hatch; S8. Remap the forward point cloud of the cargo hold hatch and its surrounding area into a two-dimensional image, and record the spatial information of the point cloud that is mapped to the pixel values of the four corner points of the detected cargo hold hatch, so as to obtain the spatial position information of the four angles of the cargo hold hatch detected by the forward lidar and the forward camera; Similarly, obtain the spatial position information of the four angles of the cargo hold hatch detected by the side lidar and the side camera; S9. Use the forward point cloud of the cargo hold hatch and its surrounding area and the side point cloud of the cargo hold hatch and its surrounding area for point cloud matching to obtain the spatial pose relationship between the forward lidar and the side lidar; S10. Utilize the spatial pose relationship between the forward lidar and the side lidar, and combine the spatial position information of the four corner points of the cargo hold hatch to obtain the final position information of the cargo hold hatch; The specific content of step S10 is: Convert the spatial position information of the four angles of the cargo hold hatch detected by the forward lidar and the forward camera using the following formula: It is the spatial position information of the four angles of the hatch of the cargo hold detected by the forward lidar and the forward camera. It is the spatial position information of the four angles of the hatch of the cargo hold detected by the side lidar and the side camera. Then, calculate the difference in the spatial positions of the four angles after conversion, that is, calculate and the difference. Compare the difference with a preset threshold. When the differences of all four corner points are less than the threshold, then use the four corner point spatial positions of as the spatial position information of the four corner points of the measured hatch of the cargo hold. If any one or several of the differences of the four corner points are greater than or equal to the threshold, re-detect and issue a warning.

2. A hatch detection method for cargo holds based on lidar and monocular camera according to claim 1, characterized in that, In step S2, the template matching specifically is: adopt standard square difference matching, traverse the image to be detected, calculate the matching degree between the template image and the overlapping sub-image, so as to obtain the area where the cargo hold hatch is located; The template image is an image of the cargo hold hatch range collected by the camera in advance and specified manually.

3. A hatch detection method for cargo holds based on lidar and monocular camera according to claim 1 or 2, characterized in that, The hatch detection in steps S3 and S7 is specifically as follows: A1. First, perform filtering and enhanced edge detection on the image, and then search for rectangles in each color channel of the image; A2. Separate the channels of the image, set a threshold, then use canny to extract edges, perform dilation processing on the image, and then search for contours; A3. Check the found contours, perform polygon fitting using the image contour points, calculate the contour area to obtain the four vertices of the rectangle, and finally obtain the maximum cosine of the angle between the contour edges, filter according to the previously set threshold, and finally combine the information of each channel to obtain the positions of the four corner points of the final cargo hatch.

4. A method for detecting a cargo hatch based on a lidar and a monocular camera according to claim 1 or 2, characterized in that, The conversion formula for mapping the pixel position of the corner point to the point cloud data is: where z c is a preset estimated height value, u and v are the pixel coordinates of the corner point in the image, is the 3*3 intrinsic matrix K of the camera, is the position coordinate of the point in the point cloud data after the corner point detected in the image is mapped back to the point cloud, R is the rotation matrix of the transformation between the lidar and the camera, and t is the displacement matrix of the transformation between the lidar and the camera, is the three-dimensional coordinate in the camera coordinate system obtained after transformation with the camera coordinate system as the reference.

5. A method for detecting a cargo hatch based on a lidar and a monocular camera according to claim 3, characterized in that, The specific formula for mapping the point cloud to a two-dimensional image is: Among them, u cw and v cw are the pixel coordinates of the point cloud mapped to the image, is the 3×3 intrinsic matrix K of the camera, is the position information of each point of the laser point cloud after dilation processing.

6. A system for detecting a cargo hatch based on a lidar and a monocular camera, characterized in that, It runs the method for detecting a cargo hatch based on a lidar and a monocular camera as described in claim 1. The system includes a forward lidar, a forward camera, a side lidar, a side camera, a first angle sensor, and a second angle sensor. The forward camera and the side camera are both monocular cameras; the forward lidar and the forward camera are installed on a stabilized pan-tilt, and the stabilized pan-tilt is installed at the elephant trunk of the gantry crane. The side lidar and the side camera are installed above the driver's cab of the gantry crane; and when the elephant trunk of the gantry crane moves above the cargo hatch, the forward lidar and the side lidar collect complete point cloud data of the cargo hatch, and the forward camera and the side camera collect complete images of the cargo hatch; the forward lidar and the side lidar obtain the pose relationship between the two lidars through off-line calibration; the forward lidar and the forward camera are jointly calibrated; the side lidar and the side camera are jointly calibrated.

Images