Bulk cargo cabin real-time fusion detection method based on laser point cloud

Abstract

The invention discloses a bulk cargo cabin real-time fusion detection method based on a laser point cloud, and the method comprises the steps: obtaining a real-time three-dimensional scene point cloud of a port scene, and obtaining a deck plane equation and a normal vector; projecting the three-dimensional scene point cloud to a two-dimensional deck plane, and constructing an intensity image taking point cloud intensity as information and a depth image taking projection distance as information; obtaining an intensity mask image from the intensity image; acquiring a depth edge image according to the depth image, and fusing the depth edge image and the intensity mask image to obtain a fused edge image; contour extraction is carried out on the fused edge image, polygon fitting is carried out, a quadrangle in the fused edge image is reserved as a candidate cabin, and a minimum enclosing rectangle of the quadrangle is obtained; and obtaining four vertex coordinates of the minimum bounding rectangle, and back-projecting the four vertex coordinates to the three-dimensional space to obtain three-dimensional space rectangular coordinate points of the final cabin. The method is suitable for a complex environment and a dynamic scene in a port operation area, and the cabin in the dynamic scene can be obtained in real time.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection of port machinery and equipment, and in particular to a real-time fusion inspection method for bulk carrier holds based on laser point clouds. Background Technology

[0002] With the gradual development of artificial intelligence technology and the continuous improvement of the performance of software and hardware equipment, the traditional manual operation mode of port loading and unloading equipment is gradually moving towards semi-automatic or even fully automatic operation mode. At the same time, technologies such as perception, positioning and collision avoidance by fusion of visual multi-sensor such as lidar and cameras are receiving more and more attention.

[0003] Ship hold inspection is a key prerequisite for realizing intelligent loading and unloading of port loading and unloading equipment. It provides the loading and unloading equipment with the perception of the operating environment and obtains the distribution of cargo in the ship hold; it constructs an accurate three-dimensional operation model to support grab bucket path planning and grab point positioning; it can effectively prevent collisions between grab buckets and ship holds, and improve the safety of intelligent loading and unloading operations.

[0004] The existing ship cabin detection technologies have the following shortcomings: (1) The use of deep learning technology or multi-sensor fusion technology results in high algorithm complexity and complex process. When multiple algorithms run on one device, it is difficult to meet the real-time requirements, which leads to an increase in port equipment costs; (2) Ship cabins can be detected in a static environment with only ship data. However, in the complex environment and dynamic scenarios of port operation areas, the position of ship cabins cannot be effectively detected in real time; (3) The failure to integrate information such as the depth and intensity of lidar results in certain defects in algorithm robustness, which is not conducive to real-time detection of ship cabins in complex port environments. Summary of the Invention

[0005] To overcome the aforementioned problems in the existing technology, this invention proposes a real-time fusion detection method for bulk carrier holds based on laser point clouds.

[0006] The technical solution adopted by this invention to solve its technical problem is: a real-time fusion detection method for bulk carrier holds based on laser point clouds, comprising the following steps: Step 1: Obtain the real-time 3D point cloud of the port scene, and obtain the deck plane equation and normal vector; Step 2: Project the 3D scene point cloud onto the 2D deck plane to construct an intensity image with point cloud intensity as information and a depth image with projection distance as information; Step 3: Binarize the intensity image to obtain a binary intensity image, and then perform morphological dilation on the binary image to obtain an intensity mask image; Step 4: Obtain a depth edge image based on the depth image obtained in Step 2, and fuse the depth edge image with the intensity mask image obtained in Step 3 to obtain a fused edge image; Step 5: Extract the contours from the fused edge image and perform polygon fitting, retaining the quadrilaterals as candidate cabins, and obtain the minimum bounding rectangle of the quadrilaterals. Step 6: Obtain the coordinates of the four vertices of the smallest bounding rectangle, and back-project them into three-dimensional space to obtain the coordinates of the final three-dimensional space rectangle of the cabin.

[0007] The aforementioned method for real-time fusion detection of bulk carrier holds based on laser point clouds, in step 1, utilizes the SLAM algorithm to perform real-time 3D scene scanning of the port area to acquire laser radar point cloud data. Where x, y, z are coordinate values. The intensity value is used to obtain the deck plane equation from the point cloud data using a planar detection algorithm. and its unit normal vector : ; And determine the unit normal vector and Axial unit vector included angle Less than a given threshold If satisfied If so, the normal vector extraction is correct.

[0008] The above-mentioned real-time fusion detection method for bulk carrier holds based on laser point clouds, specifically step 2 of projecting the 3D scene point cloud onto the 2D deck plane includes: projecting the 3D scene point cloud data... Projected onto deck plane superior, ; with the unit normal vector of the deck plane as for Axis, on the plane and orthogonal unit vectors for Axis, with and The unit vector of the cross product Construct a deck coordinate system with the y-axis as the axis. Then the coordinates of any point in the point cloud data Projection point on the deck plane for: ； Origin of point cloud data coordinates Projection point on the deck plane for , will point Defined as deck coordinate system If the origin of the coordinate system is given, then any point in the point cloud data can be considered a point. In the deck coordinate system The coordinates below are: .

[0009] The above-mentioned real-time fusion detection method for bulk carrier holds based on laser point clouds, wherein the specific process of constructing the intensity image in step 2 includes: in the deck coordinate system Only (x,y) coordinates are retained and discretized to construct a point cloud intensity map. For information intensity image Each sampled pixel Preserve 3D scene points Concentrated intensity projected onto the pixel coordinates The minimum value corresponds to the strength value. for: ; in, Indicates falling pixel 3D scene points Number of discretized sampling points.

[0010] The above-mentioned real-time fusion detection method for bulk carrier holds based on laser point clouds, wherein the specific process of constructing the depth image in step 2 includes: in the deck coordinate system By retaining only the (x,y) coordinates and discretizing them, a depth image with projection distance d as information is constructed. Each sampled pixel Preserve 3D scene points The maximum distance d projected onto the set of points at that pixel coordinate is the corresponding depth value. for: ; Where N represents the falling pixel. 3D scene points Number of discretized sampling points.

[0011] The above-mentioned real-time fusion detection method for bulk carrier holds based on laser point clouds, specifically includes step 5: Contour extraction is performed on the fused edge image, and polygon fitting is performed on each contour. Quadrilateral contours are retained as candidate cabins, and the minimum bounding rectangle of the quadrilateral contours is obtained. Its height is Width is Remove graphics whose aspect ratio does not meet the following conditions: ; in, To set a threshold.

[0012] The aforementioned real-time fusion detection method for bulk carrier holds based on laser point clouds, specifically includes step 6: obtaining the coordinates of the four vertices of the minimum bounding rectangle of the candidate hold. , Located in the deck coordinate system On the xy plane, using the deck coordinate system unit vector and Back-project it onto the three-dimensional deck coordinate system Below, it is represented as: ； Transform the three-dimensional vertices from the three-dimensional deck coordinate system Transform to 3D point cloud coordinate system , represented as: .

[0013] The beneficial effects of the present invention are: (1) The present invention reduces the dimension of three-dimensional point cloud data to two-dimensional images, which greatly reduces the algorithm complexity and realizes real-time detection; (2) This invention is applicable to complex environments and dynamic scenarios in port operation areas. It can acquire ship cabins in dynamic scenarios in real time, rather than detecting ship cabins in a static environment with only ship data, which is more effective. (3) The present invention integrates the depth and intensity information of lidar, which is superior to the existing technology in terms of algorithm robustness and can detect the cabin in real time in complex environments. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the process of this invention; Figure 2 This is a schematic diagram of the cabin point cloud data and detection results in an embodiment of the present invention; Figure 3 This is an intensity image from an embodiment of the present invention; Figure 4 This is an intensity binary image in an embodiment of the present invention; Figure 5 This is the intensity mask image in the embodiment of the present invention; Figure 6 This is a depth image in an embodiment of the present invention; Figure 7 This is a depth edge image in an embodiment of the present invention; Figure 8 This is an edge-blended image in an embodiment of the present invention; Figure 9 These are candidate cabin detection images in an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] This invention provides a real-time fusion detection method for bulk cargo ship holds based on laser point clouds. This method addresses the technical problems in existing technologies for ship hold detection, such as high algorithm complexity, limited effectiveness in static environments, and poor robustness in complex environments, which are caused by using deep learning or multi-sensor fusion techniques. By fusing depth and intensity data from laser point clouds, real-time ship hold detection can be achieved in complex port environments.

[0017] This invention provides a real-time fusion detection method for bulk carrier holds based on laser point clouds, such as... Figure 1 As shown, the process includes the following steps: Real-time 3D point cloud of the port scene is scanned using LiDAR SLAM; the deck plane equation and its normal vector are obtained using the RANSAC algorithm; the 3D point cloud is projected onto the 2D deck plane, and an intensity image with point cloud intensity as information is constructed; threshold segmentation is performed on the intensity image to obtain a binary intensity image, and morphological dilation is performed on the binary image to obtain an intensity mask image; the 3D point cloud is projected onto the 2D deck plane, and a depth image with projection distance as information is constructed; Canny edge detection is performed on the depth image to obtain a depth edge image, and the intensity mask image is fused with the depth edge image to obtain a fused edge image; contour extraction is performed on the fused edge image, and polygon fitting is performed to obtain the coordinates of the four vertices of the minimum bounding rectangle of the candidate cabin; the coordinates of the four vertices of the candidate cabin are back-projected into 3D space to obtain the final 3D space rectangular coordinate points of the cabin. Specifically, as follows: (1) Real-time 3D scene scanning of the port scene is performed using the fast-lio2 SLAM algorithm to obtain lidar point cloud data. Where x, y, z are coordinate values. For example, the intensity value, Figure 2 As shown, the overall scene size is greater than The area is square meters, with the land side on the left and the sea side on the right. The lidar is mounted vertically downwards on the trunk-like structure of the gantry crane. The bulk carrier is moored on the sea side, with two cabins located on the ship, for point cloud data analysis. The equation of the deck plane is obtained using the RANSAC plane detection algorithm. and its unit normal vector , .

[0018] And determine the unit normal vector and Axial unit vector included angle Less than a given threshold If satisfied If so, the normal vector extraction is correct. The value is 15 degrees, meaning the two are nearly parallel; .

[0019] (2) Transfer the 3D scene point cloud data Projected onto deck plane Above, with the unit normal vector of the deck plane as for Axis, on the plane and orthogonal unit vectors for Axis, with and The unit vector of the cross product Construct a deck coordinate system with the y-axis as the axis. , ; Then the coordinates of any point in the point cloud data Projection point on the deck plane for, .

[0020] Therefore, the origin of the point cloud data coordinates Projection point on the deck plane for , will point Defined as deck coordinate system If the origin of the coordinate system is given, then any point in the point cloud data can be considered a point. In the deck coordinate system The coordinates below are, .

[0021] 3D scene point cloud data Dimensional reduction to a two-dimensional deck plane The method changes the detection of ship cabins from point cloud data to image data, which greatly reduces the computational load of ship cabin detection, thereby reducing the complexity of the algorithm and improving the real-time performance of the operation.

[0022] (3) In the deck coordinate system Only (x,y) coordinates are retained and discretized to construct a point cloud intensity map. For information intensity image ,like Figure 3 As shown, each sampled pixel Preserve 3D scene points Concentrated intensity projected onto the pixel coordinates The minimum value is significant because when a laser beam is projected onto the edge of a scene, the intensity value changes due to the change in the scene's incident angle. Often smaller, assuming there is 3D scene points Discretized sampling points fall into pixels The corresponding strength value is... for, .

[0023] (4) Intensity image Otsu thresholding is used to obtain a binary intensity image, such as... Figure 4 As shown, it preserves the point cloud edge region data and removes the complex background on the left landside, improving the overall robustness of the algorithm. However, due to the fragmentation of the edge region data, further morphological dilation of the binary image is needed to enhance the edge region data, ultimately obtaining the intensity mask image, as shown. Figure 5 As shown, it increases the original range of edge region data.

[0024] (5) In the deck coordinate system By retaining only the (x,y) coordinates and discretizing them, a depth image with projection distance d as information is constructed. ,like Figure 6 As shown, each sampled pixel Preserve 3D scene points The maximum value of the distance d projected onto the point set at this pixel coordinate is significant because when observing the point cloud data from the top of the deck plane, only the point cloud data at the very top can be observed, while other data will be occluded by the topmost point. Assume there are N 3D scene points. Discretized sampling points fall into pixels The corresponding depth value for, .

[0025] The depth value is then normalized to 0~255 for display. The larger the value, the higher the height and the brighter the brightness.

[0026] (6) Depth images Canny edge detection is used to obtain a depth edge image, such as... Figure 7 As shown, it retains edge regions with large depth variations and removes flat data with small depth variations. At this point, Figure 5 The intensity mask image in Figure 7 Data fusion is performed on the depth edge images to obtain the final fused edge image, such as... Figure 8As shown, this graph retains only the edge data of ships, gantry cranes, and hoppers, while removing complex background data such as storage yards and conveyor belts, thus improving the robustness of the algorithm.

[0027] (7) Extract contours from the fused edge image and perform polygon fitting on each contour, retaining the quadrilateral contours as candidate cabins, such as... Figure 9 As shown, obtain the minimum bounding rectangle of the quadrilateral's outline. Its height is Width is The coordinates of the four vertices are respectively , And remove those whose aspect ratio does not meet the following conditions. ; Here The value is 0.85, meaning the hatch opening is approximately rectangular.

[0028] (8) Obtain the coordinates of the four vertices of the minimum bounding rectangle of the candidate ship cabin. Located in the deck coordinate system On the xy plane, the deck coordinate system is used. unit vector and Back-project it onto the three-dimensional deck coordinate system Below can be represented as, ; Then, change the three-dimensional vertices from the three-dimensional deck coordinate system. Transform to 3D point cloud coordinate system This can be represented as the final three-dimensional cabin extraction result, as follows: Figure 2 As shown in the middle pink box, .

[0029] This invention integrates depth and intensity data from laser point clouds to achieve real-time cabin detection in complex environments, improving the robustness of cabin detection in dynamic environments. It projects three-dimensional point cloud data onto a two-dimensional deck plane, extracts quadrilaterals from the image to fit the cabin edges, and uses cabin dimensions to filter out the actual cabins.

[0030] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A real-time fusion detection method for bulk carrier holds based on laser point clouds, characterized in that, Includes the following steps: Step 1: Obtain the real-time 3D point cloud of the port scene, and obtain the deck plane equation and normal vector; Step 2: Project the 3D scene point cloud onto the 2D deck plane to construct an intensity image with point cloud intensity as information and a depth image with projection distance as information; Step 3: Binarize the intensity image to obtain a binary intensity image, and then perform morphological dilation on the binary image to obtain an intensity mask image; Step 4: Obtain a depth edge image based on the depth image obtained in Step 2, and fuse the depth edge image with the intensity mask image obtained in Step 3 to obtain a fused edge image; Step 5: Extract the contours from the fused edge image and perform polygon fitting, retaining the quadrilaterals as candidate cabins, and obtain the minimum bounding rectangle of the quadrilaterals. Step 6: Obtain the coordinates of the four vertices of the smallest bounding rectangle, and back-project them into three-dimensional space to obtain the coordinates of the final three-dimensional space rectangle of the cabin.

2. The real-time fusion detection method for bulk carrier holds based on laser point clouds according to claim 1, characterized in that, In step 1, the SLAM algorithm is used to perform real-time 3D scene scanning of the port scene to acquire lidar point cloud data. ,in These are coordinate values. The intensity value is used to obtain the deck plane equation from the point cloud data using a planar detection algorithm. and its unit normal vector : ; And determine the unit normal vector and Axial unit vector included angle Less than a given threshold If satisfied If so, the normal vector extraction is correct.

3. The real-time fusion detection method for bulk carrier holds based on laser point clouds according to claim 1, characterized in that, Step 2, projecting the 3D scene point cloud onto the 2D deck plane, specifically includes: transferring the 3D scene point cloud data... Projected onto deck plane superior, ; with the unit normal vector of the deck plane as for Axis, in plane with orthogonal unit vectors for Axis, with and The unit vector of the cross product Construct a deck coordinate system with the y-axis as the axis. Then the coordinates of any point in the point cloud data Projection point on the deck plane for: ； Origin of point cloud data coordinates Projection point on the deck plane for , will point Defined as deck coordinate system If the origin of the coordinate system is given, then any point in the point cloud data can be considered a point. In the deck coordinate system The coordinates below are: 。 4. The real-time fusion detection method for bulk carrier holds based on laser point clouds according to claim 3, characterized in that, The specific process of constructing the intensity image in step 2 includes: in the deck coordinate system Only the (x,y) coordinates are retained and discretized to construct a point cloud intensity map. For information intensity image Each sampled pixel Preserve 3D scene points Concentrated intensity projected onto the pixel coordinates The minimum value corresponds to the strength value. for: ; in, Indicates falling pixel 3D scene points Number of discretized sampling points.

5. The real-time fusion detection method for bulk carrier holds based on laser point clouds according to claim 1, characterized in that, The specific process of constructing the depth image in step 2 includes: in the deck coordinate system By retaining only the (x,y) coordinates and discretizing them, a depth image with projection distance d as information is constructed. Each sampled pixel Preserve 3D scene points The maximum distance d projected onto the set of points at that pixel coordinate is the corresponding depth value. for: ； Where N represents the falling pixel. 3D scene points Number of discretized sampling points.

6. The real-time fusion detection method for bulk carrier holds based on laser point clouds according to claim 1, characterized in that, Step 5 specifically includes: Contour extraction is performed on the fused edge image, and polygon fitting is performed on each contour. Quadrilateral contours are retained as candidate cabins, and the minimum bounding rectangle of the quadrilateral contours is obtained. Its height is Width is Remove graphics whose aspect ratio does not meet the following conditions: ; in, To set a threshold.

7. The real-time fusion detection method for bulk carrier holds based on laser point clouds according to claim 3, characterized in that, Step 6 specifically includes: obtaining the coordinates of the four vertices of the minimum bounding rectangle of the candidate cabin. , Located in the deck coordinate system On the xy plane, using the deck coordinate system unit vector and Back-project it onto the three-dimensional deck coordinate system Below, is represented as, ； Transform the three-dimensional vertices from the three-dimensional deck coordinate system Transform to 3D point cloud coordinate system , is represented as: 。

Images