Bulk cargo ship cabin real-time fusion detection method based on laser point cloud

Abstract

The application discloses a bulk cargo ship cabin real-time fusion detection method based on a laser point cloud, which comprises the following steps: acquiring a real-time three-dimensional scene point cloud of a port scene, acquiring a deck plane equation and a normal vector; projecting the three-dimensional scene point cloud to a two-dimensional deck plane to construct an intensity image taking point cloud intensity as information and a depth image taking a projection distance as information; obtaining an intensity mask image from the intensity image; acquiring a depth edge image according to the depth image, fusing the depth edge image and the intensity mask image to obtain a fusion edge image; performing contour extraction on the fusion edge image and performing polygon fitting to retain quadrilaterals therein as candidate cabins to acquire minimum circumscribed rectangles of the quadrilaterals; acquiring four vertex coordinates of the minimum circumscribed rectangles and back-projecting the four vertex coordinates to a three-dimensional space to acquire three-dimensional space rectangular coordinate points of final cabins. The application is suitable for a complex environment and a dynamic scene in a port operation area and can acquire the cabins in the dynamic scene in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent detection of port machinery equipment, and in particular to a bulk cargo ship cabin real-time fusion detection method based on laser point cloud. BACKGROUND

[0002] With the gradual development of artificial intelligence technology and the continuous improvement of hardware and software device performance, the traditional port loading and unloading equipment manual operation mode is gradually moving towards semi-automatic or even fully automatic operation mode, and at the same time, the perception, positioning and anti-collision technologies of visual multi-sensor fusion such as laser radar and camera are attracting more and more attention.

[0003] Ship cabin detection is a key prerequisite for realizing intelligent loading and unloading of port loading and unloading equipment, which provides loading and unloading equipment with operation environment perception and obtains the distribution of goods in the ship cabin; it constructs an accurate three-dimensional operation model to support the path planning of the grab bucket and the positioning of the grabbing point; it can effectively prevent collision between the grab bucket and the ship cabin, and improve the safety of intelligent loading and unloading operation.

[0004] The existing ship cabin detection technology mainly has the following shortcomings: (1) using deep learning technology or multi-sensor fusion technology, the algorithm complexity is high and the process is complex, when multiple algorithms run on a device, it is difficult to meet real-time requirements, resulting in rising costs of port equipment; (2) it can detect the ship cabin in a static environment with only ship data, but it cannot effectively and real-time detect the ship cabin position in a complex port operation environment and dynamic scene; (3) it fails to fuse the depth and intensity information of laser radar, which has certain defects in algorithm robustness, and is not conducive to real-time detection of the ship cabin in a complex port environment. SUMMARY

[0005] In order to overcome the above problems in the prior art, the present application provides a bulk cargo ship cabin real-time fusion detection method based on laser point cloud.

[0006] The technical scheme adopted by the present application to solve its technical problems is: a bulk cargo ship cabin real-time fusion detection method based on laser point cloud, comprising the following steps:

[0007] Step 1, obtaining real-time three-dimensional scene point cloud of port scene, obtaining deck plane equation and normal vector;

[0008] Step 2, projecting the three-dimensional scene point cloud to the two-dimensional deck plane to construct an intensity image with point cloud intensity information and a depth image with projected distance information;

[0009] Step 3, binarizing the intensity image to obtain an intensity binary image, and performing morphological dilation on the binary image to obtain an intensity mask image;

[0010] Step 4: Obtain a depth edge image based on the depth image obtained in Step 2, and fuse the depth edge image with the intensity mask image obtained in Step 3 to obtain a fused edge image;

[0011] Step 5: Extract the contours from the fused edge image and perform polygon fitting, retaining the quadrilaterals as candidate cabins, and obtain the minimum bounding rectangle of the quadrilaterals.

[0012] Step 6: Obtain the coordinates of the four vertices of the smallest bounding rectangle, and back-project them into three-dimensional space to obtain the coordinates of the final three-dimensional space rectangle of the cabin.

[0013] The aforementioned method for real-time fusion detection of bulk carrier holds based on laser point clouds, in step 1, utilizes the SLAM algorithm to perform real-time 3D scene scanning of the port area to acquire laser radar point cloud data. Where x, y, z are coordinate values. The intensity value is used to obtain the deck plane equation from the point cloud data using a planar detection algorithm. and its unit normal vector :

[0014] ;

[0015] And determine the unit normal vector and Axial unit vector included angle Less than a given threshold If satisfied If so, the normal vector extraction is correct.

[0016] The above-mentioned real-time fusion detection method for bulk carrier holds based on laser point clouds, specifically step 2 of projecting the 3D scene point cloud onto the 2D deck plane includes: projecting the 3D scene point cloud data... Projected onto deck plane superior, ; with the unit normal vector of the deck plane as for Axis, in plane with orthogonal unit vectors for Axis, with and The unit vector of the cross product Construct a deck coordinate system with the y-axis as the axis. Then the coordinates of any point in the point cloud data Projection point on the deck plane for:

[0017] ；

[0018] Point cloud data coordinate origin Projection point on deck plane For , the point is defined as the coordinate origin of the deck coordinate system , then the coordinates of the arbitrary point of the point cloud data coordinates in the deck coordinate system are:

[0019] .

[0020] The above-mentioned bulk cargo ship cabin real-time fusion detection method based on laser point cloud, the specific construction process of the intensity image in step 2 includes: only retaining (x, y) coordinates and performing discrete sampling in the deck coordinate system , constructing an intensity image with point cloud intensity as information , each sampling pixel retains the minimum value of the intensity of the three-dimensional scene point projected into the point set under the pixel coordinates, and the corresponding intensity value is:

[0021] ;

[0022] Wherein, N represents the number of discrete sampling points of the three-dimensional scene point falling into the pixel .

[0023] The above-mentioned bulk cargo ship cabin real-time fusion detection method based on laser point cloud, the specific construction process of the depth image in step 2 includes: only retaining (x, y) coordinates and performing discrete sampling in the deck coordinate system , constructing a depth image with projection distance d as information , each sampling pixel retains the maximum value of the distance d of the three-dimensional scene point projected into the point set under the pixel coordinates, and the corresponding depth value is:

[0024] ;

[0025] Wherein, N represents the number of discrete sampling points of the three-dimensional scene point falling into the pixel .

[0026] The above-mentioned bulk cargo ship cabin real-time fusion detection method based on laser point cloud, step 5 specifically includes:

[0027] Contour extraction is performed on the fused edge image, and polygon fitting is performed on each contour, and the quadrilateral contour is retained as a candidate cabin, and the minimum circumscribed rectangle of the quadrilateral contour is obtained , the height is , the width is , and the graphics whose aspect ratio does not satisfy the following condition are removed:

[0028] ;

[0029] , wherein is a set threshold.

[0030] The bulk cargo cabin real-time fusion detection method based on laser point cloud, the step 6 specifically comprises: , , the four vertex coordinates of the minimum circumscribed rectangle of the candidate cabin are obtained , located on the xy plane of the deck coordinate system , the unit vectors and of the deck coordinate system are used to project them to the three-dimensional deck coordinate system

[0031] ；

[0032] The three-dimensional vertex is transformed from the three-dimensional deck coordinate system to the three-dimensional point cloud coordinate system , and is expressed as:

[0033] .

[0034] The beneficial effects of the present application are that (1) the present application reduces the three-dimensional point cloud data to a two-dimensional image, greatly reducing the algorithm complexity and realizing real-time detection;

[0035] (2) the present application is suitable for complex environments and dynamic scenes in the port operation area, and can obtain the cabin in the dynamic scene in real time, rather than detecting the cabin in a static environment with only ship data, and the effect is better;

[0036] (3) the present application fuses the depth and intensity information of the laser radar, and is superior to the prior art in algorithm robustness, and can detect the cabin in real time in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of the present application;

[0038] Figure 2 is a schematic diagram of cabin point cloud data and detection results in an embodiment of the present application;

[0039] Figure 3 is a strength image in the embodiment of the application;

[0040] Figure 4 is a strength binary image in the embodiment of the application;

[0041] Figure 5 is a strength mask image in the embodiment of the application;

[0042] Figure 6 is a depth image in the embodiment of the application;

[0043] Figure 7 is a depth edge image in the embodiment of the application;

[0044] Figure 8 is an edge fusion image in the embodiment of the application;

[0045] Figure 9 is a candidate cabin detection image in the embodiment of the application. DETAILED DESCRIPTION

[0046] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments.

[0047] The embodiment of the present application provides a bulk cargo cabin real-time fusion detection method based on laser point cloud, which is used to solve the technical problems of high algorithm complexity, detection only in static environment and poor robustness in complex environment caused by deep learning technology or multi-sensor fusion technology in the prior art. The depth data and intensity data of the laser point cloud are fused to realize real-time cabin detection in a complex port environment.

[0048] The embodiment of the present application provides a bulk cargo cabin real-time fusion detection method based on laser point cloud, which is used to solve the technical problems of high algorithm complexity, detection only in static environment and poor robustness in complex environment caused by deep learning technology or multi-sensor fusion technology in the prior art. The depth data and intensity data of the laser point cloud are fused to realize real-time cabin detection in a complex port environment. Figure 1As shown, comprising the following steps: real-time three-dimensional scene point cloud of port scene is scanned by SLAM depending on light radar, deck plane equation and its normal vector are obtained by using RANSAC algorithm; three-dimensional scene point cloud is projected onto two-dimensional deck plane, and an intensity image taking point cloud intensity as information is constructed; intensity binary image is obtained by threshold segmentation of intensity image, and intensity mask image is obtained by morphological dilation of binary image; three-dimensional scene point cloud is projected onto two-dimensional deck plane, and a depth image taking projected distance as information is constructed; depth edge image is obtained by Canny edge detection of depth image, and fusion edge image is obtained by fusing intensity mask image and depth edge image; four vertex coordinates of the minimum circumscribed rectangle of the candidate cabin are obtained by contour extraction and polygon fitting of the fusion edge image; three-dimensional space rectangular coordinate points of the final cabin are obtained by back projecting the four vertex coordinates of the candidate cabin to three-dimensional space. Specifically as follows:

[0049] (1) Real-time three-dimensional scene scanning of port scene is performed by SLAM algorithm depending on fast-lio2, and laser radar point cloud data is obtained , wherein x, y and z are coordinate values, is an intensity value, as shown in Figure 2 , wherein the entire scene size is greater than square meters, the left side is the land side, the right side is the sea side, the laser radar is installed on the elephant trunk beam of the door machine and is vertically downward, the bulk cargo ship is parked on the sea side, two cabins are located on the ship, and the point cloud data is obtained by using RANSAC plane detection algorithm to obtain the deck plane equation and the unit normal vector

[0050] .

[0051] and the unit normal vector is less than a given threshold , if , the normal vector is correct, and the value of is 15 degrees, that is, they are close to parallel;

[0052] .

[0053] (2) Project three-dimensional scene point cloud data onto the deck plane , take the unit normal vector of the deck plane as as axis, and take the unit vector orthogonal to on the plane​​​ For axis, to and unit vector of cross product For y axis, construct deck coordinate system ,

[0054] ;

[0055] The point cloud data coordinates of an arbitrary point The projection point on the deck plane is,

[0056] .

[0057] The point cloud data coordinates of the origin point The projection point on the deck plane is Define the point as the coordinate origin of the deck coordinate system , then the coordinates of an arbitrary point in the point cloud data in the deck coordinate system are,

[0058] .

[0059] The three-dimensional scene point cloud data is reduced to a two-dimensional deck plane , and the detected cabin in the point cloud data is changed to the detected cabin in the image data, so that the operation amount of the cabin detection is greatly reduced, and the complexity of the algorithm is reduced and the real-time operation is improved.

[0060] (3) Only the (x, y) coordinates are retained and discretely sampled in the deck coordinate system , and an intensity image with point cloud intensity as information is constructed , as shown in Figure 3 , each sampling pixel retains the minimum value of the intensity of the three-dimensional scene point projected to the point set under the pixel coordinates , which means that when the laser is projected to the edge area of the scene, the intensity value is often small due to the change of the scene incident angle. Assuming that there are three-dimensional scene points discretely sampled into the pixel , the corresponding intensity value is,

[0061] .

[0062] (4) Intensity image Otsu threshold segmentation to obtain intensity binary image, as shown in Figure 4 , which retains the point cloud edge region data, eliminates the complex background on the left land side, and improves the robustness of the overall algorithm. However, due to the existence of edge region data, further morphological dilation is needed for the binary image to enhance the edge region data, and finally obtain the intensity mask image, as shown in Figure 5 , which increases the range of the original edge region data.

[0063] (5) In the deck coordinate system , only (x, y) coordinates are retained and discretely sampled to construct a depth image with projection distance d as information , as shown in Figure 6 , each sampling pixel retains the maximum value of the distance d of the three-dimensional scene point projected to the pixel coordinate set, which means that when observing point cloud data from the top of the deck plane, only the point cloud data at the top can be observed and other data will be blocked by the top. Assuming that there are N three-dimensional scene points discretely sampled points fall into the pixel , then the corresponding depth value is

[0064] .

[0065] The depth value is then normalized to 0-255 for display. The larger the value, the higher the height and the brighter the brightness.

[0066] (6) Canny edge detection is performed on the depth image to obtain a depth edge image, as shown in Figure 7 , which retains the edge region with large depth changes and eliminates the flat data with small depth changes. At this time, the intensity mask image in Figure 5 is fused with the depth edge image in Figure 7 to obtain a fused edge image, as shown in Figure 8 , which only retains the edge data of the ship, the door machine and the funnel, and eliminates the complex background data such as the yard and the conveyor belt, improving the robustness of the algorithm.

[0067] (7) Contour extraction is performed on the fused edge image, and polygon fitting is performed on each contour to retain the quadrilateral contour as a candidate cabin, as shown in Figure 9 , the minimum circumscribed rectangle of the quadrilateral contour is obtained , with a height of and a width of , and the four vertex coordinates are , And remove those whose aspect ratio does not meet the following conditions.

[0068] ;

[0069] Here The value is 0.85, meaning the hatch opening is approximately rectangular.

[0070] (8) Obtain the coordinates of the four vertices of the minimum bounding rectangle of the candidate ship cabin. Located in the deck coordinate system On the xy plane, the deck coordinate system is used. unit vector and Back-project it onto the three-dimensional deck coordinate system Below can be represented as,

[0071] ;

[0072] Then, change the three-dimensional vertices from the three-dimensional deck coordinate system. Transform to 3D point cloud coordinate system This can be represented as the final three-dimensional cabin extraction result, as follows: Figure 2 As shown in the middle pink box,

[0073] .

[0074] This invention integrates depth and intensity data from laser point clouds to achieve real-time cabin detection in complex environments, improving the robustness of cabin detection in dynamic environments. It projects three-dimensional point cloud data onto a two-dimensional deck plane, extracts quadrilaterals from the image to fit the cabin edges, and uses cabin dimensions to filter out the actual cabins.

[0075] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A bulk cargo ship cabin real-time fusion detection method based on laser point cloud, characterized in that, The method comprises the following steps: Step 1, acquiring a real-time three-dimensional scene point cloud of a port scene, acquiring a deck plane equation and a normal vector; Step 2, projecting the three-dimensional scene point cloud to a two-dimensional deck plane, and constructing an intensity image taking point cloud intensity as information and a depth image taking projected distance as information; Step 3, binarizing the intensity image to obtain an intensity binary image, and performing morphological dilation on the binary image to obtain an intensity mask image; Step 4, acquiring a depth edge image according to the depth image obtained in step 2, fusing the depth edge image and the intensity mask image obtained in step 3 to obtain a fused edge image; Step 5, performing contour extraction on the fused edge image, and performing polygon fitting to retain quadrilaterals therein as candidate cabins, and acquiring a minimum circumscribed rectangle of the quadrilaterals; Step 6, acquiring coordinates of four vertexes of the minimum circumscribed rectangle, and back-projecting the four vertexes to a three-dimensional space to acquire three-dimensional space rectangular coordinate points of a final cabin; The specific construction process of the intensity image in step 2 includes: in the deck coordinate system Only the coordinates are reserved and discrete sampling is performed to construct an intensity image with point cloud intensity information Each sampling pixel reserves a three-dimensional scene point Project the minimum intensity of the point set in the pixel coordinates The corresponding intensity value is: wherein, represents a three-dimensional scene point falling into a pixel of a three-dimensional scene point number of discretized sampling points; The depth image construction process in step 2 includes: in the deck coordinate system Only the Coordinates are kept and discretely sampled to construct a depth image with the projection distance As information Each sampling pixel Keeps the three-dimensional scene point Projecting to the distance Maximum value in the point set under the pixel coordinates, then the corresponding depth value Is: ； where N denotes the number of discretized sampling points falling into the pixel of the three-dimensional scene point .

2. The laser point cloud-based bulk cargo ship cabin real-time fusion detection method according to claim 1, characterized in that, In the step 1, the port scene is scanned in real time by using a SLAM algorithm to obtain laser radar point cloud data wherein is a coordinate value, is an intensity value, a deck plane equation is obtained by using a plane detection algorithm on the point cloud data and a unit normal vector thereof : and discriminant unit normal vector and axis unit vector angle less than a given threshold , if then the normal vector extraction is correct.

3. The laser point cloud-based real-time bulk cargo hold fusion detection method according to claim 1, characterized in that, The step 2 of projecting the three-dimensional scene point cloud to the two-dimensional deck plane specifically comprises: projecting the three-dimensional scene point cloud data to the deck plane , ; taking the unit normal vector of the deck plane as the axis, taking the unit vector on the plane orthogonal to the axis, taking the unit vector of the cross product of and the axis, taking the unit vector of the cross product of and the axis, and taking the unit vector of the cross product of and the axis to construct the deck coordinate system , then the projection point of the point cloud data coordinate arbitrary point on the deck plane is ​ ； Point cloud data coordinate origin The projection point on the deck plane For , the point is defined as the coordinate origin of the deck coordinate system , the coordinates of the arbitrary point of the point cloud data in the deck coordinate system are: 。 4. The laser point cloud-based real-time bulk cargo hold fusion detection method according to claim 1, characterized in that, The step 5 specifically comprises: Contour extraction is performed on the fused edge image, and polygon fitting is performed on each contour, and the quadrilateral contours among them are retained as candidate cabins, and the minimum circumscribed rectangle of the quadrilateral contour is obtained , the height of which is , the width of which is , and the graphics whose aspect ratio does not satisfy the following condition are removed: wherein is a set threshold value.

5. The laser point cloud based bulk cargo hold real-time fusion detection method according to claim 3, characterized in that, The step 6 specifically comprises: acquiring four vertex coordinates of the minimum circumscribed rectangle of the candidate cabin , , located on the plane of the deck coordinate system , , using the unit vectors of the deck coordinate system , and , and back-projecting them to the three-dimensional deck coordinate system , denoted as ； transforming three-dimensional vertices from a three-dimensional deck coordinate system to a three-dimensional point cloud coordinate system is represented as: 。

Images