Unloading area retaining wall dumping line real-time detection method based on point cloud segmentation

Abstract

The invention relates to the technical field of automatic driving perception, in particular to an unloading area retaining wall dumping line real-time detection method based on point cloud segmentation, and the method comprises the steps: obtaining laser radar point cloud data through a backward laser radar on an unmanned mine car; performing preprocessing and semantic segmentation on the laser radar point cloud data to obtain a point cloud containing semantic information; performing coordinate transformation on the point cloud containing the semantic information according to the combined navigation pose and the external parameter calibration parameters to obtain point cloud information in a world coordinate system; determining a virtual track based on the reversing direction of the vehicle, and searching the point cloud information in the world coordinate system based on the virtual track to obtain boundary points; clustering and fitting the boundary points to obtain a dumping line; according to the invention, the barricade identification accuracy in multiple scenes can be improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology, and in particular to a real-time detection method for a dumping line of a retaining wall in an unloading area based on point cloud segmentation. Background Art

[0002] In special scenarios such as mining areas, since there are no lane markings, retaining walls are a key factor in restricting the driving of autonomous transport vehicles. Accurate identification of retaining walls can ensure the safe driving of autonomous transport vehicles in various mining environments, especially when high-precision maps are not updated in a timely manner. The unloading area is a key area where the roads in the mining area change frequently. When problems such as the accumulation of residual materials during the unloading process cause the retaining wall to shift, which in turn leads to inaccurate pre-parking positions, safety accidents are likely to occur. Therefore, accurate retaining wall identification plays an important role in ensuring the safe driving of autonomous vehicles.

[0003] The shape of the retaining wall in the mining area is complex and changeable. In the unloading area, the terrain changes significantly and there are many piles of residual soil. Therefore, how to accurately identify the retaining wall is a huge challenge for the autonomous driving environment perception system. In addition, the shape fitting of complex retaining walls is also of great help for accurate docking. LiDAR can perceive the surrounding environment all day long. Through LiDAR, the task of retaining wall detection in complex scenes can be realized, but sparse point clouds and lack of semantic information also bring challenges to the detection of retaining walls in mining areas.

[0004] At present, the retaining wall detection methods for mining environments mainly include traditional methods and deep learning methods. The Chinese invention patent application with publication number CN113933859A discloses a method for detecting the road surface and retaining walls on both sides of an unmanned mining truck driving scene. By obtaining a triangular mesh model in the driving scene, the point cloud data is plane fitted, and the road surface points and retaining wall points are distinguished according to the variance. This method can effectively detect the road surface and retaining walls on the driving road, but it is difficult to achieve ideal results in the complex environment of the unloading area. The Chinese invention patent application with publication number CN117233792A discloses a retaining wall detection method, a retaining wall detection device, and a storage medium. By dividing the point clouds in different directions of different partitions through multiple partitions, the retaining wall point cloud detection is realized, but this method can only reduce the interference of the recognition results caused by the complex terrain to a certain extent. The Chinese invention patent application with publication number CN115457496A discloses a method for detecting retaining wall point clouds based on point cloud segmentation. It uses a deep learning detection model to detect targets, extracts geometric features of different point cloud targets for feature representation, and performs binary classification based on the feature results to achieve retaining wall recognition. This method has no requirements for the scene, but the target detection results affect the recognition accuracy of retaining wall point clouds, and the complex and changeable retaining wall shapes are prone to interfere with the detection results. Summary of the invention

[0005] In view of the above problems, the present invention provides a real-time detection method for the dumping line of the retaining wall in the unloading area based on point cloud segmentation, which solves the technical problem that the retaining wall recognition algorithm in the prior art is difficult to meet the requirements of accurate recognition of retaining walls in multiple scenarios.

[0006] The present invention provides a real-time detection method for the dumping line of a retaining wall in an unloading area based on point cloud segmentation, comprising the following steps:

[0007] Step S1, obtaining laser radar point cloud data through the rear-facing laser radar on the unmanned mining vehicle;

[0008] Step S2: preprocessing and semantic segmenting the laser radar point cloud data to obtain a point cloud containing semantic information;

[0009] Step S3, performing coordinate transformation on the point cloud containing semantic information according to the combined navigation posture and the external parameter calibration parameters to obtain point cloud information in the world coordinate system;

[0010] Step S4, determining a virtual track based on the reversing direction of the vehicle, and searching the point cloud information in the world coordinate system based on the virtual track to obtain boundary points;

[0011] Step S5: clustering and fitting the boundary points to obtain the soil discharge line.

[0012] Preferably, the step S2 specifically includes:

[0013] Step S2-1, mapping the three-dimensional point cloud in the laser radar point cloud data into a two-dimensional image by spherical projection;

[0014] Step S2-2: Use a convolutional attention module and a multi-scale residual down / up sampling module to extract features from the two-dimensional image to obtain semantic information of each point in the two-dimensional image as the point cloud containing semantic information.

[0015] Preferably, the step S2-1 specifically includes:

[0016] The three-dimensional point cloud is spherically projected into a two-dimensional image, which is a range view image. The mapping formula is:

[0017]

[0018] Among them, x, y, z are the three-dimensional coordinates of a point in the point cloud, w and h are the width and height of the RV image, r represents the Euclidean distance from the origin to the lidar scanning point, and f down represents the minimum vertical field of view that the LiDAR can scan, and f represents the vertical field of view of the LiDAR.

[0019] Preferably, the step S2-2 specifically includes:

[0020] Using SalsaNext as the benchmark network, a convolutional attention module and a multi-scale residual down / up sampling module are used to extract features from the two-dimensional image;

[0021] The convolutional attention module includes 1×k convolution and k×1 convolution, and the relationship between channels is modeled by 1×1 convolution, and its output is used as the attention weight to give the input of the convolutional attention module;

[0022] The mathematical expression of the multi-scale residual down / up sampling module is:

[0023]

[0024] Among them, F in Conv is the input feature map. 1×1 Indicates that a 1×1 convolution kernel is used for convolution operation; Scale i Indicates the use of asymmetric convolutions of different scales; Cat is the feature concatenation operation, F out Represents the output point cloud containing semantic information.

[0025] Preferably, the step S3 specifically includes:

[0026] Step S3-1, obtain the real-time position and posture T of the integrated navigation B , obtain the calibration parameter matrix T between the laser radar and the integrated navigation LB ;

[0027] Step S3-2: Use formula T L =T LB *T B Calculate the position T of the laser radar in the world coordinate system L ;

[0028] Step S3-3: Using T L The point cloud containing semantic information at each moment is converted into the world coordinate system.

[0029] Preferably, the step S4 specifically includes:

[0030] Step S4-1, taking the heading angle θ of the vehicle when reversing as a reference, rotating 90° clockwise as the virtual trajectory construction direction, uniformly constructing trajectory points at a certain interval d in the virtual trajectory construction direction, and the total length of the virtual trajectory is l;

[0031] Step S4-2, construct a hash index, divide the point cloud in the world coordinate system into a rectangular grid, in the rectangular grid, take each trajectory point in the virtual trajectory as the search starting point, search for retaining wall points on both sides of the virtual trajectory, select the coordinate extreme point in the direction perpendicular to the retaining wall as the boundary feature point, after completing the search of all trajectory points, determine all boundary feature points as boundary points.

[0032] Preferably, the step S5 specifically includes:

[0033] Step S5-1, performing Euclidean clustering on the boundary points to obtain a plurality of boundary point clusters;

[0034] Step S5-2: Perform boundary fitting based on B-spline on the boundary points in each boundary point cluster to obtain the soil discharge line.

[0035] Preferably, the step S5-2 specifically includes:

[0036] Boundary fitting based on B-spline is performed on the boundary points in each boundary point cluster. The mathematical expression of the B-spline curve can be expressed as:

[0037]

[0038] Among them, C(t) is the defined B-spline curve, P j is the jth control point, K is the total number of control points, N j,k (t) is the B-spline basis function corresponding to the kth order of the ith control point, and t represents the position parameter on the curve.

[0039] Compared with the prior art, the present invention has at least the following beneficial effects:

[0040] (1) The present invention significantly improves the real-time detection accuracy of retaining wall point clouds by introducing an improved point cloud semantic segmentation algorithm. Combined with the convolutional attention module and the multi-scale residual sampling module, the segmentation accuracy and inference speed are ensured, and high-precision retaining wall recognition and boundary detection can be achieved.

[0041] (2) In the process of point cloud mapping, the present invention uses the real-time posture obtained by combined navigation to accurately convert the lidar point cloud into the world coordinate system, and solves the problem of sparse point cloud by superposition of multiple frames, thereby improving the density of point cloud and ensuring the accuracy of trajectory construction.

[0042] (3) The present invention uses a method of fitting boundary points with a B-spline curve to achieve accurate acquisition of the boundary information of the retaining wall, which is of great help to the update of the parking position. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0044] Figure 1 A flow chart of a method for real-time detection of dumping lines of retaining walls in unloading areas based on point cloud segmentation provided by the present invention;

[0045] Figure 2 This is a schematic diagram of the architecture of the real-time detection method for the dumping line of the retaining wall in the unloading area based on point cloud segmentation provided by the present invention.

[0046] Figure 3 This is a diagram of the detection effect provided by the present invention.

[0047] Figure 4 Schematic diagram of the virtual track and rear retaining wall provided by the present invention. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0049] The present invention focuses on improving the parking accuracy and efficiency of unmanned transport vehicles in the unloading area. In this scenario, the transport vehicle can accurately identify the position of the retaining wall dumping line based on the rear sensor data and send it to other modules for real-time update of the docking point. The road and retaining wall are segmented by the semantic segmentation method, and the retaining wall dumping line information is automatically extracted based on the retaining wall data, so as to update the docking point and achieve accurate docking.

[0050] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 1 , Figure 2 As shown, a real-time detection method for the dumping line of the retaining wall in the unloading area based on point cloud segmentation is disclosed, and the specific implementation steps are as follows:

[0051] Step S1: Obtaining laser radar point cloud data through the rear-facing laser radar on the unmanned mining vehicle.

[0052] In some embodiments, a rear sensor is installed on the unmanned mining vehicle, and the rear sensor is a laser radar, which obtains three-dimensional point cloud data of the rear environment of the unmanned mining vehicle. The point cloud data includes three-dimensional coordinate information, such as Figure 3 shown.

[0053] Step S2: preprocess and semantically segment the laser radar point cloud data to obtain a point cloud containing semantic information.

[0054] In this step, the point cloud is first preprocessed, and the three-dimensional point cloud is spherically projected into a two-dimensional image. The two-dimensional image is a range view (RV) image, and the mapping formula is:

[0055]

[0056] Among them, x, y, z are the three-dimensional coordinates of a point in the point cloud, w and h are the width and height of the RV image, r represents the Euclidean distance from the origin to the lidar scanning point, and f down represents the minimum vertical field of view that the LiDAR can scan, and f represents the vertical field of view of the LiDAR.

[0057] The preprocessed two-dimensional image is semantically segmented using a segmentation model, and semantic information is added to each point. Taking SalsaNext as the benchmark network, a convolutional attention module (CAM) and a multi-scale residual down / up sampling module (RDM / RUM) are designed to improve the network segmentation accuracy and model inference speed, and to achieve fast point cloud semantic segmentation. In order to improve the segmentation accuracy, the present invention provides a convolutional attention module, which awakens spatial attention through simple element multiplication. At the same time, in order to capture multi-scale features, a convolution with small computational complexity and large receptive field is used. This convolution decomposes the k×k convolution into 1×k and k×1 convolutions, which reduces the amount of computation while ensuring the accuracy of feature extraction, so that the model can accept receptive fields of different scales without increasing additional computational complexity. Finally, the relationship between channels is modeled by a 1×1 convolution, and its output is assigned to the input of the convolutional attention module as the attention weight.

[0058] In order to improve the inference speed, the present invention provides a multi-scale residual down / up sampling module, which obtains semantic information of different scales through asymmetric convolution. The mathematical expression is:

[0059]

[0060] Among them, F in Conv is the input feature map. 1×1 Indicates that a 1×1 convolution kernel is used for convolution operation; Scale i Indicates the use of asymmetric convolutions of different scales. The padding size of the input feature map of Scale0 is 1, and a 3×3 convolution kernel is used for feature extraction. The padding size of the input feature map of Scale1 is 2, and 1×5 and 5×1 convolution kernels are used for feature extraction. The padding size of the input feature map of Scale2 is 3, and 1×7 and 7×1 convolution kernels are used for feature extraction. Cat is the feature concatenation operation, and F out Represents the output point cloud containing semantic information.

[0061] Step S3: performing coordinate transformation on the point cloud containing semantic information according to the combined navigation posture and the external parameter calibration parameters to obtain point cloud information in the world coordinate system.

[0062] Integrated navigation refers to the use of multiple navigation technologies in combination to improve the accuracy and reliability of positioning and attitude estimation. Integrated navigation can include inertial navigation, global navigation satellite system or other combinations of sensors.

[0063] In this step, first obtain the real-time position and posture T of the integrated navigation B , the calibration parameter matrix between the laser radar and the integrated navigation is recorded as T LB , T LB Describes the fixed position and attitude of the lidar relative to the integrated navigation system, using the formula T L =T LB *T B Calculate the position T of the laser radar in the world coordinate system L , using T L The point cloud containing semantic information at each moment is converted into the world coordinate system.

[0064] In some embodiments, the present invention superimposes multiple frames of point cloud data to construct a local map. By combining multiple frames of data, the sparse point cloud problem of a single frame can be alleviated, and a denser and more detailed map can be generated.

[0065] Step S4: determining a virtual track based on the reversing direction of the vehicle, and searching the point cloud information in the world coordinate system based on the virtual track to obtain boundary points.

[0066] like Figure 4 As shown in the figure, during the reversing process, the vehicle trajectory is generally perpendicular to the direction of the retaining wall. At this time, it is difficult to associate the retaining wall point with the trajectory point. Therefore, it is necessary to construct a virtual trajectory. Taking the heading angle θ of the vehicle during reversing as the reference, rotate 90° clockwise as the direction of virtual trajectory construction. In this direction, the trajectory points are uniformly constructed with a certain spacing d. The total length of the virtual trajectory is l.

[0067] A hash index is constructed, and the point cloud in the world coordinate system is divided into a rectangular grid. In the rectangular grid, each trajectory point in the virtual trajectory is used as the search starting point, and retaining wall points are searched on both sides of the virtual trajectory. The coordinate extreme value points in the direction perpendicular to the retaining wall are selected as boundary feature points. After completing the search of all trajectory points, all boundary feature points are determined as boundary points.

[0068] Step S5: clustering and fitting the boundary points to obtain the soil discharge line.

[0069] In this step, firstly, Euclidean clustering is performed on the boundary points to obtain different boundary point clusters.

[0070] Boundary fitting based on B-spline is performed on the boundary points in each boundary point cluster. The mathematical expression of the B-spline curve can be expressed as:

[0071]

[0072] Among them, C(t) is the defined B-spline curve, P j is the jth control point, K is the total number of control points, N j,k (t) is the B-spline basis function corresponding to the kth order of the jth control point, and t represents the position parameter on the curve.

[0073] After fitting is completed, boundary line extraction is realized to obtain the soil discharge line.

[0074] Although the specific embodiments of the present invention depict various actions or steps in a specific order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring all illustrated actions or steps to be performed to obtain the desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination. The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0075] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A real-time detection method for the dumping line of the retaining wall in the unloading area based on point cloud segmentation, characterized in that: The following steps are involved: Step S1, obtaining laser radar point cloud data through the rear-facing laser radar on the unmanned mining vehicle; Step S2: preprocessing and semantic segmenting the laser radar point cloud data to obtain a point cloud containing semantic information; Step S3, performing coordinate transformation on the point cloud containing semantic information according to the combined navigation posture and the external parameter calibration parameters to obtain point cloud information in the world coordinate system; Step S4, determining a virtual track based on the reversing direction of the vehicle, and searching the point cloud information in the world coordinate system based on the virtual track to obtain boundary points; Step S5: clustering and fitting the boundary points to obtain the soil discharge line.

2. The real-time detection method for the dumping line of the unloading area retaining wall based on point cloud segmentation according to claim 1 is characterized in that: The step S2 specifically includes: Step S2-1, mapping the three-dimensional point cloud in the laser radar point cloud data into a two-dimensional image by spherical projection; Step S2-2: Use a convolutional attention module and a multi-scale residual down / up sampling module to extract features from the two-dimensional image to obtain semantic information of each point in the two-dimensional image as the point cloud containing semantic information.

3. The real-time detection method for the dumping line of the unloading area retaining wall based on point cloud segmentation according to claim 2 is characterized in that: The step S2-1 specifically includes: The three-dimensional point cloud is spherically projected into a two-dimensional image, which is a range view image. The mapping formula is: Among them, x, y, z are the three-dimensional coordinates of a point in the point cloud, w and h are the width and height of the RV image, r represents the Euclidean distance from the origin to the lidar scanning point, and f down represents the minimum vertical field of view that the LiDAR can scan, and f represents the vertical field of view of the LiDAR.

4. The real-time detection method for the dumping line of the unloading area retaining wall based on point cloud segmentation according to claim 3 is characterized in that: The step S2-2 specifically includes: Using SalsaNext as the benchmark network, a convolutional attention module and a multi-scale residual down / up sampling module are used to extract features from the two-dimensional image; The convolutional attention module includes 1×k convolution and k×1 convolution, and the relationship between channels is modeled by 1×1 convolution, and its output is assigned to the input of the convolutional attention module as the attention weight; The mathematical expression of the multi-scale residual down / up sampling module is: Among them, F in Conv is the input feature map. 1×1 Indicates that a 1×1 convolution kernel is used for convolution operation; Scale i Indicates the use of asymmetric convolutions of different scales; Cat is the feature concatenation operation, F out Represents the output point cloud containing semantic information.

5. The real-time detection method for the dumping line of the unloading area retaining wall based on point cloud segmentation according to claim 4 is characterized in that: The step S3 specifically includes: Step S3-1, obtain the real-time position and posture T of the integrated navigation B , obtain the calibration parameter matrix T between the laser radar and the integrated navigation LB ; Step S3-2: Use formula T L =T LB *T B Calculate the position T of the laser radar in the world coordinate system L ; Step S3-3: Using T L The point cloud containing semantic information at each moment is converted into the world coordinate system.

6. The real-time detection method for the dumping line of the unloading area retaining wall based on point cloud segmentation according to claim 5 is characterized in that: The step S4 specifically includes: Step S4-1, taking the heading angle θ of the vehicle when reversing as a reference, rotating 90° clockwise as the virtual trajectory construction direction, uniformly constructing trajectory points at a certain interval d in the virtual trajectory construction direction, and the total length of the virtual trajectory is l; Step S4-2, construct a hash index, divide the point cloud in the world coordinate system into a rectangular grid, in the rectangular grid, take each trajectory point in the virtual trajectory as the search starting point, search for retaining wall points on both sides of the virtual trajectory, select the coordinate extreme point in the direction perpendicular to the retaining wall as the boundary feature point, after completing the search of all trajectory points, determine all boundary feature points as boundary points.

7. The method for real-time detection of dumping line of retaining wall in unloading area based on point cloud segmentation according to claim 6 is characterized in that: The step S5 specifically includes: Step S5-1, performing Euclidean clustering on the boundary points to obtain a plurality of boundary point clusters; Step S5-2: Perform boundary fitting based on B-spline on the boundary points in each boundary point cluster to obtain the soil discharge line.

8. The real-time detection method for the dumping line of the unloading area retaining wall based on point cloud segmentation according to claim 7 is characterized in that: The step S5-2 specifically includes: Boundary fitting based on B-spline is performed on the boundary points in each boundary point cluster. The mathematical expression of the B-spline curve can be expressed as: Among them, C(t) is the defined B-spline curve, P j is the jth control point, K is the total number of control points, N J,k (t) is the B-spline basis function corresponding to the kth order of the jth control point, and t represents the position parameter on the curve.

Images