A laser point cloud 3D target detection method based on unmanned system

Abstract

The present invention provides a method for three-dimensional target detection based on laser point clouds in an unmanned system, comprising the following steps: Step S1: obtaining a laser point cloud dataset based on the unmanned system, preprocessing the laser point cloud dataset, and dividing the preprocessed laser point cloud dataset into a training set and a test set according to a preset ratio of 1:1; Step S2: constructing a three-dimensional target detection model based on a laser point cloud in the unmanned system; Step S3: inputting the training set into the three-dimensional target detection model to obtain a trained three-dimensional target detection model; Step S4: inputting the test set into the trained three-dimensional target detection model to obtain a three-dimensional target detection result. The dynamic edge convolution module and adaptive feature fusion network proposed in the present invention provide richer and more refined feature representations, thereby improving the accuracy of three-dimensional target detection in laser point clouds.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional target detection in unmanned systems, and in particular to a three-dimensional target detection method based on laser point clouds in unmanned systems. Background Art

[0002] In recent years, with the continuous development of autonomous driving technology, the requirements for the intelligence of unmanned systems have also been continuously increasing. Currently, autonomous vehicles are moving towards highly automated and fully automated directions. Considering the safety of occupants and driving routes, unmanned systems need to accurately perceive the external environment to achieve timely avoidance of external obstacles and accurate driving route planning. If the position of external vehicles and pedestrians is inaccurate during driving, the possibility of traffic accidents will be greatly increased, seriously threatening the safety and stability of urban transportation. Therefore, unmanned systems need to perform three-dimensional detection of external scenes, including vehicles, pedestrians and other targets, and accurately estimate their position and distance to ensure the safe operation of urban transportation systems.

[0003] Generally speaking, the most widely used visual sensor in unmanned systems is the camera, which perceives the external scene by acquiring color and position information in the environment. However, autonomous vehicles face the problem of low visibility at night and in bad weather conditions. The image quality obtained using the camera as a visual sensor is poor and lacks depth information in the scene, which seriously affects the unmanned system's perception of the surrounding scene. LiDAR can obtain depth information in the scene and has low requirements for weather conditions. Therefore, it is very necessary to use LiDAR to obtain point cloud information for three-dimensional target detection.

[0004] With the rapid development of deep learning in recent years, deep learning-based laser point cloud target detection networks have been gradually used to achieve three-dimensional detection of vehicles, pedestrians, etc. However, the current implementation process has the following difficulties:

[0005] 1. Point cloud information is sparse and irregular, making it difficult for the network to process directly;

[0006] 2. Currently, general point cloud object detection networks only use point information without considering the topological relationship between points, that is, the characteristics of edges;

[0007] 3. After the point cloud information passes through the stacked convolutional layers, the quality of the low-level geometric features decreases, and the geometric features with rich spatial information are not fully utilized. Summary of the Invention

[0008] The present invention provides a laser point cloud three-dimensional target detection method based on an unmanned system, the purpose of which is to solve the technical problems in the background technology that the point cloud information network is difficult to directly process, the point cloud target detection network does not consider the topological relationship between points, and the quality of low-level geometric features of the point cloud information deteriorates after passing through stacked convolution layers.

[0009] In order to achieve the above-mentioned purpose, the present invention provides a laser point cloud 3D target detection method based on an unmanned system, such as Figure 1 As shown, the following steps are included:

[0010] Step S1: obtaining a laser point cloud dataset based on an unmanned system, preprocessing the laser point cloud dataset, and dividing the preprocessed laser point cloud dataset into a training set and a test set according to a preset ratio of 1:1;

[0011] Step S2: constructing a laser point cloud 3D target detection model based on an unmanned system, wherein the laser point cloud 3D target detection model includes a point cloud voxelization module, a dynamic edge feature encoder, a two-dimensional encoder, an adaptive feature fusion module, and a multi-task detection head prediction module;

[0012] The point cloud voxelization module performs regularization and densification on the point cloud data. The dynamic edge feature encoder is used to extract the edge features of the point cloud data within the voxel and perform a two-dimensional pseudo-image representation. The point cloud data includes the position information and angle information of the point cloud, specifically (x, y, z, r), where (x, y, z) is the three-dimensional coordinate of the point cloud data and r is the reflection intensity of the point cloud data. The two-dimensional encoder extracts multi-level feature information of spatial features and semantic features from the two-dimensional pseudo-image. The adaptive feature fusion module adaptively assigns weights to features at different levels through the multi-level feature information to perform feature fusion. The multi-task detection head prediction module predicts the three-dimensional target detection frame through the fused features.

[0013] Step S3: Input the training set in step S1 into the laser point cloud 3D object detection model in step S2, perform backpropagation on the laser point cloud 3D object detection model through the point cloud detection box of the training set, the 3D object detection box output, and a preset loss function, and update the model parameters, repeat the above steps in step S3, and finally obtain a trained laser point cloud 3D object detection model;

[0014] Step S4: Input the test set in step S1 into the trained laser point cloud three-dimensional target detection model to obtain a three-dimensional target detection result.

[0015] Preferably, the point cloud voxelization module in step S2 includes the following steps:

[0016] Step S221: With the laser radar as the center of the coordinate axis, the vehicle's forward direction is the x-axis, the vehicle's left direction is the y-axis, and the vehicle's upward direction is the z-axis. The point cloud data in the training set of the acquired laser point cloud dataset is represented by a four-dimensional tensor, specifically (x, y, z, r), where (x, y, z) is the three-dimensional coordinate of the point cloud data and r is the reflection intensity of the point cloud data.

[0017] Step S222: The range of the point cloud scene on the (x, y, z) coordinate axis is specified as [(0, 69.12), (-39.68, 39.68), (-3, 1)], the size of each point cloud voxel is (0.16, 0.16, 4), and the entire point cloud scene is divided into 432 × 496 × 1 voxels of equal size;

[0018] Step S223: The number of points in each voxel does not exceed 32. If it is less than 32, zero is padded. If it exceeds 32, random sampling is performed to 32. The coordinates and reflection intensity of the 32 point clouds in each voxel are used as the features of the point cloud voxel, which is mathematically represented as a tensor of (432×496×1,32,4).

[0019] Preferably, the dynamic edge feature encoder in step S2 mainly includes a nearest neighbor sampling module, an edge convolution module and a two-dimensional pseudo image generation module, which are used to extract edge features of the point cloud;

[0020] Among them, the nearest neighbor sampling module performs sampling specifically as follows:

[0021] The edge convolution module includes a first edge feature encoder and a second edge feature encoder, which respectively extract edge features of the point cloud; the first edge feature encoder includes two multi-layer perceptrons and a maximum pooling layer, and the samples obtained by the nearest neighbor sampling module are sequentially passed through the two multi-layer perceptrons and the maximum pooling layer for feature extraction and encoding output; the second edge feature encoder includes a nearest neighbor sampling layer, a multi-layer perceptron and a maximum pooling layer, and the output data of the first edge feature encoder is subjected to a second nearest neighbor sampling through the nearest neighbor sampling layer to complete dynamic updating to obtain deeper edge features, which are input into the multi-layer perceptron of the second edge convolution for feature extraction, and then a maximum pooling operation is performed to output a feature tensor.

[0022] Preferably, step S2 includes the following steps:

[0023] Step S231: Sampling by the nearest neighbor sampling module; specifically:

[0024] First, input the voxel representation of the 432×496×1 point cloud into the nearest neighbor sampling module. Each voxel contains 32 points. The feature of each point is the 4-dimensional feature of (x, y, z) coordinates and reflection intensity. The 32 points in each voxel are used as vertices. Each vertex is sampled to 5 points in each voxel. Specifically, the nearest neighbor sampling module calculates the distance between each vertex and the remaining 31 vertices, obtains the 5 closest vertices as the nearest neighbor point set of the current vertex, repeats 32 times to traverse the 32 vertices in each voxel, and connects the 32 vertices in each voxel (X i ,Y i ,Z i )(i=0,1,2,…,32) and the coordinates of the points in the nearest neighbor sampling point set (X ij ,Y ij ,Z ij )(j=0,1,…,5), build the edge relationship between points (X i -X ij ,Y i -Y ij ,Z i -Z ij ), which is mathematically represented as a tensor of (432×496×1,32×5,4), where i is the number of vertices in each voxel, totaling 32, and j is the number of points sampled by the nearest neighbor of each vertex, totaling 5;

[0025] Step S232: The edge convolution module convolves to obtain edge features of the point cloud and encodes and outputs them; specifically, it includes:

[0026] Step S2321: The coordinates (X i ,Y i ,Z i ) and the edge relationship it represents (X i -X ij ,Y i -Y ij ,Z i -Z ij ) is used as the input of the edge convolution module, specifically a tensor of (432×496×1,32×(5+1),4). It first enters the first edge convolution module and uses two multi-layer perceptrons to extract features, converting the original 4-dimensional features into 64-dimensional features. It is then encoded and outputted through the maximum pooling layer, and the output is a tensor of (432×496×1,32,64).

[0027] Step S2322: Perform a second nearest neighbor sampling on the output of the first edge convolution module to complete dynamic update to obtain deeper edge features, and input the output to the second edge convolution module to use a multi-layer perceptron to perform feature extraction, converting the 64-dimensional features into deeper 64-dimensional features, and then perform a maximum pooling operation to output a feature tensor of (432×496×1,1,64);

[0028] Step S233: The two-dimensional pseudo image generation module generates a two-dimensional pseudo image; specifically:

[0029] The two-dimensional pseudo image generation module performs a tensor conversion operation to convert the feature tensor form of (432×496×1,1,64) into a standard two-dimensional image tensor form of (432,496,64), where H=432, W=496, C=64, H is the height of the two-dimensional pseudo image, W is the width of the two-dimensional pseudo image, and C is the number of channels of the two-dimensional pseudo image.

[0030] Preferably, the two-dimensional encoder in step S2 is mainly composed of three blocks, the first block is composed of three 3×3 two-dimensional convolutional layers, the second block is composed of five 3×3 two-dimensional convolutional layers, and the third block is composed of five 3×3 two-dimensional convolutional layers, including the following steps:

[0031] Step S241: The first block extracts spatial features and / or semantic features from the input two-dimensional pseudo image to generate a 216×248×64 two-dimensional feature representation;

[0032] Step S242: The two-dimensional feature expression in step S241 is continuously input into the second block for feature extraction of spatial features and / or semantic features to generate a 108×124×128 two-dimensional feature expression;

[0033] Step S243: The two-dimensional feature expression in step S242 is input into the third block to extract spatial features and / or semantic features to generate a 54×62×256 two-dimensional feature expression;

[0034] Step S244: Two-dimensional encoder output: The three two-dimensional feature expressions output by the three blocks are the outputs of the two-dimensional encoder.

[0035] Preferably, the adaptive feature fusion module in step S2 comprises three parts: an upsampling module, a fully connected module, and a feature fusion module, and includes the following steps:

[0036] Step S251: Upsampling module upsampling: First, the three upsampling convolution layers, 1×1 (deconv) upsampling convolution, 2×2 upsampling (deconv) upsampling convolution, and 4×4 upsampling (deconv) upsampling convolution, are input to the upsampling module. The three outputs of the two-dimensional encoder, namely the 216×248×64 two-dimensional feature expression, the 108×124×128 two-dimensional feature expression, and the 54×62×256 two-dimensional feature expression, are respectively input to the three upsampling convolution layers;

[0037] Step S252: Fully connected module splicing output: The fully connected module includes two layers of fully connected layers connected in sequence. The output of the upsampling convolution layer in step S251 is then input to the fully connected module. The three inputs pass through the two fully connected layers in sequence and are spliced to output a 216×248×3 feature representation.

[0038] The first fully connected layer contains three inputs and three outputs; the three inputs are the outputs of three upsampling convolutional layers, and the three outputs are all 216×248×128 feature expressions;

[0039] The second fully connected layer includes three inputs and three outputs. The three inputs are the outputs of the first fully connected layer, and the three outputs are all 216×248×1 feature expressions. The three outputs of the second fully connected layer are concatenated to form a single output of 216×248×3.

[0040] Step S253: Feature fusion module Feature fusion: The 216×248×3 feature expression output in step S252 continues to enter the feature fusion module and performs adaptive weight allocation through a softmax layer, and the adaptive weights are respectively allocated to the three-way output of the first layer of the fully connected layer of the fully connected module to complete the feature fusion.

[0041] Preferably, the adaptive feature fusion module is specifically:

[0042] Y=(w1(deconv(X)))·softmax(w2(w1(deconv(X))))

[0043] Wherein, X is the input of the adaptive feature fusion module, Y is the output of the adaptive feature fusion module, deconv is upsampling convolution, w1 and w2 are two fully connected layers respectively, and softmax is the softmax layer.

[0044] Preferably, the multi-task detection head prediction module in step S2 includes classification of prediction frames, regression of prediction frames, and classification of prediction frame directions; the classification of prediction frames mainly includes classification of three types of targets: cars, pedestrians, and bicycles; the purpose of regression of prediction frames is to generate accurate three-dimensional target detection frames, mainly regressing the center point coordinates, length, width, height, and angle of the detection frame, and its vector is represented as (x, y, z, w, h, l, r); the classification of prediction frame directions is used to classify the true direction of the prediction frame and the true opposite direction, so as to avoid excessive angle regression loss during prediction frame regression and facilitate model training.

[0045] Preferably, the loss function preset in step S3 is specifically:

[0046]

[0047] Among them, w=1.0, μ=2.0, λ=0.2, N pos is the number of positive samples, L cls is the prediction box classification loss, L box is the bounding box regression loss, L dir is the direction classification loss, and L is the total loss.

[0048] Preferably, step S3 further comprises the following steps: performing method validation and evaluation;

[0049] The evaluation methods used in method validation and evaluation include AP average precision and AOS average directional similarity. AP is mainly used to quantitatively analyze model accuracy, and AOS is mainly used to measure the directional similarity between the detection results and the ground truth detection frame.

[0050] Preferably, the AP mainly includes AP bbox 、AP bev 、AP 3d 、AP aos Four evaluation indicators, AP bbox Indicates the accuracy of the two-dimensional detection box, AP bev Indicates the accuracy of the detection box under the bird's eye view, AP 3d Indicates the accuracy of the three-dimensional detection frame, AP aos Indicates the accuracy of the detection frame angle;

[0051] The accuracy of the 3D detection model's predicted position and depth is defined by the degree of overlap between the 3D prediction frame and the ground truth frame, specifically:

[0052]

[0053] Among them, IOU is the intersection-over-union ratio, which is used to measure the degree of overlap between the predicted box and the GT box. Pre is the predicted box, GT is the real box, S is the area or volume of the box, and the IOU threshold is 0.7;

[0054] The AP value is used to evaluate the accuracy of the model. Precision (P) is the ratio of the number of true positive samples (TP) to the sum of TP and false positive samples (FP). Recall (R) is the ratio of the number of TP to the sum of TP and false negative samples (FN). Different thresholds will have different precision and recall rates. AP is the average value of the precision when the threshold t is different discrete values. Specifically:

[0055]

[0056]

[0057]

[0058] ρ interp (r)=max r′:r′≥r (r′)

[0059] Among them, TP is the number of true positive samples, that is, the prediction is the true box and the IOU is greater than or equal to the preset threshold, FP is the number of false positive samples, that is, the prediction is the true box but the IOU is less than the preset threshold, FN is the false negative sample, that is, the prediction is the wrong box and the IOU is less than the preset threshold, AP| R is the average precision, ρ interp (r) is the interpolation function. When calculating the average precision, only the prediction results of targets with a height greater than 25 pixels are evaluated. The easily confused categories are regarded as the same category to reduce the false positive rate, and the average of the precise values on 41 equally spaced recalls is used to approximate the AP of the classifier.

[0060] Use the average direction similarity to predict the target direction, specifically:

[0061]

[0062]

[0063] Among them, AOS is the average directional similarity, r is the recall rate of the target prediction, s(r) is the directional similarity under the condition of recall rate r, directional similarity s∈[0,1] is the normalized cosine distance between all predicted samples and GT, D(r) is the set of all predicted positive samples under recall rate r, Indicates the difference between the predicted angle of target i and GT. If it is detected that i has matched GT, then δ i =1, otherwise δ i =0.

[0064] The present invention provides a method for three-dimensional target detection based on laser point clouds in unmanned systems. The method mainly comprises a point cloud voxelization module for regularized representation of point clouds, and then an edge feature extractor is used to enhance the representation capability of point clouds by extracting edge features of point clouds within voxels. This allows the model to learn the topological relationships between points, which is conducive to extracting richer point cloud information. A two-dimensional encoder is used as the backbone network, which is more efficient than the sparse convolution of a three-dimensional encoder. An adaptive feature fusion module introduces an adaptive fusion ratio allocation mechanism. The feature maps output by different layers of the two-dimensional encoder are used as references for feature learning to generate adaptive weights for weighted fusion of multiple layers of feature maps. This further achieves the effective combination of low-level features and high-level features of point cloud information. Finally, the detection head predicts the detection box, achieving high-precision and high-efficiency three-dimensional detection of targets. The present invention is mainly applicable to the field of three-dimensional target detection in unmanned systems. The proposed dynamic edge convolution module and adaptive feature fusion network extract features from point cloud information from different perspectives, enhancing the spatial expression capability of point cloud information and the ability to combine multi-level features, thereby obtaining a richer and more refined feature representation and improving the accuracy of three-dimensional target detection in laser point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flowchart of a preferred embodiment of a laser point cloud 3D target detection method based on an unmanned system according to the present invention;

[0066] Figure 2 This is an overall framework diagram of a 3D detection model of a preferred embodiment of a laser point cloud 3D target detection method based on an unmanned system of the present invention;

[0067] Figure 3 This is a specific model diagram of an edge feature extractor of a preferred embodiment of a laser point cloud 3D target detection method based on an unmanned system of the present invention;

[0068] Figure 4 This is a specific model diagram of the adaptive feature fusion module of a preferred embodiment of the laser point cloud three-dimensional target detection method based on an unmanned system of the present invention. DETAILED DESCRIPTION

[0069] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0070] The terms "first," "second," and so on, used in this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly indicating the number or order of the technical features being referenced. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this disclosure, "plurality" means at least two, such as two or three, unless otherwise specifically defined.

[0071] In response to the existing problems, the present invention provides a laser point cloud three-dimensional target detection method based on an unmanned system.

[0072] Reference Figure 1 The present invention provides a method for detecting three-dimensional targets using laser point clouds based on an unmanned system, comprising the following steps:

[0073] Step S1: Obtain a laser point cloud dataset based on an unmanned system, preprocess the laser point cloud dataset, and divide the preprocessed laser point cloud dataset into a training set and a test set according to a preset ratio of 1:1.

[0074] Specifically, experimental verification was conducted on public datasets for autonomous driving: the method was verified and evaluated on the KITTI 3D public dataset (a dataset jointly created by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America, a public dataset obtained by collecting data from actual traffic scenes using an assembled and fully equipped collection vehicle). The KITTI dataset is a public dataset for three-dimensional object detection in the field of autonomous driving, mainly including three-dimensional object detection of categories such as vehicles, pedestrians, and bicycles, including 7481 training set samples and 7518 test set samples. On this basis, the training set is divided into a training set of 3712 samples and a validation set of 3769 samples.

[0075] Before validation, we performed three types of data augmentation. The first type randomly flipped the input point cloud left and right, or front and back. The second type rotated the input point cloud, typically within the range of [-45°, 45°] for the KITTI dataset, and then scaled the input point cloud, typically within the range of [0.95, 1.05] for the KITTI dataset. The third type, if the current sample has a small number of ground truth objects, sampled some ground truth objects from the entire KITTI dataset to populate the current sample.

[0076] Step S2: Construct a laser point cloud 3D target detection model based on an unmanned system, wherein the laser point cloud 3D target detection model includes a point cloud voxelization module, a dynamic edge feature encoder, a two-dimensional encoder, an adaptive feature fusion module, and a multi-task detection head prediction module.

[0077] Specifically, the schematic diagram of the laser point cloud 3D target detection model is as follows: Figure 2 shown.

[0078] The point cloud voxelization module regularizes and densifies the point cloud data, and the dynamic edge feature encoder is used to extract the edge features of the point cloud data within the voxel and perform a two-dimensional pseudo-image representation, wherein the point cloud data includes the position information and angle information of the point cloud, specifically (x, y, z, r), (x, y, z) is the three-dimensional coordinate of the point cloud data, and r is the reflection intensity of the point cloud data. The two-dimensional encoder extracts multi-level feature information of spatial features and semantic features from the two-dimensional pseudo-image. The adaptive feature fusion module adaptively assigns weights to different levels of features through the multi-level feature information for feature fusion. The multi-task detection head prediction module predicts the three-dimensional target detection box through the fused features.

[0079] Step S3: Input the training set in step S1 into the laser point cloud 3D target detection model in step S2, perform backpropagation on the laser point cloud 3D target detection model through the point cloud detection box of the training set, the 3D target detection box output and the preset loss function, and update the model parameters, repeat the above steps in step S3, and finally obtain a trained laser point cloud 3D target detection model.

[0080] In one embodiment, the point cloud voxelization module in step S2 includes the following steps:

[0081] Step S221: With the laser radar as the center of the coordinate axis, the vehicle's forward direction is the x-axis direction, the vehicle's left direction is the y-axis direction, and the vehicle's upward direction is the z-axis direction, the point cloud data in the training set of the acquired laser point cloud dataset is represented by a four-dimensional tensor, specifically (x, y, z, r), where (x, y, z) is the three-dimensional coordinate of the point cloud data, and r is the reflection intensity of the point cloud data. Step S222: The range of the point cloud scene on the (x, y, z) coordinate axis is specified as [(0, 69.12), (-39.68, 39.68), (-3, 1)], and the specification of each point cloud voxel is (0.16, 0.16, 4), so the entire original point cloud scene is divided into 432×496×1 voxels of equal size;

[0082] Step S222: The range of the point cloud scene on the (x, y, z) coordinate axis is specified as [(0, 69.12), (-39.68, 39.68), (-3, 1)], the size of each point cloud voxel is (0.16, 0.16, 4), and the entire point cloud scene is divided into 432 × 496 × 1 voxels of equal size;

[0083] Step S223: The number of points in each voxel does not exceed 32. If it is less than 32, zero is padded. If it exceeds 32, random sampling is performed to 32. The coordinates and reflection intensity of the 32 point clouds in each voxel are used as the features of the point cloud voxel, which is mathematically represented as a tensor of (432×496×1,32,4).

[0084] like Figure 3 As shown, in one embodiment, the dynamic edge feature encoder in step S2 mainly includes a nearest neighbor sampling module, an edge convolution module, and a two-dimensional pseudo image generation module, which are used to extract edge features of point clouds;

[0085] The nearest neighbor sampling module performs sampling as follows:

[0086] The edge convolution module includes a first edge feature encoder and a second edge feature encoder, which respectively extract edge features of the point cloud; the first edge feature encoder includes two multi-layer perceptrons and a maximum pooling layer, and the samples obtained by the nearest neighbor sampling module are sequentially passed through the two multi-layer perceptrons and the maximum pooling layer for feature extraction and encoding output; the second edge feature encoder includes a nearest neighbor sampling layer, a multi-layer perceptron and a maximum pooling layer, and the output data of the first edge feature encoder is subjected to a second nearest neighbor sampling through the nearest neighbor sampling layer to complete dynamic updating to obtain deeper edge features, which are input into the multi-layer perceptron of the second edge convolution for feature extraction, and then a maximum pooling operation is performed to output a feature tensor.

[0087] Step S2 includes the following steps:

[0088] Step S231: Sampling by the nearest neighbor sampling module; specifically:

[0089] First, input the voxel representation of the 432×496×1 point cloud into the nearest neighbor sampling module. Each voxel contains 32 points. The feature of each point is the 4-dimensional feature of (x, y, z) coordinates and reflection intensity. The 32 points in each voxel are used as vertices. Each vertex is sampled to 5 points in each voxel. Specifically, the nearest neighbor sampling module calculates the distance between each vertex and the remaining 31 vertices, obtains the 5 closest vertices as the nearest neighbor point set of the current vertex, repeats 32 times to traverse the 32 vertices in each voxel, and connects the 32 vertices in each voxel (X i ,Y i ,Z i )(i=0,1,2,…,32) and the coordinates of the points in the nearest neighbor sampling point set (X ij ,Y ij ,Z ij )(j=0,1,…,5), build the edge relationship between points (X i -Xij ,Y i -Y ij ,Z i -Z ij ), which is mathematically represented as a tensor of (432×496×1,32×5,4), where i is the number of vertices in each voxel, totaling 32, and j is the number of points sampled by the nearest neighbor of each vertex, totaling 5;

[0090] The focus of these five points is on the number. Since the sampling algorithm used is the nearest neighbor sampling algorithm, the positions of these five points are the distances from the vertex (X) calculated by the nearest neighbor sampling algorithm of the 32 points in the voxel. i ,Y i ,Z i )(i=0,1,2,…,32) the five nearest points, whose position information depends on the vertex position and the nearest neighbor sampling algorithm.

[0091] Step S232: The edge convolution module convolves to obtain edge features of the point cloud and encodes and outputs them; specifically, it includes:

[0092] Step S2321: The coordinates (X i ,Y i ,Z i ) and the edge relationship it represents (X i -X ij ,Y i -Y ij ,Z i -Z ij ) is used as the input of the edge convolution module, specifically a tensor of (432×496×1,32×(5+1),4). It first enters the first edge convolution module and uses two multi-layer perceptrons to extract features, converting the original 4-dimensional features into 64-dimensional features. It is then encoded and outputted through the maximum pooling layer, and the output is a tensor of (432×496×1,32,64).

[0093] Step S2322: Perform a second nearest neighbor sampling on the output of the first edge convolution module to complete dynamic update to obtain deeper edge features, and input the output to the second edge convolution module to use a multi-layer perceptron to perform feature extraction, converting the 64-dimensional features into deeper 64-dimensional features, and then perform a maximum pooling operation to output a feature tensor of (432×496×1,1,64);

[0094] Step S233: The two-dimensional pseudo image generation module generates a two-dimensional pseudo image; specifically:

[0095] The two-dimensional pseudo image generation module performs a tensor conversion operation to convert the feature tensor form of (432×496×1,1,64) into a standard two-dimensional image tensor form of (432,496,64), where H=432, W=496, C=64, H is the height of the two-dimensional pseudo image, W is the width of the two-dimensional pseudo image, and C is the number of channels of the two-dimensional pseudo image.

[0096] In one embodiment, the two-dimensional encoder in step S2 mainly consists of three blocks, the first block consists of three 3×3 two-dimensional convolutional layers, the second block consists of five 3×3 two-dimensional convolutional layers, and the third block consists of five 3×3 two-dimensional convolutional layers, including the following steps:

[0097] Step S241: The first block extracts spatial features and / or semantic features from the input two-dimensional pseudo image to generate a 216×248×64 two-dimensional feature representation;

[0098] Step S242: The two-dimensional feature expression in step S241 is continuously input into the second block for feature extraction of spatial features and / or semantic features to generate a 108×124×128 two-dimensional feature expression;

[0099] Step S243: The two-dimensional feature expression in step S242 is input into the third block to extract spatial features and / or semantic features to generate a 54×62×256 two-dimensional feature expression;

[0100] Step S244: Two-dimensional encoder output: The three two-dimensional feature expressions output by the three blocks are the outputs of the two-dimensional encoder.

[0101] like Figure 4 As shown, in one embodiment, the adaptive feature fusion module in step S2 includes three parts: an upsampling module, a fully connected module, and a feature fusion module, which adaptively fuses low-level geometric features with high-level semantic features, including the following steps:

[0102] Step S251: Upsampling module: First, the three outputs of the two-dimensional encoder, namely the 216×248×64 two-dimensional feature expression, the 108×124×128 two-dimensional feature expression and the 54×62×256 two-dimensional feature expression, are respectively input into the upsampling module consisting of three upsampling convolution layers: 1×1 upsampling (deconv) convolution, 2×2 upsampling (deconv) convolution and 4×4 upsampling (deconv) convolution. The three outputs of the two-dimensional encoder, namely the 216×248×64 two-dimensional feature expression, the 108×124×128 two-dimensional feature expression and the 54×62×256 two-dimensional feature expression, are respectively passed through the three upsampling convolution layers;

[0103] Step S252: Fully connected module splicing output: The fully connected module includes two layers of fully connected layers connected in sequence. The output of the upsampling convolution layer in step S251 is input to the fully connected module. The outputs of the three upsampling convolution layers are used as inputs to the fully connected module and pass through the two fully connected layers in sequence and are spliced to output a 216×248×3 feature representation.

[0104] The first fully connected layer contains three inputs and three outputs; the three inputs are the outputs of three upsampling convolutional layers, and the three outputs are all 216×248×128 feature expressions;

[0105] The second fully connected layer includes three inputs and three outputs. The three inputs are the outputs of the first fully connected layer, and the three outputs are all 216×248×1 feature expressions. The three outputs of the second fully connected layer are concatenated to form a single output of 216×248×3.

[0106] Step S253: Feature Fusion Module Feature Fusion: The 216×248×3 feature expression output in step S252 continues to enter the feature fusion module through a softmax layer for adaptive weight allocation. The adaptive weights are respectively assigned to the three outputs of the first layer of the fully connected module, completing feature fusion. In one embodiment, the adaptive feature fusion module is specifically:

[0107] Y=(w1(deconv(X)))·softmax(w2(w1(deconv(X))))

[0108] Wherein, X is the input of the adaptive feature fusion module, Y is the output of the adaptive feature fusion module, deconv is upsampling convolution, w1 and w2 are two fully connected layers respectively, and softmax is the softmax layer.

[0109] In one embodiment, the multi-task detection head prediction module in step S2 includes classification of prediction boxes, regression of prediction boxes, and classification of prediction box directions.

[0110] The classification of prediction frames mainly includes the classification of three types of targets: cars, pedestrians, and cyclists. The purpose of prediction frame regression is to generate accurate three-dimensional target detection frames. It mainly regresses the center point coordinates, length, width, height, and angle of the detection frame. Its vector is represented as (x, y, z, w, h, l, r). The classification of the prediction frame direction is used to classify the true direction of the prediction frame and the true opposite direction, avoiding excessive angle regression loss during prediction frame regression and facilitating model training.

[0111] In one embodiment, the loss function preset in step S3 is specifically:

[0112]

[0113] Among them, w=1.0, μ=2.0, λ=0.2, N pos is the number of positive samples, L cls is the prediction box classification loss, L box is the bounding box regression loss, L dir is the direction classification loss, and L is the total loss.

[0114] The evaluation methods used in method validation and evaluation include Average Precision (AP) and Average Orientation Similarity (AOS). AP is mainly used to quantitatively analyze model accuracy, and AOS is mainly used to measure the directional similarity between the detection results and the ground truth detection frame.

[0115] Furthermore, AP mainly includes AP bbox 、AP bev 、AP 3d 、AP aos Four evaluation indicators, AP bbox Indicates the accuracy of the two-dimensional detection box, AP bev Indicates the accuracy of the detection box under the bird's eye view, AP 3d Indicates the accuracy of the three-dimensional detection frame, AP aos Indicates the accuracy of the detection frame angle.

[0116] Furthermore, the accuracy of the 3D detection model's predicted position and depth is defined by the degree of overlap between the 3D prediction frame and the ground truth frame, specifically:

[0117]

[0118] Among them, IOU is the intersection-over-union ratio, which is used to measure the degree of overlap between the predicted box and the GT box. Pre is the predicted box, gt is the real box, S is the area or volume of the box, and the IOU threshold is 0.7.

[0119] Furthermore, the AP value is used to evaluate the accuracy of the model. Precision (P) is the ratio of the number of true positive samples (TP) to the sum of TP and false positive samples (FP). Recall (R) is the ratio of the number of TP to the sum of TP and false negative samples (FN). Different thresholds will have different precision and recall rates. AP is the average precision when the threshold t is different discrete values. Specifically:

[0120]

[0121]

[0122] ρ interp (r)=max r′:′≥r (r ′ )

[0123] Among them, TP is the number of true positive samples, that is, the prediction is the true box and the IOU is greater than or equal to the preset threshold, FP is the number of false positive samples, that is, the prediction is the true box but the IOU is less than the preset threshold, FN is the false negative sample, that is, the prediction is the wrong box and the IOU is less than the preset threshold, AP| R is the average precision, ρ interp (r) is the interpolation function. When calculating the average precision, only the prediction results of targets with a height greater than 25 pixels are evaluated. The easily confused categories are regarded as the same category to reduce the false positive rate, and the average of the precise values on 41 equally spaced recalls is used to approximate the AP of the classifier.

[0124] Furthermore, the average direction similarity is used to predict the target direction, specifically:

[0125]

[0126]

[0127] Among them, AOS is the average directional similarity, r is the recall rate of the target prediction, s(r) is the directional similarity under the condition of recall rate r, directional similarity sε[0,1] is the normalized cosine distance between all predicted samples and GT, D(r) is the set of all predicted positive samples under recall rate r, Indicates the difference between the predicted angle of target i and GT. If it is detected that i has matched GT, then δ i =1, otherwise δ i =0.

[0128] Furthermore, the model training uses the ADAM optimizer and is trained on a single 3090 graphics card with a batch size of 2, with an actual training epoch of 160.

[0129] Step S4: Input the test set in step S1 into the trained laser point cloud three-dimensional target detection model to obtain a three-dimensional target detection result.

[0130] The present invention provides a method for three-dimensional target detection based on laser point clouds in unmanned systems. The method mainly comprises a point cloud voxelization module for regularized representation of point clouds, and then an edge feature extractor is used to enhance the representation capability of point clouds by extracting edge features of point clouds within voxels. This allows the model to learn the topological relationships between points, which is conducive to extracting richer point cloud information. A two-dimensional encoder is used as the backbone network, which is more efficient than the sparse convolution of a three-dimensional encoder. An adaptive feature fusion module introduces an adaptive fusion ratio allocation mechanism. The feature maps output by different layers of the two-dimensional encoder are used as references for feature learning to generate adaptive weights for weighted fusion of multiple layers of feature maps. This further achieves the effective combination of low-level features and high-level features of point cloud information. Finally, the detection head predicts the detection box, achieving high-precision and high-efficiency three-dimensional detection of targets. The present invention is mainly applicable to the field of three-dimensional target detection in unmanned systems. The proposed dynamic edge convolution module and adaptive feature fusion network extract features from point cloud information from different perspectives, enhancing the spatial expression capability of point cloud information and the ability to combine multi-level features, thereby obtaining a richer and more refined feature representation and improving the accuracy of three-dimensional target detection in laser point clouds.

[0131] Beneficial effects of the present invention:

[0132] 1. The point cloud 3D detection model designed by this invention avoids the use of a 3D sparse convolution module. After point cloud sparsification, it uses edge convolution and 2D convolution for feature extraction, avoiding heavy computations and improving inference speed, making it more suitable for industrial 3D detection models.

[0133] 2. The dynamic edge feature extractor designed in this paper not only focuses on point cloud feature extraction but also pays attention to the topological relationship between points. This effectively solves the problem of incomplete or lost point cloud information caused by target occlusion. By constructing the edge relationship in the target point cloud, the geometric information representation capability of the point cloud data is improved, which effectively improves the detection accuracy.

[0134] 3. The adaptive feature fusion module designed in the present invention avoids the rigid addition of high-level semantic features and low-level geometric features, and realizes the flexible fusion of high-level semantic features and low-level geometric features through weight distribution through an adaptive factor, effectively solving the problem of the quality degradation of low-level geometric features after the point cloud information passes through the stacked convolution layers, and improving the representation ability of geometric information. The present invention achieves a simple detection difficulty AP of 88.13% for cars and 82.13% for bicycles on the public dataset KITTI dataset, and has a fast inference speed. It is a high-speed three-dimensional target detection method with relatively excellent detection effect.

[0135] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A laser point cloud 3D target detection method based on an unmanned system, characterized in that: The steps include: Step S1: obtaining a laser point cloud dataset based on an unmanned system, preprocessing the laser point cloud dataset, and dividing the preprocessed laser point cloud dataset into a training set and a test set according to a preset ratio of 1:1; Step S2: constructing a laser point cloud 3D target detection model based on an unmanned system, wherein the laser point cloud 3D target detection model includes a point cloud voxelization module, a dynamic edge feature encoder, a two-dimensional encoder, an adaptive feature fusion module, and a multi-task detection head prediction module; The point cloud voxelization module performs regularization and densification on the point cloud data. The dynamic edge feature encoder is used to extract the edge features of the point cloud data within the voxel and perform a two-dimensional pseudo-image representation. The point cloud data includes the position information and angle information of the point cloud, specifically (x, y, z, r), where (x, y, z) is the three-dimensional coordinate of the point cloud data and r is the reflection intensity of the point cloud data. The two-dimensional encoder extracts multi-level feature information of spatial features and semantic features from the two-dimensional pseudo-image. The adaptive feature fusion module adaptively assigns weights to features at different levels through the multi-level feature information to perform feature fusion. The multi-task detection head prediction module predicts the three-dimensional target detection frame through the fused features. Step S3: Input the training set in step S1 into the laser point cloud 3D object detection model in step S2, perform backpropagation on the laser point cloud 3D object detection model through the point cloud detection box of the training set, the 3D object detection box output, and a preset loss function, and update the model parameters, repeat the above steps in step S3, and finally obtain a trained laser point cloud 3D object detection model; Step S4: Input the test set in step S1 into the trained laser point cloud three-dimensional target detection model to obtain a three-dimensional target detection result.

2. The laser point cloud 3D target detection method based on an unmanned system according to claim 1, characterized in that: The point cloud voxelization module in step S2 includes the following steps: Step S221: With the laser radar as the center of the coordinate axis, the vehicle's forward direction is the x-axis, the vehicle's left direction is the y-axis, and the vehicle's upward direction is the z-axis. The point cloud data in the training set of the acquired laser point cloud dataset is represented by a four-dimensional tensor, specifically (x, y, z, r), where (x, y, z) is the three-dimensional coordinate of the point cloud data and r is the reflection intensity of the point cloud data. Step S222: The range of the point cloud scene on the (x, y, z) coordinate axis is specified as [(0, 69.12), (-39.68, 39.68), (-3, 1)], the size of each point cloud voxel is (0.16, 0.16, 4), and the entire point cloud scene is divided into 432 × 496 × 1 voxels of equal size; Step S223: The number of points in each voxel does not exceed 32. If it is less than 32, zero is padded. If it exceeds 32, random sampling is performed to 32. The coordinates and reflection intensity of the 32 point clouds in each voxel are used as the features of the point cloud voxel, which is mathematically represented as a tensor of (432×496×1,32,4).

3. The laser point cloud 3D target detection method based on an unmanned system according to claim 2, characterized in that: The dynamic edge feature encoder in step S2 includes a nearest neighbor sampling module, an edge convolution module, and a two-dimensional pseudo image generation module, which are used to extract edge features of the point cloud; Among them, the nearest neighbor sampling module performs sampling specifically as follows: The edge convolution module includes a first edge feature encoder and a second edge feature encoder, which respectively extract edge features of the point cloud; the first edge feature encoder includes two multi-layer perceptrons and a maximum pooling layer, and the samples obtained by the nearest neighbor sampling module are sequentially passed through the two multi-layer perceptrons and the maximum pooling layer for feature extraction and encoding output; the second edge feature encoder includes a nearest neighbor sampling layer, a multi-layer perceptron and a maximum pooling layer, and the output data of the first edge feature encoder is subjected to a second nearest neighbor sampling through the nearest neighbor sampling layer to complete dynamic updating to obtain deeper edge features, which are input into the multi-layer perceptron of the second edge convolution for feature extraction, and then a maximum pooling operation is performed to output a feature tensor.

4. The laser point cloud 3D target detection method based on an unmanned system according to claim 1, characterized in that: Step S2 includes the following steps: Step S231: Sampling by the nearest neighbor sampling module; specifically: First, input the voxel representation of the 432×496×1 point cloud into the nearest neighbor sampling module. Each voxel contains 32 points, and the feature of each point is the 4-dimensional feature of (x, y, z) coordinates and reflection intensity. The 32 points in each voxel are used as vertices, and each vertex is sampled to 5 points in each voxel. Specifically, the nearest neighbor sampling module calculates the distance between each vertex and the remaining 31 vertices, obtains the 5 closest vertices as the nearest neighbor point set of the current vertex, repeats 32 times to traverse the 32 vertices in each voxel, and connects the 32 vertices in each voxel. The coordinates of the points in the set of its nearest neighbor sampling points Construct edge relationships between points , which is mathematically represented as a tensor of (432×496×1,32×5,4), where i is the number of vertices in each voxel, totaling 32, and j is the number of points sampled by the nearest neighbor of each vertex, totaling 5; Step S232: The edge convolution module convolves to obtain edge features of the point cloud and encodes and outputs them; specifically, it includes: Step S2321: The coordinates of each vertex The edge relationship it represents The input of the edge convolution module is a tensor of (432×496×1,32×(5+1),4). It first enters the first edge convolution module and uses two multi-layer perceptrons to extract features, converting the original 4-dimensional features into 64-dimensional features. It is then encoded and outputted through the maximum pooling layer, and the output is a tensor of (432×496×1,32,64). Step S2322: Perform a second nearest neighbor sampling on the output of the first edge convolution module to complete dynamic update to obtain deeper edge features, and input the output to the second edge convolution module to use a multi-layer perceptron to perform feature extraction, converting the 64-dimensional features into deeper 64-dimensional features, and then perform a maximum pooling operation to output a feature tensor of (432×496×1,1,64); Step S233: The two-dimensional pseudo image generation module generates a two-dimensional pseudo image; specifically: The two-dimensional pseudo image generation module performs a tensor conversion operation to convert the feature tensor form of (432×496×1,1,64) into a standard two-dimensional image tensor form of (432,496,64), where H=432, W=496, C=64, H is the height of the two-dimensional pseudo image, W is the width of the two-dimensional pseudo image, and C is the number of channels of the two-dimensional pseudo image.

5. The laser point cloud 3D target detection method based on an unmanned system according to claim 1, characterized in that: Step S2: The two-dimensional encoder consists of three blocks, the first block consists of three 3×3 two-dimensional convolutional layers, the second block consists of five 3×3 two-dimensional convolutional layers, and the third block consists of five 3×3 two-dimensional convolutional layers. The following steps are included: Step S241: The first block extracts spatial features and / or semantic features from the input two-dimensional pseudo image to generate a 216×248×64 two-dimensional feature representation; Step S242: The two-dimensional feature expression in step S241 is continuously input into the second block for feature extraction of spatial features and / or semantic features to generate a 108×124×128 two-dimensional feature expression; Step S243: The two-dimensional feature expression in step S242 is input into the third block to extract spatial features and / or semantic features to generate a 54×62×256 two-dimensional feature expression; Step S244: Two-dimensional encoder output: The three two-dimensional feature expressions output by the three blocks are the outputs of the two-dimensional encoder.

6. The laser point cloud 3D target detection method based on an unmanned system according to claim 5, characterized in that: The adaptive feature fusion module in step S2 comprises three parts: an upsampling module, a fully connected module, and a feature fusion module, and includes the following steps: Step S251: Upsampling module: First, the three outputs of the two-dimensional encoder, namely the 216×248×64 two-dimensional feature expression, the 108×124×128 two-dimensional feature expression, and the 54×62×256 two-dimensional feature expression, are input to the upsampling module composed of three upsampling convolution layers: 1×1 upsampling deconv convolution, 2×2 upsampling deconv convolution, and 4×4 upsampling deconv convolution. The three outputs of the two-dimensional encoder, namely the 216×248×64 two-dimensional feature expression, the 108×124×128 two-dimensional feature expression, and the 54×62×256 two-dimensional feature expression, are respectively passed through the three upsampling convolution layers; Step S252: Fully connected module splicing output: The fully connected module includes two layers of fully connected layers connected in sequence. The output of the upsampling convolution layer in step S251 is input to the fully connected module. The outputs of the three upsampling convolution layers are used as inputs to the fully connected module and pass through the two fully connected layers in sequence and are spliced to output a 216×248×3 feature representation. The first fully connected layer contains three inputs and three outputs; the three inputs are the outputs of three upsampling convolutional layers, and the three outputs are all 216×248×128 feature expressions; The second fully connected layer includes three inputs and three outputs. The three inputs are the outputs of the first fully connected layer, and the three outputs are all 216×248×1 feature expressions. The three outputs of the second fully connected layer are concatenated to form a single output of 216×248×3. Step S253: Feature fusion module Feature fusion: The 216×248×3 feature expression output in step S252 continues to enter the feature fusion module and performs adaptive weight allocation through a softmax layer, and the adaptive weights are respectively allocated to the three-way output of the first layer of the fully connected layer of the fully connected module to complete the feature fusion.

7. The method for detecting three-dimensional targets using laser point clouds based on an unmanned system according to claim 5, wherein: The adaptive feature fusion module is specifically: ； in, is the input of the adaptive feature fusion module, is the output of the adaptive feature fusion module, deconv is the upsampling convolution, 、 There are two fully connected layers and softmax is the softmax layer.

8. The laser point cloud 3D target detection method based on an unmanned system according to claim 1, characterized in that: The multi-task detection head prediction module described in step S2 includes prediction box classification, prediction box regression, and prediction box direction classification; the prediction box classification includes classification of three types of targets: cars, pedestrians, and bicycles; the purpose of prediction box regression is to generate an accurate three-dimensional target detection box, and regress the center point coordinates, length, width, height, and angle of the detection box, whose vector is represented as (x, y, z, w, h, l, r). The classification of the prediction box direction is used to classify the true direction of the prediction box and the true opposite direction, avoiding excessive angle regression loss during prediction box regression, and facilitating model training.

9. The laser point cloud 3D target detection method based on an unmanned system according to claim 1, characterized in that: The loss function preset in step S3 is specifically: ； in, , , , is the number of positive samples, is the prediction box classification loss, is the bounding box regression loss, is the direction classification loss, For the total loss.

10. The method for three-dimensional target detection based on laser point cloud of unmanned system according to claim 1, characterized in that: Step S3 also includes the following steps: performing method validation and evaluation; The evaluation methods used in method validation and evaluation include AP average precision and AOS average directional similarity. AP is used to quantitatively analyze model accuracy, and AOS is used to measure the directional similarity between the detection results and the ground truth detection frame.

11. The method for detecting three-dimensional targets using laser point clouds based on an unmanned system according to claim 10, wherein: AP includes 、 、 、 Four evaluation indicators, Indicates the accuracy of the two-dimensional detection frame, Indicates the accuracy of the detection frame under the bird's eye view, Indicates the accuracy of the 3D detection frame, Indicates the accuracy of the detection frame angle; The accuracy of the 3D detection model's predicted position and depth is defined by the degree of overlap between the 3D prediction frame and the ground truth frame, specifically: ； Among them, IOU is the intersection-over-union ratio, which is used to measure the degree of overlap between the predicted box and the GT box. Pre is the predicted box, GT is the real box, S is the area or volume of the box, and the IOU threshold is 0.7; The AP value is used to evaluate the accuracy of the model. The precision P is the ratio of the number of true positive samples TP to the sum of the number of TP and false positive samples FP. The recall rate R is the ratio of the number of TP to the sum of the number of TP and false negative samples FN. Different thresholds will have different precision and recall rates. AP is the average precision rate when the threshold t is different discrete values. Specifically: ； ； ； ； Among them, TP is the number of true positive samples, that is, the prediction is a true box and the IOU is greater than or equal to the preset threshold, FP is the number of false positive samples, that is, the prediction is a true box but the IOU is less than the preset threshold, FN is the false negative sample, that is, the prediction is an error box and the IOU is less than the preset threshold, is the average precision, It is an interpolation function. When calculating the average precision, only the prediction results of the target height greater than 25 pixels are evaluated. The easily confused categories are regarded as the same category to reduce the false positive rate, and the average of the precise values on 41 equally spaced recalls is used to approximate the AP of the classifier. Use the average direction similarity to predict the target direction, specifically: ； ； Among them, AOS is the average directional similarity, r is the recall rate of target prediction, s(r) is the directional similarity under the condition of recall rate r, and directional similarity is the normalized cosine distance between all predicted samples and GT, D(r) is the set of all predicted positive samples under the recall rate r, Indicates the difference between the predicted angle of target i and GT. If it is detected that i has matched GT, then ,otherwise .

Images