A Deep Learning-Based 3D Object Detection Method

Abstract

This invention relates to a deep learning-based 3D object detection method. Specifically, it involves loading a point cloud dataset as training sample images, representing each point cloud using vectors, dividing the scene into multiple columnar structures, and using tensors to represent the entire scene. The images in the scene are then subjected to grayscale conversion and mean filtering. A deep learning convolutional neural network is constructed, including a columnar structure extraction network, an image preprocessing module, a 2D feature extraction network, and a multi-branch detection head network. The point cloud dataset is loaded as a training set to train the neural network model, resulting in a trained neural network model. In the usage phase, the image is fed into the pre-trained neural network model to obtain the corresponding target output, and the 3D position and category of each target are calculated. This invention enables accurate localization and classification of targets in autonomous driving scenarios while maintaining speed.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving methods, specifically to a 3D target detection method for autonomous driving perception modules, and more particularly to a 3D target detection method based on deep learning. Background Technology

[0002] Currently, object detection technology based on 2D images is very mature and has been widely used in real life, such as facial recognition, industrial product defect detection, and security monitoring. However, 2D images cannot detect the depth information of objects, so 2D object detection is not suitable for some 3D scenarios, especially in the field of autonomous driving, where the 3D information of the target is very important.

[0003] In recent years, with the development of hardware and deep learning, deep learning-based 3D object detection has achieved high accuracy and fast detection speed, playing a role in various fields, especially in autonomous driving. Current automobiles are equipped with high-precision LiDAR and other sensors. Even in adverse weather conditions, LiDAR can still collect point clouds with rich spatial information. Therefore, utilizing LiDAR point clouds for 3D object detection has become a research hotspot in recent years.

[0004] A representative example is PointNet, a method based on raw point clouds proposed by Lang et al. It uses PointNet to learn point cloud representations organized in columns, which are voxels with a finite spatial range in the z-direction. PointNet can utilize all the information represented by the point cloud and perform operations on columns rather than voxels, making it very efficient for computation on GPUs.

[0005] The advantage of this type of model, which converts point cloud features into 2D image features, is its fast detection speed. However, one drawback is that it uses random sampling of the point cloud within each column, which can lead to some important points not being captured, resulting in a loss of accuracy. Furthermore, the feature extraction and fusion methods are too simplistic. When compressing the collected point cloud into a feature map using a columnar feature network, a simple max-pooling method is employed, which results in the loss of much feature information, causing target detection to be missed and thus affecting the detection accuracy. Summary of the Invention

[0006] To address the issue of significant feature loss caused by downsampling and feature extraction of point cloud information into feature maps, this invention employs a series of improved methods to achieve accurate localization and classification of targets in autonomous driving scenarios while maintaining speed.

[0007] This invention proposes that during the point cloud downsampling stage, point clouds in high-density regions exhibit high similarity, while point clouds in low-density regions should be preserved as much as possible. This approach aims to retain as much information as possible about distant, sparse point clouds, thereby improving the accuracy of distant target detection. Subsequently, by leveraging the advantages of image denoising and attention mechanisms, features are extracted from the point cloud information. Then, the preprocessed image and its corresponding point cloud are efficiently extracted and fused. The image preprocessing step optimizes image quality and reduces noise, providing a better foundation for subsequent target detection. This series of improvements enhances the system's feature extraction and fusion capabilities, resulting in higher detection accuracy.

[0008] Therefore, this invention proposes the following technical solution: a 3D object detection method based on deep learning, the method comprising the following steps:

[0009] Step (1): Read the KITTI dataset, divide the KITTI dataset into training set and test set, load the training set to train the model, preprocess the point cloud data in the loaded training set, use vectors to represent each point cloud, divide the scene into multiple columns, and use tensors to represent the entire scene.

[0010] Step (2): Load the image corresponding to the point cloud scene in step (1), perform grayscale processing on the image in the scene, optimize the processing process, improve the target detection accuracy, and then perform mean filtering to suppress noise and improve image quality.

[0011] Step (3): Construct a deep learning convolutional neural network, including a column extraction network, an image preprocessing module, a 2D feature extraction network, and a multi-branch detection head network;

[0012] Step (4): The training set data is forward propagated to obtain the output of the deep learning convolutional neural network, the loss is calculated, backpropagation is performed, the network parameters are updated, and the trained neural network model is obtained.

[0013] Step (5), usage stage: receive test set point cloud data, send the image into the pre-trained neural network model, obtain the corresponding output target, and calculate the position and category of each 3D target.

[0014] Furthermore, in step (1), the original radar point cloud data is read, and the point cloud data is divided into grids according to the X and Y axes where the point cloud data is located in the scene. Any point cloud data that falls into a grid is considered to be in a columnar body.

[0015] Read the data of each point cloud in the columnar structure, including the point cloud coordinates (x, y, z) and the reflection intensity r of the point cloud. Assume that the number of point clouds in each columnar structure is N. sumThen the information of each point cloud can be represented by a vector of D=5, namely D=(x, y, z, r, l), where x, y, z, r are the point cloud information, and l is the relative position of the geometric center of the point cloud column.

[0016]

[0017] Where, x p y p , z p These represent the coordinates of the center point of the column;

[0018] Retain N point clouds from each column, if N > N sum Fill with zeros, or randomly select a point cloud. The specific method is as follows:

[0019] Assume each sample has P non-empty columns, and a certain column grid has N columns. sum Given a point cloud dataset, randomly select a point cloud i with coordinates i = (x, y, z). Set a range length d and count the number N points in the point cloud within the interval ([xd, x+d], [yd, y+d], [zd, z+d]). around The probability that this point will be discarded is... This process is repeated until N point clouds are selected. Each sample can then be represented by a tensor of (D, P, N).

[0020] Furthermore, in step (2), the image in the scene is converted to grayscale. After grayscale processing, its color changes will be removed, and only the trend of gray intensity changes will be retained.

[0021] Using G(r, r, r) in the grayscale model and corresponding to RGB(R, G, B) in the color image model, the weighted average of RGB is calculated using the weighted average method, where the weights of the three primary colors are obtained as follows:

[0022] r=(Q r R+Q g G+Q b B)

[0023] Among them, Q r Q represents the weight of R. g Q represents the weight of G. b This indicates the weight of B;

[0024] After grayscale processing of the image, mean filtering is used to suppress noise and improve image quality. The acquired image is divided into several templates of the same size. Assuming the original image resolution is X*Y, a new filtered image is obtained after mean filtering. The calculation process is as follows:

[0025]

[0026] Where j(x, y) represents the image before filtering, k(a, b) represents the new filtered image, ∑∑ represents the summation of all pixels in the window, and M×N is the window size.

[0027] Furthermore, in step (3), the sample tensor (D, P, N) obtained in step (1) is transformed into a tensor (C, P, N) by feature extraction through a simplified PointNet network. Max pooling, average pooling, and random pooling are then performed on the dimension containing the column N in the tensor to obtain three feature maps of the (C, P) dimensions, F. max = (C, P), F avg = (C, P), F ran = (C, P) Then add and fuse the three feature maps F fin =F max +F avg +F ran Dividing the P dimension into H and W, i.e., P = H * W, we can obtain the feature map (C, H, W) formed after transformation and fusion.

[0028] The above feature map is divided into multiple 4*4 pixel smaller feature maps, each called a patch. The feature map is then transformed into a (16*C, H / 4, W / 4) format. Each patch's feature map (16*C, 4, 4) is converted into a one-dimensional feature vector. Therefore, the entire feature map is represented by PN 16*C*H / 4*W / 4 feature vectors, where PN is the number of patches. The feature vector corresponding to each patch is then added to the positional encoding vector to obtain the final feature vector to be passed to the attention feature extraction module. The formula for obtaining the positional encoding vector is as follows:

[0029]

[0030] Where t represents the position of each patch in the feature map. d represents the position vector corresponding to each patch. model The dimension corresponding to each patch is 16*C*H / 4*W / 4 in this case. i is the location number;

[0031] By adding the aforementioned positional encoding vector to the vector corresponding to each patch, the final feature vector F to be passed to the attention feature extraction module is obtained. patch =PN*(16*C*H / 4*W / 4);

[0032] In the attention feature extraction module, the vector corresponding to each input patch is first calculated, and the vectors corresponding to each patch are combined into a matrix represented by X, i.e., X = (PN, 16*C*H / 4*W / 4), to obtain Q, K, and V, where:

[0033] Q = X * W q

[0034] K = X * W k

[0035] V = X * W v

[0036] W q W k and W v It is a learnable linear transformation matrix. After calculating Q, K, and V using the self-attention formula, we can obtain a feature vector that integrates the upper and lower features. The formula for calculating the overall self-attention formula is:

[0037]

[0038] Where, d k It is the number of columns in the Q and K matrices, i.e., the vector dimension;

[0039] After using the attention feature extraction module, we can obtain the result related to F. patch Vectors of the same dimension, i.e., PN (16*C*H / 4*W / 4) feature vectors, are represented as a (PN, 16*C*H / 4*W / 4) feature matrix. By reorganizing the feature matrix, a feature map of (C, H, W) is obtained. Finally, a new feature map F is obtained by convolving the feature matrix. pcl = (4C, H / 2, H / 2);

[0040] The image processed in step (2) is subjected to feature extraction using a 2D feature extraction network, and the feature map of the image is obtained with dimension F. picture = (2C, H / 2, W / 2), to obtain the point cloud features F above. pcl With image features F picture Feature map F obtained by feature concatenation mix =(6C,H / 2,W / 2) is fed into the multi-branch detection head network for training;

[0041] The multi-branch detection head network module corresponds to the input feature map F. mixThree downsampling operations are performed to obtain three feature maps of different sizes. After performing a 1*1 convolution operation on each of the three feature maps, a top-down upsampling feature fusion is performed. The feature maps of different sizes are responsible for the classification and regression prediction tasks of targets of different sizes. Usually, the top layer feature map has the largest receptive field, so it has better classification and regression prediction results for large targets. The middle layer is responsible for the classification and regression prediction tasks of medium-sized targets, and the bottom layer is responsible for the classification and regression prediction tasks of small targets.

[0042] Furthermore, in step (4), the total loss function of the network is defined as:

[0043]

[0044]

[0045]

[0046] L CLS =-α a (1-p α ) γ logp α

[0047] Where L is the total loss function, L CLS For the loss function of classification, L Ioc Let L be the loss function for location. dir β is the orientation classification loss function; Ioc ,β CLS ,β dir These are the coefficients of the location loss function, the classification loss function, and the orientation classification loss function, respectively; P is the predicted bounding box, G is the ground truth bounding box, and IoU represents the proportion of overlap between the predicted and ground truth bounding boxes; p 2 (P, G) represents the square of the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box, c 2 This refers to the squared distance between the diagonals of the enclosing regions of the two bounding boxes; parameter p is the probability that the sample is positive; a a Here, α and γ are the weighting coefficients, α and γ are the system hyperparameters, and N is the weighting coefficient. pos To effectively predict the number of boxes.

[0048] Furthermore, in step (5), after inputting the KITTI test set data into the trained network model, for each target, the classification prediction branch will predict the type of the target, and the regression prediction branch will output the x, y, z, w, h, l, θ of the target; where (x, y, z) represents the position of the target center point, w represents the width of the predicted box, h represents the height of the predicted box, l represents the length of the predicted box, and θ represents the orientation angle of the predicted box.

[0049] The beneficial effects of this invention are as follows: Compared with the prior art, the 3D object detection method based on deep learning of this invention has the following advantages:

[0050] (1) Point clouds of distant objects are sparse, so random sampling or sampling of the farthest point is abandoned when downsampling the point cloud. Probabilistic algorithms are used to preserve information of sparse point clouds in the distance as much as possible;

[0051] (2) Use a network structure based on attention feature extraction module to extract features to obtain feature vectors that better integrate context information, and reshape the feature vectors into feature maps so as to facilitate subsequent fusion with image features;

[0052] (3) When compressing the extracted point cloud features into a feature map, three different compression methods are used and then fused together to retain as many features as possible.

[0053] (4) The point cloud features are fused with the corresponding image features. Through grayscale conversion and mean filtering in the image preprocessing step, the noise and color changes of the image are reduced, which helps to improve the accuracy of target detection.

[0054] (5) By adopting a multi-branch decoupled detection head, each branch is responsible for objects of different sizes. At the same time, the regression and classification prediction are separated to obtain a more accurate prediction model, and the model training convergence speed is also faster. Attached Figure Description

[0055] Figure 1 Here is a flowchart of a 3D object detection method;

[0056] Figure 2 This is a schematic diagram of a columnar extraction network;

[0057] Figure 3 This is a schematic diagram of the feature transformation module;

[0058] Figure 4 A schematic diagram of an attention mechanism feature extraction network;

[0059] Figure 5 This is a schematic diagram of the attention feature extraction module;

[0060] Figure 6 This is a schematic diagram of the detection head module. Detailed Implementation

[0061] In response to the aforementioned issues and objectives, this invention provides a method for 3D object detection based on a deep learning algorithm, and also proposes a detection system. Figure 1This diagram displays the flowchart of the 3D object detection method. The system mainly consists of six parts: a columnar extraction network, a feature transformation module, an attention mechanism feature extraction network, an image preprocessing module, a 2D feature extraction network, and a detection head composed of a multi-branch decoupling network. The specific steps of the system are as follows:

[0062] Step (1): Read the KITTI dataset and divide it into a training set and a test set. Load the training set to train the model, preprocess the loaded point cloud data, use vectors to represent each point cloud, divide the scene into multiple columns, and use tensors to represent the entire scene;

[0063] Step (2): Load the image corresponding to the point cloud scene in step (1), perform grayscale processing on the image in the scene, optimize the processing process, improve the target detection accuracy, and then perform mean filtering to suppress noise and improve image quality.

[0064] Step (3): Construct a deep learning convolutional neural network, including a column extraction network, an image preprocessing module, a 2D feature extraction network, and a multi-branch detection head network;

[0065] Step (4): Load the training set. The data is forward propagated to obtain the output of the deep learning convolutional neural network. Calculate the loss, backpropagate, update the network parameters, and obtain the trained neural network model.

[0066] Step (5), usage stage: receive test set point cloud data, send the image into the pre-trained neural network model, obtain the corresponding output target, and calculate the 3D position and category of each target.

[0067] In the further step (1), the original radar point cloud data is read. In the scene, the point cloud data is divided into grids according to the X and Y axes (excluding the Z axis). Any point cloud data that falls into a grid is considered to be in a columnar body, or it can be understood that they constitute a columnar body.

[0068] Read the data of each point cloud in the columnar structure, including the point cloud coordinates (x, y, z) and the reflection intensity r of the point cloud. Assume that the number of point clouds in each columnar structure is N. sum Then the information of each point cloud can be represented by a vector of D=5, namely D=(x, y, z, r, l), where x, y, z, r are the input point cloud data information, and l is the relative position of the geometric center of the point cloud column.

[0069]

[0070] Where, x p y p , z pThese represent the coordinates of the center point of the column.

[0071] To preserve N point clouds in each column, if N > N sum Padding with zeros, or randomly selecting point clouds, preserving point clouds from areas of lower density as much as possible. The specific steps are as follows:

[0072] Assume each sample has P non-empty columns, and each column grid has N columns. sum Given a point cloud dataset, randomly select a point cloud i with coordinates i = (x, y, z). Set a range length d and count the number N points in the point cloud within the interval ([xd, x+d], [yd, y+d], [zd, z+d]). around The probability that this point will be discarded is... This process is repeated until N point clouds are selected. The sample can then be represented by a tensor of (D, P, N).

[0073] Furthermore, in step (2), the image in the 3D scene is subjected to grayscale processing to optimize the processing and improve the accuracy of target detection. After grayscale processing, the color changes will be removed, and only the trend of gray intensity changes will be retained.

[0074] After completing the grayscale processing of the image, the mean filtering method is used to suppress noise and improve image quality. Since the output of the center pixel of the mean template is mainly determined by the low-pass filtering form, and most of the details of the image target are distributed in the high-frequency region of the image, the choice of template determines the noise suppression capability and the clarity of its details. Therefore, the selection of template size needs to take into account both noise reduction capability and the preservation of image details.

[0075] In the further step (3), the original columnar extraction network is used, and the simplified PointNet network is used to process the point cloud data and extract features. After processing, the representation of the point cloud changes from 5-dimensional D to C-dimensional, so the entire scene feature tensor becomes (C, P, N). Directly performing max pooling on the (C, P, N) tensor in the N dimension directly yields the (C, P) tensor, but this would lose a lot of information. Therefore, this system adds two other pooling methods: random pooling and average pooling. Random pooling selects based on the magnitude of the values ​​in the entire dimension, with larger values ​​having a higher probability of being selected. Average pooling, as the name suggests, calculates the average value of the entire dimension and takes the average value as the final result. The three pooling methods can extract different types of features, such as Figure 2 As shown.

[0076] like Figure 3As shown, max pooling, average pooling, and random pooling are performed on the dimension containing the column, resulting in three (C, P)-dimensional feature maps F. max = (C, P), F avg = (C, P), F ran = (C, P) Then add and fuse the three feature maps F fin =F max +F avg +F ran Dividing the P dimension into H and W, i.e., P = H * W, yields the feature map (C, H, W) formed after transformation and fusion.

[0077] like Figure 4 As shown, the above feature map is divided into multiple 4*4 pixel small feature maps, each called a patch. The original feature map is then transformed into a (16*C, H / 4, W / 4) format, where the number of patches is PN = H / 4*W / 4. Each patch's feature map (16*C, 4, 4) is transformed into a one-dimensional feature vector. Therefore, the entire feature map can be transformed into PN 16*C*H / 4*W / 4 feature vectors. The vector corresponding to each patch, plus positional encoding, yields the final feature vector to be passed to the attention feature extraction module.

[0078] The formula for obtaining the position encoding vector is as follows:

[0079]

[0080] Where t represents the position of each patch in the feature map. d represents the position vector corresponding to each patch. model The dimension corresponding to each patch is 16*C*H / 4*W / 4 in this case. i represents the position number.

[0081] By adding the positional encoding vector to the vector corresponding to each patch, the final feature vector F to be passed to the attention feature extraction module is obtained. patch = (PN, 16*C*H / 4*W / 4).

[0082] like Figure 5 As shown, in the attention feature extraction module, the vector corresponding to each input patch is first calculated, and the vectors corresponding to each patch are combined into a matrix represented by X, i.e., X = (PN, 16*C*H / 4*W / 4), to obtain Q, K, and V, where:

[0083] Q = X * W q

[0084] K = X * W k

[0085] V = X * W v

[0086] W q W k and W v It is a learnable linear transformation matrix. After calculating Q, K, and V using the self-attention formula, we can obtain a feature vector that fuses the upper and lower features. The overall formula for calculating self-attention is:

[0087]

[0088] Where, d k It is the number of columns in the Q and K matrices, i.e., the vector dimension.

[0089] After using the attention feature extraction module, we can obtain the result related to F. patch Given PN vectors of the same dimension (16*C*H / 4*W / 4), this can be represented as a feature matrix of (PN, 16*C*H / 4*W / 4). Reorganizing this feature matrix yields a feature map of (C,H,W). Finally, convolution is performed on the feature matrix to obtain a new feature map F. pcl = (4C, H / 2, H / 2);

[0090] The image processed in step (2) is subjected to feature extraction using a 2D feature extraction network, and the feature map of the image is obtained with dimension F. picture = (2C, H / 2, W / 2), to obtain the point cloud features F above. pcl With image features F picture Feature map F obtained by feature concatenation mix = (6C, H / 2, W / 2) is fed into the multi-branch detection head network for training.

[0091] The multi-branch detection head network module corresponds to the input feature map F. mix Three downsampling operations are performed to obtain three feature maps of different sizes. Each of these feature maps is then subjected to a 1x1 convolution operation followed by top-down upsampling feature fusion. The feature maps of different sizes are responsible for classification and regression prediction of targets of different sizes. Typically, the topmost feature map has the largest receptive field, thus providing better classification and regression prediction results for large targets. The middle layers are responsible for classification and regression prediction of medium-sized targets, and the bottom layer is responsible for classification and regression prediction of small targets.

[0092] Furthermore, for step (4), the total loss function of the network is defined as:

[0093]

[0094]

[0095]

[0096] L CLS =-α a (1-p α ) γ logp α

[0097] Where L is the total loss function, L CLS For the loss function of classification, L Ioc Let L be the loss function for location. dir β is the orientation classification loss function; Ioc ,β CLS ,β dir These are the coefficients of the location loss function, the classification loss function, and the orientation classification loss function, respectively; P is the predicted bounding box, G is the ground truth bounding box, and IoU represents the overlap ratio between the predicted and ground truth bounding boxes. 2 (P, G) represents the square of the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box, c 2 This refers to the squared distance between the diagonals of the enclosing regions of two rectangles, where p is the probability that the sample is positive; a a Here, α and γ are the weighting coefficients, α and γ are the system hyperparameters, and N is the weighting coefficient. pos To determine the number of effective prediction boxes;

[0098] Further, in step (5), after inputting the KITTI test set data into the trained network model, the final target classification prediction branch will predict the type of the target, and the regression prediction branch will output the target's x, y, z, w, h, l, θ; where (x, y, z) represents the target center point position, w represents the predicted box width, h represents the predicted box height, l represents the predicted box length, and θ represents the vehicle yaw angle.

[0099] The following description, in conjunction with the accompanying drawings and embodiments, provides a further explanation of a specific embodiment of the present invention. These embodiments are merely illustrative of the technical solutions of the present invention and should not be construed as limiting the scope of protection of this application. Figure 1 The flowchart shown is for this 3D object detection method. The system mainly consists of six parts: a columnar extraction network, a feature transformation module, an attention mechanism feature extraction network, image preprocessing, a 2D feature extraction network, and a detection head composed of a multi-branch decoupling network.

[0100] The deep learning-based 3D object detection method of this invention includes the following steps:

[0101] (1) Loading the dataset: Divide the dataset into training set, test set and validation set. Read and save the real labels of the training set and tensile the read training set point cloud to obtain the point cloud data. Specifically, first divide the input point cloud data into a network according to the X and Y axes to form a columnar structure. The bottom width and length of each columnar structure are 0.16 meters.

[0102] Assume that the number of point clouds in each column is N. sum Then the information of each point cloud can be represented by a vector of D=5, namely D=(x, y, z, r, l), where x, y, z, r are the input point cloud data information, and l is the relative position of the point to the geometric center of the column.

[0103]

[0104] Where, x p y p , z p These represent the coordinates of the center point of the column.

[0105] Retain N = 35 point clouds in each column; if N > N sum Padding with zeros, or randomly selecting point clouds, preserving point clouds from areas of lower density as much as possible. The specific steps are as follows:

[0106] Assume each sample has P non-empty columns, and each column grid has N columns. sum Given a point cloud dataset, randomly select a point cloud i with coordinates i = (x, y, z). Set a range length d and count the number N points in the point cloud within the interval ([xd, x+d], [yd, y+d], [zd, z+d]). around The probability that this point will be discarded is... This process is repeated until N point clouds are selected. The sample can then be represented by a tensor of (D, P, N).

[0107] (2) Extract columnar features: such as Figure 2 As shown, the network will extract features from the tensor (D, P, N) obtained in the previous step.

[0108] Using the tensor (D, P, N) obtained in the first step as input, the D dimension in the original tensor is generated into the C dimension through a fully connected network layer to obtain more point cloud feature information.

[0109] (3) Compressed columnar features: based on Figure 3 The features obtained in the previous step will be transformed into a two-dimensional feature map.

[0110] Max pooling, average pooling, and random pooling are performed on the dimension containing the columnar structures in the tensor, resulting in three (C, P)-dimensional feature maps F. max = (C, P), F avg = (C, P), F ran = (C, P) Then add and fuse the three feature maps F fin =F max +F avg +F ran = (C, P).

[0111] Dividing the P dimension into H and W, i.e., P = H * W, yields the feature map (C, H, W) formed after transformation and fusion.

[0112] (4) Use an attention mechanism to extract features from the obtained feature map: such as Figure 4 As shown.

[0113] The feature map is divided into multiple 4x4 pixel smaller feature maps, each called a Patch. The original feature map is then transformed into a (16*C, H / 4, W / 4) format, where the number of Patches is PN = H / 4 * W / 4. The 2D feature map (16*C, 4, 4) within each Patch is converted into a feature vector. Therefore, the entire feature map can be transformed into PN 16*C*H / 4*W / 4 feature vectors. The feature vector corresponding to each Patch is then combined with a positional encoding to obtain the final feature vector to be passed to the attention feature extraction module. The formula for obtaining the positional encoding vector is as follows:

[0114]

[0115] Where t represents the position of each patch in the feature map. d represents the position vector corresponding to each patch. model The dimension corresponding to each patch is 16*C*H / 4*W / 4 in this case. i represents the position number. By adding the above position encoding features to the feature vector corresponding to each patch, the final feature vector (PN, 16*C*H / 4*W / 4) to be passed to the attention feature extraction module is obtained.

[0116] A schematic diagram of the attention feature extraction module is shown below. Figure 5 As shown.

[0117] In the attention feature extraction module, the feature vector corresponding to each input patch is first calculated. The feature vectors corresponding to each patch are combined into a matrix represented by X, i.e., X = (PN, 16*C*H / 4*W / 4), and Q, K, and V are obtained, where:

[0118] Q = X * W q

[0119] K = X * W k

[0120] V = X * W v

[0121] W q W k and W v It is a learnable linear transformation matrix.

[0122] After calculating Q, K, and V using Self-Attention, we can obtain a feature vector that integrates the upper and lower features. The overall formula for calculating Self-Attention is:

[0123]

[0124] Where, d k It is the number of columns in the Q and K matrices, i.e., the vector dimension.

[0125] Following the attention feature extraction module in the previous step, we obtain vectors with the same dimension as the input, namely PN (16*C*H / 4*W / 4) vectors, which can be represented as a (PN, 16*C*H / 4*W / 4) feature matrix. By reorganizing the feature matrix, we obtain a (C, H, W) feature map. Finally, we perform a convolution operation on the feature map to obtain a new feature map F. pcl = (4C, H / 2, H / 2).

[0126] (5) Preprocessing the acquired images: The images in the 3D scene are converted to grayscale to optimize the processing and improve the accuracy of target detection. After grayscale processing, the color changes will be removed, and only the trend of gray intensity changes will be retained.

[0127] Using G(r, r, r) from the grayscale model, corresponding to RGB(R, G, B) in the color image model, the grayscale value ranges from 0 to 255, where 0 represents a completely black image and 255 represents a completely white image. Based on the weighted average method, the weighted average of RGB values ​​is calculated as follows:

[0128] R = G = B = (Q) r R+Q g G+Q b B)

[0129] Among them: Q r Q represents the weight of R. g Q represents the weight of G.b This represents the weight of B. Referring to human visual habits and meeting the actual needs of application scenarios, extensive experimental data shows that a ratio of R:G:B = 0.3:0.59:0.1 best matches the subjective perception of the human eye. Therefore, the grayscale value (Gray) can be calculated using a weighted average as follows:

[0130] Gray = 0.3R + 0.59G + 0.11B

[0131] After grayscale processing of the image, mean filtering is used to suppress noise and improve image quality. The acquired image is divided into several templates of the same size. Assuming the original image resolution is X*Y, a new filtered image is obtained after mean filtering. The calculation process is as follows:

[0132]

[0133] Where j(x, y) represents the image before filtering, k(a, b) represents the new filtered image, ∑∑ represents the summation of all pixels in the window, and M×N is the window size.

[0134] Since the output of the center pixel of the mean template is mainly determined by the low-pass filtering method, and most of the details of the image target are distributed in the high-frequency region of the image, the choice of template determines the noise suppression capability and the clarity of its details. Therefore, the selection of template size needs to take into account both noise reduction capability and preservation of image details.

[0135] (6) Use a 2D feature extraction network to extract features from the processed image: Here, the ResNet-50 network is generally used as the image feature extraction network, and the final feature map of the image has a dimension of F. picture =(2C,H / 2,W / 2) is used to facilitate subsequent stitching with point cloud features.

[0136] like Figure 6 As shown, the point cloud feature map F above is obtained. pcl With image feature map F picture Feature F obtained by feature concatenation mix = (6C, H / 2, W / 2) is fed into the multi-branch decoupled detection head network for training.

[0137] like Figure 6 As shown, the multi-branch detection head network module corresponds to the input feature map F. mixThree downsampling operations are performed to obtain three feature maps of different sizes. Each of these feature maps is then subjected to a 1x1 convolution operation followed by top-down upsampling feature fusion. The feature maps of different sizes are responsible for classification and regression prediction of targets of different sizes. Typically, the topmost feature map has the largest receptive field, thus providing better classification and regression prediction results for large targets. The middle layers are responsible for classification and regression prediction of medium-sized targets, and the bottom layer is responsible for classification and regression prediction of small targets.

[0138] The output detection results mainly include the category information of the detected target's bounding box, center point coordinates, width, height, and length, as well as the vehicle's yaw angle (x, y, z, w, h, l, θ). The network is trained using a loss function to achieve convergence. The specific loss function is as follows:

[0139]

[0140]

[0141]

[0142] L CLS =-α a (1-p α ) γ logp α

[0143] Where L is the total loss function, L CLS For the loss function of classification, L Ioc Let L be the loss function for location. dir β is the orientation classification loss function; Ioc ,β CLS ,β dir ρ represents the coefficients of the location loss function, the classification loss function, and the orientation classification loss function, respectively; P is the predicted bounding box, G is the ground truth bounding box, and IoU represents the proportion of overlap between the predicted and ground truth bounding boxes. 2 (P, G) represents the square of the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box, c 2 This refers to the squared distance between the diagonals of the enclosing regions of the two rectangles. The parameter p is the probability that the sample is positive; a a Here, α and γ are the weighting coefficients, and α and γ are the system's hyperparameters. Primarily targeting the regression module, this system employs an IoU loss function designed to improve the accuracy of the predicted box's position and length by using the ratio between the predicted and ground truth boxes and the diagonal of the closure region between boxes for location regression.

[0144] (7) Usage phase: After the KITTI test set data is input into the trained network model, the final target classification prediction branch will predict the type of the target, and the regression prediction branch will output the x, y, z, w, h, l, θ of the target; where (x, y, z) represents the position of the target center point, w represents the width of the prediction box, h represents the height of the prediction box, l represents the length of the prediction box, and θ represents the vehicle yaw angle.

[0145] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep learning-based 3D object detection method, characterized in that, The method comprises the following steps: Step (1), reading the KITTI dataset, dividing the KITTI dataset into a training set and a test set, loading the training set to train the model, preprocessing the loaded point cloud data, using a vector to represent each point cloud, dividing the scene into multiple columnar bodies, and using a tensor to represent the entire scene; Step (2), loading the image corresponding to the point cloud scene in step (1), performing grayscale processing on the image in the scene, optimizing the processing process, improving the target detection accuracy, then performing mean filtering processing to suppress noise and improve image quality; Step (3), constructing a deep learning convolutional neural network, including a columnar extraction network, an image preprocessing module, a 2D feature extraction network, and a multi-branch detection head network; Step (4), loading the training set, the data is obtained after forward propagation, the output of the deep learning convolutional neural network is calculated, the loss degree is calculated, the network parameters are updated, and the trained neural network model is obtained; Step (5), in the use stage, the point cloud data and the image are sent into the pre-trained neural network model, and the position and category of each 3D target are calculated; In step (1), the original radar point cloud data is read, and the point cloud data is divided into a grid according to the X and Y axes in the scene. Any point cloud data falling into a grid is considered to be in a columnar body; Reading data of each point cloud in the columnar body, including point cloud coordinates ) and reflection intensity of the point cloud , assuming the number of point clouds in each columnar body is ， Then the information of each point cloud can be represented by a vector of = 5, respectively , where is the point cloud information, is the relative position of the geometric center of the point cloud columnar body; wherein respectively represent the coordinate positions of the center points of the columnar bodies; Reserve a point cloud in each columnar body If > Zero fill, otherwise randomly extract a point cloud, the specific method is as follows: Suppose each sample has P non-empty columns, and a certain column grid has From a point cloud dataset, randomly select one point cloud. The coordinates of this point cloud are Set the range length. Statistics in the interval Number of inner point clouds The probability that this point will be discarded is... This process is repeated until N point clouds are selected. Each sample can then be represented by a tensor of (D, P, N). In step (2), the image in the scene is subjected to grayscale processing. After grayscale processing, the color change is removed, and only the gray intensity change trend is retained; Using the gray scale model corresponding to the color image model According to the weighted average method, the weighted average value of RGB is calculated as follows: wherein represents the weight of , represents the weight of , represents the weight of . After the gray processing of the image is completed, the mean filter method is used to suppress noise and improve the image quality. The collected image is divided into several templates with the same size. It is assumed that the original image resolution is After the mean filter processing is completed, a new filtered image is obtained, and the calculation process is as follows: wherein, denotes the image before filtering, denotes the new filtered image, denotes the sum over all pixels in the window, is the window size.

2. The method of claim 1, wherein, In step (3), the sample tensor (D, P, N) obtained in step (1) is subjected to feature extraction by the simplified PointNet network, and is converted into a tensor (C, P, N). Through maximum value pooling, average value pooling and random pooling on the dimension of the columnar body N in the tensor, three feature maps of (C, P) dimensions are obtained, respectively = (C, P), = (C, P), = (C, P) Then the three feature maps are added and fused , the P dimension is divided into H and W, that is, P = H W, a feature map of (C, H, W) formed after conversion and fusion is obtained; The feature map is divided into a plurality of 4 4 pixels of the feature map, each small feature map is called a patch, and the feature map is converted into (16 C, H / 4, W / 4) format, and each patch feature map (16 C,4,4) is converted into a one-dimensional feature vector, and the entire feature map is converted into PN 16 C H / 4 W / 4 feature vector representation, wherein PN is the number of patches, and each patch corresponds to a feature vector plus a position encoding vector to obtain a feature vector to be input into the attention feature extraction module; wherein the formula for obtaining the position encoding vector is as follows: where t is the position of each patch in the feature map, represents the position vector corresponding to each patch, is the dimension corresponding to each patch, here it refers to 16 C H / 4 W / 4, = , i is the position number, By adding the above position encoding vector and the vector corresponding to each patch, the feature vector to be finally transmitted into the attention feature extraction module is obtained = PN (16 C H / 4 W / 4) In the attention feature extraction module, firstly, the vector corresponding to each patch is calculated, and the vector corresponding to each patch is combined into a matrix represented by X, that is, X= (PN, 16 C H / 4 W / 4), Q, K, and V are obtained, wherein: and is a learnable linear transformation matrix, and after Q, K, and V are calculated through the self-attention formula, a feature vector that integrates the upper and lower features is obtained. The calculation formula of the total self-attention is: wherein, is the number of columns of Q, K matrices, i.e. the vector dimension; After the attention feature extraction module, a vector with the same dimension as PN(16 C H / 4 W / 4) feature vectors can be obtained, represented as a feature matrix of (PN, 16 C H / 4 W / 4), and a feature map of (C, H, W) is obtained by reorganizing the feature matrix, and finally a new feature map = (4C, H / 2, H / 2) is obtained by convolution operation on the feature matrix. The 2D feature extraction network is used for feature extraction on the image processed in step (2), and the dimension of the feature map of the image is = (2C, H / 2, W / 2), and the feature map obtained by splicing the point cloud feature and the image feature is = (6C, H / 2, W / 2) is sent to the multi-branch detection head network module for training. The multi-branch detection head network module corresponds to the incoming feature map Three downsampling operations are respectively performed to obtain three different specifications of feature maps 1After the convolution operation, the features are fused from top to bottom. Different specifications of feature maps are respectively responsible for classification prediction and regression prediction tasks of different sizes of targets. Generally, the feature map at the top layer has the largest receptive field, so it has better classification prediction and regression prediction effect for large targets. The middle layer is responsible for the classification prediction and regression prediction tasks of medium-sized targets, and the lower layer is responsible for the classification prediction and regression prediction tasks of small targets.

3. The method of claim 1, wherein, In step (4), the total loss function of the network is defined as: wherein L is a total loss function, is a classification loss function, is a position loss function, is a direction classification loss function; , , are a coefficient of the position loss function, a coefficient of the classification loss function and a coefficient of the direction classification loss function, respectively; P is a predicted bounding box, G is a ground truth bounding box, and IoU represents a proportion of overlap between the predicted bounding box and the ground truth bounding box; represents a square of a distance between a center point of the predicted bounding box and a center point of the ground truth bounding box, refers to a square of a distance of a diagonal line of a closed region of the two bounding boxes; and parameters is a probability that a sample is positive; is a weight coefficient, and α, is a hyperparameter of the system, is a number of effective predicted bounding boxes.

4. The method of claim 1, wherein, In step (5), after the KITTI test set data is transmitted into the trained network model, finally, for each target, the classification prediction branch predicts the type of the target, and the regression prediction branch outputs the ; wherein (x, y) represents the target center point position, represents the predicted box width, represents the predicted box height, represents the predicted box length, represents the predicted box orientation angle.​

Images