A millimeter-wave radar point cloud classification method

Abstract

This paper discloses a method for millimeter-wave radar point cloud classification. After data preprocessing, a millimeter-wave radar point cloud dataset is input into a classifier, ultimately outputting radar target points with classification information. This paper implements a method for millimeter-wave radar target point cloud classification using a deep neural network. This method achieves classification accuracy exceeding 90%, with an IoU (Interference over Union) of foreground points reaching 0.4, effectively filtering out unused targets and improving target detection.

Description

Technical Field

[0001] The present invention is applied to the field of autonomous driving or robotics, and specifically relates to a millimeter-wave radar point cloud classification method. Background Art

[0002] With the rapid development of autonomous driving technology, the various sensors used for environmental perception have also undergone technological and product upgrades. Millimeter-wave radar, as a mature automotive sensor, offers advantages such as long detection range, accurate speed measurement, and stable operation in adverse weather conditions. However, due to frequency band limitations, millimeter-wave radar has limited detection resolution and suffers from a limited number of targets. Recent products have significantly improved resolution, and the increased number of targets detectable by radar has also enabled further advancements in point cloud classification.

[0003] In the past two years, researchers have proposed neural networks that can directly learn point clouds based on the unique permutation invariance and rotation invariance of point clouds, achieving good segmentation and detection results. However, there has been no extensive research in deep learning for two-dimensional millimeter-wave radar target point clouds. Although millimeter-wave radar can only detect x and y coordinates, which is two fewer dimensions than the x, y, z, and intensity of laser radar, its unique relative velocity v r The radar cross-sectional area (RCS) information can increase the input dimension and improve the final point cloud segmentation effect. Therefore, if the millimeter-wave radar point cloud can be classified, it will play an important role in filtering invalid target points and extracting key information. Summary of the Invention

[0004] The purpose of the present invention is to provide a millimeter-wave radar point cloud classification method, which can assign category attributes to each millimeter-wave radar target point, playing an important role in filtering invalid target points and extracting key information.

[0005] To achieve the purpose of the present invention, the present invention provides a millimeter-wave radar point cloud classification method. The millimeter-wave radar point cloud dataset is input into a classifier after data preprocessing, and finally outputs radar target points with category information.

[0006] in,

[0007] The dataset is an open source dataset or a proprietary dataset created based on the radar model used and the driving scenario.

[0008] in,

[0009] The data preprocessing is used to ensure that the data in the millimeter wave radar point cloud data set contains at least the horizontal coordinate x, vertical coordinate y, and relative speed v of the obstacle target. rAnd radar cross section area rcs four target attributes.

[0010] in,

[0011] The data preprocessing is also used to resample each frame of data to ensure the consistency of the number of point clouds in each frame. The above resampling is performed according to the average number, rather than the maximum number.

[0012] in,

[0013] The network structure of the classifier is as follows:

[0014] The input data dimension is (n,4), and the output data dimension is (n,), where n represents the number of point clouds and 4 represents the number of input channels, respectively: horizontal coordinate x, vertical coordinate y, relative speed v r and radar cross section rcs, k represents the number of output categories;

[0015] (a) Point cloud classification network: The input data undergoes two one-dimensional convolution operations, with the dimension becoming (n, 64). Here, a multi-layer perceptron is used to represent this convolution operation. After three layers of MLP, the point cloud dimension is increased to (n, 1024). Max pooling is performed on this layer to obtain the global features of this frame of point cloud. Finally, a fully connected operation is performed on the 1024 global features to output k category scores for a single frame of point cloud.

[0016] (b) Point cloud segmentation network

[0017] In order to obtain the category information of each point, the (n, 64) middle layer in the above classification network is selected, and the global features and category scores are attached to each point cloud to obtain the input data of the segmentation network with a dimension of (n, 64+1024+k). The category score is added as a reference, and finally the category probability of each point cloud is obtained after five layers of MLP.

[0018] (c) Loss function

[0019] The loss function consists of two parts. One part is to use the cross entropy loss function for the accuracy of the category. The calculation formula is as follows:

[0020]

[0021] Among them, y i Indicates the label of sample i, the foreground point is 1 and the background point is 0; p i Indicates the probability that sample i is predicted to be positive;

[0022] The other part is to include the intersection of union (IoU) of foreground points in the calculation of the loss function. The calculation formula is as follows:

[0023]

[0024] Among them, c gt represents the true value of the labeled category, c pred represents the predicted category,

[0025] The total loss function is

[0026] L total =L1+α·L2

[0027] Among them, α is the proportion of L2 loss, and the default value is 0.01.

[0028] Compared with the existing technology, the beneficial effect of the present invention is that it realizes a method for millimeter-wave radar target point cloud classification using deep neural networks. The classification accuracy of this method can reach more than 90%, and the IoU of foreground points reaches 0.4, which plays a good role in filtering out useless targets and improving target detection effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of a millimeter-wave radar point cloud classification method for this application.

[0030] Figure 2 Label a schematic diagram for a scene.

[0031] Figure 3 The frequency distribution histogram of the ambiguity state at the foreground and background points is shown.

[0032] Figure 4 Shown is the network structure diagram of the classifier in this application. DETAILED DESCRIPTION

[0033] It should be noted that, unless there is any conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] The present invention proposes a millimeter-wave two-dimensional point cloud classification method for vehicle-mounted millimeter-wave radar sensors. This method can assign category attributes to each millimeter-wave radar target point, which plays an important role in filtering invalid target points and extracting key information.

[0036] The present invention inputs the original data of the two-dimensional detection target of the millimeter wave radar, and after data preprocessing, it passes through the forward propagation of the classification network and the segmentation network respectively, and finally outputs the radar target point with category information. The overall design framework is as follows: Figure 1 As shown, how to design and train the classifier network is the key to the present invention.

[0037] (1) Dataset creation

[0038] The dataset can use an open source dataset or create a proprietary dataset based on the radar model and driving scenario used. Since the point cloud of the millimeter-wave radar is sparse, it is impossible to determine the exact size and position of obstacles during annotation. Therefore, it is recommended to collect the three-dimensional point cloud of the lidar and the image data of the camera at the same time during the data collection process. Figure 2 The figure shows the annotation results of a scene equipped with five millimeter-wave radars and one lidar. The millimeter-wave radar point cloud within the bounding box is determined to be the detection target of the object and is labeled with the corresponding category. The categories described in this invention include two categories, denoted by k, that is, k = 2, namely foreground obstacles (people, cars, cyclists, etc.) and background obstacles (trees, railings, etc.).

[0039] (2) Data preprocessing

[0040] The millimeter-wave radar product data of different manufacturers are slightly different. In order to ensure the output effect of the classifier described in the invention, it is stipulated that the millimeter-wave radar raw data at least contains the horizontal coordinate x, vertical coordinate y, relative speed v of the obstacle target. r and radar cross section rcs. Some radar products return target attributes that are richer. In addition to the above attributes, they may also contain the measurement status of obstacles, which can be selectively included during training. The selection method can be judged by considering whether the statistical characteristics of the attribute in different categories are similar. For example, a radar product will also give the Doppler radial velocity uncertainty state (ambiguity state) of each target. However, the frequency distribution of this attribute in the foreground and background is almost the same, such as Figure 3 As shown in the figure, the mean and variance are almost equal, which is not recommended as an input feature for the classifier.

[0041] Furthermore, the number of targets returned by millimeter-wave radars may vary depending on their operating modes. To facilitate matrix operations during subsequent model training, each frame of data needs to be resampled to ensure a consistent number of point clouds per frame. Considering the excessive number of useless background target points in millimeter-wave radars, this invention resamples based on the average number, rather than the maximum number.

[0042] (3) Classifier

[0043] like Figure 4 The network structure of the classifier of the present invention is shown as follows. The input data dimension is (n, 4) and the output data dimension is (n,), where n represents the number of point clouds and 4 represents the number of input channels, respectively: horizontal coordinate x, vertical coordinate y, relative speed vr If there are other channels with significantly different statistical characteristics, they can also be added to the network training. k represents the number of output categories.

[0044] a. Point cloud classification network

[0045] After two one-dimensional convolutions, the input data becomes (n, 64). This convolution operation is represented here using a multi-layer perceptron (MLP). Three more MLP layers increase the point cloud dimension to (n, 1024). Max pooling is performed on this layer to obtain the global features of the point cloud frame. Finally, a fully connected operation is performed on the 1024 global features to output k category scores for the single-frame point cloud.

[0046] b. Point cloud segmentation network

[0047] To obtain the category information for each point, we then select the (n, 64) intermediate layer in the classification network and append the global features and category scores to each point cloud, obtaining the input data for the segmentation network with a dimension of (n, 64 + 1024 + k). This combination of three is used as the input to the segmentation network because we believe that the intermediate layer and global feature layer of the network best reflect the shape structure of the point cloud in that frame. The category scores are then added as a reference, and finally, after five layers of MLP, the category probability of each point cloud is obtained.

[0048] c. Loss function

[0049] The loss function consists of two parts. One part is to use the cross entropy loss function for the accuracy of the category. The calculation formula is as follows:

[0050]

[0051] Among them, y i Indicates the label of sample i, the foreground point is 1 and the background point is 0; p i represents the probability that sample i is predicted to be positive.

[0052] The other part is to include the intersection of union (IoU) of foreground points in the calculation of the loss function. The calculation formula is as follows:

[0053]

[0054] Among them, c gt represents the true value of the labeled category, c pred Indicates the predicted category.

[0055] The total loss function is

[0056] LtOtAl =L1+α·L2

[0057] Among them, α is the proportion of L2 loss, and the default value is 0.01.

[0058] The present invention implements a method for millimeter-wave radar target point cloud classification using a deep neural network. The classification accuracy of this method can reach over 90%, and the IoU of foreground points reaches 0.4, which plays a good role in filtering out useless targets and improving target detection effects.

[0059] It should be noted that the technical solutions not described in detail in this application adopt publicly known technologies.

[0060] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A millimeter wave radar point cloud classification method, characterized in that: The millimeter-wave radar point cloud dataset, after data preprocessing, is input into the classifier, and finally outputs radar target points with category information; The network structure of the classifier is as follows: The input data dimension is (n, 4), and the output data dimension is (n, k), where n represents the number of point clouds and 4 represents the number of input channels, which are: horizontal coordinate x, vertical coordinate y, relative speed v r and radar cross section rcs, k represents the number of output categories; (a) Point cloud classification network: After two one-dimensional convolution operations, the input data becomes (n, 64) in dimension. This convolution operation is represented by a multi-layer perceptron. After three layers of MLP, the point cloud dimension is increased to (n, 1024). Max pooling is performed on this layer to obtain the global features of this frame of point cloud. Finally, a fully connected operation is performed on the 1024 global features to output k category scores for a single frame of point cloud. (b) Point cloud segmentation network The (n, 64) middle layer in the above classification network is selected, and the global features and category scores are attached to each point cloud to obtain the input data of the segmentation network with a dimension of (n, 64+1024+k). The category scores are added as a reference, and finally the category probability of each point cloud is obtained after five layers of MLP. (c) Loss function The loss function consists of two parts. One part is to use the cross entropy loss function for the accuracy of the category. The calculation formula is as follows: Among them, y i Indicates the label of sample i, the foreground point is 1 and the background point is 0; p i Indicates the probability that sample i is predicted to be positive; The other part is to include the intersection of union (IoU) of foreground points in the calculation of the loss function. The calculation formula is as follows: Among them, c gt represents the true value of the labeled category, c pred Indicates the predicted category; The total loss function is L total =L1+α·L2 Among them, α is the proportion of L2 loss, and the default value is 0.

01.

2. The millimeter wave radar point cloud classification method according to claim 1, characterized in that: The dataset is an open source dataset or a proprietary dataset created based on the radar model used and the driving scenario.

3. The millimeter wave radar point cloud classification method according to claim 1, characterized in that: The data preprocessing is used to ensure that the data in the millimeter wave radar point cloud data set contains at least the horizontal coordinate x, vertical coordinate y, and relative speed v of the obstacle target. r And radar cross section area rcs four target attributes.

4. The millimeter wave radar point cloud classification method according to claim 3, characterized in that: The data preprocessing is also used to resample each frame of data to ensure the consistency of the number of point clouds in each frame. The above resampling is performed according to the average number, rather than the maximum number.

Images