Automatic driving-oriented ViT-based millimeter wave radar point cloud segmentation method
By preprocessing and extracting features from millimeter-wave radar point clouds using a ViT-based neural network architecture, the challenge of sparse data segmentation is solved, achieving efficient semantic segmentation and meeting the detection performance requirements of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH OF CHINA
- Filing Date
- 2023-07-28
- Publication Date
- 2026-07-14
Smart Images
Figure CN117475142B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of autonomous driving and deep learning, and specifically relates to a millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving. Background Technology
[0002] Autonomous vehicles need to understand their surrounding environment and road conditions in order to navigate safely in dynamic real-world environments. To achieve reliable overall environmental perception and improve driving safety, the sensor suites of autonomous vehicles are often multifunctional. Different sensor modes (such as cameras, LiDAR, or radar) are usually combined to complement each other. Especially in autonomous driving scenarios, the failure of one mode can lead to fatal consequences. Therefore, multimodal information acquisition is essential. Cameras and LiDAR sensors, which are widely used on vehicles, can accurately capture the environment, but their perception mechanisms limit information acquisition in adverse weather conditions such as fog, rain, and snow.
[0003] Therefore, additional information is needed to supplement support in this situation, and this information can be obtained through radar sensors. Because radar operates in the millimeter-wave band, it can still stably perceive the surrounding environment even in adverse weather conditions. In addition, from a hardware cost perspective, millimeter-wave radar is low-cost and widely used, and has been integrated into various series of vehicles for several years. Therefore, radar sensors are crucial for achieving safe autonomous driving.
[0004] Point cloud segmentation is the task of assigning a class label or a class probability vector to each point in a point cloud. With the development of millimeter-wave radar and the gradual maturation of autonomous driving technology, millimeter-wave radar has received increasing attention for semantic segmentation tasks. The excellent performance of neural networks in image-based semantic segmentation tasks has driven many scholars to study point cloud-based semantic segmentation tasks. Point cloud-based semantic segmentation tasks can be summarized into three categories: projection-based, voxel-based, and point-based methods.
[0005] Projection-based methods, still inspired by convolutional neural networks (CNNs), project point clouds onto a frontal view or 2D distance image to utilize 2D convolution. However, projection-based methods face problems such as discrete errors and occlusion between objects due to the representation of intermediate features. Voxel-based methods, on the other hand, convert point clouds into dense cubes and utilize CNNs and multilayer perceptrons (MLPs) for computation, significantly improving computational speed. Since point clouds are sparse and have varying densities, only a small portion of the voxels are occupied by the point cloud. This makes applying dense CNNs inefficient. Point-based methods, however, process the point cloud directly, preserving the integrity of geometric information, and are equally applicable to sparse point clouds.
[0006] Among the many point cloud-based semantic segmentation methods, there are still some challenges, such as: (1) Compared with image and lidar point clouds, millimeter wave point clouds are sparser, the number of reflection points of the target is very small, and some small targets may only have one or two reflection points; (2) The amount of information contained in millimeter wave point clouds is limited, and the feature extraction method will affect the semantic features of the points; (3) Point clouds contain a large number of background points, and simple neighborhood aggregation features may overwhelm the feature learning of the target points.
[0007] Therefore, it is necessary to design a detection algorithm with practical value that can handle sparse data formats and fully extract radar information. Summary of the Invention
[0008] In view of this, the main objective of the present invention is to provide a millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving.
[0009] To achieve the above objectives, the technical solution of the present invention is implemented as follows:
[0010] This invention provides a ViT-based millimeter-wave radar point cloud segmentation method for autonomous driving. The method is as follows:
[0011] Preprocess the point cloud data and generate features with a time dimension to obtain training data;
[0012] A neural network is constructed using the training data;
[0013] The point cloud data is forward-propagated in a neural network architecture to generate predicted classification labels for the point cloud; the predicted classification labels for the point cloud include real labels and predicted labels.
[0014] The loss function is determined by comparing the true labels with the predicted labels on the classification loss function.
[0015] The parameters of the neural network are backpropagated using the loss function, and the parameters are iteratively updated using the SGD optimizer to obtain a trained neural network.
[0016] The trained neural network is used to perform semantic segmentation on the prediction dataset to determine the category label of each point.
[0017] In the above scheme, the preprocessing of point cloud data and the generation of features with a time dimension to obtain training data specifically involves: given a frame of point cloud data P0 = {p0, p1, ..., p...} with timestamp t0. n As the first frame of data, any point in the point cloud The system has six-dimensional features: position (horizontal and vertical coordinates), distance, azimuth, velocity, and radar cross-section. It continues to read point cloud frames with a time interval of 500ms, using the frame number as the temporal feature. The features of each point are then updated accordingly. Aggregate all frames into one frame P = {P0, P1, ... P} k}, obtain a point cloud training set with added time information; for each frame of point cloud data in the point cloud training set, process each frame of point cloud to fix the number of points in each frame to 3072. If the number of points is insufficient, randomly copy existing points to fill them. If the number of points is more than the specified number, randomly remove points of the background category until the number of points is equal to the specified number. The processed point cloud frame is used as the input data of the network, i.e., the training data.
[0018] In the above scheme, the construction of the neural network using the training data specifically involves: constructing the network in a hierarchical structure, with the following connection order: feature fusion layer, two-level sampling feature extraction layer, two-level feature propagation layer, and classifier; each sampling feature extraction layer includes a downsampling module, a neighborhood generation module, and a neighborhood feature extraction module; wherein the neighborhood feature extraction module includes a perception module and a spatiotemporal attention module, and each feature propagation layer includes an interpolation module and a feature post-fusion module.
[0019] In the above scheme, the feature fusion layer extracts features from the radar observation information of the input point cloud, that is, the radar feature grouping of the point cloud. Specifically, it divides the seven-dimensional information of the point cloud into two groups, (x, y, t) and (r, a, υ). r ,rcs,t), respectively, use a one-dimensional convolutional network to extract 32-dimensional features, and obtain The two sets of features are then fused together using a one-dimensional convolutional network to obtain 64-dimensional features, resulting in the final fused features. As the output of the feature fusion layer.
[0020] In the above scheme, each of the two-level sampling feature extraction layers SA1 and SA2 constructs a downsampling module sblock, a neighborhood generation module gblock, and a neighborhood feature extraction module ablock. The neighborhood feature extraction module is composed of a spatiotemporal attention submodule ViT and a perception submodule RA. The spatiotemporal attention submodule ViT performs attention operations on the neighborhood point cloud and outputs neighborhood features. The perception submodule RA performs convolution operations on the radar features of the neighborhood point cloud and outputs new features. The outputs of the spatiotemporal attention submodule ViT and the perception submodule RA are concatenated and then fused using convolutional operations via MLP. The number of samples in each downsampling module is N1 and N2, respectively; the neighborhood sizes generated in each layer are K1 and K2, which are the neighborhood features output by the neighborhood feature extraction module in the second level.
[0021] In the above scheme, the interpolation module of each feature propagation layer in the two feature propagation layers FP1 and FP2 will downsample the point cloud P. i i = 0, 1, 2 and the point cloud P before samplingi-1 To perform point-to-point distance metric calculations, for each point in the point cloud before sampling, select the M closest points from the downsampled points, based on the distance d. j For j = 1, 2, ..., m, calculate the reciprocal of the distance and normalize it as the weight feature a of the point. j j = 1, 2, ..., m; the features of the M nearest points are multiplied by their respective weights and added together to form the propagated features of that point. In the feature fusion module, these features are connected with the sampled features extracted from the point cloud in the sampling feature extraction layer. The final propagated features are then obtained through further fusion via MLP.
[0022] In the above scheme, the classifier is a classification predictor composed of a multilayer perceptron (MLP), and the result is normalized by the Softmax activation function; there are 6 categories, including background, car, pedestrian and bicycle, which are used to classify the final propagation features. The dimensions of each layer of the MLP are [256, 64, 6].
[0023] In the above scheme, the step of determining the loss function by inputting the real label data and the predicted label data into the loss function specifically involves: in each training epoch n, the training batch size (batchsize) B n The total number of training samples is S, and the predicted probability of each sample in each category is... The true category is y n Focal loss is used as the final loss function.
[0024] In the above scheme, the backpropagation of the neural network parameters through the loss function and the iterative updating of the parameters using the SGD optimizer to obtain a trained neural network specifically involves: optimizing all parameters of the network using the SGD optimizer, and setting the initial learning rate to l. r0 The training data includes the total number of training iterations (E), the learning rate decay step size (stepsize), and the decay rate (decay). One batch of training data is read in, fed into the network for forward propagation, and the loss function is calculated and then backpropagated. If the number of training steps meets the decay condition, the learning rate is decayed, and training stops after the preset total number of training iterations is reached.
[0025] In the above scheme, the step of performing semantic segmentation on the prediction dataset using a trained neural network to determine the category label of each point is specifically: taking the category with the highest prediction probability as the prediction label, thus obtaining the semantic segmentation result of the point cloud.
[0026] Compared with existing technologies, this invention utilizes the generated temporal features to expand the dimension of the point cloud input to the neural network, enhancing the representational ability of the point cloud and the network's ability to classify targets from the background. This invention fuses the radar information contained in each neighborhood of the point cloud and uses time encoding of the point cloud neighborhood. It uses ViT to notice the spatial distribution changes within the point cloud neighborhood at different times, outputting the features of points of interest within the neighborhood, thus improving the discriminative power of the point cloud neighborhood features. The network designed in this invention has a small number of parameters, is easy to train, and its detection performance basically meets the daily needs of autonomous driving. Attached Figure Description
[0027] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and, together with their descriptions, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:
[0028] Figure 1 This invention provides a flowchart of a millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving, as an embodiment of the present invention.
[0029] Figure 2 This invention provides a structural diagram of the neural network in a ViT-based millimeter-wave radar point cloud segmentation method for autonomous driving, as provided in an embodiment of the present invention.
[0030] Figure 3 This invention provides a schematic diagram of the sampling feature extraction layer in a ViT-based millimeter-wave radar point cloud segmentation method for autonomous driving, as shown in the embodiments of the present invention.
[0031] Figure 4 This invention provides a schematic diagram of the feature propagation layer in a ViT-based millimeter-wave radar point cloud segmentation method for autonomous driving, as provided in an embodiment of the present invention. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0033] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, article, or apparatus that includes that element.
[0034] This invention provides a ViT-based millimeter-wave radar point cloud segmentation method for autonomous driving, such as... Figure 1 As shown, the method is as follows:
[0035] Step 101: Preprocess the point cloud data and generate features with a time dimension to obtain training data;
[0036] Specifically, given a frame of point cloud data P0 = {p0, p1, ..., p...} with timestamp t0, ... n As the first frame of data, any point in the point cloud With six-dimensional features including position (horizontal and vertical coordinates), distance, azimuth, velocity, and radar cross-section, point cloud frames with a time interval of 500ms are read forward. The frame number of each point is used as the temporal feature, and the features of each point are updated to p. i =(x i y i r i a i υ ri rcs i , t i Aggregate all frames into one frame P = {P0, P1, ..., P2}. k}, obtain the point cloud training set with added time information.
[0037] For each frame of point cloud data, the processing ensures that the number of points in each frame is fixed at 3072. If the number of points is insufficient, existing points are randomly copied to fill the gaps. If the number of points exceeds the specified number, points classified as background are randomly removed until the number of points equals the specified number. The processed point cloud frame serves as the input data for the network.
[0038] Currently, most millimeter-wave point cloud datasets are insufficient in terms of scale, target diversity, or the quality of the radar sensors used, limiting the development of radar point cloud-based perception tasks. In some embodiments, RadarScenes is used as a novel large-scale public dataset; therefore, this invention adopts the RadarScenes dataset as the dataset for millimeter-wave point cloud semantic segmentation tasks.
[0039] The RadarScenes dataset is a large-scale, publicly available dataset for automotive radar point cloud perception tasks. It utilizes sensors including a color camera, four GHz automotive radar sensors, an odometer, and a DGPS system. It records data in various formats, including images, point clouds, and vehicle motion data, and is widely used for tasks such as classification, semantic segmentation, and instance segmentation.
[0040] The initial labels include 11 target categories: cars, large vehicles, trucks, buses, trains, bicycles, electric two-wheelers, pedestrians, pedestrian groups, animals, and others. In addition to these 11 regular categories, a mapping function is provided that, along with the dataset tools, projects most classes into a coarser label set containing only: cars, large vehicles, two-wheelers, pedestrians, and pedestrian groups. Each point cloud also has a tracking label; point clouds belonging to the same target share the same tracking label.
[0041] This invention uses sequences from the RadarScenes dataset consisting of 158 annotated sequences. In experiments, 130 recommended sequences were used for training, and the remaining 28 sequences were divided into validation (sequence numbers: 6, 42, 58, 85, 99, 122) and test sets. Segmentation was performed using coarse labels from the dataset annotations for cars, pedestrians, pedestrian groups, bicycles, trucks, and background.
[0042] Set the batch size to 16, read in the training data, and synchronize all data to the coordinate system at time 0. The feature dimension of the point cloud data used is C0=7, including the X-coordinate, Y-coordinate, distance to the vehicle, azimuth angle to the vehicle, compensated speed, radar cross-section, and time. In addition, since each frame of point cloud data is processed to a fixed size, an additional number of actual points N is added. point This information is not input into the network; it is only used to remove duplicate data points when calculating the loss function later.
[0043] Step 102: Construct a neural network using the training data;
[0044] Specifically, such as Figure 2 As shown, the network is constructed in a hierarchical structure, consisting of a feature fusion layer, two levels of sampling feature extraction layers, two levels of feature propagation layers, and a classifier, in the order of connection. Each level of sampling feature extraction layers contains a downsampling module, a neighborhood generation module, and a neighborhood feature extraction module; where, as... Figure 3 As shown, the neighborhood feature extraction module further includes a perception module and a spatiotemporal attention module. Figure 4 As shown, each feature propagation layer contains an interpolation module and a feature post-fusion module.
[0045] A feature fusion layer is constructed. This layer takes the radar observation information of the input point cloud, i.e., the radar feature grouping of the point cloud, and extracts features. Specifically, it divides the seven-dimensional information of the point cloud into two groups, (x, y, t) and (r, a, υ). r ,rcs,t), respectively, use a one-dimensional convolutional network to extract 32-dimensional features, and obtain The two sets of features are then fused together using a one-dimensional convolutional network to obtain 64-dimensional features, resulting in the final fused features. As the output of the feature fusion layer;
[0046] Construct two-level sampling feature extraction layers SA1 and SA2, as follows: Figure 3 As shown, each level of the sampling feature extraction layer constructs a downsampling module (sblock), a neighborhood generation module (gblock), and a neighborhood feature extraction module (ablock). The neighborhood feature extraction module is constructed by combining a spatiotemporal attention submodule (ViT) and a perception submodule (RA). The spatiotemporal attention submodule (ViT) performs attention operations on the neighborhood point cloud and outputs neighborhood features. The perception submodule RA performs convolution operations on the radar features of the neighborhood point cloud and outputs new features. The outputs of the two sub-modules are concatenated and then fused using convolutional operations via MLP. The number of samples in each downsampling module is N1 and N2, respectively; the neighborhood sizes generated by each layer are K1 and K2, respectively. In other words, the input to the two-level cascaded sampling feature extraction layers is the fused feature of the radar features of the point cloud and the output of the feature fusion layer, and the output is the neighborhood feature output by the second-level neighborhood feature extraction module;
[0047] For example, let N1 = 512, and output the radar features of point cloud P1 after downsampling. and fusion features The input is fed into the constructed neighborhood generation module, which generates a neighborhood of 15 points for each point, and the radar features of the neighborhood can be obtained separately. and fusion features Radar features and fused features from the neighborhood are input into the neighborhood feature extraction module. Radar features are input into the perception module, which outputs perception features. The fused features and radar features are fed together into the spatiotemporal attention submodule, which outputs attention features. The two features are concatenated and input into MLP1 to obtain the sampled features of the output point cloud of the first-level sampling feature extraction layer.
[0048] Let N2 = 256, then the radar features of the output point cloud P2 from the first-level sampling feature extraction layer are obtained after the second-level sampling feature extraction layer downsampling module. and sampling to extract features The input is fed into the constructed neighborhood generation module, which generates a neighborhood of 15 points for each point, and the radar features of the neighborhood are obtained from each neighborhood. and sampling to extract features Radar features and sampled features from the neighborhood are input into the neighborhood feature extraction module. Radar features are then input into the perception module, which outputs the perception features. Radar features and sampled features are fed together into the spatiotemporal attention submodule, which outputs attention features. The two features are concatenated and input into MLP2 to obtain the output features of the second-level sampling feature extraction layer. And it serves as the final output of the sampling extraction layer.
[0049] s neighi =gblock i (s i ), i = 1, 2 [1]
[0050] h neighi =gblock i (h i ), i = 1, 2 [2]
[0051]
[0052] Construct two feature propagation layers FP1 and FP2, with each feature propagation layer as follows: Figure 4 As shown. In the interpolation module, the downsampled point cloud P... i i = 0, 1, 2 and the point cloud P before sampling i-1 To perform point-to-point distance metric calculations, for each point in the point cloud before sampling, select the M closest points from the downsampled points, based on the distance d. j For j = 1, 2, ..., m, calculate the reciprocal of the distance and normalize it as the weight feature a of the point. j Let j = 1, 2, ..., m; the features of the M nearest points are multiplied by their respective weights and summed to form the propagated features of that point. These features are then concatenated with the sampled features extracted from the point cloud at the sampling feature extraction layer within the post-feature fusion module, and further fused using MLP to obtain the final propagated features. In other words, the input to the first-level feature propagation layer is the features of P2 extracted by the second-level sampling feature extraction layer and the features of input P1; the output is the propagated P1. The P1 features are then passed to P0 via the second-level feature propagation layer FP2, ultimately outputting the features of P0.
[0053] For example, the first-level feature propagation layer FP1 receives the features of P2 and P1 output by SA2, calculates the distance between points in P2 and points in P1, selects the 15 points with the smallest distance, assigns a weight to each point as the inverse of the normalized distance, and sums the weighted features of each point to obtain the propagation feature of the points in P2. This feature is then mixed with the sampled features of P2 using an MLP (Multi-Level Processing), with each dimension of the MLP being [384, 384]. The second-level feature propagation layer FP2 receives the features of P1 and P0 output by SA1, calculates the distance between points in P1 and points in P0, selects the 15 points with the smallest distance, assigns a weight to each point as the inverse of the normalized distance, and sums the weighted features of each point to obtain the propagation feature of the points in P0. This feature is then mixed with the sampled features of P0 using an MLP (Multi-Level Processing), with each dimension of the MLP being [512, 512]. In other words, the input is the features of P2 output by the sampled feature extraction layer SA2, and the final output is the features of P0.
[0054] X FPi =FP i (X sa(3-i) X sa(2-i) [4]
[0055] A classifier is constructed using a multilayer perceptron (MLP) predictor, and the probabilities are normalized using the softmax activation function. There are six categories (background, car, pedestrian, bicycle), and the probability of X is... FP2 For classification, the dimensions of each layer of the MLP are [256, 64, 6]; that is, the input is the final propagated features of the feature propagation layer, and the output is the predicted probability of each point.
[0056] Step 103: Generate predicted classification labels for the point cloud by forward propagating the point cloud data in a neural network architecture; calculate the loss function on the classification loss function using the real labels and the predicted labels;
[0057] Specifically, the predicted classification labels of the point cloud include real labels and predicted labels.
[0058] Define the total training period E and the training batch size B. n If the total number of training samples is S, then the number of training batches in each period is... The initial learning rate is set to lr0, the learning rate decay step size is stepsize, and the decay rate is decayay; the loss function is set to focal loss, and the prediction loss of the network is calculated using the loss function.
[0059] For example, let the total training cycles be E = 100 and the batch size be B. n =16, the total number of training samples is S=32258, then the number of batches trained in each period is... The initial learning rate is lr0 = 1.0, the decay step size is stepsize = 10, and the decay rate is decay = 0.9. The class corresponding to the maximum predicted probability of each point in the classifier output is taken as its class and written in one-hot encoding form. The loss is calculated by inputting the true label y into the loss function.
[0060]
[0061] Step 104: Backpropagate the parameters of the neural network using the loss function, and iteratively update the parameters using the SGD optimizer to obtain the trained neural network;
[0062] Specifically, the initial weights of all convolutional and linear connection layers in the neural network are set to 0. The mean weights of all batch normalized layers are set to 0, and the variance weights are set to 1. The SGD optimizer is used to optimize all parameters of the network, and the initial learning rate is set to lr0. One batch of training data is read in, input into the network for forward propagation, the loss function is calculated, and backpropagation is performed. It is determined whether the total number of training steps (step) is divisible by the decay step size (stepsize). If it is, the current learning rate is multiplied by the decay rate to obtain the new learning rate; if not, the current learning rate is still used. It is determined whether the total number of training epochs has been reached. If it is, the training ends, and the trained neural network is obtained; otherwise, the training of the neural network continues.
[0063] Step 105: Perform semantic segmentation on the prediction dataset using the trained neural network to determine the category label for each point.
[0064] Specifically, the semantic segmentation result of the point cloud can be obtained by taking the classifier's highest predicted probability for each point as the predicted label for each point.
[0065] This invention introduces segmentation tasks to millimeter-wave radar point clouds, and achieves semantic segmentation of point clouds through supervised training of point cloud segmentation tasks.
[0066] In this invention, the time labels generated for point cloud segmentation are obtained chronologically based on their acquisition time, requiring no additional information. It is entirely based on existing real-world label content, without introducing new label data or increasing the difficulty of obtaining training data. The scheme was ultimately tested on the RadarScenes validation dataset. The F1-scores for background, vehicle, pedestrian, pedestrian group, bicycle, and truck were 100, 82, 56, 84, 80, and 72, respectively, with IoU of 99.4, 69.3, 38.7, 72.6, 67.0, and 56.7. This invention basically meets the daily needs of autonomous driving.
[0067] Experimental data:
[0068] Determine the validation dataset and evaluation metrics;
[0069] This invention uses sequences 5, 14, 19, 24, 31, 48, 53, 63, 68, 73, 79, 89, 93, 107, 111, 130, 135, 138, 147, 148, 153, and 155 from the RadarScenes validation set as test data. The official RadarScenes evaluation criteria, the macro-average F1 score and mIoU, are used as evaluation metrics. The F1 score represents the classification accuracy of a particular class, corresponding to the harmonic mean of precision and recall, comprehensively evaluating the segmentation performance of a single class. A higher F1 score indicates better performance and greater practical value. The macro-average F1 score is the average of the scores for each class. IoU is the intersection-union ratio (IoU) between the predicted and ground truth sets for a particular class, representing the overlap between the predictions and the ground truth for a single class. mIoU is the average IoU for each class.
[0070] The F1-score metric is defined as follows:
[0071]
[0072] Where P is precision and R is recall, TP represents positive samples predicted as positive, FP represents positive samples predicted as negative, and FN represents negative samples predicted as negative, the mathematical definitions of precision and recall are as follows:
[0073]
[0074]
[0075] For each category, its Precision and Recall can be obtained. The F1-score for each category can be obtained using Formula 6.
[0076] The IoU metric is defined as follows:
[0077]
[0078] To illustrate the practicality of this invention, four comparative experiments were conducted. Experiment A used the PointNet++ algorithm for training and segmentation; Experiment B used the PointTransformer algorithm for training and segmentation; Experiment C used the GaussianRadarTransformer algorithm for training and segmentation; and Experiment D used the algorithm described above for training and segmentation. All models were trained and detected using the exact same training hyperparameters and on the same machine. The segmentation objects were vehicles, pedestrians, pedestrian groups, bicycles, and trucks, with six background categories. The final results are shown in Table 1:
[0079] Table 1: F1-score results of the comparative experiment
[0080]
[0081] Table 2: IoU Results of Comparative Experiment
[0082]
[0083] Compared with experimental groups A, B, and C, experimental group D had higher macroF1-score and mIoU scores than experimental groups A, B, and C. The macroF1-score was 5.2079, 3.6251, and 4.7076 higher, respectively, and the mIoU was 5.5728, 4.2633, and 5.6981 higher, respectively. Compared to experimental groups A and D, experimental group D improved its F1-score by 3, 17, 5, and 7 for the four categories of vehicles, pedestrians, pedestrian groups, and trucks, respectively, and improved its IoU by 0.2, 5.4, 14.2, 6.8, and 8.1 for the five categories of background, vehicles, pedestrians, pedestrian groups, and trucks, respectively. Compared to experimental groups B and D, experimental group D improved its F1-score by 1, 5, 10, and 7 for the four categories of background, vehicles, pedestrians, and pedestrian groups, and improved its IoU by 0.7, 6.3, 9.2, 9.5, and 0.8 for the five categories of background, vehicles, pedestrians, pedestrian groups, and bicycles, respectively. Compared with experimental group C, experimental group D improved the F1-score by 1, 4, 10, 11 and 4 in the five categories of background, vehicle, pedestrian, pedestrian group and bicycle, respectively, and improved the IoU by 0.5, 5.1, 8.8, 14.6 and 5.2 in the five categories of background, vehicle, pedestrian, pedestrian group and bicycle, respectively.
[0084] The above results show that the algorithm in experimental group D has better segmentation performance, and its performance advantage is more obvious in the segmentation of categories such as vehicles, pedestrians, and pedestrian groups. The difference in segmentation between bicycles and trucks is no more than 1, therefore the above invention basically meets the segmentation requirements for autonomous driving.
[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.
Claims
1. A millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving, characterized in that, The method is as follows: Preprocess the point cloud data and generate features with a time dimension to obtain training data; A neural network is constructed using the training data; The point cloud data is forward-propagated in a neural network architecture to generate predicted classification labels for the point cloud; the predicted classification labels for the point cloud include real labels and predicted labels. The loss function is determined by comparing the true labels with the predicted labels on the classification loss function. The parameters of the neural network are backpropagated using the loss function, and the parameters are iteratively updated using the SGD optimizer to obtain a trained neural network. The trained neural network is used to perform semantic segmentation on the prediction dataset to determine the category label of each point. The construction of the neural network using the training data specifically involves: constructing the network in a hierarchical structure, with the following connection order: feature fusion layer, two-level sampling feature extraction layer, two-level feature propagation layer, and classifier; each sampling feature extraction layer sequentially includes a downsampling module, a neighborhood generation module, and a neighborhood feature extraction module; wherein the neighborhood feature extraction module includes a perception module and a spatiotemporal attention module, and each feature propagation layer includes an interpolation module and a feature post-fusion module; The two-level sampling extraction feature layer A downsampling module is constructed within each level of the feature extraction feature layer. A neighborhood generation module A neighborhood feature extraction module Construct a neighborhood feature extraction module, which consists of a spatiotemporal attention submodule. and perception submodule Composition, spatiotemporal attention submodule Perform attention operations on the neighborhood point cloud and output neighborhood features. Perception submodule Perform convolution operations on the radar features of the neighborhood point cloud and output new features. The spatiotemporal attention submodule and perception submodule After the output is spliced, it is passed through By fusing features using convolution operations, the number of samples in each downsampling module is respectively The neighborhood sizes generated in each layer are respectively That is, the neighborhood features output by the second-level neighborhood feature extraction module.
2. The millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving according to claim 1, characterized in that, The preprocessing of point cloud data to generate features with a time dimension and obtain training data specifically involves: given a frame timestamp of... Point cloud data As the first frame of data, any point in the point cloud The system has six-dimensional features: position (horizontal and vertical coordinates), distance, azimuth, velocity, and radar cross-section. It continues to read point cloud frames with a time interval of 500ms, using the frame number as the temporal feature. The features of each point are then updated accordingly. Aggregate all frames into one frame A point cloud training set with added time information is obtained. For each frame of point cloud data in the point cloud training set, each frame of point cloud is processed to fix the number of points in each frame to 3072. If the number of points is insufficient, existing points are randomly copied to fill the gaps. If the number of points exceeds the specified number, points of the background category are randomly removed until the number of points is equal to the specified number. The processed point cloud frame is used as the input data of the network, i.e., the training data.
3. The millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving as described in claim 1, characterized in that, The feature fusion layer takes the radar observation information of the input point cloud, that is, extracts features by grouping the point cloud radar features. Specifically, it divides the seven-dimensional information of the point cloud into two groups. and Each 32-dimensional feature was extracted using a one-dimensional convolutional network, resulting in... , The two sets of features are then fused together using a one-dimensional convolutional network to obtain 64-dimensional features, resulting in the final fused features. This serves as the output of the feature fusion layer.
4. The millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving as described in claim 3, characterized in that, The two-level feature propagation layer The interpolation module of each feature propagation layer will downsample the point cloud. Point cloud before sampling To perform point-to-point distance calculations, for each point in the point cloud before sampling, select the closest point among the downsampled points. A number of points, based on distance Calculate the reciprocal of the distance and normalize it as the weight feature of the point. ; The features of the nearest points are multiplied by their respective weights and summed to form the propagated feature of that point. This feature is then connected to the sampled features extracted from the point cloud at the sampling feature extraction layer within the post-feature fusion module. Further integration yields the final propagation characteristics.
5. The millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving as described in claim 4, characterized in that, The classifier employs a multilayer perceptron (MLP) predictor, and the results are normalized using the softmax activation function. There are six categories: background, car, pedestrian, and bicycle. The final propagation features are classified, and the dimensions of each layer of the MLP are as follows: .
6. The millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving as described in any one of claims 1-5, characterized in that, The loss function is determined by inputting the real label data and the predicted label data into the loss function, specifically in each training cycle. The batch size used for training. The total number of training samples is The predicted probabilities of each sample in each category obtained through training are: The real category is Focal loss is used as the final loss function.
7. The millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving as described in claim 6, characterized in that, The process of backpropagating the parameters of the neural network using the loss function and iteratively updating the parameters using the SGD optimizer to obtain a trained neural network specifically involves: optimizing all parameters of the network using the SGD optimizer, and setting the initial learning rate to be... and total number of training sessions Learning rate decay step size The attenuation rate is Read in one batch of training data, input it into the network for forward propagation, calculate the loss function, and then propagate backward. If the number of training steps meets the decay condition, the learning rate is decayed, and training stops after the preset total number of training iterations is reached.
8. The millimeter-wave radar point cloud segmentation method based on ViT for autonomous driving as described in claim 7, characterized in that, The process of semantically segmenting the prediction dataset using a trained neural network to determine the category label for each point involves taking the category with the highest prediction probability as the prediction label, thereby obtaining the semantic segmentation result of the point cloud.