A 4D millimeter wave radar-based point cloud data enhancement method

Abstract

The application discloses a point cloud data enhancement method based on a 4D millimeter wave radar and belongs to the technical field of automatic driving perception. Firstly, a voxel Gaussian encoder is constructed, the density of point cloud in each voxel, the mean and standard deviation of Gaussian distribution in the vertical direction and the maximum reflection intensity are calculated, and sparse radar point cloud is encoded into an aerial view tensor which retains continuous vertical features; then, a spatial offset is dynamically predicted through an offset perception alignment mechanism, and a deformable convolution is used to correct the multipath coordinate drift of radar echo; finally, a conditional diffusion model is trained with the aligned radar features as conditions and laser radar point cloud as a supervision signal, so that reverse reconstruction from pure noise to dense point cloud is realized. While maintaining efficient calculation of two-dimensional convolution, the application systematically solves the problems of vertical information loss and spatial boundary ambiguity, and significantly improves the structure fidelity and global consistency of generated point cloud.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving and cross-modal perception technology, specifically relating to a point cloud data augmentation method based on 4D millimeter-wave radar. Background Technology

[0002] Accurate and robust 3D perception is the cornerstone of autonomous driving systems. While LiDAR can provide dense and accurate geometric representations, its performance degrades significantly in adverse weather conditions such as rain, fog, and snow. In recent years, 4D millimeter-wave radar has gained considerable attention due to its all-weather perception capabilities. However, compared to LiDAR, the original 4D radar point cloud has inherent drawbacks: firstly, the data is extremely sparse, lacking fine-grained geometric topology; secondly, it is highly susceptible to multipath effects and ghost noise, resulting in spatial drift.

[0003] Existing generative methods typically project radar data onto a Bird's-Eye View (BEV) grid for processing. However, traditional BEV projection schemes suffer from two drawbacks: First, vertical geometric information collapse. Forcing sparse 3D radar echoes onto a flat 2D BEV grid inevitably leads to the loss of geometric information in the vertical (Z-axis) direction, making it extremely difficult to reconstruct a continuous 3D shape from 2D priors. Second, spatial misalignment in the XY plane. Due to multipath drift and ghosting in radar, radar point clouds often deviate from the actual geometric boundaries of the target. Directly inputting these drift features into the model results in spatial divergence and blurred boundaries in the generated point cloud. Therefore, there is an urgent need to acquire dense radar point clouds and remove noise through generative methods to achieve point cloud data augmentation. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a point cloud data augmentation method based on 4D millimeter-wave radar. This method solves the problems of vertical height information loss during BEV projection and blurred point cloud boundaries and low accuracy caused by XY plane multipath drift in existing radar-based point cloud generation methods. This invention discloses a point cloud data augmentation method based on 4D millimeter-wave radar, comprising the following steps: Step S1: Acquire time-aligned 4D millimeter-wave radar point cloud data and lidar point cloud data; Step S2: Construct a voxel Gaussian encoder to encode the 4D millimeter-wave radar point cloud data and lidar point cloud data into bird's-eye view network tensors respectively; The encoding method is as follows: calculate the density of the point cloud within each voxel, the mean of the Gaussian distribution along the vertical direction, the standard deviation of the Gaussian distribution along the vertical direction, and the maximum reflection intensity to obtain the four-dimensional point cloud parameters, which are used as the tensor of the bird's-eye view network. Step S3: Correct spatial drift through offset-aware alignment mechanism to obtain aligned radar feature tensors; The offset-aware alignment mechanism is as follows: dynamically predict the two-dimensional spatial offset of the point cloud and pull the drifting radar echoes in the bird's-eye view network tensor back to the real physical boundary. Step S4: Using the aligned radar feature tensor as a condition and the lidar bird's-eye view network tensor as a supervision signal, train the conditional diffusion model so that it learns the inverse denoising trajectory from pure noise to dense point cloud, and obtain the trained conditional diffusion model. Step S5: The 4D millimeter-wave radar point cloud acquired in real time is processed by steps S2 and S3 and then input into the trained conditional diffusion model to guide the initial Gaussian noise to perform inverse denoising and reconstruct a dense 3D point cloud.

[0005] Optionally, step S2 specifically includes: Step S2-1: The three-dimensional spatial grid of the point cloud distributed by the millimeter-wave radar and lidar is converted into two-dimensional bird's-eye view grid cells, with each grid cell representing a voxel in the vertical direction. Step S2-2: Divide the point cloud data into each voxel; Step S2-3: Calculate the density of the point cloud within each voxel, the mean of the Gaussian distribution along the vertical direction, the standard deviation of the Gaussian distribution along the vertical direction, and the maximum reflection intensity to obtain the four-dimensional point cloud parameters. Step S2-4: Encode the four-dimensional point cloud parameters of the millimeter-wave radar and lidar into millimeter-wave radar bird's-eye view network tensors and lidar bird's-eye view network tensors, respectively.

[0006] Optionally, step S3 specifically includes: Step S3-1: Construct the offset prediction module, including a convolutional neural network and a deformable convolutional network; Step S3-2: Input the tensor of the millimeter-wave radar bird's-eye view network into the convolutional neural network and output a two-dimensional spatial offset tensor, which represents the offset of the convolution kernel of the deformable convolutional network in the plane. Step S3-3: Input the tensor of the millimeter-wave radar bird's-eye view network into the deformable convolutional network. The deformable convolutional network performs deformable convolution operation on the tensor of the millimeter-wave radar bird's-eye view network according to the two-dimensional spatial offset tensor, and outputs the aligned tensor.

[0007] Optionally, step S4 specifically includes: Step S4-1: Add random Gaussian noise multiple times to the tensor of the LiDAR bird's-eye view network to obtain a noisy tensor; Step S4-2: Concatenate the aligned radar feature tensor with the noise tensor in the channel dimension, and input it into the core network along with the number of noise additions to output the prediction tensor. Step S4-3: Calculate the loss between the prediction tensor and the bird's-eye view network tensor, optimize the network parameters through backpropagation, and enable the model to learn the backpropagation trajectory.

[0008] Optionally, step S5 specifically includes: Step S5-1: Randomly generate Gaussian noise in the target space; Step S5-2: The real-time alignment features obtained after processing the real-time millimeter-wave point cloud in steps S2 and S3 are concatenated with the Gaussian noise in the channel dimension, and input together with the total number of denoising steps into the core network of the trained conditional diffusion model. The reverse denoising is performed iteratively multiple times to obtain the final predicted tensor, which contains four-dimensional predicted point cloud parameters. Step S5-3: During decoding, invalid background rasters are filtered out based on the predicted density, and truncated Gaussian sampling is performed on the valid rasters based on the predicted vertical mean and the predicted standard deviation to restore the vertical coordinates, and the two-dimensional feature tensor is reconstructed into a dense three-dimensional point cloud.

[0009] Optionally, in step S5-3, horizontal coordinates are uniformly sampled with the grid center as the reference, and truncated Gaussian sampling is performed based on the predicted vertical mean and the predicted standard deviation to recover the vertical coordinates.

[0010] Optionally, in step S1, the 4D millimeter-wave radar point cloud data and lidar point cloud data are unified into the vehicle coordinate system and time-aligned.

[0011] Another aspect of the present invention discloses a point cloud data augmentation system based on 4D millimeter-wave radar for performing the aforementioned method, comprising: The input and encoding module is used to convert the original 4D millimeter-wave radar point cloud and lidar point cloud into a bird's-eye view feature tensor through voxel Gaussian encoding. The offset-aware deformation alignment module includes a convolutional neural network, a deformable convolutional network, and a two-dimensional spatial offset tensor. The convolutional neural network is used to extract initial features and predict the two-dimensional spatial offset tensor of the grid. The deformable convolutional network is combined with the two-dimensional spatial offset tensor to actively correct spatial misalignment in the plane. The diffusion denoising module is used to guide the core network to learn the inverse diffusion trajectory of the point cloud distribution during the training phase with radar alignment features as a condition, and to iteratively denoise the pure Gaussian noise distribution to generate a dense point cloud during the inference phase. The point cloud sampling and reconstruction module is used to reconstruct two-dimensional feature tensors into real-time dense three-dimensional point clouds by combining local density guidance mechanism and voxel Gaussian coding strategy during the inference generation stage.

[0012] Compared with the prior art, the present invention has at least the following beneficial effects: 1. This invention proposes a novel BEV encoding paradigm—voxel Gaussian encoding. This method overcomes the bottleneck of geometric truncation caused by traditional point cloud projection into BEV space. By performing Gaussian statistical modeling on the point cloud within the voxel, continuous vertical feature representation is achieved, effectively overcoming the problem of vertical geometric information collapse caused by traditional BEV projection.

[0013] 2. This invention introduces an offset-aware deformation alignment mechanism. This mechanism breaks through the limitation of existing methods that passively accept misaligned features. By dynamically predicting spatial offsets, it systematically corrects the inherent multipath coordinate drift and grid quantization errors of radar, significantly improving the structural fidelity of the generated point cloud in the XY plane.

[0014] 3. While improving the structural integrity and global distribution consistency of the generated dense 3D radar point cloud, this invention still maintains the high-efficiency computational advantage of 2D convolution and avoids the huge computational cost of 3D networks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 The point cloud data enhancement method based on 4D millimeter-wave radar of the present invention enhances millimeter-wave point cloud data. Figure 2 The present invention provides a millimeter-wave radar point cloud data augmentation architecture diagram based on the 4D millimeter-wave radar point cloud data augmentation method. Figure 3 A schematic diagram of Gaussian voxel encoding of millimeter-wave and lidar point clouds using the point cloud data augmentation method based on 4D millimeter-wave radar of the present invention; Figure 4 A schematic diagram of the denoising process during the inference stage in the point cloud data augmentation method based on 4D millimeter-wave radar of the present invention. Detailed Implementation

[0017] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0018] A specific embodiment of the present invention, such as Figures 1-4 As shown, a point cloud data augmentation method based on 4D millimeter-wave radar is disclosed, which specifically includes the following steps: S1 simultaneously acquires 4D millimeter-wave radar point cloud data and lidar point cloud data, unifies the coordinates of the two point clouds into the vehicle coordinate system, and performs time alignment.

[0019] S2: Construct a voxel Gaussian encoder to extract features from time-aligned 4D millimeter-wave radar point cloud data and lidar point cloud data to obtain the BEV network tensor. This invention uses voxel Gaussian encoding to encode the millimeter-wave and lidar point clouds within each voxel to obtain the BEV network tensor. The encoded information includes the density of point clouds within voxels. Mean of Gaussian distribution along the Z-axis Standard deviation and maximum reflection intensity 4D point cloud parameters are obtained. By using continuous vertical statistical Gaussian modeling within local spatial regions, BEV prior features that retain fine-grained vertical structures are extracted, avoiding the high computational cost of 3D convolution while preserving Z-axis information, such as... Figure 4 As shown.

[0020] The specific process is as follows: S2-1: The 3D spatial grid of the 4D millimeter-wave radar point cloud data and lidar point cloud data is rasterized into 2D BEV grid cells, with each grid cell representing a voxel in the vertical direction.

[0021] S2-2: Divide the 4D millimeter-wave radar point cloud data and lidar point cloud data acquired in S1 into each voxel.

[0022] S2-3: Calculate the density of the point cloud within each voxel. Mean of Gaussian distribution along the Z-axis and standard deviation and maximum reflection intensity I 4D point cloud parameters are obtained. .

[0023] S2-4: Transfer 4D point cloud parameters Encoded as BEV network tensor ( ), ,in, This represents the BEV tensor after encoding the millimeter-wave radar point cloud. This represents the BEV tensor of the LiDAR point cloud after encoding, where H is the height of the BEV grid and W is the width of the BEV grid. It is a real number.

[0024] S3: Spatial drift is corrected through an offset-sensing alignment mechanism to obtain aligned radar features; The offset-aware alignment mechanism constructed in this invention can dynamically predict the offset of point clouds. In the reverse denoising process, radar physical information is actively searched and extracted, and the millimeter-wave radar point cloud output in step S2 is encoded into a BEV tensor. The drifting radar echoes are pulled back to the real physical boundaries.

[0025] The specific process is as follows: S3-1: Construct an offset prediction module, which consists of a convolutional neural network (CNN) and a deformable convolutional network (DCN).

[0026] S3-2: Convert the millimeter-wave radar BEV tensor generated in step S2. Input into a convolutional neural network (CNN), output a two-dimensional spatial offset tensor. ,in, The two-dimensional XY spatial offset of the deformable convolution (DCN) kernel. , This indicates that after a deformable convolution kernel performs a convolution operation... shaft and The offset distance of the axis.

[0027] S3-3: Convert the millimeter-wave radar BEV tensor generated in step S2 Input a deformable convolutional network (DCN), the DCN network is based on the two-dimensional spatial offset tensor output by S3-2 BEV tensor for millimeter-wave radar Perform deformable convolution operations and output aligned tensors. .

[0028] S4: The tensor aligned after S3 As a condition, the real dense LiDAR point cloud BEV tensor is introduced into the BEV network tensor. As a supervisory signal, the conditional diffusion model is trained to learn the inverse denoising trajectory from pure noise to dense point cloud.

[0029] The specific processing procedure is as follows: S4-1 Forward Diffusion: Encoded BEV Tensor of the Real Dense LiDAR Point Cloud in S2-4 The noise-inducing tensor is obtained by adding t random Gaussian noises. , t is The inner random integer, where T is the total number of denoising steps; S4-2 Reverse Learning: Aligning the tensor output from step S3 and The concatenation is performed along the channel dimension and input along with step t into the core network U-Net of the diffusion model, outputting... , ; S4-3 Loss Optimization: Calculate the loss obtained in S4-2 and The loss between them is used to optimize the network parameters in all conditional diffusion models using backpropagation, so that the model learns the backdiffusion trajectory from noise to dense point cloud distribution.

[0030] S5: The reasoning generation stage of the conditional diffusion model, which is the actual deployment and reasoning stage after the conditional diffusion model has been trained.

[0031] This invention deploys and uses a trained conditional diffusion model, inputs real-time radar alignment features into the model, guides the initial Gaussian noise to perform inverse denoising, and finally reconstructs and generates a dense 3D point cloud with fine-grained geometric topology.

[0032] The specific processing procedure is as follows: S5-1 Initial Noise Construction: In a given target space Randomly generate pure Gaussian noise inside ; S5-2 Conditional Guided Denoising: Encodes real-time acquired millimeter-wave point clouds into real-time millimeter-wave radar BEV tensors. The real-time radar alignment features are obtained by inputting the deformable convolutional network (DCN) trained in stage S4. As a conditional guide, the initial noise constructed with S5-1 The concatenation is performed along the channel dimension and input into the core network U-Net of the conditional diffusion model along with the total number of denoising steps T, resulting in a tensor that has undergone the first denoising step. Next, the tensor after the first denoising... Tensor aligned with S3 The concatenation is performed, and the result is input into the U-Net along with the current denoising step T-1, yielding a tensor that has undergone two denoising steps. This process is repeated T times to obtain the final predicted tensor. , The final tensor of the prediction obtained by S5-2 Includes 4D predicted point cloud parameters .

[0033] S5-3 Probabilistic Sampling and Reconstruction: During decoding, firstly based on the predicted density... Invalid background rasters are filtered out; then, for valid rasters, horizontal coordinates are uniformly sampled with the raster center as the reference, and the vertical mean is calculated based on the predicted value. Standard deviation of the forecast The truncated Gaussian sampling is performed to recover the vertical coordinates; finally, the two-dimensional feature tensor is directly decoded and reconstructed into an enhanced real-time dense 3D point cloud.

[0034] Another aspect of the present invention discloses a point cloud data augmentation system based on 4D millimeter-wave radar, such as... Figure 2 As shown, it includes an input and encoding module, an offset-aware deformation alignment module, a diffusion denoising module, and a point cloud sampling and reconstruction module.

[0035] (1) Input and encoding module, used to convert the original sparse 4D millimeter-wave radar point cloud and the dense lidar point cloud, which serves as the ground truth of the target, into BEV feature tensors through voxel Gaussian encoding. .

[0036] (2) Offset-aware deformation alignment module: This module consists of a basic convolutional neural network (CNN), a deformable convolutional network (DCN), and a two-dimensional spatial offset tensor. Composition. The basic convolutional neural network (CNN) is used to extract... Two-dimensional spatial offset tensor of initial features and predicted raster The DCN and This is combined with a feature-level approach to actively correct spatial misalignment caused by multipath effects in the XY plane.

[0037] (3) Diffusion denoising module: During the training phase, radar-aligned features are used for denoising. As a conditional guide, the U-Net network learns the backward diffusion trajectory of the point cloud distribution by minimizing the loss function between the predicted target and the true target value.

[0038] (4) Point cloud sampling and reconstruction module: During the inference generation stage, the model starts from a pure Gaussian noise distribution and undergoes iterative denoising under the conditional constraints of radar alignment features. At the same time, combined with a local density guidance mechanism and a voxel Gaussian coding strategy, it is reconstructed into a real-time dense lidar-like 3D point cloud.

[0039] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0040] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A point cloud data augmentation method based on 4D millimeter-wave radar, characterized in that, Includes the following steps: Step S1: Acquire time-aligned 4D millimeter-wave radar point cloud data and lidar point cloud data; Step S2: Construct a voxel Gaussian encoder to encode the 4D millimeter-wave radar point cloud data and lidar point cloud data into bird's-eye view network tensors respectively; The encoding method is as follows: calculate the density of the point cloud within each voxel, the mean of the Gaussian distribution along the vertical direction, the standard deviation of the Gaussian distribution along the vertical direction, and the maximum reflection intensity to obtain the four-dimensional point cloud parameters, which are used as the tensor of the bird's-eye view network. Step S3: Correct spatial drift through offset-aware alignment mechanism to obtain aligned radar feature tensors; The offset-aware alignment mechanism is as follows: dynamically predict the two-dimensional spatial offset of the point cloud and pull the drifting radar echoes in the bird's-eye view network tensor back to the real physical boundary. Step S4: Using the aligned radar feature tensor as a condition and the lidar bird's-eye view network tensor as a supervision signal, train the conditional diffusion model so that it learns the inverse denoising trajectory from pure noise to dense point cloud, and obtain the trained conditional diffusion model. Step S5: The 4D millimeter-wave radar point cloud acquired in real time is processed by steps S2 and S3 and then input into the trained conditional diffusion model to guide the initial Gaussian noise to perform inverse denoising and reconstruct a dense 3D point cloud.

2. The method according to claim 1, characterized in that, Step S2 specifically includes: Step S2-1: The three-dimensional spatial grid of the point cloud distributed by the millimeter-wave radar and lidar is converted into two-dimensional bird's-eye view grid cells, with each grid cell representing a voxel in the vertical direction. Step S2-2: Divide the point cloud data into each voxel; Step S2-3: Calculate the density of the point cloud within each voxel, the mean of the Gaussian distribution along the vertical direction, the standard deviation of the Gaussian distribution along the vertical direction, and the maximum reflection intensity to obtain the four-dimensional point cloud parameters. Step S2-4: Encode the four-dimensional point cloud parameters of the millimeter-wave radar and lidar into millimeter-wave radar bird's-eye view network tensors and lidar bird's-eye view network tensors, respectively.

3. The method according to claim 1, characterized in that, Step S3 specifically includes: Step S3-1: Construct the offset prediction module, including a convolutional neural network and a deformable convolutional network; Step S3-2: Input the tensor of the millimeter-wave radar bird's-eye view network into the convolutional neural network and output a two-dimensional spatial offset tensor, which represents the offset of the convolution kernel of the deformable convolutional network in the plane. Step S3-3: Input the tensor of the millimeter-wave radar bird's-eye view network into the deformable convolutional network. The deformable convolutional network performs deformable convolution operation on the tensor of the millimeter-wave radar bird's-eye view network according to the two-dimensional spatial offset tensor, and outputs the aligned tensor.

4. The method according to claim 1, characterized in that, Step S4 specifically includes: Step S4-1: Add random Gaussian noise multiple times to the tensor of the LiDAR bird's-eye view network to obtain a noisy tensor; Step S4-2: Concatenate the aligned radar feature tensor with the noise tensor in the channel dimension, and input it into the core network along with the number of noise additions to output the prediction tensor. Step S4-3: Calculate the loss between the prediction tensor and the bird's-eye view network tensor, optimize the network parameters through backpropagation, and enable the model to learn the backpropagation trajectory.

5. The method according to claim 1, characterized in that, Step S5 specifically includes: Step S5-1: Randomly generate Gaussian noise in the target space; Step S5-2: The real-time alignment features obtained after processing the real-time millimeter-wave point cloud in steps S2 and S3 are concatenated with the Gaussian noise in the channel dimension, and input together with the total number of denoising steps into the core network of the trained conditional diffusion model. The reverse denoising is performed iteratively multiple times to obtain the final predicted tensor, which contains four-dimensional predicted point cloud parameters. Step S5-3: During decoding, invalid background rasters are filtered out based on the predicted density, and truncated Gaussian sampling is performed on the valid rasters based on the predicted vertical mean and the predicted standard deviation to restore the vertical coordinates, and the two-dimensional feature tensor is reconstructed into a dense three-dimensional point cloud.

6. The method according to claim 5, characterized in that, In step S5-3, horizontal coordinates are uniformly sampled with the grid center as the reference, and truncated Gaussian sampling is performed based on the predicted vertical mean and the predicted standard deviation to recover the vertical coordinates.

7. The method according to claim 1, characterized in that, In step S1, the 4D millimeter-wave radar point cloud data and lidar point cloud data are unified into the vehicle coordinate system and time-aligned.

8. A point cloud data augmentation system based on 4D millimeter-wave radar, used to execute the method according to any one of claims 1 to 7, characterized in that, include: The input and encoding module is used to convert the original 4D millimeter-wave radar point cloud and lidar point cloud into a bird's-eye view feature tensor through voxel Gaussian encoding. The offset-aware deformation alignment module includes a convolutional neural network, a deformable convolutional network, and a two-dimensional spatial offset tensor. The convolutional neural network is used to extract initial features and predict the two-dimensional spatial offset tensor of the grid. The deformable convolutional network is combined with the two-dimensional spatial offset tensor to actively correct spatial misalignment in the plane. The diffusion denoising module is used to guide the core network to learn the inverse diffusion trajectory of the point cloud distribution during the training phase with radar alignment features as a condition, and to iteratively denoise the pure Gaussian noise distribution to generate a dense point cloud during the inference phase. The point cloud sampling and reconstruction module is used to reconstruct two-dimensional feature tensors into real-time dense three-dimensional point clouds by combining local density guidance mechanism and voxel Gaussian coding strategy during the inference generation stage.

Images