A method for enhancing millimeter wave radar point clouds

Abstract

The application discloses a kind of millimeter wave radar point cloud enhancement methods.The original echo ADC data collected by millimeter wave radar is used as input, and distance-azimuth-pitch 3D heat map is generated through signal preprocessing module;Based on the architecture of generative adversarial network, millimeter wave radar point cloud is supervised using spatiotemporal synchronous laser radar point cloud;Radar features are extracted using 3D-2D asymmetric Unet network structure, then the angle resolution is improved through super-resolution module, and finally the existence probability map and distance map are obtained using existence probability prediction module and distance prediction module respectively, and the millimeter wave radar point cloud is obtained through the screening of existence probability map and the conversion of spatial geometric relationship.The application improves the accuracy and density of the generated point cloud, and realizes millimeter wave radar point cloud enhancement.

Description

Technical Field

[0001] This invention belongs to the field of millimeter-wave radar point cloud generation, specifically a method for enhancing millimeter-wave radar point clouds. Background Technology

[0002] The development of autonomous driving technology is inseparable from the advancement of sensor technology. Currently, common vehicle sensors include cameras, LiDAR, and millimeter-wave radar. Millimeter-wave radar is favored by researchers due to its advantages such as a large effective range, moderate cost, and robustness to weather conditions.

[0003] The quality of point clouds generated by millimeter-wave radar measurements has a decisive impact on subsequent perception tasks such as target detection and tracking. However, existing millimeter-wave radar point cloud generation algorithms are limited by antenna aperture, resulting in poor point cloud quality in real-world road driving scenarios, specifically manifested as sparse point clouds and high noise levels. Increasing the antenna aperture can alleviate these problems to some extent, but the cost is high and there are technical bottlenecks such as I / O performance and computational load. Therefore, software algorithms for millimeter-wave radar point cloud enhancement have attracted extensive research without changing the radar antenna aperture, transmission method, or other hardware requirements.

[0004] Millimeter-wave radar point cloud enhancement methods involve two aspects: constant false alarm rate (CFAR) algorithms and super-resolution algorithms. The former aims to accurately separate target points from noise, mainly addressing issues such as low target signal-to-noise ratio and multipath effects. The latter aims to improve imaging quality, primarily addressing issues such as extended targets and densely packed target scenes.

[0005] The rapid development of deep learning technology has also opened up new avenues for millimeter-wave radar point cloud enhancement technology. Its main idea is to train a defined neural network model through data and labels. Its performance is affected not only by the amount of data but also by the rationality of the network structure. Although this rationality does not have complete interpretability, it still needs to be designed in a targeted manner based on a deep understanding of the characteristics of the input data, so as to ensure the normal convergence of the network and that the output is consistent with the expected results within a certain error range. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a millimeter-wave point cloud enhancement method, the specific technical solution of which is as follows:

[0007] This invention includes the following steps:

[0008] S1: Raw radar echo data was acquired;

[0009] S2: Generate static range-azimuth-elevation 3D heatmaps and dynamic range-azimuth-elevation 3D heatmaps using raw radar echo data;

[0010] S3: Extract radar features using the 3D-2D asymmetric Unet network structure in the generative network, and generate super-resolution radar features through the super-resolution module;

[0011] S4: Input the extracted super-resolution radar features into the existence probability prediction module to generate an existence probability map; input the extracted super-resolution radar features into the distance prediction module to generate a distance map;

[0012] S5: Using the extracted super-resolution radar features as conditions, the generated existence probability map and the generated range map as fake data, and the existence probability label and range label generated by the spatiotemporally synchronized lidar point cloud as real data, a discrimination operation is performed in the discrimination network to obtain the data confidence.

[0013] The beneficial effects of this invention are as follows: This invention uses raw echo ADC data acquired by millimeter-wave radar as input, and generates a range-azimuth-elevation 3D heatmap through a signal preprocessing module; based on a generative adversarial network architecture, it uses spatiotemporally synchronized lidar point clouds to supervise the generation of millimeter-wave radar point clouds; it extracts radar features using a 3D-2D asymmetric Unet network structure, then improves angular resolution through a super-resolution module, and finally uses an existence probability prediction module and a range prediction module to obtain an existence probability map and a range map, respectively. Through filtering of the existence probability map and transformation of spatial geometric relationships, the millimeter-wave radar point cloud is obtained. This invention improves the accuracy and density of the generated point cloud, achieving millimeter-wave radar point cloud enhancement. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0015] Figure 1 This is a schematic diagram of a MIMO millimeter-wave radar transmitting FMCW using TDMA.

[0016] Figure 2 This is a schematic diagram of a millimeter-wave point cloud enhancement method;

[0017] Figure 3 This is a schematic diagram of a real-world test scenario for the method of this invention;

[0018] Figure 4 This is a point cloud effect diagram of the method of the present invention. Detailed Implementation

[0019] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0020] This application discloses a method for enhancing millimeter-wave radar point clouds, including the following steps:

[0021] S1: As Figure 1 As shown, the number of MIMO millimeter-wave radar transmitting antennas in this embodiment is N. TX The number of receiving antennas is N RX The transmitting antenna uses time-division multiplexing (TDMA) to transmit frequency-modulated continuous wave (FMCW); the chirp signal period is T. chirp and N within its period adc ADC sampling; antenna transmission sequence is 1, 2, ..., N TX One cycle is defined as all transmitting antennas sequentially transmitting a chirped signal once, with a period of T. loop N loop Each iteration of the loop constitutes a frame, with a period of T. frame N were finally collected. adc ×N loop ×N RX ×N TX Raw radar echo data;

[0022] S2: Generate static / dynamic range-azimuth-elevation 3D heat maps using raw radar echo data, as detailed below:

[0023] S2.1: Perform rangeFFT, dopplerFFT, TDMA phase compensation, azimuthFFT, and elevationFFT sequentially on the raw radar echo data to obtain the radar data block as shown in the following formula:

[0024] S(i r i d i a i e )∈R#(1)

[0025] Where i r i d i a i e Represents the index subscripts for each dimension, and R represents the real number field;

[0026] S2.2: Distinguish between static target echo data and dynamic target echo data along the doppler dimension;

[0027] S2.3: Extract the radar data block corresponding to zero velocity separately, then perform regularization and position encoding to generate a static range-azimuth-elevation 3D heat map, as shown in the following formula:

[0028]

[0029] Where i r i a i e N represents the index subscript for each dimension. d P represents the number of points set for the doppler-dimensional FFT operation. a ,P e These represent the position codes for azimuth and elevation, respectively. norm() represents regular expression operation, and concat() represents concatenation operation.

[0030] S2.4: Aggregate the radar data blocks corresponding to non-zero velocities, then perform regularization and position encoding to generate a dynamic range-azimuth-elevation 3D heatmap, as shown in the following formula:

[0031]

[0032] Where i r i d i a i e N represents the index subscript for each dimension. d P represents the number of points set for the doppler-dimensional FFT operation. a ,P e These represent the position codes for azimuth and elevation, respectively. norm() represents regular expression operation, and concat() represents concatenation operation.

[0033] S3: Radar features are extracted using a 3D-2D asymmetric Unet network structure, and then super-resolution radar features are generated through a super-resolution module, as detailed below:

[0034] S3.1: Extract 3D features in the distance-azimuth-pitch dimensions using 3D convolution and then downsample them;

[0035] S3.2: Merge the distance dimension into the feature dimension as a azimuth-pitch 2D feature, and use an attention strategy to compress the feature dimension;

[0036] S3.3: Based on the Feature Pyramid (FPN) concept, 2D / 3D features at different scales are aggregated through upsampling layers and blending layers;

[0037] S4: Input the super-resolution radar features extracted in step S3 into the existence probability prediction module to generate an existence probability map, as follows:

[0038] S4.1: The super-resolution radar features are aggregated and dimensionality reduced using a fully connected layer, and then an existence probability map is generated using the sigmoid function, as shown in the following equation:

[0039] M exist (ia i e )∈[0,1]#(4)

[0040] Where i a i e Index subscripts indicating azimuth and elevation dimensions.

[0041] S5: Input the super-resolution radar features extracted in step S3 into the range prediction module to generate a range map, as follows:

[0042] S5.1: The super-resolution radar features are aggregated and dimensionality reduced using a fully connected layer, and a distance center location map is generated using the softmax function;

[0043] S5.2: Use a fully connected layer to aggregate and reduce the dimensionality of super-resolution radar features, and generate a distance center offset map through the sigmoid function;

[0044] S5.3: Merge the distance center location map and the distance center offset map to obtain the distance map as shown in the following formula:

[0045]

[0046] M offset (i a i e )∈[0,1]#(6)

[0047]

[0048] Where i r i a i e M represents the index subscript for distance, azimuth, and pitch dimensions. center For the distance from the center location, M offset This is a distance-to-center offset diagram, where r represents the distance from the grid and w is the offset width.

[0049] S6: Using the super-resolution radar features extracted in step S3 as conditions, the existence probability map generated in step S4 and the range map generated in step S5 as fake data, and the existence probability labels and range labels generated by the spatiotemporally synchronized lidar point cloud as real data, a discrimination operation is performed to obtain the data confidence level. See [link to relevant documentation]. Figure 2 The details are as follows:

[0050] S6.1: Extract features of real / fake data using the Unet network;

[0051] S6.2: Based on the PatchGAN concept, the conditions are concatenated with the features extracted in step S6.1, and then the data confidence score is generated using the feature pyramid structure.

[0052] Furthermore, during the training of the generative adversarial network, using the spatiotemporally synchronized LiDAR point cloud as the ground truth, the existence probability loss function, distance loss function, and generative adversarial loss function are calculated, and the network parameters are updated using gradient descent. Once the network stabilizes and converges, the training ends, as detailed below:

[0053] S7.1: Convert the lidar point cloud into the form shown in the following formula:

[0054]

[0055] Where i a i e The index subscript represents the azimuth and elevation dimensions, Ω indicates that it falls within (i a i e The set of points in the azimuth-elevation grid corresponding to the azimuth;

[0056] S7.2: The probability label and distance label are constructed as shown in the following equation:

[0057] GT exist (i a i e )=(P(i a i e )>0)#(9)

[0058] GT range (i a i e )=P(i a i e )#(10)

[0059]

[0060] Where i r i a i e The index subscripts represent distance, azimuth, and pitch dimensions; R represents the distance grid.

[0061] S7.3: The existence probability loss function is defined based on the cross-entropy function, as shown in the following equation:

[0062]

[0063] Where N represents the total number of azimuth-elevation grids;

[0064] S7.4: The distance loss function, defined based on the cross-entropy function and the L1 loss function, is shown in the following equation:

[0065]

[0066] Where ir i a i e Index subscripts representing distance, azimuth, and elevation dimensions;

[0067] S7.5: The generative adversarial loss function based on the WGAN-GP definition is shown in the following equation:

[0068]

[0069]

[0070] in This represents the discriminant loss term obtained using dummy data. This represents the discrimination loss term obtained using real data, and gradient represents the gradient penalty term. This represents the generated loss term obtained using dummy data;

[0071] Furthermore, in the generative adversarial network inference process, the existence probability map generated in step S4 is first binarized according to the set threshold to generate an existence mask. Then, based on the existence mask, valid values ​​are selected from the distance map generated in step S5, and finally converted into a point cloud through spatial geometric relationships.

[0072] In summary, as Figure 2 As shown, the raw radar echo data is processed by a signal preprocessing module to generate static / dynamic 3D heatmaps, which are used as input to the generator network. In the generator network, a 3D-2D asymmetric Unet network is used to extract radar features. These features are then generated via a super-resolution module, followed by existence probability prediction and range prediction modules to generate existence probability maps and range maps, respectively. Finally, the two are aggregated to obtain the radar point cloud. In the discriminator network, the super-resolution radar features are used as conditions. The existence probability labels and range labels constructed from the lidar point cloud are used as true data, while the existence probability maps and range maps obtained from the generator network are used as false data. The Unet network extracts true / false data features, which, along with the conditional features, are then fed into the confidence prediction module to obtain the patch confidence score.

[0073] Verification example: Figure 3 As shown, this is a main road driving scenario with vehicle targets, used to test the effectiveness of the proposed method. Figure 4As shown in the figure, (a) is the scene lidar point cloud, used to record the position information of scene points and use it as the ground truth; (b) is the target lidar point cloud separated from the scene lidar point cloud, used to compare the imaging effect of the target point cloud; (c) is the millimeter-wave radar point cloud generated based on the traditional method. By comparing with (b), it can be found that although a point cloud is generated at the target, the point cloud is very sparse and lacks descriptive information about the target's shape and size; (d) is the millimeter-wave radar point cloud generated based on the method of the present invention. By comparing with (c), it can be observed that the density of the target point cloud is significantly improved, and the point cloud initially has descriptive information about the target's shape and size, demonstrating the effectiveness of the present invention in enhancing millimeter-wave radar point clouds.

[0074] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for enhancing point clouds in millimeter-wave radar, characterized in that, Includes the following steps: S1: Raw radar echo data was acquired; S2: Generate static range-azimuth-elevation 3D heatmaps and dynamic range-azimuth-elevation 3D heatmaps using raw radar echo data; S3: Extract radar features using the 3D-2D asymmetric Unet network structure in the generative network, and generate super-resolution radar features through the super-resolution module; S4: Input the extracted super-resolution radar features into the existence probability prediction module to generate an existence probability map; input the extracted super-resolution radar features into the distance prediction module to generate a distance map; S5: Using the extracted super-resolution radar features as conditions, the generated existence probability map and the generated range map as fake data, and the existence probability label and range label generated by the spatiotemporally synchronized lidar point cloud as real data, a discrimination operation is performed in the discrimination network to obtain the data confidence.

2. The millimeter-wave radar point cloud enhancement method according to claim 1, characterized in that, During the training of the generative adversarial network, the spatiotemporally synchronized LiDAR point cloud is used as the ground truth. The existence probability loss function, distance loss function, and generative adversarial loss function are calculated, and the network parameters are updated using the gradient descent method. The training ends when the network stabilizes and converges.

3. The millimeter-wave radar point cloud enhancement method according to claim 1, characterized in that: In the inference process of generative adversarial network, the generated existence probability map is first binarized according to the set threshold to generate an existence mask. Then, based on the existence mask, valid values ​​are selected from the generated distance map. Finally, it is transformed into a point cloud through spatial geometric relationships.

4. The millimeter-wave radar point cloud enhancement method according to claim 1, characterized in that, Two types of 3D heatmaps are generated in S2 through the following sub-steps: S2.1: Perform rangeFFT, dopplerFFT, TDMA phase compensation, azimuthFFT, and elevationFFT sequentially on the raw radar echo data to obtain the radar data block as shown in the following formula: S(i r ,i d ,i a ,i e )∈R#(1) Where i r i d i a i e Represents the index subscripts for each dimension, and R represents the real number field; S2.2: Distinguish between static target echo data and dynamic target echo data along the doppler dimension; S2.3: Extract the radar data block corresponding to zero velocity separately, perform regularization and position encoding, and generate a static range-azimuth-elevation 3D heat map, as shown in the following formula: Where i r i a i e N represents the index subscript for each dimension. d P represents the number of points set for the Doppler-dimensional FFT operation. a ,P e These represent the position codes for azimuth and elevation, respectively. norm() represents a regular expression operation, and concat() represents a concatenation operation. S2.4: Aggregate the radar data blocks corresponding to non-zero velocities, perform regularization and position encoding, and generate a dynamic range-azimuth-elevation 3D heatmap, as shown in the following formula: Where i r i d i a i e N represents the index subscript for each dimension. d P represents the number of points set for the Doppler-dimensional FFT operation. a ,P e These represent the position codes for azimuth and elevation, respectively. norm() represents regular expression operation, and concat() represents concatenation operation.

5. The millimeter-wave radar point cloud enhancement method according to claim 1, characterized in that, In step S3, radar features are extracted using a 3D-2D asymmetric Unet network, which is achieved through the following sub-steps: S3.1: Extract 3D features in the distance-azimuth-pitch dimensions using 3D convolution and then downsample them; S3.2: Merge the distance dimension into the feature dimension as a azimuth-pitch 2D feature, and use an attention strategy to compress the feature dimension; S3.3: Based on the feature pyramid, 2D / 3D features at different scales are aggregated through upsampling layers and blending layers.

6. The millimeter-wave radar point cloud enhancement method according to claim 1, characterized in that, The S4 section contains a probability prediction module, which is implemented through the following sub-steps: A fully connected layer is used to aggregate and reduce the dimensionality of super-resolution radar features; an existence probability map is generated using the sigmoid function, as shown in the following equation: M exist (i a ,i e )∈[0,1]#(4) Where i a i e Index subscripts indicating azimuth and elevation dimensions.

7. The millimeter-wave radar point cloud enhancement method according to claim 1, characterized in that, The distance prediction module in S4 is implemented through the following sub-steps: S5.1: Aggregate and reduce the dimensionality of super-resolution radar features using a fully connected layer; generate a distance center location map using the softmax function; S5.2: Aggregate and reduce the dimensionality of super-resolution radar features using a fully connected layer; generate a distance center offset map using the sigmoid function; S5.3: Merge the distance center location map and the distance center offset map to obtain the distance map as shown in the following formula: M offset (i a ,i e )∈[0,1]#(6) Where i r i a i e M represents the index subscript for distance, azimuth, and pitch dimensions. center For the distance from the center location, M offset This is a distance-to-center offset diagram, where R represents the distance from the grid and w is the offset width.

8. The millimeter-wave radar point cloud enhancement method according to claim 1, characterized in that, The generation of data confidence in S5 is achieved through the following sub-steps: S6.1: Extract features of real / fake data using the Unet network; S6.2: Based on PatchGAN, the conditions are concatenated with the features extracted in step S6.1; Data confidence scores are generated using a feature pyramid structure.

9. The millimeter-wave radar point cloud enhancement method according to claim 2, characterized in that, The calculation of the loss function in S7 is achieved through the following sub-steps: S7.1: Convert the lidar point cloud into the form shown in the following formula: Where i a i e The index subscript represents the azimuth and elevation dimensions, Ω indicates that it falls within (i a i e The set of points in the azimuth-elevation grid corresponding to the azimuth; S7.2: The probability label and distance label are constructed as shown in the following equation: GT exist (i a ,i e )=(P(i a ,i e )>0)#(9) GT range (i a ,i e )=P(i a ,i e )#(10) Where i r i a i e The index subscripts represent distance, azimuth, and pitch dimensions; R represents the distance grid. S7.3: The existence probability loss function is defined based on the cross-entropy function, as shown in the following equation: Where N represents the total number of azimuth-elevation grids; S7.4: The distance loss function, defined based on the cross-entropy function and the L1 loss function, is shown in the following equation: Where i r i a i e Index subscripts representing distance, azimuth, and elevation dimensions; S7.5: The generative adversarial loss function based on the WGAN-GP definition is shown in the following equation: in This represents the discriminant loss term obtained using dummy data. This represents the discrimination loss term obtained using real data, and gradient represents the gradient penalty term. This represents the generated loss term obtained using dummy data.

Images