A method for generating millimeter-wave radar point cloud based on diffusion model

Abstract

The present invention discloses a method for generating millimeter-wave radar point clouds based on a diffusion model, including: obtaining spatially and temporally aligned lidar point cloud data and millimeter-wave radar data; compressing the lidar point cloud data into two dimensions, performing a two-dimensional fast Fourier transform on the millimeter-wave radar data to obtain compressed point cloud data and a millimeter-wave range-azimuth heat map with the same dimension and concatenating them to obtain concatenated data; based on a Markov chain, performing forward diffusion on the concatenated data; for the concatenated data after forward diffusion, training an inverse diffusion network to predict the transition kernel of the reverse Markov chain to learn the mapping relationship from the millimeter-wave radar domain to the lidar domain; in the deployment stage, only need to obtain millimeter-wave radar data, preprocess it and input it into the trained inverse diffusion network, so as to generate lidar-like point clouds to achieve high-quality millimeter-wave radar point cloud generation.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous navigation of unmanned systems, and particularly relates to a method for generating millimeter-wave radar point clouds based on a diffusion model. Background Art

[0002] In harsh environments such as rain, fog, sand and dust, and smoke, mainstream optical sensors such as lidar and binocular cameras are prone to degradation or even failure, and it is impossible to achieve accurate environmental perception and autonomous navigation in these environments. Compared with the two, millimeter-wave radar is robust to most extreme weather environments due to its longer wavelength and is not affected by fine particles.

[0003] The radar system commonly uses a Constant False Alarm Rate (CFAR) detector to achieve target detection. The algorithm first sets guard cells around the cell under test to eliminate the energy leakage of the signal of the cell under test in the surrounding area, and then selects reference cells within a certain range outside the guard cells for power estimation to determine the threshold. To improve the detection performance or reduce the computational cost, there are also improved algorithms such as (CA)-CFAR and (OS)-CFAR.

[0004] However, when applied to scenarios with cluttered and dense environments such as autonomous driving and robot autonomous navigation, the background noise and interference levels are significantly increased and difficult to model, and the above algorithms are difficult to apply. To meet the needs of unmanned system autonomous navigation, the academic community has proposed many solutions based on deep learning algorithms for tasks such as target detection, odometry, and map construction. Among them, some methods also focus on generating millimeter-wave radar point clouds. These methods all use lidar-millimeter-wave radar data that has been time-synchronized and externally calibrated. For example, the RadarHD method is based on the U-Net network framework, uses the distance-horizontal azimuth heat map of the millimeter-wave radar as the network input, and through the supervision of cross-entropy loss and Dice loss, the network generates two-dimensional lidar-like point clouds based on the millimeter-wave radar data; the RPDNet algorithm uses the distance-velocity heat map of the millimeter-wave radar as the network input and conducts adversarial learning based on the GAN network to improve the generation performance; the DREAM-PCD algorithm combines traditional radar signal processing and deep learning algorithms. First, non-coherent accumulation and synthetic aperture accumulation are performed on multiple frames of millimeter-wave radar point clouds to improve the signal-to-noise ratio and angular resolution of the original signal respectively. Based on the DGCNN network, supervised learning is performed on the enhanced millimeter-wave radar point clouds to obtain high-resolution millimeter-wave radar point clouds.

[0005] Existing millimeter-wave radar point cloud generation technologies include Multiple Signal Classification (MUSIC), CFAR and their improved algorithms based on traditional data processing algorithms; and RadarHD, RPDNe based on deep learning ] , and solutions such as DREAM-PCD.

[0006] Compared with traditional point cloud generation methods based on digital signal processing means, existing algorithmic solutions based on deep learning can generate relatively dense point clouds. Given paired lidar point clouds, through per-pixel loss functions or adversarial learning for gradient descent, a deep neural network can learn to denoise and upsample millimeter-wave radar signals to obtain lidar-like point clouds. However, the network frameworks adopted by the above methods do not have sufficient generation capabilities, thus unable to accurately restore the point cloud structure information, and the generated millimeter-wave radar point clouds lack obstacle edge features, still far inferior to lidar point clouds in terms of accuracy and density. The millimeter-wave radar point clouds obtained by network inference are superior to those obtained by the traditional methods shown in the above figure, can depict the obstacle contours, but lose many details, lack edge features, are relatively blurred, and are still difficult to apply to complex scenarios.

[0007] In addition, there are methods such as "Gao X, Roy S, Xing G. MIMO-SAR: A hierarchical high-resolution imaging algorithm for mmWave FMCW radar in autonomous driving[J]. IEEE Transactions on Vehicular Technology, 2021, 70(8): 7322-7334." that draw on Synthetic Aperture Radar (SAR) technology in radar systems to coherently process temporally continuous multi-frame sensor data on the premise that the sensor poses are known, achieving the effect of increasing the radar aperture and thereby improving the resolution. The DREAM-PCD method also utilizes SAR technology during data preprocessing. However, this technology requires the sensor poses to be known, and the imaging effect is very sensitive to pose accuracy. In the harsh environment autonomous navigation scenario concerned by the present invention, often only millimeter-wave radar can provide reliable sensing information, unable to meet the condition of known high-precision poses. Therefore, millimeter-wave radar imaging algorithms based on SAR technology are difficult to apply to autonomous navigation in harsh environments of unmanned systems. Summary of the Invention

[0008] In view of the problems existing in the prior art, the purpose of the embodiments of the present application is to provide a method for generating millimeter-wave radar point clouds based on a diffusion model.

[0009] According to the first aspect of the embodiments of the present application, a method for generating millimeter-wave radar point clouds based on a diffusion model is provided, including:

[0010] Obtain spatio-temporally aligned lidar point cloud data and millimeter-wave radar data;

[0011] Compress the lidar point cloud data into two dimensions, perform a two-dimensional fast Fourier transform on the millimeter-wave radar data, obtain compressed point cloud data and a millimeter-wave range-azimuth heat map with the same dimensions, and concatenate them to obtain concatenated data;

[0012] Based on the Markov chain, perform forward diffusion on the concatenated data;

[0013] For the concatenated data after forward diffusion, train an inverse diffusion network to predict the transition kernel of the reverse Markov chain to learn the mapping relationship from the millimeter-wave radar domain to the lidar domain;

[0014] Obtain the millimeter-wave radar data to be processed, and after preprocessing, input it into the trained inverse diffusion network to generate lidar-like point cloud data.

[0015] Further, for a point cloud composed of N points , the forward diffusion process is modeled as a Markov chain:

[0016]

[0017] where That is, the Markov diffusion kernel, which adds noise to the points at the previous time step and models the distribution of the points at the next time step, is defined as:

[0018]

[0019] where is the variance scheduling hyperparameter that controls the diffusion rate of the process, T is the total number of noise addition steps, is the normal distribution, is the same size as identity matrix.

[0020] Further, the optimization objective during the training of the inverse diffusion network is

[0021]

[0022] where, is the true value data distribution estimated by the neural network, is the normal distribution.

[0023] Furthermore, the backbone of the inverse diffusion network is a U-Net structure, including a downsampling module and an upsampling module. The downsampling module extracts the network input as corresponding feature maps through convolutional layers and pooling layers, where a multi-head attention mechanism is applied between the convolutional layers. The upsampling module continuously upsamples to convert the feature maps into network outputs of the same size as the network input.

[0024] Furthermore, a consistency model is used to optimize the trained inverse diffusion network, and the consistency model is used to generate lidar point cloud data corresponding to the millimeter-wave radar data to be processed.

[0025] Furthermore, the consistency model training function As a probability flow ordinary differential equation solver, it enables the noisy samples to generate initial samples in a single step from time t. The training process depends on this solver and the inverse diffusion network to generate adjacent point pairs on the trajectory of the probability flow ordinary differential equation. By minimizing the difference between the point pairs and the output of the inverse diffusion network, the features of the trained inverse diffusion network are extracted into a single-step sampler, thereby distilling the inverse diffusion network into a consistency model.

[0026] According to the second aspect of the embodiments of the present application, a millimeter-wave radar point cloud generation device based on a diffusion model is provided, including:

[0027] An acquisition module, configured to acquire spatio-temporally aligned lidar point cloud data and millimeter-wave radar data;

[0028] A compression module, configured to compress the lidar point cloud data into two dimensions, perform a two-dimensional fast Fourier transform on the millimeter-wave radar data, obtain compressed point cloud data and a millimeter-wave range-azimuth heat map with the same dimension and concatenate them to obtain concatenated data;

[0029] A forward diffusion module, configured to perform forward diffusion on the concatenated data based on a Markov chain;

[0030] A training module, configured to train the inverse diffusion network to predict the transition kernel of the reverse Markov chain for the concatenated data after forward diffusion, so as to learn the mapping relationship from the millimeter-wave radar domain to the lidar domain;

[0031] A point cloud generation module, configured to acquire millimeter-wave radar data to be processed, preprocess it and input it into the trained inverse diffusion network, thereby generating lidar-like point cloud data.

[0032] According to the third aspect of the embodiments of the present application, a computer program product is provided, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the method described in the first aspect is implemented.

[0033] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including:

[0034] One or more processors;

[0035] A memory for storing one or more programs;

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0037] According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0038] The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

[0039] As can be seen from the above embodiments, through the improvement of the lidar-supervised millimeter-wave radar point cloud generation technology, the generation of millimeter-wave radar point clouds based on lidar supervision is essentially both a denoising task and a super-resolution task. Therefore, the present application combines the diffusion model algorithm and utilizes its advantages of strong representation ability and stable training to generate high-precision dense millimeter-wave radar point clouds. Specifically, the diffusion model technology is combined with the millimeter-wave radar sensing technology, and the millimeter-wave radar data is used as the network control condition for point cloud generation, so that the diffusion model can recover high-precision and dense point clouds rich in obstacle edge information and structural features from the noisy point clouds, achieving extremely low loss and extremely high consistency in comparison with the ground truth collected by the lidar. The generated point clouds can be directly used for tasks such as map construction.

[0040] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0042] Figure 1 is a flowchart of a method for generating millimeter-wave radar point clouds based on a diffusion model shown according to an exemplary embodiment.

[0043] Figure 2 is a schematic diagram of the training process of a diffusion model conditional on millimeter-wave radar data shown according to an exemplary embodiment.

[0044] Figure 3 is a comparison diagram of the millimeter-wave radar point cloud generation results shown according to an exemplary embodiment, whereFigure 3 In (a) is the RAM (i.e., the network input), Figure 3 in (b) is the lidar point cloud ground truth, Figure 3 in (c) is the point cloud generated by RedarHD, Figure 3 in (d) is the point cloud generated by this method.

[0045] Figure 4 is a comparison diagram of the accelerated inference effect shown according to an exemplary embodiment, where Figure 4 in (a) is the lidar point cloud ground truth, Figure 4 in (b) is the lidar-like point cloud obtained by the inverse diffusion network in this method after 80 steps of inference, Figure 4 in (c) is the lidar-like point cloud obtained by the consistency model in this method after 2 steps of inference.

[0046] Figure 5 is a block diagram of a millimeter-wave radar point cloud generation device based on a diffusion model shown according to an exemplary embodiment.

[0047] Figure 6 is a schematic diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0048] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application.

[0049] The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms of "a", "the" and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0051] Figure 1It is a flowchart of a millimeter-wave radar point cloud generation method based on a diffusion model shown according to an exemplary embodiment. As Figure 1 shown, the method may include the following steps:

[0052] S1: Obtain spatially and temporally aligned lidar point cloud data and millimeter-wave radar raw data;

[0053] S2: Compress the lidar point cloud data to two dimensions, perform a two-dimensional fast Fourier transform on the millimeter-wave radar data to obtain compressed point cloud data and a millimeter-wave range-azimuth heat map with the same dimension, and concatenate them to obtain concatenated data;

[0054] S3: Based on the Markov chain, perform forward diffusion on the concatenated data;

[0055] S4: For the concatenated data after forward diffusion, train an inverse diffusion network to predict the transition kernel of the inverse Markov chain to learn the mapping relationship from the millimeter-wave radar domain to the lidar domain;

[0056] S5: Obtain millimeter-wave radar data to be processed, preprocess it, and input it into the trained inverse diffusion network to generate lidar-like point cloud data.

[0057] This method addresses the inherent limitations in deep learning-based point cloud generation schemes such as variational auto-encoders (VAE), generative adversarial networks (GAN), and Normalizing Flows, such as the unstable training process in GAN. It uses a probability generation model inspired by non-equilibrium thermodynamics to generate point clouds, characterizes the point cloud generation task as an inverse diffusion process of transforming a particle noise distribution into a desired shape distribution, and innovatively proposes a denoising diffusion probability generation model that is conditional on millimeter-wave radar data and can generate high-precision point clouds. It can process and make full use of the input millimeter-wave radar data, recover the obstacle information two-dimensional point cloud from the randomly sampled noise point cloud under the consistent perspective of the millimeter-wave radar, and the denoising process of the recovered point cloud is realized by a neural network. This method greatly improves the point cloud generation effect and can be widely applied in fields such as Simultaneous Localization and Mapping (SLAM), mobile robot navigation, and autonomous driving.

[0058] In the specific implementation of step S1, obtain spatially and temporally aligned lidar point cloud data and millimeter-wave radar data;

[0059] Specifically, during the data acquisition process of the millimeter-wave radar, only the radar array in the horizontal direction is enabled, and it is necessary to ensure that the millimeter-wave radar data and the lidar point cloud data are at the same moment, with the same field of view angle and detection range.

[0060] In the specific implementation of step S2, the lidar point cloud data is compressed into two dimensions, and two-dimensional fast Fourier transform is performed on the millimeter-wave radar data to obtain compressed point cloud data and a millimeter-wave range-azimuth heat map with the same dimensions, and they are concatenated to obtain concatenated data.

[0061] Specifically, when processing the lidar point cloud data in the present invention, it is compressed into a bird's-eye view (Bird's-Eye View, BEV), and regarded as a grayscale image (with the number of channels being 1, the length being H, and the width being W), converting the point cloud generation problem into an image generation problem.

[0062] Specifically, the lidar point cloud data as the ground truth is projected into two dimensions and input in the form of a matrix containing a two-dimensional coordinate list. The data dimension is (1, 128, 128), which can be regarded as a grayscale image with the number of channels being 1, the length being 128, and the width being 128. During the data acquisition process of the millimeter-wave radar, only the radar array in the horizontal direction is enabled. Through two-dimensional fast Fourier transform (Fast Fourier transform, FFT) of the data, a millimeter-wave range-azimuth heat map (Range-Azimuth HeatMap, RAM) is obtained, and its dimension is consistent with the lidar point cloud data projected into two dimensions. The RAM and the lidar point cloud data are superimposed in the channel dimension to obtain concatenated data.

[0063] In the specific implementation of step S3, based on the Markov chain, forward diffusion is performed on the concatenated data.

[0064] For the concatenated data, that is, the point cloud composed of N points (which has been converted to the two-dimensional BEV perspective and represented by a grayscale image), each point can be regarded as independently sampled from a point distribution, and its distribution is expressed as , where z is the image condition that determines the point distribution. As time goes by, the point cloud gradually spreads into a chaotic set of points, and the target point distribution is converted into a noise distribution. This forward diffusion process can be modeled as a Markov chain:

[0065]

[0066] where That is, the Markov diffusion kernel, which adds noise to the points at the previous time step and models the distribution of the points at the next time step, is defined as:

[0067]

[0068] where is the variance scheduling hyperparameter for controlling the diffusion rate of the process, T is the total number of noise addition steps, is a normal distribution, is the same as the identity matrix of the same size.

[0069] In the specific implementation of step S4, for the concatenated data after forward diffusion, the inverse diffusion network is trained to predict the transition kernel of the reverse Markov chain to learn the mapping relationship from the millimeter-wave radar domain to the lidar domain. That is, in the training stage, the network input includes both the two-dimensional lidar point cloud data as the ground truth and the millimeter-wave radar RAM data. After the two are concatenated, forward diffusion is performed, and the network attempts to learn the inverse process of forward diffusion. In the inference stage, the network input only has Gaussian noise and millimeter-wave radar RAM data. After the two are concatenated and input into the network, the network outputs the lidar point cloud corresponding to the millimeter-wave radar RAM data to achieve high-quality point cloud generation;

[0070] The generation process of the target point cloud is regarded as the reverse of the diffusion process, and the input is the points sampled from the simple noise distribution in which the distribution is an approximation of. The reverse diffusion process of restoring the point cloud from the noise distribution to the target distribution is modeled as a Markov chain, and a neural network is introduced to learn its transition kernel so that the Markov chain can reconstruct the required point cloud shape. Since the purpose of the Markov chain is to model the point distribution, it is impossible to generate point clouds of various shapes only relying on the Markov chain. Therefore, an image characterized by millimeter-wave radar RA data is introduced as the condition of the transition kernel. Correspondingly, the training objective of the inverse diffusion network is formulated as maximizing the variational lower bound of the likelihood of the point cloud under the given image condition, and further formalizing it in a closed form to obtain a computable expression.

[0071] The present invention proposes a unique network for learning the reverse diffusion process of point clouds conditioned on millimeter-wave radar data. The lidar point cloud ground truth is forward diffused to add noise, and the millimeter-wave radar RAM data at the same moment, the same field of view angle, and detection range is used as a condition. The two are concatenated (i.e., superimposed in the channel dimension) and participate in the training of the neural network. After the concatenated data is forward diffused, for each denoising process, the network is trained to predict the transition kernel of the Markov chain. Each prediction is guided by the millimeter-wave radar data, and the noise is iteratively removed from the initial Gaussian noise moment with the goal of generating the original point cloud. Finally, the random Gaussian noise matrix is learned as the latent feature of the point cloud. The network training process is as Figure 2 .

[0072] Points that have undergone forward diffusion are restored to the lidar point cloud ground truth through an inverse Markov chain. This process requires the introduction of a neural network and training from the data. The inverse diffusion generation process is expressed as:

[0073]

[0074]

[0075] Among them, is the joint probability distribution, that is, the probability distribution of the lidar point cloud data, is the condition for controlling the network to denoise, which is the millimeter-wave radar RAM data in this problem. is estimated by a neural network with parameter The mean of the normal distribution obtained by forward diffusion to The starting distribution is set to the standard normal distribution :

[0076]

[0077]

[0078] The goal of training the inverse diffusion process is to maximize the log-likelihood of the point cloud. This problem can be transformed into maximizing its variational lower bound:

[0079]

[0080] In the formula is the mathematical expectation, is the estimate of and is the conditional probability between the lidar ground truth and the millimeter-wave radar RAM.

[0081] The above variational constraint can be rewritten as the training objective L to be minimized:

[0082]

[0083] Due to the property that the distributions of each point are independent of each other, the above formula can be expanded to:

[0084]

[0085] In the above formula, ①~⑤ are all the KL divergences of Gaussian functions, which have closed expressions and can be optimized and simplified separately for network training. Finally, the optimization objective conditional on z can be derived and expressed as:

[0086]

[0087] Among them​ is the true value data distribution estimated by the neural network, is a normal distribution, where is Gaussian noise, z is the condition for the control network to denoise, which is the millimeter-wave radar RAM data in this problem, T is the total number of noise addition steps, is the conditional probability between the millimeter-wave radar RAM and the lidar true value.

[0088] Select the inverse diffusion network parameter θ to maximize the probability of recovering the true value from the noise. This optimization objective can be equivalent to three different forms; considering the combination relationship and data requirements of different modules, the original data is predicted by the model in the form of, and this loss function characterizes the true inverse diffusion distribution and the inverse diffusion distribution to be solved of the KL divergence.

[0089] The backbone structure of the inverse diffusion network is a U-Net structure, which includes a downsampling module and an upsampling module. The former extracts the original image into a feature map of (16, 16) through convolution and pooling, while the latter continuously upsamples to obtain a network output of the same size as the network input. Among the convolutional layers, a multi-head attention mechanism is applied to focus on the significant regions of the two-dimensional matrix (i.e., the image) corresponding to the point cloud. For the sampling process, the input of the network is randomly generated Gaussian noise, and the trained model is used to denoise and generate the corresponding two-dimensional matrix of the point cloud with the specified millimeter-wave radar RAM as the condition. The input can also be the two-dimensional matrix of the point cloud obtained after certain processing of the millimeter-wave radar data. This scheme can provide richer initial values consistent with the generation target information, which can improve the efficiency of the inference process and the performance of the generated point cloud.

[0090] In specific implementation, if this application is applied to the real-time positioning and mapping task, certain requirements are put forward for its operation efficiency, and at the same time, the quality of the generated point cloud should be ensured. Therefore, the Consistency Models (CM) technology can also be used to optimize the above model. Specifically, the consistency model training function is used as a probability flow (PF) ordinary differential equation (ODE) solver, so that the Gaussian noise and the millimeter-wave radar RAM data as the generation condition can generate the initial sample (i.e., the corresponding lidar point cloud) from the denoising step T through a single-step inference. The training process depends on this solver and the pre-trained inverse diffusion network to generate adjacent point pairs on the PF ODE trajectory. By minimizing the difference between the point pairs and the model output, the features of the pre-trained inverse diffusion network are extracted into the single-step sampler, so as to distill the inverse diffusion network into a consistency model. Through optimization, the point cloud generation efficiency of the present invention meets the real-time requirements, and the point cloud generation effect does not decrease significantly.

[0091] Compared with general point cloud generation models, the neural network of the present invention has stronger representation ability. Especially for point cloud information in indoor and outdoor scenes, it can effectively extract its distribution characteristics. With the help of the powerful high-dimensional feature extraction ability of the diffusion model, the present invention effectively introduces millimeter-wave radar data to accurately and efficiently guide the generation process of point clouds. The generated point clouds have high fineness and accuracy, meeting the requirements of subsequent applications.

[0092] In the specific implementation of step S5, the millimeter-wave radar data to be processed is obtained, and after preprocessing, it is input into the trained inverse diffusion network to generate lidar-like point cloud data.

[0093] Specifically, during network training, given a millimeter-wave radar-lidar joint dataset, the neural network learns the process of gradually denoising from Gaussian noise to generate the corresponding lidar point cloud with millimeter-wave radar data as the generation condition. Therefore, during network inference, without lidar point clouds, as long as the millimeter-wave radar data is used as the generation condition, the neural network can generate lidar-like point cloud data in the same scene corresponding to it.

[0094] The present invention is stronger than existing methods in terms of millimeter-wave radar point cloud accuracy and density. Figure 3 That is, it shows the result comparison of generating point clouds between this article and the RadarHD method (《Prabhakara A, Jin T, Das A, et al. High resolution pointclouds from mmwave radar[C] / / 2023 IEEE International Conference on Roboticsand Automation (ICRA). IEEE, 2023: 4135-4142.》) based on the same frame of RAM. Figure 3 It can be seen that based on the RAM shown in (a) of Figure 3 , compared with the corresponding lidar point cloud ground truth ( Figure 3 in (b)), it can be known that the RadarHD method loses a large amount of environmental details and edge features when generating point clouds ( Figure 3 in (c)), while the method proposed in the present invention faithfully restores the lidar point cloud ground truth ( Figure 3 in (d)).

[0095] In addition, in terms of algorithm optimization, to solve the problem of huge computational overhead caused by the iterative solution of the diffusion model, the present invention first introduces a consistency model to distill the diffusion model for generating millimeter-wave radar point clouds, achieving a significant improvement in operating efficiency, enabling the model to directly run on an edge computing platform and be applied to real-time tasks. Figure 4As shown, after introducing the consistency model, the point cloud effect obtained by only iterating twice (only performing two network forward propagations) ( Figure 4 in (c)) is comparable to the result obtained by iterating 100 steps using the conventional diffusion model inference method (Karras T, Aittala M, Aila T, et al. Elucidating the designspace of diffusion-based generative models[J]. Advances in Neural InformationProcessing Systems, 2022, 35: 26565-26577.) ( Figure 4 in (b)), and at the same time, it highly faithfully restores the lidar point cloud ground truth ( Figure 4 in (a)), greatly improving the calculation efficiency while ensuring the quality of the generated point cloud.

[0096] Corresponding to the foregoing embodiments of the millimeter-wave radar point cloud generation method based on the diffusion model, the present application also provides embodiments of a millimeter-wave radar point cloud generation device based on the diffusion model.

[0097] Figure 5 is a block diagram of a millimeter-wave radar point cloud generation device based on a diffusion model shown according to an exemplary embodiment. Referring to Figure 5 , the device may include:

[0098] An acquisition module 21, configured to acquire spatio-temporally aligned lidar point cloud data and millimeter-wave radar data;

[0099] A compression module 22, configured to compress the lidar point cloud data into two dimensions, perform a two-dimensional fast Fourier transform on the millimeter-wave radar data, obtain compressed point cloud data and a millimeter-wave range-azimuth heat map with the same dimension and concatenate them to obtain concatenated data;

[0100] A forward diffusion module 23, configured to perform forward diffusion on the concatenated data based on a Markov chain;

[0101] A training module 24, configured to train an inverse diffusion network to predict the transition kernel of the reverse Markov chain for the forward-diffused concatenated data, so as to learn the mapping relationship from the millimeter-wave radar domain to the lidar domain;

[0102] A point cloud generation module 25, configured to acquire millimeter-wave radar data to be processed, preprocess it and input it into the trained inverse diffusion network, so as to generate lidar-like point cloud data.

[0103] Regarding the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0104] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions of the method embodiments. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0105] Correspondingly, this application also provides a computer program product, including computer programs / instructions, which when executed by a processor, implement the method for generating millimeter-wave radar point clouds based on a diffusion model as described above.

[0106] Correspondingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating millimeter-wave radar point clouds based on a diffusion model as described above. As Figure 6 shown, it is a hardware structure diagram of any device with data processing capabilities where the device for generating millimeter-wave radar point clouds based on a diffusion model provided by an embodiment of the present invention is located. In addition to Figure 6 the processors, memory, and network interfaces shown, any device with data processing capabilities where the device in the embodiment is located usually also includes other hardware according to the actual functions of the device with data processing capabilities, which will not be elaborated here.

[0107] Correspondingly, the present application further provides a computer-readable storage medium, on which computer instructions are stored. When the instructions are executed by a processor, the above-described method for generating millimeter-wave radar point clouds based on a diffusion model is implemented. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0108] After considering the specification and practicing the content disclosed herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application.

Claims

1. A method for generating millimeter wave radar point cloud based on a diffusion model, characterized in that: include: Obtain time-space aligned LiDAR point cloud data and millimeter wave radar raw data; Compressing the laser radar point cloud data into two dimensions, performing a two-dimensional fast Fourier transform on the millimeter wave radar data, obtaining compressed point cloud data and a millimeter wave range-azimuth heat map of the same dimension, and concatenating them to obtain concatenated data; Based on the Markov chain, forward diffusion is performed on the concatenated data; For the concatenated data after forward diffusion, the reverse diffusion network is trained to predict the transfer kernel of the reverse Markov chain to learn the mapping relationship from the millimeter wave radar domain to the lidar domain; The millimeter-wave radar data to be processed is obtained, the trained inverse diffusion network is optimized using a consistency model, and the laser radar point cloud data corresponding to the millimeter-wave radar data to be processed is generated using the consistency model.

2. The method according to claim 1, characterized in that For a point cloud composed of N points , the forward diffusion process is modeled as a Markov chain: ， in That is, the Markov diffusion kernel, which adds noise to the points at the previous time step and models the distribution of the points at the next time step, is defined as: ， in is the variance scheduling hyperparameter for controlling the diffusion rate of the process, T is the total number of noise addition steps, is a normal distribution, For Identity matrix of the same size.

3. The method according to claim 1, characterized in that The optimization goal of the inverse diffusion network training is ， in, is the true data distribution estimated by the neural network, is a normal distribution.

4. The method according to claim 1, characterized in that The backbone of the inverse diffusion network is a U-Net structure, which includes a downsampling module and an upsampling module. The downsampling module extracts the network input into the corresponding feature map through convolutional layers and pooling layers, wherein a multi-head attention mechanism is applied between convolutional layers. The upsampling module converts the feature map into a network output of the same size as the network input by continuous upsampling.

5. The method according to claim 1, characterized in that Consistency model training function As a probability flow ordinary differential equation solver, the noisy samples generate initial samples in a single step from time t. The training process relies on this solver and the inverse diffusion network to generate adjacent point pairs on the probability flow ordinary differential equation trajectory. By minimizing the difference between the point pairs and the output of the inverse diffusion network, the features of the trained inverse diffusion network are extracted into the single-step sampler, thereby distilling the inverse diffusion network into a consistent model.

6. A millimeter wave radar point cloud generation device based on a diffusion model, characterized in that: include: The acquisition module is used to obtain the time-space aligned lidar point cloud data and millimeter wave radar raw data; A compression module, used to compress the laser radar point cloud data into two dimensions, perform a two-dimensional fast Fourier transform on the millimeter wave radar data, obtain compressed point cloud data and millimeter wave range-azimuth heat map of the same dimension, and concatenate them to obtain concatenated data; A forward diffusion module, used for forward diffusion of the concatenated data based on a Markov chain; The training module is used to train the reverse diffusion network to predict the transfer kernel of the reverse Markov chain based on the concatenated data after forward diffusion, so as to learn the mapping relationship from the millimeter wave radar domain to the lidar domain; The point cloud generation module is used to obtain millimeter-wave radar data to be processed, optimize the trained inverse diffusion network using a consistency model, and generate lidar point cloud data corresponding to the millimeter-wave radar data to be processed using the consistency model.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images