Millimeter wave radar point cloud generation method based on diffusion model

Abstract

The invention discloses a millimeter-wave radar point cloud generation method based on a diffusion model. The method comprises the following steps: acquiring laser radar point cloud data and millimeter-wave radar data which are in space-time alignment; compressing the laser radar point cloud data to two dimensions, performing two-dimensional fast Fourier transform on the millimeter-wave radar data to obtain compressed point cloud data and a millimeter-wave distance-azimuth heat map with the same dimension, and serially connecting the compressed point cloud data and the millimeter-wave distance-azimuth heat map to obtain serially-connected data; performing forward diffusion on the tandem data based on a Markov chain; training a reverse diffusion network to predict a transfer kernel of a reverse Markov chain for the tandem data after forward diffusion so as to learn a mapping relation from a millimeter wave radar domain to a laser radar domain; in a deployment stage, millimeter-wave radar data are acquired, preprocessed and then input into the trained inverse diffusion network, so that laser-like radar point cloud is generated, and high-quality millimeter-wave radar point cloud generation is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous navigation of unmanned systems, and in particular relates to a method for generating millimeter wave radar point clouds based on a diffusion model. Background Art

[0002] In harsh environments such as rain, fog, dust, and smoke, mainstream optical sensors such as LiDAR and binocular cameras are prone to degradation or even failure, and cannot achieve accurate perception of the environment and autonomous navigation in these environments. Compared with the two, millimeter-wave radar is not affected by fine particles due to its longer wavelength, so it is robust to most extreme weather environments.

[0003] Radar systems often use a constant false alarm rate (CFAR) detector to detect targets. The algorithm first sets up protection cells around the unit under test to eliminate the energy leakage of the unit under test signal in the surrounding area, and then selects a reference cell within a certain range outside the protection cell to perform power estimation and determine the threshold. In order to improve detection performance or reduce computational overhead, there are also improved algorithms such as (CA)-CFAR and (OS)-CFAR.

[0004] However, when applied to cluttered and dense scenes such as autonomous driving and robot autonomous navigation, the background noise and interference levels increase significantly and are difficult to model, making the above algorithms difficult to apply. In order to meet the needs of autonomous navigation of unmanned systems, the academic community has proposed many solutions based on deep learning algorithms for tasks such as target detection, odometer, and map construction. Among them, some methods also focus on millimeter wave radar point cloud generation. These methods all use laser radar and millimeter wave radar data that have been time synchronized and calibrated with external parameters. For example, the RadarHD method is based on the U-Net network framework, and takes the distance-horizontal azimuth heat map of the millimeter wave radar as the network input. Then, through the supervision of cross entropy loss and Dice loss, the network generates a two-dimensional laser radar point cloud based on the millimeter wave radar data; the RPDNet algorithm takes the distance-speed heat map of the millimeter wave radar as the network input, and performs adversarial learning based on the GAN network to improve the generation performance; the DREAM-PCD algorithm combines traditional radar signal processing with deep learning algorithms. First, non-coherent accumulation (Non-coherent accumulation) and synthetic aperture accumulation (Synthetic Aperture Accumulation) are performed on multi-frame millimeter wave radar point clouds to improve the signal-to-noise ratio and angular resolution of the original signal respectively. Based on the DGCNN network, the enhanced millimeter wave radar point cloud is supervised and learned to obtain a high-resolution millimeter wave radar point cloud.

[0005] Existing millimeter wave radar point cloud generation technologies include multiple signal classification (MUSIC) based on traditional data processing algorithms, CFAR and its improved algorithms; and RadarHD, RPDNet based on deep learning. ] , DREAM-PCD and other solutions.

[0006] Compared with traditional point cloud generation methods based on digital signal processing, existing deep learning-based algorithm solutions can generate relatively dense point clouds. Given paired lidar point clouds, deep neural networks can learn to denoise and upsample millimeter-wave radar signals through pixel-by-pixel loss functions or adversarial learning to obtain lidar-like point clouds. However, the network framework used in the above methods does not have sufficient generation capabilities, so it is impossible to accurately restore the point cloud structure information, and the generated millimeter-wave radar point cloud lacks obstacle edge features, and is still far inferior to the lidar point cloud in terms of accuracy and density. The millimeter-wave radar point cloud obtained by network inference is better than the point cloud obtained by the traditional method shown in the figure above. It can depict the outline of obstacles, but it loses many details and lacks edge features. It is relatively fuzzy and still difficult to apply to complex scenes.

[0007] In addition, there are also methods such as "Gao X, Roy S, Xing G. MIMO-SAR: A hierarchical high-resolution imaging algorithm for mmWave FMCW radar in autonomous driving[J].IEEE Transactions on Vehicular Technology, 2021, 70(8): 7322-7334.", which draws on the synthetic aperture radar (SAR) technology in the radar system and performs coherent processing on multiple frames of temporally continuous sensor data under the premise of known sensor posture, so as to increase the radar aperture and thereby improve the resolution. The DREAM-PCD method also uses SAR technology in data preprocessing. However, this technology requires that the sensor posture is known, and the imaging effect is very sensitive to the posture accuracy. In the harsh environment autonomous navigation scenario concerned by the present invention, only millimeter-wave radar can provide reliable sensing information, which cannot meet the condition of high-precision posture known. Therefore, the millimeter-wave radar imaging algorithm based on SAR technology is difficult to apply to the autonomous navigation of unmanned systems in harsh environments. Summary of the invention

[0008] In view of the problems existing in the prior art, the purpose of the embodiments of the present application is to provide a method for generating millimeter wave radar point cloud based on a diffusion model.

[0009] According to a first aspect of an embodiment of the present application, a method for generating a millimeter-wave radar point cloud based on a diffusion model is provided, comprising:

[0010] Obtain time-space aligned LiDAR point cloud data and millimeter wave radar data;

[0011] Compressing the laser radar point cloud data into two dimensions, performing a two-dimensional fast Fourier transform on the millimeter wave radar data, obtaining compressed point cloud data and a millimeter wave range-azimuth heat map of the same dimension, and concatenating them to obtain concatenated data;

[0012] Based on the Markov chain, forward diffusion is performed on the concatenated data;

[0013] For the concatenated data after forward diffusion, the reverse diffusion network is trained to predict the transfer kernel of the reverse Markov chain to learn the mapping relationship from the millimeter wave radar domain to the lidar domain;

[0014] The millimeter-wave radar data to be processed is obtained and input into the trained inverse diffusion network after preprocessing to generate lidar-like point cloud data.

[0015] Furthermore, for a point cloud composed of N points , the forward diffusion process is modeled as a Markov chain:

[0016]

[0017] in That is, the Markov diffusion kernel, which adds noise to the points at the previous time step and models the distribution of the points at the next time step, is defined as:

[0018]

[0019] in is the variance scheduling hyperparameter for controlling the diffusion rate of the process, T is the total number of noise addition steps, is a normal distribution, For Identity matrix of the same size.

[0020] Furthermore, the optimization goal during the inverse diffusion network training is

[0021]

[0022] in, is the true data distribution estimated by the neural network, is a normal distribution.

[0023] Furthermore, the backbone of the inverse diffusion network is a U-Net structure, including a downsampling module and an upsampling module. The downsampling module extracts the network input into a corresponding feature map through a convolution layer and a pooling layer, wherein a multi-head attention mechanism is applied between the convolution layers. The upsampling module converts the feature map into a network output of the same size as the network input through continuous upsampling.

[0024] Furthermore, the trained inverse diffusion network is optimized using a consistency model, and the consistency model is used to generate lidar point cloud data corresponding to the millimeter-wave radar data to be processed.

[0025] Furthermore, the consistency model training function As a probability flow ordinary differential equation solver, the noisy samples generate initial samples in a single step from time t. The training process relies on this solver and the inverse diffusion network to generate adjacent point pairs on the probability flow ordinary differential equation trajectory. By minimizing the difference between the point pairs and the output of the inverse diffusion network, the features of the trained inverse diffusion network are extracted into the single-step sampler, thereby distilling the inverse diffusion network into a consistent model.

[0026] According to a second aspect of an embodiment of the present application, a millimeter wave radar point cloud generation device based on a diffusion model is provided, comprising:

[0027] An acquisition module, used to acquire time-space aligned lidar point cloud data and millimeter wave radar data;

[0028] A compression module, used to compress the laser radar point cloud data into two dimensions, perform a two-dimensional fast Fourier transform on the millimeter wave radar data, obtain compressed point cloud data and millimeter wave range-azimuth heat map of the same dimension, and concatenate them to obtain concatenated data;

[0029] A forward diffusion module, used for forward diffusion of the concatenated data based on a Markov chain;

[0030] The training module is used to train the reverse diffusion network to predict the transfer kernel of the reverse Markov chain based on the concatenated data after forward diffusion, so as to learn the mapping relationship from the millimeter wave radar domain to the lidar domain;

[0031] The point cloud generation module is used to obtain the millimeter-wave radar data to be processed, and input the pre-processed data into the trained inverse diffusion network to generate lidar-like point cloud data.

[0032] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0033] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0034] one or more processors;

[0035] A memory for storing one or more programs;

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0037] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0038] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0039] It can be seen from the above embodiments that this application improves the technology of generating millimeter-wave radar point clouds through laser radar supervision. The generation of millimeter-wave radar point clouds based on laser radar supervision is essentially both a denoising task and a super-resolution task. Therefore, this application combines the diffusion model algorithm and uses its advantages of strong representation ability and stable training to generate high-precision dense millimeter-wave radar point clouds. Specifically, the diffusion model technology is combined with the millimeter-wave radar perception technology, and the point cloud is generated using millimeter-wave radar data as the network control condition, so that the diffusion model can recover high-precision dense point clouds rich in obstacle edge information and structural features from the noisy point cloud, and achieves extremely low loss and extremely high consistency in comparison with the true value collected by the laser radar. The generated point cloud can be directly used for tasks such as map construction.

[0040] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 The present invention is a flowchart of a method for generating millimeter wave radar point cloud based on a diffusion model according to an exemplary embodiment.

[0043] Figure 2 It is a schematic diagram of a diffusion model training process based on millimeter-wave radar data according to an exemplary embodiment.

[0044] Figure 3 is a comparison diagram of millimeter wave radar point cloud generation results according to an exemplary embodiment, wherein Figure 3 (a) is RAM (i.e. network input), Figure 3 (b) in the figure is the true value of the laser radar point cloud. Figure 3 (c) in the figure is the point cloud generated by RedarHD. Figure 3 (d) in the figure is the point cloud generated by this method.

[0045] Figure 4 is a comparison diagram of accelerated reasoning effects according to an exemplary embodiment, wherein Figure 4 (a) in the figure is the true value of the laser radar point cloud. Figure 4 (b) is the laser-like point cloud obtained by 80 steps of inverse diffusion network reasoning in this method. Figure 4 (c) in the figure is the laser-like point cloud obtained by the two-step thrust of the consistency model in this method.

[0046] Figure 5 The present invention is a block diagram of a millimeter wave radar point cloud generating device based on a diffusion model according to an exemplary embodiment.

[0047] Figure 6 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0049] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0050] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0051] Figure 1is a flowchart of a method for generating millimeter wave radar point cloud based on a diffusion model according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps:

[0052] S1: Obtaining spatiotemporally aligned lidar point cloud data and millimeter-wave radar raw data;

[0053] S2: compressing the laser radar point cloud data into two dimensions, performing a two-dimensional fast Fourier transform on the millimeter wave radar data, obtaining compressed point cloud data and a millimeter wave range-azimuth heat map of the same dimension, and concatenating them to obtain concatenated data;

[0054] S3: forward diffusion of the concatenated data based on a Markov chain;

[0055] S4: For the concatenated data after forward diffusion, the reverse diffusion network is trained to predict the transfer kernel of the reverse Markov chain to learn the mapping relationship from the millimeter wave radar domain to the lidar domain;

[0056] S5: Obtain the millimeter-wave radar data to be processed, and input it into the trained inverse diffusion network after preprocessing, so as to generate lidar-like point cloud data.

[0057] This method aims at the inherent limitations of deep learning-based point cloud generation schemes such as variational auto-encoders (VAE), generative adversarial networks (GAN), and normalizing flows, such as the instability of the training process in GAN. It uses a probabilistic generation model inspired by non-equilibrium thermodynamics to generate point clouds, characterizes the point cloud generation task as an inverse diffusion process of transforming the particle noise distribution into the required shape distribution, and innovatively proposes a denoising diffusion probabilistic generation model that can generate high-precision point clouds based on millimeter-wave radar data. It can process and fully utilize the input millimeter-wave radar data, and restore the two-dimensional point cloud of obstacle information under the consistent perspective of the millimeter-wave radar from the randomly sampled noise point cloud. The denoising process of the restored point cloud is realized by a neural network. This method greatly improves the point cloud generation effect and can be widely used in fields such as simultaneous localization and mapping (SLAM), mobile robot navigation, and autonomous driving.

[0058] In a specific implementation of step S1, obtaining time-space aligned lidar point cloud data and millimeter wave radar data;

[0059] Specifically, during the millimeter-wave radar data collection process, only the horizontal radar array is enabled, and it is necessary to ensure that the millimeter-wave radar data and the lidar point cloud data are at the same time, with the same field of view angle and detection range.

[0060] In the specific implementation of step S2, the laser radar point cloud data is compressed into two dimensions, and the millimeter wave radar data is subjected to a two-dimensional fast Fourier transform to obtain compressed point cloud data and a millimeter wave range-azimuth heat map of the same dimension and concatenate them to obtain concatenated data;

[0061] Specifically, when processing the lidar point cloud data, the present invention compresses it to a bird's-eye view (BEV) and regards it as a grayscale image (the number of channels is 1, the length is H, and the width is W), thereby converting the point cloud generation problem into an image generation problem.

[0062] Specifically, the laser radar point cloud data as the true value is projected into two dimensions and input in the form of a matrix containing a two-dimensional coordinate list. The data dimension is (1, 128, 128), which can be regarded as a grayscale image with 1 channel, 128 length, and 128 width. During the millimeter wave radar data collection process, only the horizontal radar array is enabled. Through the two-dimensional fast Fourier transform (FFT) of the data, the millimeter wave range-azimuth heat map (RAM) is obtained, and its dimension is consistent with the laser radar point cloud data projected into two dimensions. The RAM is superimposed with the laser radar point cloud data in the channel dimension to obtain the concatenated data.

[0063] In the specific implementation of step S3, forward diffusion is performed on the concatenated data based on a Markov chain;

[0064] For the concatenated data, i.e., the point cloud composed of N points (has been converted to a two-dimensional BEV perspective and represented by a grayscale image), where each point can be considered as an independent sample from a point distribution, whose distribution is represented as , z is the image condition that determines the point distribution. As time goes by, the point cloud gradually diffuses into a chaotic point set, and the target point distribution is transformed into a noise distribution. The forward diffusion process can be modeled as a Markov chain:

[0065]

[0066] in That is, the Markov diffusion kernel, which adds noise to the points at the previous time step and models the distribution of the points at the next time step, is defined as:

[0067]

[0068] in is the variance scheduling hyperparameter for controlling the diffusion rate of the process, T is the total number of noise addition steps, is a normal distribution, For Identity matrix of the same size.

[0069] In the specific implementation of step S4, for the concatenated data after forward diffusion, the reverse diffusion network is trained to predict the transfer kernel of the reverse Markov chain to learn the mapping relationship from the millimeter-wave radar domain to the lidar domain. That is, the network input in the training phase includes both two-dimensional lidar point cloud data as the true value and millimeter-wave radar RAM data. The two are concatenated and forward diffused, and the network attempts to learn the inverse process of forward diffusion. In the inference phase, the network input is only Gaussian noise and millimeter-wave radar RAM data, which are concatenated and input into the network. The network outputs the lidar point cloud corresponding to the millimeter-wave radar RAM data to achieve high-quality point cloud generation;

[0070] The generation process of the target point cloud is regarded as the inverse of the diffusion process, and the input is a simple noise distribution The distribution of the points sampled in is The inverse diffusion process of restoring the target distribution from the noise distribution is modeled as a Markov chain, and a neural network is introduced to learn its transfer kernel so that the Markov chain can reconstruct the required point cloud shape. Since the purpose of the Markov chain is to model the point distribution, it is impossible to generate point clouds of various shapes by relying solely on the Markov chain. For this reason, the image characterized by the millimeter-wave radar RA data is introduced as the condition of the transfer kernel. Correspondingly, the training objective of the inverse diffusion network is formulated as maximizing the lower bound of the likelihood variation of the point cloud under given image conditions, which is further formalized in a closed form to obtain a computable expression.

[0071] The present invention proposes a unique network for learning the reverse diffusion process of point clouds based on millimeter-wave radar data. It performs forward diffusion on the true value of the lidar point cloud to add noise, and uses the millimeter-wave radar RAM data at the same time, field of view and detection range as a condition, and concatenates the two (i.e., superimposes them in the channel dimension) to participate in the training of the neural network. After the concatenated data undergoes forward diffusion, the training network predicts the transfer kernel of the Markov chain for each step of the denoising process. Each prediction is guided by the millimeter-wave radar data, and the noise is iteratively removed from the initial Gaussian noise moment with the goal of generating the original point cloud. Finally, the random Gaussian noise matrix is ​​learned as the hidden feature of the point cloud. The network training process is as follows: Figure 2 .

[0072] The forward diffused points are restored from Gaussian noise to the true value of the lidar point cloud through the reverse Markov chain. This process requires the introduction of a neural network and training from data. The reverse diffusion generation process is expressed as:

[0073]

[0074]

[0075] in, is the joint probability distribution, that is, the probability distribution of the lidar point cloud data, In order to control the conditions of network denoising, in this problem it is the millimeter wave radar RAM data. The parameter is The neural network estimates Forward diffusion to The normal distribution mean of Set to standard normal distribution :

[0076]

[0077]

[0078] The goal of training the inverse diffusion process is to maximize the log-likelihood of the point cloud, which can be transformed into maximizing its variational lower bound:

[0079]

[0080] In the formula is the mathematical expectation, For The estimate, is the conditional probability between the true value of the lidar and the millimeter-wave radar RAM.

[0081] The above variational constraint can be rewritten as a training objective L to be minimized:

[0082]

[0083] According to the independent nature of the distribution of each point, the above formula can be expanded to:

[0084]

[0085] In the above formula, ①~⑤ are the KL divergences of Gaussian functions, which have closed expressions and can be optimized and simplified respectively to facilitate network training. Finally, the optimization target with z as the condition can be derived, which is expressed as:

[0086]

[0087] in is the true data distribution estimated by the neural network, is a normal distribution, where is Gaussian noise, z is the condition for controlling network denoising, which is the millimeter-wave radar RAM data in this problem, T is the total number of noise addition steps, is the conditional probability between the millimeter-wave radar RAM and the lidar true value.

[0088] The inverse diffusion network parameter θ is selected to maximize the probability of recovering the true value from the noise. This optimization objective can be equivalent to three different forms; considering the combination relationship and data requirements of different modules, the original data is predicted by the model. The loss function represents the true inverse diffusion distribution and the inverse diffusion distribution to be determined The KL divergence of .

[0089] The backbone structure of the inverse diffusion network is the U-Net structure, which includes a downsampling module and an upsampling module. The former extracts the original image into a (16, 16) feature map through convolution and pooling, while the latter continuously upsamples to obtain a network output of the same size as the network input. Between the convolution layers, a multi-head attention mechanism is applied to focus on the salient areas of the two-dimensional matrix (i.e., image) corresponding to the point cloud. For the sampling process, the input of the network is randomly generated Gaussian noise. Using the trained model, the corresponding point cloud two-dimensional matrix is ​​denoised and generated based on the specified millimeter-wave radar RAM. The input can also be a point cloud two-dimensional matrix obtained after a certain processing of the millimeter-wave radar data. This solution can provide richer initial values ​​consistent with the generated target information, which can improve the efficiency of the reasoning process and the performance of the generated point cloud.

[0090] In specific implementation, if this application is applied to real-time positioning and map building tasks, certain requirements are put forward for its operation efficiency, and the quality of the generated point cloud should be guaranteed. Therefore, the consistency model (CM) technology can also be used to optimize the above model. Specifically, the consistency model training function As a probability flow (PF) ordinary differential equation (ODE) solver, Gaussian noise and millimeter-wave radar RAM data as the generation condition can generate initial samples (i.e., the corresponding lidar point cloud) from the denoising step number T through single-step reasoning. The training process relies on the solver and the pre-trained inverse diffusion network to generate adjacent point pairs on the PF ODE trajectory. By minimizing the difference between the point pairs and the model output, the features of the pre-trained inverse diffusion network are extracted into the single-step sampler, thereby distilling the inverse diffusion network into a consistency model. Through optimization, the point cloud generation efficiency of the present invention meets the real-time requirements, and there is no obvious decrease in the point cloud generation effect.

[0091] Compared with the general point cloud generation model, the neural network of the present invention has stronger representation ability, especially for point cloud information in indoor and outdoor scenes, and can effectively extract its distribution characteristics. With the help of the diffusion model's powerful extraction ability for high-dimensional features, the present invention effectively introduces millimeter-wave radar data to accurately and efficiently guide the point cloud generation process. The generated point cloud has high precision and accuracy, meeting the needs of subsequent applications.

[0092] In the specific implementation of step S5, the millimeter wave radar data to be processed is obtained, and after preprocessing, it is input into the trained inverse diffusion network to generate laser radar point cloud data;

[0093] Specifically, during network training, given a millimeter-wave radar-lidar joint dataset, the neural network learns the process of using millimeter-wave radar data as a generation condition and gradually denoising from Gaussian noise based on a diffusion model to generate the corresponding lidar point cloud. Therefore, during network reasoning, there is no need for lidar point cloud, but only the millimeter-wave radar data as a generation condition, and the neural network can generate the corresponding lidar-like point cloud data in the same scene.

[0094] The present invention is superior to existing methods in terms of millimeter wave radar point cloud accuracy and density. Figure 3 This shows the comparison of the point cloud generated by this paper and RadarHD method (Prabhakara A, Jin T, Das A, et al. High resolution pointclouds from mmwave radar[C] / / 2023 IEEE International Conference on Roboticsand Automation (ICRA). IEEE, 2023: 4135-4142.) based on the same frame RAM. Figure 3 It can be seen that based on Figure 3 The RAM shown in (a) and the corresponding LiDAR point cloud ground truth ( Figure 3 By comparison with (b) in the figure, the point cloud generated by the RadarHD method loses a lot of environmental details and edge features ( Figure 3 (c) in the figure), while the method proposed in the present invention restores the true value of the lidar point cloud with high fidelity ( Figure 3 (d) in the figure.

[0095] In addition, in terms of algorithm optimization, in order to solve the problem of huge computational overhead caused by iterative solution of the diffusion model, the present invention first introduces a consistency model to distill the diffusion model generated by the millimeter-wave radar point cloud, which greatly improves the operating efficiency and enables the model to run directly on the edge computing platform and be applied to real-time tasks. Figure 4As shown in Figure 2, after the consistency model is introduced, the point cloud effect obtained by only iterating twice (only two network forward propagations) is as follows: Figure 4 (c) in the figure and the result obtained by iterating 100 steps by introducing the conventional diffusion model inference method (Karras T, Aittala M, Aila T, et al. Elucidating the designspace of diffusion-based generative models[J]. Advances in Neural InformationProcessing Systems, 2022, 35: 26565-26577.) Figure 4 (b) in the figure, and also restores the true value of the LiDAR point cloud with high fidelity ( Figure 4 (a) in the figure greatly improves the computational efficiency while ensuring the quality of the generated point cloud.

[0096] Corresponding to the aforementioned embodiment of the millimeter-wave radar point cloud generation method based on the diffusion model, the present application also provides an embodiment of the millimeter-wave radar point cloud generation device based on the diffusion model.

[0097] Figure 5 is a block diagram of a millimeter wave radar point cloud generation device based on a diffusion model according to an exemplary embodiment. Figure 5 , the device may include:

[0098] An acquisition module 21 is used to acquire time-space aligned lidar point cloud data and millimeter wave radar data;

[0099] A compression module 22 is used to compress the laser radar point cloud data into two dimensions, perform a two-dimensional fast Fourier transform on the millimeter wave radar data, obtain compressed point cloud data and millimeter wave range-azimuth heat map of the same dimension, and concatenate them to obtain concatenated data;

[0100] A forward diffusion module 23, used for forward diffusion of the concatenated data based on a Markov chain;

[0101] The training module 24 is used to train the reverse diffusion network to predict the transfer kernel of the reverse Markov chain for the concatenated data after forward diffusion, so as to learn the mapping relationship from the millimeter wave radar domain to the lidar domain;

[0102] The point cloud generation module 25 is used to obtain the millimeter wave radar data to be processed, and input the data into the trained inverse diffusion network after preprocessing, so as to generate laser radar point cloud data.

[0103] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0104] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0105] Accordingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned millimeter-wave radar point cloud generation method based on the diffusion model.

[0106] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned millimeter wave radar point cloud generation method based on the diffusion model. Figure 6 As shown in FIG. 1 , a hardware structure diagram of a millimeter wave radar point cloud generation device based on a diffusion model provided by an embodiment of the present invention is provided in any device with data processing capability, except Figure 6 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0107] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the millimeter-wave radar point cloud generation method based on the diffusion model as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0108] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. A method for generating millimeter wave radar point cloud based on a diffusion model, characterized in that: include: Obtain time-space aligned LiDAR point cloud data and millimeter wave radar data; Compressing the laser radar point cloud data into two dimensions, performing a two-dimensional fast Fourier transform on the millimeter wave radar data, obtaining compressed point cloud data and a millimeter wave range-azimuth heat map of the same dimension, and concatenating them to obtain concatenated data; Based on the Markov chain, forward diffusion is performed on the concatenated data; For the concatenated data after forward diffusion, the reverse diffusion network is trained to predict the transfer kernel of the reverse Markov chain to learn the mapping relationship from the millimeter wave radar domain to the lidar domain; The millimeter-wave radar data to be processed is obtained and input into the trained inverse diffusion network after preprocessing to generate lidar-like point cloud data.

2. The method according to claim 1, characterized in that For a point cloud composed of N points , the forward diffusion process is modeled as a Markov chain: ， in That is, the Markov diffusion kernel, which adds noise to the points at the previous time step and models the distribution of the points at the next time step, is defined as: ， in is the variance scheduling hyperparameter for controlling the diffusion rate of the process, T is the total number of noise addition steps, is a normal distribution, For Identity matrix of the same size.

3. The method according to claim 1, characterized in that The optimization goal of the inverse diffusion network training is ， in, is the true data distribution estimated by the neural network, is a normal distribution.

4. The method according to claim 1, characterized in that: The backbone of the inverse diffusion network is a U-Net structure, which includes a downsampling module and an upsampling module. The downsampling module extracts the network input into the corresponding feature map through convolutional layers and pooling layers, wherein a multi-head attention mechanism is applied between convolutional layers. The upsampling module converts the feature map into a network output of the same size as the network input by continuous upsampling.

5. The method according to claim 1, characterized in that The trained inverse diffusion network is optimized using a consistency model, and the consistency model is used to generate lidar point cloud data corresponding to the millimeter-wave radar data to be processed.

6. The method according to claim 5, characterized in that Consistency model training function As a probability flow ordinary differential equation solver, the noisy samples generate initial samples in a single step from time t. The training process relies on this solver and the inverse diffusion network to generate adjacent point pairs on the probability flow ordinary differential equation trajectory. By minimizing the difference between the point pairs and the output of the inverse diffusion network, the features of the trained inverse diffusion network are extracted into the single-step sampler, thereby distilling the inverse diffusion network into a consistent model.

7. A millimeter wave radar point cloud generation device based on a diffusion model, characterized in that: include: An acquisition module, used to acquire time-space aligned lidar point cloud data and millimeter wave radar data; A compression module, used to compress the laser radar point cloud data into two dimensions, perform a two-dimensional fast Fourier transform on the millimeter wave radar data, obtain compressed point cloud data and millimeter wave range-azimuth heat map of the same dimension, and concatenate them to obtain concatenated data; A forward diffusion module, used for forward diffusion of the concatenated data based on a Markov chain; The training module is used to train the reverse diffusion network to predict the transfer kernel of the reverse Markov chain based on the concatenated data after forward diffusion, so as to learn the mapping relationship from the millimeter wave radar domain to the lidar domain; The point cloud generation module is used to obtain the millimeter-wave radar data to be processed, and input the pre-processed data into the trained inverse diffusion network to generate lidar-like point cloud data.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images