Super-resolution of radar point clouds using diffusion-based generative modeling

Abstract

The disclosed embodiments use a diffusion-based generative model for radar point cloud super-resolution. The disclosed embodiments use diffusion modeled arithmetical operations to generate higher resolution radar point cloud data from lower resolution radar point cloud data.

Description

Technical Field

[0001] The present disclosure relates generally to the field of artificial intelligence. More specifically, the disclosed embodiments relate to employing a diffusion model to generate high-resolution radar data from data received by a low-resolution radar. Background Art

[0002] Radar has been used in vehicles and is beginning to gain research interest and attention as a sensor useful for autonomous vehicles. The continued pursuit of higher levels of autonomy and higher levels of safety assurance requires improved perception. Unlike cameras that are sensitive to lighting changes and LiDAR sensors that are sensitive to adverse weather conditions such as fog, rain, and snow, radar sensors are known to be robust to these environmental changes and are known to have much higher distances (i.e., ~200 meters). However, radar systems are currently not as popular as sensors for autonomous vehicles (such as RGB cameras and LiDAR sensors). One of the reasons for this unpopularity is the noisy data output and sparse data output (i.e., low resolution) when converted to point clouds. Another reason for the unpopularity of radar systems is the computational and algorithmic costs involved in the signal processing algorithms used to extract point clouds from noisy raw radar measurements.

[0003] Super-resolution techniques aim to improve the resolution of a given input that has a coarse resolution for various reasons, some of which include being compressed for the purpose of low-bandwidth transmission, or being obtained from a low-cost sensor that would sense at a lower resolution than desired due to sensor design trade-offs (e.g., temporal resolution vs. spatial resolution) or other system design, cost, or efficiency considerations. Super-resolution has been explored for RGB images, and some deep learning-based algorithms have been proposed for this task. Summary of the invention

[0004] Recently, diffusion-based generative models have been introduced as a potential solution for RGB image super-resolution, which can generate high-resolution and photo-realistic images from low-resolution images. These models represent an improvement over previous work on generative modeling and more traditional super-resolution methods (e.g., optimization, compressed sensing, signal processing, etc.) because their training is stable, they do not suffer from mode collapse, and they can generate high-resolution and high-quality synthetic data. Prior art has only been developed and tested on synthetic shape point clouds (e.g., from the ShapeNet dataset), RGB images obtained from cameras, and LiDAR point clouds.

[0005] The combination of diffusion-based generative modeling and super-resolution is an attractive approach that helps address the sparsity problem of radar point cloud data with the goal of improving the resolution of point clouds obtained from low-cost radar sensors. High-resolution radar point clouds can be used as an information source that is at least comparable to cameras and LiDAR sensors for downstream detection and path planning algorithms.

[0006] So far, some work has been done on inpainting, generating LiDAR point clouds, and super-resolution for images. However, there is little or no work on applying diffusion-based generative models to LiDAR point clouds for generative modeling applications.

[0007] The disclosed embodiments use a diffusion-based generative model for radar point cloud super-resolution. Known diffusion modeling algorithms may be slow due to the iterative generation process and therefore may not be suitable for real-time applications. The disclosed embodiments may speed up the generation process by distillation of the diffusion model, allowing as few as a single generation step. The disclosed embodiments may provide the option of trading off data quality for more computational cost by using more than one generation step.

[0008] Some disclosed embodiments include a method comprising: sampling a radar point cloud dataset to generate a mini-batch of samples from the dataset, wherein the radar point cloud dataset corresponds to a first resolution; for each sample in the mini-batch of samples, calculating a noise data sample; for each sample in the mini-batch of samples, calculating a conditioning input, wherein the conditioning input is derived from a low-resolution radar point cloud sample corresponding to each sample in the mini-batch, wherein the low-resolution sample corresponds to a second resolution lower than the first resolution; and training the diffusion model on the mini-batch of samples and the conditioning input.

[0009] Some disclosed embodiments include a method comprising: receiving a first sample of a radar point cloud, the first sample corresponding to a first resolution, the first sample comprising a first noise level; computing a conditioned input from a radar point cloud corresponding to a second resolution, wherein the second resolution is lower than the first resolution; and applying a trained diffusion model to the first sample and the conditioned input to produce a second sample, the second sample corresponding to the first resolution and comprising a noise level lower than the first noise level.

[0010] Some disclosed embodiments include a system comprising: one or more processors; and a memory comprising processor-executable instructions, which when executed by the one or more processors cause the system to perform operations, the operations comprising: sampling a radar point cloud dataset to generate a mini-batch of samples from the dataset, wherein the radar point cloud dataset corresponds to a first resolution; for each sample in the mini-batch of samples, calculating a noise data sample; for each sample in the mini-batch of samples, calculating a conditioning input, wherein the conditioning input is derived from a low-resolution radar point cloud sample corresponding to each sample in the mini-batch, wherein the low-resolution sample corresponds to a second resolution lower than the first resolution; and training the diffusion model on the mini-batch of samples and the conditioning input. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flow chart of an example method according to the disclosed embodiments is shown.

[0012] Figure 2 A flow chart of an example method according to the disclosed embodiments is shown.

[0013] Figure 3 An example embodiment of a general purpose computer system according to the present disclosure is shown. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the present invention in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the drawings may be combined with the features illustrated in one or more other drawings to produce embodiments that are not explicitly illustrated or described. The combination of features shown provides representative embodiments of typical applications. However, for specific applications or implementations, various combinations and modifications of features consistent with the teachings of the present disclosure may be required.

[0015] The disclosed embodiments advantageously exploit the mathematical principles of diffusion-based generative modeling. In deep generative modeling, a deep neural network is trained to approximate the data distribution of a given data set and can be used to generate new data samples that share similar properties with data samples from the original data set. Diffusion-based models typically have a stable training process, do not suffer from mode collapse, and have good mode coverage. Therefore, diffusion-based modeling can overcome the problems that plague prior art deep generative modeling techniques. In addition, some diffusion-based models can be used to generate very high quality data samples. Research on diffusion-based models has mainly focused on RGB image data. Some studies have attempted to extend diffusion-based models to other data types, such as audio and point clouds.

[0016] A common variation of the diffusion model used in the literature is called the Denoised Diffusion Probabilistic Model (DDPM). In the DDPM framework, samples from a given dataset are corrupted by gradually adding Gaussian noise over a large number of steps, so that the samples at the end can be assumed to be drawn from a Gaussian distribution. A neural network is then trained to reverse this denoising process and gradually denoise the data samples over a large number of steps to approximate the original samples from the given dataset. Consider a data sample Z0, where subscript 0 denotes the original clean data sample, to which we gradually add Gaussian noise over a number of steps {1, 2, 3, …, N} to obtain Z N When N is large enough, we can assume that the final noise sample Z N Has a standard normal distribution The joint distribution of noise samples is given by: in, is called the transition probability, and α i <1 is a fixed sequence of scalars that determines the amount of noise added and the amount of signal removed at each step. Due to the mathematical simplicity of the corruption process (i.e., it is linear because at each step we simply add noise to the samples from the previous step), the following equation can be used to sample a sample of the partially corrupted data at any arbitrary step i:

[0017] To generate data samples, we need to learn the reverse destruction process, which starts from p(z N )~N(0,I) extracted noise samples. This is also an iterative process, where N The noise starts to be gradually removed, and the joint distribution of the reverse process trajectory is given by: in, and μ θ(z i+1 , i+1) is the learning function approximated by a deep neural network parameterized by weights θ. In order to train the network and learn μ θ (z i+1 , i+1), the DDPM framework optimizes the lower bound of the log-likelihood of the data sample. The following two loss functions can be derived from the log-likelihood limit: (i) μ θ (z i+1 , i+1) and the target The mean square error between (ii)∈ θ (z i+1 , i+1) and use formula (2) to obtain Z from Z0 i The mean square error between the desired noise ε and the predicted noise ∈ θ The best performance is achieved when the above loss function (ii) is used between ε and the actual noise ε.

[0018] The process for training diffusion models as described above leads to so-called unconditional diffusion models. These models can learn the data distribution, and once trained using equation (3) above, a back-diffusion process can be used to diffuse from the original data set to approximate data samples. However, in unconditional diffusion models, there is no mechanism to control or guide this back-diffusion process. For example, this mechanism is needed to obtain samples from a specific class or corresponding to a certain user-provided input. One way to guide the diffusion model is to regulate the back-diffusion process using some input that the user can provide, which is first processed by another neural network and then fed to the model ∈ at each step. θ (z i+1 , i+1, c). Here, c=f φ (user input) is fed to the model ∈ θ The adjustment input, φ is calculated ∈ θ The network weights are conditioned on the inputs, and the user input can be anything the user chooses to provide to the diffusion process, such as a text prompt, an image, or data from another sensor.

[0019] The disclosed embodiments use the mathematics of diffusion modeling as described above to generate higher resolution radar point cloud data equivalent to using a higher resolution radar sensor. Consider one such radar point cloud sample as x0, using the same notation as in the description above. A radar point cloud can be represented as a list with dimensions (N, m), where N is the number of points in the point cloud and m is the length of the feature vector corresponding to each point. In the most basic setting, we would have a feature vector of length m=5 per point consisting of the following real numbers: (x, y, z, r v, rcs), where (x, y, z) form the 3D Cartesian coordinates of the point, r v is the signed scalar radial velocity of the point, and rcs is a real number in the interval [0, 1] that depends on the properties of the object from which the radar reflection occurred. Another popular way to represent point cloud data is to use the so-called distance map, which is a tensor of radar data obtained through a voxelization process. The distance values ​​are obtained by converting the Cartesian (x, y, z) radar reflections into spherical coordinates of (range, azimuth, elevation). The distance values ​​are used to obtain a distance map by discretizing the azimuth and elevation coordinates of the reflection. The voxelization process also converts the data from a list of N points (i.e., an irregular grid) to a regular grid, where the rows represent the discretized elevation axis and the columns represent the discretized azimuth axis. The advantage of the regular grid representation is that it enables one to use convolutional neural network (CNN)-based architectures developed in diffusion modeling studies for RGB image data. The disadvantage is that, depending on the discretization, some data may be lost. A finer grid will reduce this error, but may also result in a sparse grid, which will then require specialized architectures, such as sparse CNNs, for memory efficiency. In summary, the choice of data representation affects the conditional diffusion model ∈ θ (z i+1 , i+1, c). For list-based (irregular grid) representation, you can choose from the following architectures: PointNet++, Point-GNN, Pointpillars, and Point-Voxel CNN, while for distance map (regular grid) representation, you can choose AttentionU-Net or UViT architecture.

[0020] For training a diffusion model to perform super-resolution, the conditioning c can be a lower resolution radar point cloud subsampled from the corresponding higher resolution radar point cloud data sample x0, or input from a lower resolution radar sensor mounted on the same vehicle as the expensive higher resolution radar sensor, whose timestamp is closest to the higher resolution data sample. Using this conditioning, a diffusion model can be trained to generate a higher resolution radar point cloud given a lower resolution radar point cloud.

[0021] In the disclosed embodiments, conditioning need not be limited to a single modality. In addition to the low-resolution radar point cloud, data from other modalities may be used, such as RGB images from a camera, data from an event camera, and / or a sonar sensor mounted on the same vehicle. In this way, additional context may be provided that may potentially aid the diffusion process in determining the plausibility of predicted points in the high-resolution point cloud output.

[0022] The problem of performing super-resolution can also be viewed as a point cloud completion problem. The problem can be thought of as inpainting of a point cloud. Prior work has applied this idea to the shape completion problem on data from the ShapeNet dataset, where a diffusion model is given an incomplete point cloud of a specific shape (e.g., airplane, car, chair, etc.) and the task is to complete or fill in the rest of the point cloud. The proposed framework fixes the total number of points (N) in the point cloud and then freezes a subset of the points provided as input to the diffusion model. The rest of the points is randomly initialized by sampling from a single Gaussian distribution, and the diffusion model learns to diffuse only the unfrozen portions of the point cloud The frozen portion can be considered as a regulation of the diffusion process. Although the prior art does not provide any additional regulation input to the diffusion model, the disclosed embodiments may also consider providing RGB images from a camera, or data from an event camera, or data from a sonar sensor mounted to the vehicle as additional regulation input.

[0023] According to the disclosed embodiments, the diffusion model can be trained in the following manner: Starting with a radar point cloud dataset D, where samples from the dataset D are treated as higher resolution samples received from a higher resolution sensor; a deep neural network ∈ initialized with random weights θ θ ; Neural network, used to calculate the conditioned input c = f parameterized by weights φ φ ; and other samples, including radar point cloud samples from one or more sensors, having a lower resolution than the samples in dataset D. In some embodiments, the other samples may include RGB image data, LiDAR data, data from an event camera, and / or sonar data. Each of the other samples corresponds to a sample in dataset D. The following steps may be performed: i. Randomly sample mini-batches of higher resolution radar point cloud samples from D; ii. A random sampling step from {1, 2, 3, ..., N} of the same size as the mini-batch in step (i); iii. Use equation (2) to calculate the corresponding noise data sample for each sample in the mini-batch, where the amount of corruption depends on the sampling step of the corresponding mini-batch element; iv. For each element in the mini-batch, compute the corresponding conditioning input c from the corresponding low-resolution radar point cloud, and (if present) the corresponding RGB camera image, data from the event camera, and / or the corresponding sonar data; v. Using the results of steps (iii) and (iv), calculate ∈ for each mini-batch element θ (z i+1 , i+1, c), as described above; vi. Calculate the loss function, which is ∈ θ (z i+1 , i+1, c) and the noise ε used in step (iii) averaged over all mini-batch elements; vii. Calculate the gradient of the loss function with respect to the weights θ and update the weights in the opposite direction of the gradient scaled by a predetermined learning rate or learning rate scheduler; viii. Repeat the steps starting from (i) until the convergence criteria are met.

[0024] Figure 1 A flow chart of an example method 100 according to the disclosed embodiment is shown. According to the disclosed embodiment, the method 100 can be used to train a diffusion model. At operation 102, the method 200 samples a radar point cloud data set to generate a mini-batch of samples from the data set, wherein the radar point cloud data set corresponds to a first resolution. At operation 104, the method 200 calculates a noise data sample for each sample in the mini-batch samples. At operation 106, the method 200 calculates an adjustment input for each sample in the mini-batch, wherein the adjustment input is derived from a low-resolution radar point cloud sample corresponding to each sample in the mini-batch, wherein the low-resolution sample corresponds to a second resolution lower than the first resolution. At operation 108, the method 100 trains a diffusion model on the mini-batch samples and the adjustment input.

[0025] The diffusion model trained according to the disclosed embodiments can be used to generate higher resolution radar point clouds. For example, the disclosed embodiments can perform the following operations: i. Random sampling of noisy radar point cloud Its resolution is the same as the required clean higher resolution radar point cloud samples; ii. Compute the conditioned input c using input from a lower resolution sensor, any other low-cost sensor (e.g., RGB camera, event camera, or sonar, if available), and compute ∈ θ (z N , N, c); iii. Starting from i=N-1 to i=0, go back in reverse order and perform the following operations: a. Calculate ∈ θ (z i+1 , i+1, c); b. Use z i+1 and ∈ θ (z i+1 , i+1, c) calculate z i , which is the same as z using the DDPM equation i+1 Fewer data samples than noise; iv. The output at the end of the iterative process in step (iii) is z0, which is a clean higher resolution radar point cloud that corresponds to the input from the lower resolution radar sensor.

[0026] Figure 2 A flow chart of an example method 200 according to the disclosed embodiments is shown. According to the disclosed embodiments, the method 200 can be used to generate a higher resolution radar point cloud. At operation 202, the method 200 receives a first sample of a radar point cloud, the first sample corresponding to a first resolution, the first sample including a first noise level. At operation 204, the method 200 calculates an adjusted input from a radar point cloud corresponding to a second resolution, wherein the second resolution is lower than the first resolution. In addition, at operation 204, the method 200 can calculate additional adjusted inputs from other lower cost sensors (e.g., RGB cameras, event cameras, or sonar, if available). At operation 206, the method 200 applies the trained diffusion model to the first sample and the adjusted input to produce a second sample corresponding to the first resolution and including a noise level lower than the first noise level.

[0027] Figure 3 A block diagram of an example embodiment of a general purpose computer system 300 is shown. The computer system 300 may include a set of instructions that may be executed to cause the computer system 300 to perform any one or more of the methods or computer-based functions disclosed herein. For example, the computer system 300 may include an executable program to execute Figure 1 and Figure 2 The computer system 300 may be connected to other computer systems or peripheral devices via a network. In addition, the computer system 300 may include other computing devices or be included in other computing devices.

[0028] like Figure 3As shown, the computer system 300 may include one or more processors 302. The one or more processors 302 may include, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), or both. The computer system 300 may include a main memory 304 and a static memory 306, which may communicate with each other via a bus 308. As shown, the computer system 300 may also include a video display unit 310, such as a liquid crystal display (LCD), a projection television display, a flat panel display, a plasma display, or a solid-state display. In addition, the computer system 300 may include an input device 312, such as a remote control device with a wireless keyboard, a keyboard, a microphone coupled to a speech recognition engine, a camera such as a video camera or a static camera, or a cursor control device 314 such as a mouse device. The computer system 300 may also include a disk drive unit 316, a signal generating device 318 such as a speaker, and a network interface device 320. The network interface 320 may enable the computer system 300 to communicate with other systems via a network 328. For example, the network interface 320 may enable the machine learning system 120 to communicate with a database server (not shown) or a controller (not shown) in a manufacturing system.

[0029] In some embodiments, Figure 3 As shown, the disk drive unit 316 may include one or more computer-readable media 322, in which one or more sets of instructions 324, such as software, may be embedded. For example, the instructions 324 may embody one or more methods or functions, such as those disclosed herein. In certain embodiments, during execution by the computer system 300, the instructions 324 may reside completely or at least partially within the main memory 304, the static memory 306, and / or the processor 302. The main memory 304 and the processor 302 may also include computer-readable media.

[0030] In some embodiments, dedicated hardware implementations (e.g., application specific integrated circuits, programmable logic arrays, and other hardware devices) may be configured to implement one or more methods or functions described herein. The applications that may include the devices and systems of various embodiments may include a wide variety of electronic and computer systems. One or more embodiments described herein may use two or more specific interconnected hardware modules or devices with related control and data signals that may be transmitted between modules and through modules to implement functions, or as part of an application specific integrated circuit. Therefore, the present system includes software, firmware, and hardware implementations, or a combination thereof.

[0031] Although the computer-readable medium is shown as a single medium, the term "computer-readable medium" includes a single medium or multiple media, such as a centralized or distributed database, and / or associated caches and servers that store one or more sets of instructions. The term "computer-readable medium" should also include any medium that can store or encode a set of instructions for execution by a processor or cause a computer system to perform any one or more of the methods or functions disclosed herein.

[0032] In some embodiments, some or all of the computer-readable media will be non-transitory media. In a specific non-limiting exemplary embodiment, the computer-readable medium may include a solid-state memory, such as a memory card or other package that accommodates one or more non-volatile read-only memories. In addition, the computer-readable medium may be a random access memory or other volatile rewritable memory. In addition, the computer-readable medium may include a magneto-optical or optical medium, such as a disk or tape or other storage device, to capture a carrier signal, such as a signal transmitted through a transmission medium.

[0033] The disclosed embodiments can be used by receiving sensor signals from one or more specific sensors (i.e., lower resolution radar sensors) to calculate control signals for controlling a physical system (e.g., a computer-controlled machine such as a robot or autonomous vehicle). This is done by generating a higher resolution output conditioned on a lower resolution input via a learning iterative denoising process. The final result can then be fed to downstream detection and path planning algorithms for safety planning and control. The present invention can also be extended to receive signals from other sensors (e.g., cameras or sonars), which can be combined with lower resolution radar inputs to condition the generation of higher resolution radar outputs. In summary, the main application of the present invention is to generate radar point cloud data, which is equivalent to using a higher resolution radar sensor, but at the cost of more computation and using inputs from (one or more) lower resolution sensors.

[0034] Another application is to significantly limit the number of expensive higher resolution radar sensors required for the operation of a fleet of autonomous agents. A single vehicle equipped with higher resolution radar sensor(s) as well as a camera and lower resolution radar can potentially be used to collect data that can then be used to train a diffusion model to generate data produced by the higher resolution radar sensor(s) using only input from the lower resolution radar sensor and camera or sonar (if available). This is therefore expected to significantly save hardware costs for training and operating a fleet of autonomous vehicles that would be equipped only with lower resolution sensors and / or other low cost sensors.

[0035] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms included in the claims. The words used in the specification are descriptive rather than restrictive, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously mentioned, the features of the various embodiments can be combined to form other embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments have been described as providing advantages or preferred over other embodiments or prior art implementations in terms of one or more desired characteristics, it should be recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties include, but are not limited to, strength, durability, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Therefore, embodiments described as being less ideal than other embodiments or prior art implementations for one or more features are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method for training a diffusion model, comprising: Sampling a radar point cloud dataset to generate a small batch of samples from the dataset, wherein the radar point cloud dataset corresponds to a first resolution; For each sample in the mini-batch, calculate the noise data sample; computing a conditioning input for each sample in the mini-batch, wherein the conditioning input is derived from a low-resolution radar point cloud sample corresponding to each sample in the mini-batch, wherein the low-resolution sample corresponds to a second resolution lower than the first resolution; and The diffusion model is trained on mini-batches of samples and conditioned inputs.

2. The method according to claim 1, wherein: The conditioning input is also derived from data other than the low resolution radar point cloud samples.

3. The method according to claim 2, wherein: Additional data includes RGB images, data from an event camera, LiDAR data, or sonar data.

4. The method according to claim 1, wherein: Calculating the conditioning input is performed by a neural network.

5. The method according to claim 3, wherein: Calculating the conditioning input is performed by a neural network.

6. A method comprising: receiving a first sample of a radar point cloud, the first sample corresponding to a first resolution, the first sample comprising a first noise level; conditioning input from a radar point cloud calculation corresponding to a second resolution, wherein the second resolution is lower than the first resolution; and The trained diffusion model is applied to the first samples and a conditioning input to produce second samples corresponding to the first resolution and including a noise level lower than the first noise level.

7. The method according to claim 6, wherein: The conditioning input is also calculated from data outside the radar point cloud corresponding to the second resolution.

8. The method according to claim 7, wherein: Additional data includes RGB images, data from an event camera, LiDAR data, or sonar data.

9. The method according to claim 6, wherein: Calculating the conditioning input is performed by a neural network.

10. The method according to claim 8, wherein: Calculating the conditioning input is performed by a neural network.

11. A system comprising: one or more processors; and A memory including processor-executable instructions that, when executed by one or more processors, cause the system to perform operations comprising: Sampling a radar point cloud dataset to generate a small batch of samples from the dataset, wherein the radar point cloud dataset corresponds to a first resolution; For each sample in the mini-batch, calculate the noise data sample; computing a conditioning input for each sample in the mini-batch, wherein the conditioning input is derived from a low-resolution radar point cloud sample corresponding to each sample in the mini-batch, wherein the low-resolution sample corresponds to a second resolution lower than the first resolution; and The diffusion model is trained on mini-batches of samples and conditioned inputs.

12. The system according to claim 11, wherein: The conditioning input is also derived from data other than the low resolution radar point cloud samples.

13. The method according to claim 12, wherein: Additional data includes RGB images, data from an event camera, LiDAR data, or sonar data.

14. The method according to claim 11, wherein: Calculating the conditioning input is performed by a neural network.

15. The method according to claim 13, wherein: Calculating the conditioning input is performed by a neural network.

Images