A radar sensor data synthesis method and system based on diffusion model

Abstract

The invention discloses a radar sensor data synthesis method and system based on a diffusion model, which relates to the technical field of radar signal processing. A real picture is input into a generative adversarial network to output high-quality synthetic radar data. The generative adversarial network is trained as follows: a real picture and real radar data corresponding to the real picture are obtained; an encoder is used to process the real picture to obtain a semantic vector; forward diffusion processing is performed on the real radar data, and state data when standard Gaussian noise is added at each step is recorded; the semantic vector and the state data at each step are input into a decoder of a generator to generate synthetic radar data; the real radar data and the synthetic radar data are input into a discriminator to generate a real radar feature vector and a predicted radar feature vector. The radar sensor data synthesis method and system can generate realistic radar data under a variety of complex environments, and provide reliable data support for intelligent systems.

Description

Technical Field

[0001] The present invention relates to the field of radar signal processing technology, and in particular to a radar sensor data synthesis method and system based on a diffusion model. Background Art

[0002] Radar sensors play a key role in modern technology, especially in areas such as autonomous driving, intelligent transportation systems, and drone navigation. Radar sensors measure the distance, speed, and angle of objects by emitting electromagnetic waves and receiving reflected waves. Compared with other sensors, radar has the ability to work in various weather conditions and lighting environments, making it widely favored in practical applications. However, obtaining a large amount of high-quality radar sensor data to train and test various intelligent systems is a time-consuming and expensive task, which limits the promotion and optimization of many applications.

[0003] In the field of autonomous driving, radar sensors are used to detect vehicles, pedestrians, obstacles, etc. in the surrounding environment. These data, combined with data from cameras and lidar, can provide a 360-degree field of view of the vehicle, helping to achieve functions such as path planning, obstacle avoidance, and lane keeping. However, data collection in actual road environments requires a lot of manpower and material resources, and data acquisition in different weather and road conditions is particularly difficult. In addition, in order to capture enough environmental changes and complex scenes, a large amount of data must be collected under various conditions, which further increases costs and complexity.

[0004] Intelligent transportation systems rely on a large amount of sensor data to monitor and manage traffic flow, optimize traffic signal control, and provide real-time traffic information services. Radar sensors have obvious advantages in these applications and can work normally under adverse weather conditions. However, to cover a wide range of traffic networks, a large number of radar sensors need to be deployed, which is not only costly but also complex to maintain. Therefore, how to effectively obtain and utilize high-quality radar data has become an urgent problem to be solved.

[0005] Drone navigation is another area that is highly dependent on radar data. When performing tasks such as logistics distribution, environmental monitoring, and disaster relief, drones need to rely on radar data for navigation and obstacle avoidance. Drone operating environments are varied, including complex terrains such as cities, mountains, and forests. In order to ensure that drones operate safely and reliably in various environments, a large amount of high-quality radar data is required for training and testing. However, collecting this data in the field faces many technical and operational challenges, such as the diversity and uncertainty of the environment, resulting in missing, incomplete, noisy, and sparse radar data, which makes the quality of radar data low. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a radar sensor data synthesis method and system based on a diffusion model, which has high computational efficiency and generation stability, can generate realistic radar data in a variety of complex environments, and provide reliable data support for intelligent systems.

[0007] The present invention proposes a radar sensor data synthesis method based on a diffusion model, which inputs real images into a generative adversarial network to output high-quality synthetic radar data;

[0008] The training of the generative adversarial network is as follows:

[0009] Step 1: Obtain real images and real radar data corresponding to the real images to construct a training data set;

[0010] Step 2: Use the encoder to linearly embed the real picture and encode the position information to obtain a high-dimensional feature vector. Use the convolutional neural network to extract the local features of the high-dimensional feature vector, and combine the attention mechanism to capture the global features of the high-dimensional feature vector to map the high-dimensional feature vector to a semantic vector.

[0011] Step 3: Perform forward diffusion processing on the real radar data, gradually add standard Gaussian noise until the real radar data evolves into pure noise, and record the state data when adding standard Gaussian noise at each step;

[0012] Step 4: input the semantic vector and the state data when adding standard Gaussian noise at each step into the decoder of the generator, and the decoder generates synthetic radar data from the semantic vector and the state data when adding standard Gaussian noise at each step under the guidance of real picture conditions;

[0013] Step 5: Input the real radar data and the synthetic radar data into the discriminator, and construct a loss function based on the generated real radar feature vector and the predicted radar feature vector combined with the KL divergence to optimize the trainable parameters of the generative adversarial network.

[0014] Furthermore, in step 2, the encoder is used to linearly embed the real image and encode the position information to obtain a high-dimensional feature vector, specifically:

[0015] The real image is obtained through the visual sensor, and the real image is divided into image blocks of fixed size. Each image block is converted into a linearly embedded feature vector after linear embedding;

[0016] Add position information to each image block and use the cosine function to generate a set of position codes;

[0017] The positional encoding is added to the linearly embedded feature vector to obtain a high-dimensional feature vector.

[0018] Furthermore, in step three, the real radar data is forward diffused and standard Gaussian noise is gradually added until the real radar data evolves into pure noise, specifically:

[0019] Using the forward diffusion process of the diffusion model, the real radar data 0 to Step 1 adds standard Gaussian noise to obtain the noise-added data set , For the The noisy data is obtained by adding standard Gaussian noise. Approximately obeys the mean of 0 and the covariance matrix is ​​the unit matrix Multivariate Gaussian noise distribution of , and as the real radar data evolves into pure noise.

[0020] Furthermore, pure noise The generation formula is as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] in, is the cumulative noise scaling factor, representing the noise from step 1 to step The cumulative noise impact of the steps, For the The noise scaling factor of the step, No. The noise factor of the step, Indicates The index of the step to add standard Gaussian noise.

[0025] Furthermore, in step 4, the decoder is a conditional diffusion model, which consists of a semantic vector carrying the real image features. Guidance, prediction of noise Defined as:

[0026] ;

[0027] in, In the forward diffusion process The state data when standard Gaussian noise is added step by step, are the learnable parameters of the generator, is the semantic vector, Represents the conditional diffusion model based on the current state data and semantic vector To predict the output of the noise, Indicates the current status data The gradient of represents the distribution of the conditional diffusion model, For Step to The noise scaling factor of the step accumulation;

[0028] The formula for generating synthetic radar data is:

[0029] ;

[0030] ;

[0031] in, The synthetic radar data.

[0032] Furthermore, in step 5, the loss function The details are as follows:

[0033]

[0034] in, is the feature vector dimension, and They are The real radar feature vector and predicted radar feature vectors, represents the hyperparameter, Indicates that in the conditional training step and State data when standard Gaussian noise is added The expectations below, represents the distribution of the conditional diffusion model, Table 1 shows the distribution of real image features. represents the KL divergence.

[0035] A radar sensor data synthesis system based on a diffusion model that inputs real images into a generative adversarial network to output high-quality synthetic radar data;

[0036] The training of the generative adversarial network is as follows:

[0037] Step 1: Obtain real images and real radar data corresponding to the real images to construct a training data set;

[0038] Step 2: Use the encoder to linearly embed the real picture and encode the position information to obtain a high-dimensional feature vector. Use the convolutional neural network to extract the local features of the high-dimensional feature vector, and combine the attention mechanism to capture the global features of the high-dimensional feature vector to map the high-dimensional feature vector to a semantic vector.

[0039] Step 3: Perform forward diffusion processing on the real radar data, gradually add standard Gaussian noise until the real radar data evolves into pure noise, and record the state data when adding standard Gaussian noise at each step;

[0040] Step 4: input the semantic vector and the state data when adding standard Gaussian noise at each step into the decoder of the generator, and the decoder generates synthetic radar data from the semantic vector and the state data when adding standard Gaussian noise at each step under the guidance of real picture conditions;

[0041] Step 5: Input the real radar data and the synthetic radar data into the discriminator, and construct a loss function based on the generated real radar feature vector and the predicted radar feature vector combined with the KL divergence to optimize the trainable parameters of the generative adversarial network.

[0042] The advantages of the radar sensor data synthesis method and system based on the diffusion model provided by the present invention are: it aims to solve the problems of missing, incomplete, noisy, sparsity, etc. in radar data, and improve the real-time and effectiveness of data synthesis; extract real picture features through the encoder, combine forward diffusion and reverse denoising processes to generate high-authenticity radar data, and optimize the generation model through the discriminator. This method not only improves the authenticity and diversity of the generated data, but also performs well in computational efficiency and generation stability. It has broad application prospects, especially in the fields of autonomous driving, intelligent transportation systems and drone navigation, and can significantly improve the system's environmental perception ability and operational safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of the present invention;

[0044] Figure 2 Training flowchart for generating adversarial networks. DETAILED DESCRIPTION

[0045] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0046] like Figure 1 and 2 As shown, the present invention proposes a radar sensor data synthesis method based on a diffusion model, which inputs a real picture into a generative adversarial network to output high-quality synthetic radar data;

[0047] The training of the generative adversarial network is as follows:

[0048] Step 1: Obtain real images and real radar data corresponding to the real images to construct a training data set;

[0049] High-resolution visual sensors (such as cameras) are used to collect data from real scenes. The collected images cover a variety of environments, such as day, night, sunny, rainy, etc., to ensure the diversity and adaptability of real pictures and real radar data.

[0050] Step 2: Use the encoder to linearly embed the real picture and encode the position information to obtain a high-dimensional feature vector. Use the convolutional neural network to extract the local features of the high-dimensional feature vector, and combine the attention mechanism to capture the global features of the high-dimensional feature vector to map the high-dimensional feature vector to a semantic vector.

[0051] First, the encoder is used to extract features from real images, including color features, texture features, context features, and global features, as follows:

[0052] The encoder collects data of real scenes through visual sensors and divides real images into fixed-size image blocks (patches), for example, 16×16 pixel blocks. Each image block is converted into a linearly embedded feature vector through linear embedding processing, and position information is added to each image block. A set of position codes is generated using the cosine function, and the position code is added to the linearly embedded feature vector to obtain a high-dimensional feature vector for the encoder to use.

[0053] Linearly embedded feature vector Generation of: ,in, is the weight matrix, is the bias vector, is the image block after segmentation of the real image.

[0054] In generating a set of position codes using the cosine function, assuming the position is , the dimension is , position encoding The formula is:

[0055]

[0056]

[0057] in, is the position index, is the dimension index of the positional encoding. Combine the positional encoding with the linearly embedded feature vector: ;in is a high-dimensional feature vector, which is then input to the encoder.

[0058] The encoder is implemented by a convolutional neural network (CNN) with an attention mechanism. CNN is mainly composed of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. It can extract color features, texture features, context features, and global features. The convolutional layer is used to extract local features, while the pooling layer is used to reduce dimensions and prevent overfitting. The encoder uses a convolutional neural network (CNN) to extract local features of the image and combines it with an attention mechanism to capture global dependencies. Finally, the encoder converts the input high-dimensional feature vector into Mapping to semantic vectors :

[0059]

[0060] in, For the encoder, is a learnable parameter, It contains all the extracted features of the real image and provides data input for the generation of synthetic radar data in the subsequent decoder.

[0061] Step 3: Perform forward diffusion processing on the real radar data, gradually add standard Gaussian noise until the real radar data evolves into pure noise, and record the state data when adding standard Gaussian noise at each step;

[0062] A set of real radar data is selected as the initial data. These data cover a variety of scenarios and conditions to ensure the generalization ability of the generated model. The forward diffusion process of the diffusion model is used to 0 to Step 1 adds standard Gaussian noise to obtain the noise-added data set , For the The noisy data is obtained by adding standard Gaussian noise. Approximately obeys the mean of 0 and the covariance matrix is ​​the unit matrix Multivariate Gaussian noise distribution of , and as the real radar data evolves into pure noise.

[0063] The noise adding process consists of a series of noise factors Parameterized, pure noise The generation formula is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] in, , is the cumulative noise scaling factor, representing the noise from step 1 to step The cumulative effect of noise, For the The noise scaling factor of the step, For the The noise factor of the step, Indicates The index of the step to add standard Gaussian noise.

[0068] for ,according to , can be directly introduced , For the The noise factor of the step.

[0069] In this process, real radar data Gradually add noise, and after multiple iterations, it eventually evolves into pure noise The state data when adding standard Gaussian noise at each step is recorded for use in the reverse denoising process of the decoder. The diffusion process is usually divided into multiple steps (for example, 1000 steps), and a predefined proportion of noise is added in each step until the data is completely Gaussian noise. This multi-step iterative process ensures that the model can learn the gradual change process of the data distribution.

[0070] Step 4: input the semantic vector and the state data when adding standard Gaussian noise at each step into the decoder of the generator, and the decoder generates synthetic radar data from the semantic vector and the state data when adding standard Gaussian noise at each step under the guidance of real picture conditions;

[0071] Understandably, the real picture Obtain semantic vector through encoder The corresponding real radar data is obtained through the forward diffusion process to obtain the noise data group , that is to say, get Each state data is input into the decoder as a training step, and the semantic vector is used to represent the state data when standard Gaussian noise is added. Guide the generation of synthetic radar data, input the synthetic radar data generated in each training step and the real radar data into the discriminator for guided generation; that is, in the reverse diffusion, the U-net network is used to predict the first State data when standard Gaussian noise is added As the decoder in the denoising process During the entire training process, the denoising process can be guided based on real image training, so that the diffusion model can be applied to the generation of synthetic radar data in different scenarios.

[0072] First, the decoder receives the semantic vector obtained by the encoder , to obtain synthetic radar data The decoder is a conditional diffusion model consisting of a semantic vector carrying real image features. Guided by, using the real picture encoding information to obtain the distribution of new synthetic radar data. The prediction noise is defined as:

[0073]

[0074] in, In the forward diffusion process The state data when standard Gaussian noise is added step by step, are the learnable parameters of the generator, is the semantic vector, Represents the conditional diffusion model based on the current state data and semantic vector To predict the output of the noise, Indicates the current status data The gradient of represents the distribution of the conditional diffusion model, For Step to The noise scaling factor for the step accumulation.

[0075] In the forward diffusion process of step three, The trainable U-net network is used to predict the state data when standard Gaussian noise is added. ,in represents the learnable parameters, and the U-net network is a CNN-based image segmentation network. Based on this, given the first State data when standard Gaussian noise is added To predict synthetic radar data The process can be formulated as:

[0076] ;

[0077] ;

[0078] Specifically, the number of steps when adding standard Gaussian noise is used as the training step, and at each training step On the network, we use a trainable U-net network To predict the noise , For training step The decoder uses the conditional diffusion model to generate radar data by step-by-step denoising starting from the noise data. Each step predicts the noise component in the current data and removes it, gradually restoring the real radar data. Through multiple steps of reverse denoising, the final generated synthetic radar data Gradually approaching the original real radar data distribution.

[0079] The decoder uses a conditional diffusion model to pass the semantic vector of the real picture. Guidance Synthetic Radar Data In order to generate effective synthetic radar data in the conditional diffusion model, a suitable loss function is defined to optimize the conditional diffusion model. The loss is the conditional loss , is used to ensure that the generated synthetic radar data can match the conditional information (real image features), and the KL divergence is used to measure the difference between the distribution of generated data and the distribution of conditional information.

[0080] Conditional loss The definition is as follows: ;

[0081] in is the distribution of the conditional diffusion model, that is, given the condition Time semantic vector The posterior distribution of is the distribution of real image features, that is, the semantic vector The prior distribution of Indicates that the condition and The expectations below, is the KL divergence.

[0082] Step 5: Input the real radar data and the synthetic radar data into the discriminator, and construct a loss function based on the generated real radar feature vector and the predicted radar feature vector combined with the KL divergence to optimize the trainable parameters of the generative adversarial network.

[0083] The discriminator is used to compare the real radar data With synthetic radar data , the discriminator is implemented by a one-dimensional convolutional neural network (DCNN), which mainly consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer, which can be expressed as a function ,in is the input radar data, and the discriminator outputs a feature vector:

[0084] ;

[0085] in, and are the real radar feature vector and the predicted radar feature vector respectively.

[0086] In order to measure the difference between real radar data and synthetic radar data, a discriminative loss is defined , using mean square error (MSE) as the discriminant loss , represents the difference between real radar data and synthetic radar data; discriminant loss The formula is:

[0087] ;

[0088] The loss function set in this embodiment is combined with the discriminant loss With conditional loss , to measure the difference between the real radar feature vector and the predicted radar feature vector under conditional information constraints: that is, the loss function as follows:

[0089] ;Right now,

[0090] ;

[0091] in, is the feature vector dimension, and They are The real radar feature vector and predicted radar feature vectors, represents a hyperparameter used to balance the reconstruction loss and the conditional loss to generate radar data more effectively.

[0092] By calculating the loss function , we can get the difference measure between real radar data and synthetic radar data. We can get the minimization loss function , thereby improving the authenticity of the generated data and obtaining the final synthetic radar data . Back propagation process:

[0093] ;

[0094] in are the parameters of the generator, is the learning rate; by continuously optimizing the parameters of the generator, the generator can generate more and more realistic synthetic radar data, thereby obtaining the final synthetic radar data During the training process, the generator and the discriminator conduct adversarial training, the generator tries to generate more realistic data, and the discriminator continuously improves its ability to distinguish.

[0095] Through steps one to five, high-quality radar sensor data synthesis is achieved. The encoder extracts real image features, combines forward diffusion and reverse denoising processes to generate highly realistic radar data, and optimizes the generated model through the discriminator. This method not only improves the authenticity and diversity of generated data, but also performs well in terms of computational efficiency and generation stability.

[0096] The present invention realizes high-quality radar sensor data synthesis through the above steps. The encoder extracts real picture features, combines forward diffusion and reverse denoising processes to generate high-authenticity radar data, and optimizes the generation model through the discriminator. This method not only improves the authenticity and diversity of the generated data, but also performs well in terms of computational efficiency and generation stability. It has broad application prospects, especially in the fields of autonomous driving, intelligent transportation systems, and drone navigation, and can significantly improve the system's environmental perception ability and operational safety.

[0097] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A radar sensor data synthesis method based on a diffusion model, characterized in that: Input real images into the generative adversarial network to output high-quality synthetic radar data; The training of the generative adversarial network is as follows: Step 1: Obtain real images and real radar data corresponding to the real images to construct a training data set; Step 2: Use the encoder to linearly embed the real picture and encode the position information to obtain a high-dimensional feature vector. Use the convolutional neural network to extract the local features of the high-dimensional feature vector, and combine the attention mechanism to capture the global features of the high-dimensional feature vector to map the high-dimensional feature vector to a semantic vector. Step 3: Perform forward diffusion processing on the real radar data, gradually add standard Gaussian noise until the real radar data evolves into pure noise, and record the state data when adding standard Gaussian noise at each step; Step 4: input the semantic vector and the state data when adding standard Gaussian noise at each step into the decoder of the generator, and the decoder generates synthetic radar data from the semantic vector and the state data when adding standard Gaussian noise at each step under the guidance of real picture conditions; Step 5: Input the real radar data and the synthetic radar data into the discriminator, and construct a loss function based on the generated real radar feature vector and the predicted radar feature vector in combination with the KL divergence to optimize the trainable parameters of the generative adversarial network; Loss Function The details are as follows: in, is the feature vector dimension, and They are The real radar feature vector and predicted radar feature vectors, represents the hyperparameter, Indicates that in the conditional training step and State data when standard Gaussian noise is added The expectations below, represents the distribution of the conditional diffusion model, Table 1 shows the distribution of real image features. represents the KL divergence.

2. The radar sensor data synthesis method based on the diffusion model according to claim 1 is characterized in that: In step 2, the encoder is used to linearly embed the real image and encode the position information to obtain a high-dimensional feature vector, specifically: The real image is obtained through the visual sensor, and the real image is divided into image blocks of fixed size. Each image block is converted into a linearly embedded feature vector after linear embedding; Add position information to each image block and use the cosine function to generate a set of position codes; The positional encoding is added to the linearly embedded feature vector to obtain a high-dimensional feature vector.

3. The radar sensor data synthesis method based on the diffusion model according to claim 1 is characterized in that: In step 3, the real radar data is forward diffused and standard Gaussian noise is gradually added until the real radar data evolves into pure noise. Specifically: Using the forward diffusion process of the diffusion model, the real radar data 0 to Step 1 adds standard Gaussian noise to obtain the noise-added data set , For the The noisy data is obtained by adding standard Gaussian noise. Approximately obeys the mean of 0 and the covariance matrix is ​​the unit matrix Multivariate Gaussian noise distribution of , and as the real radar data evolves into pure noise.

4. The radar sensor data synthesis method based on the diffusion model according to claim 3 is characterized in that: Pure noise The generation formula is as follows: ； ； ； in, is the cumulative noise scaling factor, representing the noise from step 1 to step The cumulative noise impact of the steps, For the The noise scaling factor of the step, No. The noise factor of the step, Indicates The index of the step to add standard Gaussian noise.

5. The radar sensor data synthesis method based on the diffusion model according to claim 1, characterized in that: In step 4, the decoder is a conditional diffusion model consisting of a semantic vector carrying real image features. Guidance, prediction of noise Defined as: ； in, In the forward diffusion process The state data when standard Gaussian noise is added step by step, are the learnable parameters of the generator, is the semantic vector, Represents the conditional diffusion model based on the current state data and semantic vector To predict the output of the noise, Indicates the current status data The gradient of represents the distribution of the conditional diffusion model, For Step to The noise scaling factor of the step accumulation; The formula for generating synthetic radar data is: ； ； in, The synthetic radar data.

6. A radar sensor data synthesis system based on a diffusion model, characterized in that: Input real images into the generative adversarial network to output high-quality synthetic radar data; The training of the generative adversarial network is as follows: Step 1: Obtain real images and real radar data corresponding to the real images to construct a training data set; Step 2: Use the encoder to linearly embed the real picture and encode the position information to obtain a high-dimensional feature vector. Use the convolutional neural network to extract the local features of the high-dimensional feature vector, and combine the attention mechanism to capture the global features of the high-dimensional feature vector to map the high-dimensional feature vector to a semantic vector. Step 3: Perform forward diffusion processing on the real radar data, gradually add standard Gaussian noise until the real radar data evolves into pure noise, and record the state data when adding standard Gaussian noise at each step; Step 4: input the semantic vector and the state data when adding standard Gaussian noise at each step into the decoder of the generator, and the decoder generates synthetic radar data from the semantic vector and the state data when adding standard Gaussian noise at each step under the guidance of real picture conditions; Step 5: Input the real radar data and the synthetic radar data into the discriminator, and construct a loss function based on the generated real radar feature vector and the predicted radar feature vector in combination with the KL divergence to optimize the trainable parameters of the generative adversarial network; Loss Function The details are as follows: in, is the feature vector dimension, and They are The real radar feature vector and predicted radar feature vectors, represents the hyperparameter, Indicates that in the conditional training step and State data when standard Gaussian noise is added The expectations below, represents the distribution of the conditional diffusion model, Table 1 shows the distribution of real image features. represents the KL divergence.

Images