Method for generating multi-class special vehicle sar image based on denoising diffusion model

Abstract

This invention discloses a method for generating multi-category SAR images of special vehicles based on a denoising diffusion model. Specifically, the method comprises: Step 1, acquiring samples of special vehicle SAR images and preprocessing them; Step 2, creating category labels for the preprocessed special vehicle SAR images from Step 1 according to their categories, and converting the category labels into learnable category encoding vectors; Step 3, obtaining the time-step encoding vector at time t; adding noise to the special vehicle SAR images with the pre-defined category labels from Step 2; Step 4, feeding the time-step encoding vector at time t, the noisy special vehicle SAR images, and the category encoding vector into a self-attention U-Net network for training to obtain a denoising diffusion model; Step 5, generating special vehicle SAR images based on the denoising diffusion model. This invention solves the problems of poor stability during training and excessively large network parameters in existing technologies using adversarial mechanisms to generate special vehicle SAR images, which are prone to mode collapse.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to a method for generating SAR images of multiple types of special vehicles based on a denoising diffusion model. Background Technology

[0002] Synthetic Aperture Radar (SAR) is a widely used technology for long-range sensing and image acquisition. It generates high-resolution images of the Earth's surface by emitting microwave signals and receiving reflected signals. SAR images have important applications in fields such as feature identification, geographic information systems, environmental monitoring, and astronomy. However, in the field of feature identification, the acquisition of SAR images from special vehicles is often subject to several limitations, specifically: First, SAR images must be acquired using radar equipment mounted on an aircraft, making the data acquisition process expensive and complex; second, due to the rarity and confidentiality of special vehicles, data acquisition is strictly limited and cannot be carried out arbitrarily, further hindering the possibility of large-scale SAR image acquisition; furthermore, certain environments, such as severe weather, atmospheric interference, and nighttime lighting, may limit the quality and availability of SAR images from special vehicles, leading to uncertainty in data acquisition. Moreover, the missions of special vehicles often need to be performed in complex and challenging environments, further emphasizing the need for high-quality SAR images from special vehicles. These difficulties and limitations have led to the finite nature of SAR image datasets for special vehicles, which in turn poses a significant challenge to deep learning models in SAR image recognition and detection tasks for ground features.

[0003] With the development of deep learning, remarkable progress has been made in the field of SAR image generation, giving rise to a number of outstanding algorithms. Among them, generation algorithms based on Generative Adversarial Networks (GANs) have gained favor among many researchers. GANs consist of a generator network and a discriminator network. Through adversarial training, the generator and discriminator jointly optimize, eventually reaching Nash equilibrium, making the generated data indistinguishable from real data. GANs can learn complex mappings to generate realistic images from noisy data, which is very useful for generating SAR images. Existing GAN-based SAR image generation algorithms include: AttentionResidual-Wasserstein Generative Adversarial Network (AR-WGAN) and Multi-Scale Discriminator Generative Adversarial Network (Multi-GAN), both of which have achieved good results in the generation of special vehicle SAR images. However, due to their unique adversarial training method, their training process is often unstable. Furthermore, due to excessive network depth and inappropriate activation function selection, they are highly susceptible to gradient explosion or vanishing gradients, leading to mode collapse. In addition, because GAN networks include both generators and discriminators, the number of network model parameters is typically large, resulting in excessively long training times. These challenges limit their application in high-quality special vehicle SAR image generation tasks. Even with the addition of multiple components to stabilize training, it remains difficult to fundamentally solve these problems caused by the training mechanism.

[0004] The denoising diffusion model is a type of generative model that has attracted widespread attention in recent years due to its advantages such as high generation quality, strong diversity, and stable training. It comprises two main process domains: forward diffusion and inverse denoising, such as... Figure 1As shown. In the forward diffusion stage, noise distribution is gradually introduced into the image until the image becomes completely random noise. In the reverse denoising stage, a series of Markov chains are used to gradually remove noise at each time step, thereby recovering the image from Gaussian noise. Diffusion models are widely used in the field of optical image generation due to their strong pattern coverage and the quality of generated samples. However, in the field of SAR image generation, denoising diffusion models have a series of limitations: first, for SAR images severely affected by speckle noise, the global information of the feature map is ignored during the denoising training process; second, it cannot achieve accurate generation of SAR images of specific categories. This means that it may perform poorly when dealing with specific application scenarios or tasks. Based on the above analysis, the shortcomings of existing methods are:

[0005] (1) The method of generating SAR images of special vehicles using GAN has low stability during training due to its special adversarial training mechanism, and is prone to mode collapse. In addition, due to its complex network structure, the number of computational parameters is too large.

[0006] (2) With the widespread application of GAN, researchers have expanded the SAR image dataset for special vehicles using GAN-based methods. However, these methods all generate images for a single target category. To generate images for multiple categories, multiple networks must be trained.

[0007] (3) Existing denoising diffusion models are only applied to optical images. In the denoising training process for special vehicle SAR images that are severely affected by speckle noise, the global information of the feature map is ignored and the accurate generation of SAR images of specific types of special vehicles cannot be achieved. Summary of the Invention

[0008] The purpose of this invention is to provide a method for generating SAR images of multiple categories of special vehicles based on a denoising diffusion model. This method solves the problems in the prior art where GANs using adversarial mechanisms to generate SAR images of special vehicles are prone to mode collapse due to poor stability during the training process and excessive network parameters.

[0009] The technical solution adopted in this invention is a method for generating SAR images of multiple categories of special vehicles based on a denoising diffusion model, which is implemented according to the following steps:

[0010] Step 1: Obtain samples of SAR images of special vehicles and preprocess them;

[0011] Step 2: Create category labels for the special vehicle SAR images preprocessed in Step 1 according to the categories, and convert the category labels into learnable category coding vectors;

[0012] Step 3: Obtain the time step encoding vector at time t; add noise to the special vehicle SAR image with the preset category label from Step 2;

[0013] Step 4: The time step encoding vector at time t, the noisy special vehicle SAR image, and the category encoding vector are fed into the self-attention U-Net network for training to obtain the denoising diffusion model.

[0014] Step 5: Generate SAR images of special vehicles based on the denoising diffusion model.

[0015] The invention is further characterized in that,

[0016] The specific steps for processing the sample data in step 1 are as follows:

[0017] The acquired SAR images of special vehicles are normalized, with pixel values ​​scaled to the range of -1 to 1. Then, they are centrally cropped to a size of 128*128, and finally, image enhancement is performed to obtain the original training image data.

[0018] The image enhancement specifically involves performing random brightness adjustment, random mirror flipping, and histogram equalization sequentially on the centrally cropped SAR image of special vehicles.

[0019] Step 2 is as follows:

[0020] For each type of special vehicle in the original training image data obtained in step 1, an index dictionary is created for the SAR images. The key value corresponding to each category index is transformed into a learnable category parameter tensor through the Embedding embedding layer function in the torch library. The category parameter tensor is then transformed into a learnable category encoding vector σ through the category encoding layer.

[0021] The category encoding layer consists of a fully connected layer, a SiLU activation layer, and another fully connected layer connected in sequence. The category parameter tensor is transformed into a learnable category encoding vector σ by passing through the fully connected layer, the SiLU activation layer, and the fully connected layer in sequence.

[0022] Step 3 specifically involves:

[0023] Step 3.1: Time t is obtained by discrete sampling within the range of 1 to T. The time step tensor is obtained by concatenating the sin and cosine of sin and cosine of t. Then, the time step encoding layer is passed through to output the time step encoding vector τ at time t.

[0024] Step 3.2: Randomly select a certain category of special vehicle SAR images processed in Step 2, and randomly sample Gaussian noise ε. Obtain the noisy special vehicle SAR image at time t using the noisy formula of the diffusion model, where the noisy formula is as follows:

[0025]

[0026] Where x0 is the SAR image of the special vehicle without noise after step 2, α t Let x be the variance parameter of the direct uniform sampling from 0.9999 to 0.98 generated by the linear scheduler, t∈(0,T). t The SAR image of the special vehicle after adding noise at time t.

[0027] The time-step encoding layer consists of a fully connected layer, a SiLU activation layer, and another fully connected layer connected in sequence. The time-step tensor is processed sequentially through the fully connected layer, the SiLU activation layer, and the fully connected layer to output the time-step encoded vector τ at time t.

[0028] Step 4 specifically involves:

[0029] Step 4.1: The image with added noise obtained in step 3.2 at time t is fed into the first convolutional layer of the encoder of the self-attention U-Net network for 3*3 convolution operation to obtain feature map F1;

[0030] Step 4.2: The category encoding vector σ obtained in Step 2 and the time step encoding vector τ obtained in Step 3.1 are fed into the fully connected layer of the encoder of the self-attention U-Net network to expand into a conditional encoding with the same dimension as the feature map F1 obtained in Step 4.1. The conditional encoding is then concatenated with the feature map F1 obtained in Step 4.1 to obtain a conditional feature map F2 containing conditional information.

[0031] Step 4.3: The conditional feature map F2 is fed into the remaining convolutional layers of the encoder in the self-attention U-Net network for further downsampling. After downsampling to its fourth convolutional layer, a self-attention module is introduced to construct the relationships between pixels through a self-attention mechanism, resulting in the encoded self-attention feature information F. encoder ;

[0032] Step 4.4: Encode the self-attention feature information F encoder The signal is fed into a decoder that is symmetrical to the encoder structure, and deconvolution is performed through the deconvolution block in the decoder to obtain the prediction noise ε′ at that moment.

[0033] Step 4.5: Calculate the mean squared error loss between the predicted noise ε′ and the Gaussian noise ε added to the image in step 3.2. This loss is used for backpropagation to update the network weights. The formula for calculating the mean squared error loss is as follows:

[0034] MSE_loss=(ε-ε′) 2

[0035] Where ε is the Gaussian noise randomly sampled at time t in step 3.2, and ε′ is the prediction of the Gaussian noise added at time t by the self-attention U-Net network;

[0036] Step 4.6: Randomly select any special vehicle SAR image processed in step 2 and its corresponding label, and repeat steps 4.1 to 4.5 at any sampling time. Set the learning rate to 0.0001, and train the network through multiple iterations until the preset number of iterations is reached to obtain the denoising diffusion model.

[0037] Step 5 specifically involves:

[0038] Step 5.1: Preset the initial iteration time T, and randomly sample Gaussian noise ε at the initial iteration time T and time T. T The specified category label is input into the denoising diffusion model trained in step 4.6, and the network output data is generated, i.e., the predicted noise ε′ at time T. T ;

[0039] Step 5.2, based on the network output data ε′ T Calculate the network input data ε at time T–1. T-1 , ε T-1 =ε T -ε′ T ;

[0040] After the calculation is completed, we enter time T–1, at which point the network input data should be ε. T-1 The specified category label, and the iteration time T–1;

[0041] Step 5.3: Repeat the above steps for a total of T iterations until the iteration time T is zero, and obtain the SAR image ε0 of the corresponding category of special vehicle.

[0042] The beneficial effects of this invention are:

[0043] (1) The present invention provides a method for generating multi-category special vehicle SAR images based on a denoising diffusion model. It uses a U-Net network based on self-attention to generate high-fidelity special vehicle SAR images. The number of network parameters is relatively small compared to SAR images generated based on GAN. Furthermore, the present invention eliminates the adversarial training method, which improves stability and solves the problems of unstable training process, easy mode collapse, and large number of parameters in the training process of SAR image generation by GAN.

[0044] (2) By embedding category coding during training, the model can automatically adjust the generated results according to the category information to meet the requirements of a specific category, thus achieving the effect of generating SAR images of special vehicles of a specific category, thereby avoiding the complexity and resource consumption of training multiple independent models for each category.

[0045] (3) Based on the traditional denoising diffusion model, the present invention effectively processes special vehicle SAR images that are severely affected by speckle noise by introducing a Self-attention module, thereby improving the quality of the generated images. At the same time, a category label encoding layer is introduced outside the backbone U-Net network, enabling the network to read label information during training and achieve accurate generation of SAR images of specific categories of special vehicles. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the diffusion trajectories of forward and reverse diffusion in the existing denoising diffusion model;

[0047] Figure 2 This is a flowchart of the method for generating SAR images of multiple categories of special vehicles based on a denoising diffusion model according to the present invention;

[0048] Figure 3 This is a schematic diagram of the Self-attention module in the method for generating SAR images of multiple categories of special vehicles based on a denoising diffusion model in this invention.

[0049] Figure 4 This is a schematic diagram of a complete iterative training process of the self-attention U-Net network in the method for generating SAR images of multiple special vehicles based on the denoising diffusion model of the present invention.

[0050] Figure 5 This is a schematic diagram of the method for generating SAR images of special vehicles based on a denoising diffusion model according to the present invention, which generates SAR images of special vehicles through a complete iterative inference process.

[0051] Figure 6a This is a schematic diagram of samples of category T72 in the MSTAR special vehicle dataset used in Embodiment 2 of the present invention;

[0052] Figure 6b This is a schematic diagram of samples of category 762 in the MSTAR special vehicle dataset used in Embodiment 2 of the present invention;

[0053] Figure 6c This is a schematic diagram of the sample of category D7 in the MSTAR special vehicle dataset used in Embodiment 2 of the present invention;

[0054] Figure 7a This is a schematic diagram of generating the T72 sample in Embodiment 3 of the present invention;

[0055] Figure 7b This is a schematic diagram of generating the T62 sample in Embodiment 3 of the present invention;

[0056] Figure 7c This is a schematic diagram of generating the D7 sample in Embodiment 3 of the present invention;

[0057] Figure 8 This is a comparison of the generation effects of the method for generating SAR images of multiple categories of special vehicles based on the denoising diffusion model of the present invention with the existing SAR image generation model based on GAN. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] Example 1

[0060] This invention relates to a method for generating SAR images of multiple categories of special vehicles based on a denoising diffusion model, the process of which is as follows: Figure 2 As shown, the specific steps are as follows:

[0061] Step 1: Acquire and preprocess SAR image samples of special vehicles. Specifically, the sample data processing involves: normalizing the acquired SAR images of special vehicles, scaling the pixel values ​​to the range of -1 to 1, then performing a central crop to a size of 128*128, followed by image enhancement to obtain the original training image data. The image enhancement process involves sequentially performing random brightness adjustment, random mirror flipping, and histogram equalization on the centrally cropped SAR images of special vehicles.

[0062] Step 2: Create category labels for the special vehicle SAR images preprocessed in Step 1 according to their categories, and convert the category labels into learnable category coding vectors; specifically:

[0063] For each class of special vehicles in the original training image data obtained in step 1, an index dictionary is created for the SAR images. The key value corresponding to each class index is transformed into a learnable class parameter tensor through the Embedding embedding layer function in the torch library. The class parameter tensor is then transformed into a learnable class encoding vector σ through the class encoding layer. The class encoding layer includes a fully connected layer, a SiLU activation layer, and a fully connected layer connected in sequence. The class parameter tensor is transformed into a learnable class encoding vector σ through the fully connected layer, the SiLU activation layer, and the fully connected layer in sequence.

[0064] Step 3: Obtain the time step encoding vector at time t; add noise to the special vehicle SAR image with the preset category label from Step 2; specifically:

[0065] Step 3.1: Time t is obtained by discrete sampling within the range of 1 to T. The time step tensor is obtained by concatenating the sine and cosine of sin and cosine, and then passed through the time step coding layer to output the time step coding vector τ at time t. The time step coding layer includes a fully connected layer, a SiLU activation layer, and a fully connected layer connected in sequence. The time step tensor is processed by the fully connected layer, the SiLU activation layer, and the fully connected layer in sequence to output the time step coding vector τ at time t.

[0066] Step 3.2: Randomly select a certain category of special vehicle SAR images processed in Step 2, and randomly sample Gaussian noise ε. Obtain the noisy special vehicle SAR image at time t using the noisy formula of the diffusion model, where the noisy formula is as follows:

[0067]

[0068] Where x0 is the SAR image of the special vehicle without noise after step 2, α t Let x be the variance parameter of the direct uniform sampling from 0.9999 to 0.98 generated by the linear scheduler, t∈(0,T). t The SAR image of the special vehicle after adding noise at time t.

[0069] Step 4: Based on the time step encoding vector at time t, the noisy special vehicle SAR image, and the category encoding vector, the denoising diffusion model is trained using a self-attention U-Net network; for example... Figure 4 The diagram shown illustrates the overall process of one complete iterative training cycle of the self-attention U-Net network.

[0070] Step 4.1: The image with added noise obtained in step 3.2 at time t is fed into the first convolutional layer of the encoder of the self-attention U-Net network for 3*3 convolution operation to obtain feature map F1;

[0071] Step 4.2: The category encoding vector σ obtained in Step 2 and the time step encoding vector τ obtained in Step 3.1 are fed into the fully connected layer of the encoder of the self-attention U-Net network to expand into a conditional encoding with the same dimension as the feature map F1 obtained in Step 4.1. The conditional encoding is then concatenated with the feature map F1 obtained in Step 4.1 to obtain a conditional feature map F2 containing conditional information.

[0072] Step 4.3: The conditional feature map F2 is fed into the remaining convolutional layers of the encoder in the self-attention U-Net network for further downsampling. After downsampling to its fourth convolutional layer, a self-attention module is introduced, the structure of which is as follows: Figure 3 As shown, the relationship between pixels is constructed through a self-attention mechanism, resulting in the encoded self-attention feature information F. encoder ;

[0073] Step 4.4: Encode the self-attention feature information F encoder The signal is fed into a decoder that is symmetrical to the encoder structure, and deconvolution is performed through the deconvolution block in the decoder to obtain the prediction noise ε′ at that moment.

[0074] Step 4.5: Calculate the mean squared error loss between the predicted noise ε′ and the Gaussian noise ε added to the image in step 3.2. This loss is used for backpropagation to update the network weights. The formula for calculating the mean squared error loss is as follows:

[0075] MSE_loss=(ε-ε′) 2

[0076] Where ε is the Gaussian noise randomly sampled at time t in step 3.2, and ε′ is the prediction of the Gaussian noise added at time t by the self-attention U-Net network;

[0077] Step 4.6: Randomly select any special vehicle SAR image processed in step 2 and its corresponding label, and repeat steps 4.1 to 4.5 at any sampling time. Set the learning rate to 0.0001, and train the network through multiple iterations until the preset number of iterations is reached to obtain the denoising diffusion model.

[0078] Step 5: Generate SAR images of special vehicles based on the denoising diffusion model. The process is as follows: Figure 5 As shown, specifically:

[0079] Step 5.1: Preset the initial iteration time T, and randomly sample Gaussian noise ε at the initial iteration time T and time T. T The specified category label is input into the denoising diffusion model trained in step 4.6, and the network output data is generated, i.e., the predicted noise ε′ at time T. T ;

[0080] Step 5.2, based on the network output data ε′ T Calculate the network input data ε at time T–1. T-1 , ε T-1 =ε T -ε′ T ;

[0081] After the calculation is completed, we enter time T–1, at which point the network input data should be ε. T-1 The specified category label, and the iteration time T–1;

[0082] Step 5.3: Repeat the above steps for a total of T iterations until the iteration time T is zero, and obtain the SAR image ε0 of the corresponding category of special vehicle.

[0083] Example 2

[0084] Based on Example 1, the special vehicle SAR image dataset used in this invention is a publicly available dataset provided by NASA: Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset. This dataset was acquired using a high-resolution focused synthetic aperture radar with a resolution of 0.3m × 0.3m, operating in the X-band with HH polarization. The MSTAR dataset contains SAR images of 10 target types for special vehicles: T72 (tank), T62 (tank), D7 (bulldozer), 2S1 (self-propelled howitzer), BMP_2 (infantry fighting vehicle), BRDM_2 (armored reconnaissance vehicle), BTR_60 (armored personnel carrier), BTR_70 (armored personnel carrier), ZIL_131 (freight truck), and ZSU_234 (self-propelled anti-aircraft gun). Each target type was imaged at elevation angles of 17° and 15°. The specific data distribution of the MSTAR dataset is shown in Table 1.

[0085] Table 1

[0086]

[0087] 2743 images captured under a 17° elevation angle condition were selected as samples of special vehicle SAR images from this dataset, and the true category of special vehicle to which each SAR image belongs was labeled. Some images in the sample are shown below. Figure 3 The special vehicle target is imaged in the center of the image, with a simple background and some terrain texture. Figure 6a For category T72 samples, Figure 6b For category 762 samples, Figure 6c This is a sample of category D7.

[0088] In terms of data preprocessing, this invention normalizes the original image training data samples and scales the pixel values ​​to the range of -1 to 1, so as to make the neural network training more stable and achieve a faster convergence speed.

[0089] To ensure that all sample SAR images have the same resolution, the original sample images are cropped to 128*128 pixels at the center to ensure the consistency of the input data.

[0090] In terms of image enhancement, each sample image was randomly mirrored to enhance data diversity and prevent overfitting in training neural networks with small batches of data.

[0091] The image after random mirror flipping is processed for brightness variation and histogram equalization is performed to further improve image contrast and reduce the impact of uneven brightness.

[0092] The SiLU activation layer of this invention uses the SiLU activation function, which is defined as follows:

[0093] SiLU(x)=x ★ sigmoid(x)

[0094] Here, x is the feature encoding of the fully connected layer output. The smoothness and non-linearity of the SiLU activation function help the model better capture complex features in the data.

[0095] The Self-attention module of this invention constructs the relationship between pixels through a self-attention mechanism, wherein the attention calculation formula is as follows:

[0096]

[0097] Where Q, K, and V are matrices obtained by linear transformation of the input, d k These are the dimensions of matrices Q and K. Softmax() represents the normalization exponential function, and Attention() is the function that calculates the attention value.

[0098] Example 3

[0099] Based on Examples 1 and 2, the SAR images of special vehicles in Example 2 are used as samples, and the corresponding special vehicle models, such as “T72” and “T62” in Table 1, are used as labels. Steps 1 to 5 are executed to train the denoising and diffusion model.

[0100] The hardware platform used in this embodiment is an Intel(R) Xeon(R) CPU E5-2650 v4@2.20GHz CPU, 12GB RAM, and NVIDIA TITAN V GPU. The operating system is Ubuntu 20.04.4, and the simulation software platform is Python 3.8 and PyTorch 1.10.

[0101] The training set images in step 1 are all cropped to a size of 128*128, and the random Gaussian noise sampled in step 5 is also 128*128 in size;

[0102] Example generated results are as follows Figures 7a-7c The image shown is the SAR image result for three types of special vehicles. Figure 7a The generated SAR image results for T72, Figure 7b The generated SAR image results for T62, Figure 7cThe SAR image results for the generated D7 are shown below, along with the SAR image results for three special vehicles.

[0103] This embodiment generates SAR images for ten categories of special vehicles. The number of SAR images generated for each category is consistent with the number of images for each category shown in Table 1. The difference between the real SAR images and the generated SAR images is evaluated, specifically including the following steps:

[0104] The generated SAR image and the real SAR image are input into the Inception network for processing to obtain the FID (FrchetInception Distance) and IS (Inception Score) indicators;

[0105] The generated SAR images and real SAR images are matched together to conduct user preference research and obtain the UPS (User Preference Score) index.

[0106] The difference assessment is expressed by the FID index, IS index, and UPS index, as well as visual qualitative comparison.

[0107] In this embodiment, the performance metrics of the method for generating special vehicle SAR images based on the denoising diffusion model provided in this application are compared with those of existing GAN methods, as shown in Table 2 below:

[0108] Table 2

[0109] FID↓ 88.42 62.93 36.73 23.47 IS↑ 5.62±0.84 6.49±0.09 8.82±0.05 10.43±0.12 UPS↑ 3.41 3.37 20.07 73.15

[0110] As shown in the table above, the present invention achieves an FID of 23.47, an IS of 10.43±0.12, and a UPS of 73.15 on the MSTAR special vehicle SAR dataset. Compared with the previous GAN-based SAR image generation model, it is the best in all three metrics.

[0111] In Example 1, the SAR image generated by AR-WGAN with the best visual effect was selected and compared with the image generated by the method of this invention for visual qualitative analysis. Taking T62 as an example, the results are as follows: Figure 8 As shown;

[0112] As shown in the figure, the image of the same type and angle generated by the method of this invention produces speckle noise around the target vehicle based on AR-WGAN, causing the target to become blurred. Figure 8 As shown in the second row of boxes, this affects the realistic quality of the generated image, and it also fails to generate the terrain texture details hidden in the realistic image in the background, such as... Figure 8 As shown in the first row of boxes;

[0113] The target vehicle generated by the method of the present invention is visually consistent with the real image, and generates inconspicuous terrain texture information on the background. The main difference from the real image lies in the difference in speckle noise around the target.

[0114] The number of parameters in the generative model is shown in Table 3:

[0115] Table 3

[0116] Parameters 108.9642M 83.0870M

[0117] The generation process used in this invention only requires training a single U-Net based on Self-attention to achieve the effect of controlling category labels and generating SAR images of 10 categories of special vehicles. Compared with AR-WGAN, which also uses the Self-attention structure, this invention not only has fewer model parameters than AR-WGAN, but also learns category labels to achieve accurate generation in multi-category target generation tasks, avoiding the complexity and resource consumption of training multiple independent models for each category.

[0118] This embodiment shows that:

[0119] (1) In the method for generating multi-category special vehicle SAR images based on the denoising diffusion model provided by the present invention, the image generation model includes a category label embedding layer, a temporal distribution embedding layer, an encoder layer based on a self-attention module, and a decoder layer based on a self-attention module. The network structure is relatively simple and has fewer parameters, which helps to reduce the risk of overfitting and increase the diversity of multi-category generated special vehicle SAR images. Furthermore, the self-attention layer introduces a self-attention mechanism, which can effectively improve the expressive power of the feature map and thus improve the quality of the generated special vehicle SAR images.

[0120] (2) This invention eliminates the adversarial training mechanism in the previous GAN training process, and the network model is simpler than GAN, which reduces the probability of gradient explosion or gradient vanishing during training, making the training process of the network model more stable.

[0121] (3) The special vehicle SAR image generation method provided by the present invention utilizes category information. During the training process, the category information contained in the SAR image is added to the network to be trained using the embedding mechanism, linear layer and SiLU activation layer. The trained image generation model can generate special vehicle SAR images of any specified category without training multiple models. This avoids the complexity and resource consumption of training multiple independent models for each category and has a wider range of applications.

Claims

1. A method for generating a multi-class special vehicle SAR image based on a denoising diffusion model, characterized in that, The specific steps are as follows: Step 1: Obtain samples of SAR images of special vehicles and preprocess them; Step 2: Create category labels for the special vehicle SAR images preprocessed in Step 1 according to the categories, and convert the category labels into learnable category coding vectors; Step 3: Obtain the time step encoding vector at time t. Randomly sampled Gaussian noise is added to the special vehicle SAR images labeled with the preset categories in step 2 using a diffusion model. We obtain the SAR image of the special vehicle after adding noise at time t; Step 4: Based on the time step encoding vector at time t, the noisy special vehicle SAR image, and the category encoding vector, the denoising diffusion model is trained using a self-attention U-Net network. Specifically: Step 4.1: The image with added noise obtained in Step 3 at time t is fed into the first convolutional layer of the encoder of the self-attention U-Net network for a 3*3 convolution operation to obtain the feature map. ; Step 4.2, use the category encoding vector obtained in Step 2. and the time step encoding vector obtained in step 3 The fully connected layer of the encoder in the self-attention U-Net network is fed into the network and expanded to the feature map obtained in step 4.

1. Conditional encoding of the same dimension, and compared with the feature map obtained in step 4.

1. By concatenating the features, a conditional feature map containing conditional information is obtained. ; Step 4.3, conditional feature map The data is fed into the remaining convolutional layers of the encoder in the self-attention U-Net network for further downsampling. After downsampling to its fourth convolutional layer, a self-attention module is introduced to construct relationships between pixels through a self-attention mechanism, resulting in the encoded self-attention feature information. ; Step 4.4, encode the self-attention feature information into a decoder symmetric to the encoder structure, and deconvolve through a deconvolution block in the decoder to obtain the prediction noise at this time ; Step 4.5, compute the predicted noise The mean squared error loss of the Gaussian noise added to the image in step 3 for backpropagation to update the network weights, the mean squared error loss is computed as follows: wherein, is the random sampling of Gaussian noise at time t in step 3, is the prediction of the added Gaussian noise at time t by the self-attention U-Net network; Step 4.6: Randomly select any special vehicle SAR image processed in step 2 and its corresponding label, repeat steps 4.1 to 4.5 at any sampling time, set the learning rate to 0.0001, and train the network through multiple iterations until the preset number of iterations is reached to obtain the denoising diffusion model. Step 5: Generate SAR images of special vehicles based on the denoising diffusion model.

2. The method for generating multi-category special vehicle SAR images based on a denoising diffusion model according to claim 1, characterized in that, The specific steps for processing the sample data in step 1 are as follows: The acquired SAR images of special vehicles are normalized, with pixel values ​​scaled to the range of -1 to 1. Then, they are centrally cropped to a size of 128*128, and finally, image enhancement is performed to obtain the original training image data.

3. The method for generating multi-class special vehicle SAR images based on a denoising diffusion model according to claim 2, characterized in that, The image enhancement specifically involves performing random brightness adjustment, random mirror flipping, and histogram equalization sequentially on the centrally cropped SAR image of special vehicles.

4. The method for generating multi-class special vehicle SAR images based on a denoising diffusion model according to claim 1, characterized in that, Step 2 specifically involves: For each class of special vehicles in the original training image data obtained in step 1, an index dictionary is created for the SAR images. The key value corresponding to each class index is transformed into a learnable class parameter tensor using the Embedding embedding layer function in the torch library. This class parameter tensor is then transformed into a learnable class encoding vector through a class encoding layer. .

5. The method for generating multi-class special vehicle SAR images based on a denoising diffusion model according to claim 4, characterized in that, The category encoding layer comprises a fully connected layer, a SiLU activation layer, and another fully connected layer connected in sequence. The category parameter tensor is converted into a learnable category encoding vector through the fully connected layer, SiLU activation layer, and another fully connected layer in sequence. .

6. The method for generating multi-class special vehicle SAR images based on a denoising diffusion model according to claim 5, characterized in that, Step 3 specifically involves: Step 3.1: Time is obtained by discrete sampling within the range of 1 to T. The time step tensor is then obtained by concatenating the sine and cosine of sin and cosine values ​​of t. This tensor is then passed through a time step coding layer to output the time step coded vector at time t. ; Step 3.2, randomly select a special vehicle SAR image of a certain category processed by step 2, and randomly sample Gaussian noise , obtain the special vehicle SAR image after adding noise at time t through the noise adding formula of the diffusion model, wherein the noise adding formula is as follows: in, For the SAR image of special vehicles without noise after step 2 processing, The variance parameter is the direct uniform sampling from 0.9999 to 0.98 generated by the linear scheduler. , The SAR image of the special vehicle after adding noise at time t.

7. The method for generating multi-category special vehicle SAR images based on a denoising diffusion model according to claim 6, characterized in that, The time-step encoding layer comprises a fully connected layer, a SiLU activation layer, and another fully connected layer connected in sequence. The time-step tensor is processed sequentially through the fully connected layer, the SiLU activation layer, and the fully connected layer to output the time-step encoded vector at time t. .

8. The method for generating multi-class special vehicle SAR images based on a denoising diffusion model according to claim 1, characterized in that, Step 5 specifically involves: Step 5.1: Preset the initial iteration time T, and randomly sample Gaussian noise at the initial iteration time T and time T. The specified category labels are input into the denoising diffusion model trained in step 4.6, and the network output data, i.e., the predicted noise at time T, is generated. ; Step 5.2, computing network input data at time T - 1 from network output data , , ; After the calculation is completed, that is, at time T - 1, the input data of the network at this time should be , the specified category label, iteration time T - 1; Step 5.3, repeat the above steps, iterating a total of T times, until iteration time is zero, resulting in a special vehicle SAR image corresponding to the class.

Images