High-Fidelity Omnidirectional SAR Image Generation Method Based on Deep Generation Model

Abstract

The present invention discloses a high-fidelity omnidirectional SAR image generation method based on a deep generation model, including: inputting a preset category, a preset azimuth angle, and a preset noise vector of the SAR image to be generated into an image generation model; the image generation model includes a linear layer, a first generation residual module, a self-attention module, a second generation residual module, and a generation module; after the linear layer expands the dimension of the noise vector, the first generation residual module extracts features from the noise vector with expanded dimension and generates a first feature map; the self-attention module generates a self-attention feature map according to the first feature map; based on the self-attention feature map, the generation module generates a SAR image under the preset category and the preset azimuth angle. The image generation model used in the present invention has a simple network structure and a small number of parameters, which is beneficial to reducing the risk of overfitting, and the self-attention model introduces a self-attention mechanism, which can improve the expression ability of the feature map, and further improve the quality of the generated SAR image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and particularly relates to a high-fidelity omnidirectional SAR image generation method based on a deep generation model. Background Art

[0002] Currently, the deep generation model has developed rapidly, and the image data augmentation based on the deep generation model is the mainstream of the current development. GAN (Generative Adversarial Networks) is a model widely used for image generation, which consists of a generator and a discriminator. Through the method of adversarial training, the generator and the discriminator are jointly optimized to finally reach the Nash equilibrium, and finally achieve the effect that the generated data is indistinguishable from the real data.

[0003] With the good performance and continuous development of GAN in the field of optical image data generation, the SAR image augmentation method based on GAN has developed rapidly. The existing SAR image augmentation methods include: an end-to-end SAR image generation method, and a clutter regularization method is used to improve the training stability; a SAR image generation method based on Stack GAN; a SAR image generation method based on MulticonstraintGAN.

[0004] However, the existing SAR image generation methods still have deficiencies: First, the generative adversarial network used in the above methods ignores the global information of the feature map during the training process; Second, the utilization of the imaging mechanism characteristics of SAR itself needs to be strengthened; Third, the network proposed in the above methods has unstable problems during the training process. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a high-fidelity omnidirectional SAR image generation method based on a deep generation model. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0006] The present invention provides a high-fidelity omnidirectional SAR image generation method based on a deep generation model, including:

[0007] Inputting a preset category, a preset azimuth angle, and a preset noise vector of the SAR image to be generated into an image generation model; the image generation model includes a linear layer, a first generation residual module, a self-attention module, a second generation residual module, and a generation module connected in sequence;

[0008] After the linear layer expands the dimension of the noise vector, the first generation residual module extracts features from the noise vector with the expanded dimension and generates a first feature map;

[0009] The self-attention module generates a self-attention feature map based on the first feature map;

[0010] Based on the self-attention feature map, the generation module generates a SAR image at a preset class and a preset azimuth angle by using a preset activation function.

[0011] In an embodiment of the present invention, the step of the self-attention module generating a self-attention feature map based on the first feature map includes:

[0012] The self-attention module respectively convolves the first feature map by using three 1×1 convolution kernels to obtain a first sub-map, a second sub-map, and a third sub-map;

[0013] The first sub-map and the second sub-map are multiplied point by point and then input into a Softmax function to generate an attention map;

[0014] The attention map and the third sub-map are multiplied point by point and then convolved to obtain a self-attention feature map.

[0015] In an embodiment of the present invention, the image generation model is trained according to the following steps:

[0016] Obtain a training data set, where the training data set includes a plurality of training samples, each training sample includes a sample image, a true class corresponding to the sample image, and a true azimuth angle, and the sample image is a SAR image;

[0017] Generate a noise vector, and input the generated noise vector into a to-be-trained generative adversarial network, so that a to-be-trained generator network in the to-be-trained generative adversarial network generates a fake image corresponding to the true class and the true azimuth angle based on the second preset noise vector;

[0018] Generate instance noise and add the instance noise to the fake image and the sample image respectively;

[0019] Input the fake image added with instance noise and the sample image added with instance noise into a to-be-trained discriminator network in the to-be-trained generative adversarial network respectively, and determine a loss value of a preset loss function according to a discrimination result of the to-be-trained discriminator network, the true class, and the true azimuth angle;

[0020] Judge whether the to-be-trained generative adversarial network converges according to the loss value; if so, use the to-be-trained generator network in the to-be-trained generative adversarial network as an image generation model; if not, after alternately adjusting network parameters of the to-be-trained generator network and network parameters of the to-be-trained discriminator network through backpropagation, return to the step of generating a noise vector and inputting the generated noise vector into the to-be-trained generative adversarial network.

[0021] In one embodiment of the present invention, the preset loss function includes a first preset loss function for the generator network to be trained and a second preset loss function for the discriminator network to be trained.

[0022] In one embodiment of the present invention, the step of inputting the fake image added with instance noise and the sample image into the discriminator network to be trained in the generative adversarial network to be trained, and determining the loss value of the preset loss function according to the discrimination result of the discriminator network to be trained, the true category, and the true azimuth angle includes:

[0023] Input the fake image added with instance noise into the discriminator network to be trained, and determine the first loss value according to the first discrimination result of the discriminator network to be trained, the true category, the true azimuth angle, and the first preset loss function;

[0024] Input the sample image added with instance noise into the discriminator network to be trained, and determine the second loss value according to the second discrimination result of the discriminator network to be trained, the true category, the true azimuth angle, and the second preset loss function;

[0025] Calculate the sum of the first loss value and the second loss value to obtain the loss value of the preset loss function.

[0026] In one embodiment of the present invention, the step of obtaining the training data set includes:

[0027] Crop the sample images in the training data set to a size of 128*128;

[0028] Obtain the pixel values of the cropped sample images, where the pixel values are floating-point data;

[0029] Set the pixel values greater than 0.5 in the cropped sample images to 0.5, and transform the pixel values between 0 and 0.5 to 0-255 to obtain the grayscale processed and quantized sample images.

[0030] In one embodiment of the present invention, the preset noise vector is a one-dimensional vector and follows a Gaussian distribution.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] (1) In the high-fidelity omnidirectional SAR image generation method based on a deep generation model provided by the present invention, the image generation model includes a linear layer, a first generation residual module, a self-attention module, a second generation residual module, and a generation module connected in sequence. The network structure is simple and has a small number of parameters. Therefore, it is beneficial to reduce the risk of overfitting. Moreover, the self-attention model introduces a self-attention mechanism, which can improve the expressive ability of the feature map and thus improve the quality of the generated SAR images.

[0033] (2) The SAR image generation method provided by the present invention utilizes the mechanism characteristics of SAR imaging. During the training process, the azimuth angle information and category information of the targets included in the SAR images are added to the generator to be trained using conditional batch normalization. The image generation model obtained after training can generate SAR images with arbitrary azimuth angles and categories, and has a wider application range.

[0034] (3) By adding instance noise, the present invention can increase the intersection between the two distributions of fake images and real SAR images (sample images), and after using spectral norm normalization, the network training can be made more stable.

[0035] The following will further elaborate on the present invention in conjunction with the accompanying drawings and embodiments. Description of the Drawings

[0036] Figure 1 is a flowchart of a high-fidelity omnidirectional SAR image generation method based on a deep generation model provided by an embodiment of the present invention;

[0037] Figure 2 is a schematic structural diagram of a self-attention module provided by an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of the training process of an image generation model provided by an embodiment of the present invention;

[0039] Figure 4 is a schematic structural diagram of a generator adversarial network to be trained provided by an embodiment of the present invention;

[0040] Figure 5 is a schematic structural diagram of a generation residual module provided by an embodiment of the present invention;

[0041] Figure 6 is an example diagram of an image generation model provided by an embodiment of the present invention. Detailed Embodiments

[0042] The following further describes the present invention in detail in conjunction with specific embodiments, but the implementation manners of the present invention are not limited thereto.

[0043] Figure 1It is a flowchart of a high-fidelity omnidirectional SAR image generation method based on a deep generation model provided by an embodiment of the present invention. As Figure 1 shown, an embodiment of the present invention provides a high-fidelity omnidirectional SAR image generation method based on a deep generation model, including:

[0044] S1. Input a preset category, a preset azimuth angle, and a preset noise vector of the SAR image to be generated into an image generation model; the image generation model includes a linear layer, a first generation residual module, a self-attention module, a second generation residual module, and a generation module connected in sequence;

[0045] S2. After the linear layer expands the dimension of the noise vector, the first generation residual module extracts features from the dimension-expanded noise vector and generates a first feature map;

[0046] S3. The self-attention module generates a self-attention feature map according to the first feature map;

[0047] S4. Based on the self-attention feature map, the generation module uses a preset activation function to generate a SAR image under the preset category and the preset azimuth angle.

[0048] In this embodiment, the preset noise vector is a one-dimensional vector and follows a Gaussian distribution.

[0049] Figure 2 It is a schematic structural diagram of the self-attention module provided by an embodiment of the present invention. As Figure 2 shown, in the above step S3, the step in which the self-attention module generates a self-attention feature map according to the first feature map includes:

[0050] S301. The self-attention module uses three 1×1 convolution kernels to perform convolution on the first feature map respectively to obtain a first sub-map, a second sub-map, and a third sub-map;

[0051] S302. Multiply the first sub-map and the second sub-map and then input them into the Softmax function to generate an attention map;

[0052] S303. Multiply the attention map and the third sub-map and then perform convolution to obtain a self-attention feature map.

[0053] It should be noted that when the self-attention module performs convolution on the first feature map, the parameters of the three convolution kernels used are different.

[0054] After obtaining the first sub-map, the second sub-map, and the third sub-map, the attention map is generated based on the first sub-map and the second sub-map according to the following formula:

[0055]

[0056] where s ij= f(x i ) T g(x j ), represents the first feature map input to the self-attention module, and x i represents the i-th row of the first feature map x, and x j represents the j-th column of the first feature map x. f(x i ) = W f x i , and g(x j ) = W g x j . β j,i represents the degree of attention paid to the i-th feature point when generating the j-th feature point of the attention map.

[0057] Step S303, calculate the output of the attention layer according to the attention map and the third sub-map to obtain the self-attention feature map o. In this embodiment wherein h(x i ) = W h x i , and v(x i ) = W v x i .

[0058] Exemplarily, calculate the final output y of the attention module:

[0059] y = γo + x

[0060] where γ represents the scale parameter of the output self-attention feature map.

[0061] Figure 3 is a schematic diagram of the training process of the image generation model provided by the embodiment of the present invention, Figure 4 is a schematic diagram of the structure of the to-be-trained generative adversarial network provided by the embodiment of the present invention. As Figure 3-4 shown, in this embodiment, the image generation model can be trained according to the following steps:

[0062] Obtain a training data set, the training data set includes a plurality of training samples, each training sample includes a sample image, the true category corresponding to the sample image, and the true azimuth angle, and the sample image is a SAR image;

[0063] Generate a noise vector and input the generated noise vector into the to-be-trained generative adversarial network, so that the to-be-trained generator network in the to-be-trained generative adversarial network generates a fake image corresponding to the true category and the true azimuth angle based on the second preset noise vector;

[0064] Generate instance noise and add the instance noise to the fake image and the sample image respectively;

[0065] Input the fake image added with instance noise and the sample image added with instance noise into the discriminator network to be trained in the generative adversarial network to be trained respectively, and determine the loss value of the preset loss function according to the discrimination result of the discriminator network to be trained, the true category and the true azimuth angle;

[0066] Judge whether the generative adversarial network to be trained converges according to the loss value; if so, use the generator network to be trained in the generative adversarial network to be trained as an image generation model; if not, after alternately adjusting the network parameters of the generator network to be trained and the network parameters of the discriminator network to be trained by backpropagation, return to the above step of generating a noise vector and inputting the generated noise vector into the generative adversarial network to be trained.

[0067] Specifically, in this embodiment, the MSTAR dataset is selected as the training dataset. Table 1 shows the number of training samples, category information, and azimuth angle information of the training dataset:

[0068] Table 1

[0069] True category True azimuth Quantity BMP2 17 233 BTR70 17 233 T72 17 232 BTR60 17 256 2S1 17 299 BRDM2 17 298 D7 17 299 T62 17 299 ZIL131 17 299 ZSU234 17 299

[0070] It should be understood that in the high-fidelity omnidirectional SAR image generation method based on a deep generation model provided by the present invention, the category refers to the category of the target included in the SAR image, and the azimuth angle refers to the azimuth angle of the synthetic aperture radar that generates the SAR image.

[0071] Optionally, the step of obtaining the training dataset includes:

[0072] Crop the sample images in the training dataset to a size of 128*128;

[0073] Obtain the pixel values of the cropped sample images, and the pixel values are floating-point data;

[0074] Set the pixel values greater than 0.5 in the cropped sample images to 0.5, and transform the pixel values between 0 and 0.5 to 0-255 to obtain the sample images after gray processing and quantization.

[0075] In this embodiment, to ensure that the resolutions of all sample images are the same, the center of the original sample image can be cropped to a size of 128*128. And since the pixel values of the MSTAR sample images are floating-point data with different sizes, the part of the pixel values greater than 0.5 in the cropped sample image is set to 0.5, and the pixel values in the range of floating-point data 0-0.5 are transformed into the integer range of 0-255, completing the grayscale processing and quantization of the sample image, which is beneficial to ensuring that the background brightness of the sample image is the same.

[0076] Optionally, after obtaining the true category and true azimuth angle of the sample image, the true category and true azimuth angle together with a randomly generated one-dimensional array are input into the generator to be trained as a noise vector, so that the generator to be trained generates a fake image; specifically, as Figure 5 shown, in this embodiment, conditional batch normalization (CBN) is used to add the true category and true azimuth angle of the sample image to the generation residual module of the generator network to be trained. Each generation residual module has two conditional batch normalization operations, so that the generation residual module can be combined with the category and azimuth angle information. x is the feature map extracted by the previous layer of the network, and label refers to conditional information such as category or azimuth angle. The embedding layer embed is used to encode the conditional information into two vectors σ and β, and the feature map is transformed into the output out using σ and β, so that the conditional information is included in the feature map. Its expression is:

[0077] CBN(b) = σ·BN(b) + β

[0078] where b is the feature map information, and BN(b) is the result of batch normalization.

[0079] After the spectral norm normalization is introduced into the GAN network to be trained, the spectral norm normalization acts on the weight matrix of each layer of the generator to be trained and the discriminator to be trained, avoiding abnormal gradients caused by too many parameters of the generator and making the entire training process more stable and efficient.

[0080] During each iterative training process, instance noise that follows a Gaussian distribution with a mean of 0 and a variance changing with the iteration number k needs to be generated. When the iteration number is k, the variance is expressed as follows:

[0081]

[0082] where σ0 represents the initial variance of the instance noise, and I noise represents the training iteration number. Therefore, when the iteration number is k, the instance noise can be expressed as:

[0083]

[0084] Furthermore, at the i-th iteration, instance noise is first added to the fake image and sample image input to the discriminator. For the sake of convenience, the input image of the discriminator network to be trained, that is, the fake image with instance noise added and the sample image with instance noise added, are both denoted as t. Then, adding instance noise can be expressed as:

[0085] t k ′=t+IN k .

[0086] Furthermore, the fake image with added instance noise and the sample image with added instance noise are respectively input into the discriminator network to be trained in the generative adversarial network to be trained, and the loss value of the preset loss function is determined according to the discrimination result of the discriminator network to be trained, the true category and the true azimuth, including:

[0087] Inputting a fake image with instance noise added thereto into a discriminator network to be trained, and determining a first loss value according to a first discrimination result of the discriminator network to be trained, a true category, a true azimuth, and a first preset loss function;

[0088] Inputting the sample image with the added instance noise into the discriminator network to be trained, and determining a second loss value according to the second discrimination result of the discriminator network to be trained, the true category, the true azimuth, and the second preset loss function;

[0089] Calculate the sum of the first loss value and the second loss value to obtain a loss value of a preset loss function.

[0090] In this embodiment, a fake image with instance noise added is input into the discriminator to be trained together with the true category and the true azimuth. The loss value of the preset function is calculated based on the output result of the discriminator network to be trained, and backpropagation is performed.

[0091] In this embodiment, the preset loss function includes a first preset loss function of the generator network to be trained and a second preset loss function of the discriminator network to be trained. For example, the preset loss function is:

[0092]

[0093] Where t represents the fake image with instance noise added and the sample image with instance noise added that are input to the discriminator network to be trained, G(z) represents the fake image of the generator to be trained, y represents the category, a represents the azimuth angle, and the Adam optimizer is used to optimize the discriminator parameters.

[0094] In order to stabilize the model training process, this embodiment gives different learning rates to the generator to be trained and the discriminator to be trained during the training process, which are 0.0001 and 0.0004 respectively.

[0095] Figure 6 It is an example diagram of the image generation model provided by the embodiments of the present invention. As Figure 6 shown, the high-fidelity omnidirectional SAR image generation method based on the deep generation model provided by the present invention can generate SAR images of a specified type and azimuth angle.

[0096] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows:

[0097] (1) In the high-fidelity omnidirectional SAR image generation method based on the deep generation model provided by the present invention, the image generation model includes a linear layer, a first generation residual module, a self-attention module, a second generation residual module, and a generation module connected in sequence. The network structure is simple and the number of parameters is small. Therefore, it is beneficial to reduce the risk of overfitting. Moreover, the self-attention model introduces a self-attention mechanism, which can improve the expression ability of the feature map, and further improve the quality of the generated SAR images.

[0098] (2) The SAR image generation method provided by the present invention utilizes the mechanism characteristics of SAR imaging. During the training process, the azimuth angle information and category information of the targets included in the SAR images are added to the generator to be trained using conditional batch normalization. The image generation model obtained after training can generate SAR images of any azimuth angle and category, and has a wider application range.

[0099] (3) By adding instance noise, the present invention can increase the intersection between the two distributions of fake images and real SAR images (sample images), and after using spectral norm normalization, the network training can be made more stable.

[0100] In the description of the present invention, the terms "first" and "second" are only used for descriptive purposes, and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more, unless otherwise specifically defined.

[0101] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0102] Although the present application has been described in connection with various embodiments, those skilled in the art will understand and realize other variations of the disclosed embodiments by viewing the accompanying drawings, the disclosure, and the appended claims, in the course of practicing the claimed application.

[0103] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A high-fidelity omnidirectional SAR image generation method based on a deep generative model, characterized in that Including: Input the preset category, preset azimuth angle, and preset noise vector of the SAR image to be generated into the image generation model; The image generation model includes a linear layer, a first generation residual module, a self-attention module, a second generation residual module, and a generation module connected in sequence; After the linear layer expands the dimension of the noise vector, the first generation residual module extracts features from the noise vector with expanded dimension and generates a first feature map; The self-attention module generates a self-attention feature map according to the first feature map; Based on the self-attention feature map, the generation module generates a SAR image under the preset category and preset azimuth angle by using a preset activation function; The step that the self-attention module generates a self-attention feature map according to the first feature map includes: The self-attention module performs convolution on the first feature map by using three 1*1 convolution kernels respectively to obtain a first sub-map, a second sub-map, and a third sub-map; Multiply the first sub-map and the second sub-map and then input the result into the Softmax function to generate an attention map; Multiply the attention map and the third sub-map and then perform convolution to obtain a self-attention feature map; The image generation model is trained according to the following steps: Obtain a training data set, where the training data set includes multiple training samples, and each training sample includes a sample image, the true category corresponding to the sample image, and the true azimuth angle, and the sample image is a SAR image; Generate a noise vector and input the generated noise vector into the generator network to be trained in the generative adversarial network to be trained, so that the generator network to be trained in the generative adversarial network to be trained generates a fake image corresponding to the true category and true azimuth angle based on a second preset noise vector; Generate instance noise and add the instance noise to the fake image and the sample image respectively; Input the fake image with instance noise and the sample image with instance noise into the discriminator network to be trained in the generative adversarial network to be trained respectively, and determine the loss value of the preset loss function according to the discrimination result of the discriminator network to be trained, the true category, and the true azimuth angle; Judge whether the generative adversarial network to be trained converges according to the loss value; if so, use the generator network to be trained in the generative adversarial network to be trained as the image generation model; if not, after alternately adjusting the network parameters of the generator network to be trained and the network parameters of the discriminator network to be trained through backpropagation, return to the step of generating a noise vector and inputting the generated noise vector into the generative adversarial network to be trained; 2. The high-fidelity omnidirectional SAR image generation method based on a deep generation model according to claim 1, wherein The preset loss function includes a first preset loss function of the generator network to be trained and a second preset loss function of the discriminator network to be trained; 3. The high-fidelity omnidirectional SAR image generation method based on a deep generation model according to claim 2, wherein The step of inputting the fake image with instance noise and the sample image with instance noise into the discriminator network to be trained in the generative adversarial network to be trained respectively, and determining the loss value of the preset loss function according to the discrimination result of the discriminator network to be trained, the true category, and the true azimuth angle includes: Input the fake image added with instance noise into the discriminator network to be trained, and determine the first loss value according to the first discrimination result of the discriminator network to be trained, the true category, the true azimuth angle, and the first preset loss function; Input the sample image added with instance noise into the discriminator network to be trained, and determine the second loss value according to the second discrimination result of the discriminator network to be trained, the true category, the true azimuth angle, and the second preset loss function; Calculate the sum of the first loss value and the second loss value to obtain the loss value of the preset loss function.

4. The high-fidelity omnidirectional SAR image generation method based on a deep generation model according to claim 1, wherein The step of obtaining the training data set includes: Crop the sample images in the training data set to a size of 128*128; Obtain the pixel values of the cropped sample images, and the pixel values are floating-point data; Set the pixel values greater than 0.5 in the cropped sample images to 0.5, and transform the pixel values between 0 and 0.5 to 0-255 to obtain the gray-processed and quantized sample images.

5. The high-fidelity omnidirectional SAR image generation method based on a deep generation model according to claim 1, wherein The preset noise vector is a one-dimensional vector and follows a Gaussian distribution.

Images