SAR image generation method based on style migration

Abstract

The invention provides an SAR image generation method based on style migration, and belongs to the technical field of synthetic aperture radar imaging and artificial intelligence. According to the method, firstly, an SAR image data set is used for training a convolutional neural network, so that the network can accurately extract the features of an SAR image, then, a style migration method is utilized, the neural network obtained through training is combined, an optical image is taken as a content image, the SAR image is taken as a style image, and the generation of a migration image is realized by optimizing the total migration loss. According to the method, the convolutional neural network and the style migration technology are utilized, the visual features are extracted from the optical image and converted into the features conforming to the SAR image style, and automatic generation from the optical image to the SAR image is achieved. The SAR image generated by the method can be used for data set expansion, data diversity is enhanced, and the method has a wide application prospect in the field of remote sensing image analysis and data processing.

Description

Technical Field

[0001] The present invention belongs to the field of synthetic aperture radar (SAR) imaging and artificial intelligence technology, and in particular to a SAR image generation method based on style migration. Background Art

[0002] Synthetic Aperture Radar (SAR) imaging technology is an important detection method that can provide high-resolution images in various weather conditions. However, obtaining real SAR images is usually costly and complicated. Therefore, in order to expand the diversity of SAR image datasets and generate deception templates, it is particularly important to study a method to generate SAR images from existing optical images.

[0003] In the article "SAR and Optical Image Registration Algorithm Based on Style Transfer Invariant Features" published by Chen Shiwei et al. in Systems Engineering and Electronics, a style transfer network was trained to convert SAR images into optical images, and the difference between the two was used as a feature to achieve accurate matching between the original SAR image and the optical reference image. Although this method generates optical images from SAR images, it shows the possibility of using style transfer for conversion.

[0004] In the article "SAR and optical image matching based on style transfer and edge features" published by Pan Ming et al. in Beijing Surveying and Mapping, a cycle-consistent adversarial generative network was used to achieve the change of optical images and SAR images. However, the style transfer using a deep neural network based on a convolutional neural network is more flexible and convenient for style transfer for a specified image.

[0005] In the patent application number CN202311095451.3 filed by Beijing University of Aeronautics and Astronautics, "A SAR Image Simulation Method Based on Multi-category Landform Style Transfer", semantic segmentation and adversarial generative network are combined to use visible light and SAR image data to achieve multi-category landform style transfer of SAR images. However, the processing process of this method is relatively complicated. Summary of the invention

[0006] In view of this, the present invention provides a SAR image generation method based on style transfer. The present invention converts the visual information of the optical image into a style that conforms to the characteristics of the SAR image by combining a convolutional neural network and style transfer technology, thereby simulating the SAR image, generating more data samples, and enhancing the diversity of the samples.

[0007] The technical solution adopted by the present invention is:

[0008] A SAR image generation method based on style transfer includes the following steps:

[0009] Step 1: construct a training data set and a test data set based on SAR image data;

[0010] Step 2: Build a deep neural network for SAR image classification based on graph neural network;

[0011] Step 3, setting the loss function and optimizer, batch training the SAR image classification deep neural network through the training data set, and using the test data set for evaluation to obtain the trained SAR image classification deep neural network;

[0012] Step 4: Obtain the style image and content image respectively, randomly generate an initial migration image, and use an optimizer to optimize the initial migration image based on the style image, content image and trained SAR image classification deep neural network to obtain a style-migrated SAR image.

[0013] Furthermore, the specific method of step 1 is:

[0014] Obtain N SAR image data, which are divided into M categories, where each SAR image sample The dimension is , the corresponding category label is , ∈M; Construct a sample set based on SAR image data , build a label set .

[0015] According to the proportion The sample set and label set are split to obtain a training data set and a test data set, where the training data set includes a training sample data set. And training label dataset , including data, the test data set includes the test sample set and the test label set , including data.

[0016] Furthermore, in step 2, the deep neural network for SAR image classification based on graph neural network is composed of 3 convolutional layers, 1 pooling layer, the first Relu activation function, 2 fully connected layers, and the second Relu activation function in series. The five parts are respectively , , , , Indicates that the input of the SAR image classification deep neural network is sample , , , , The outputs are:

[0017] .

[0018] Furthermore, the specific method of step 3 is:

[0019] Step 3-1, the loss function uses the cross entropy loss function, and the optimizer uses the Adam optimizer;

[0020] Step 3-2, set the training rate ;

[0021] Step 3-3, setting the SAR image classification deep neural network to training mode;

[0022] Step 3-4, from the training sample data set And training label dataset Randomly extract B sample data and label data from the dataset to form a batch training sample data set and batch training labeled dataset ;

[0023] Step 3-5, clear the optimizer's network parameters for the SAR image classification deep neural network The cumulative gradient value of

[0024] Steps 3-6, Input SAR image classification deep neural network, and get ,Will and Input the loss function and obtain the loss value loss;

[0025] Step 3-7, use the loss value to back propagate and use the optimizer to adjust the network parameters Make updates;

[0026] Step 3-8, repeat steps 3-4 to 3-7 until the data set is emptied;

[0027] Step 3-9, setting the SAR image classification deep neural network to evaluation mode;

[0028] Step 3-10: Set the test sample set Input SAR image classification deep neural network, and get , get The position corresponding to the maximum value of the second dimension in the test label set The same probability , if the probability Less than threshold , then return to step 3-3 to retrain, otherwise, save the network parameters , and obtain the trained SAR image classification deep neural network.

[0029] Furthermore, the specific method of step 4 is:

[0030] Step 4-1: Load network parameters ;

[0031] Step 4-2: Randomly extract a SAR image sample from the SAR image data as the style image , the dimension is ;

[0032] Step 4-3: Load the optical image to be converted and convert it into grayscale with the dimension , as the content image ;

[0033] Step 4-4: Construct a random image with a mean of 0 and a standard deviation of 1 , the dimension is ; Set weights , get the initial migration image ;

[0034] Step 4-5: Use Adam optimizer to optimize the migration image Optimize and set the training rate ;

[0035] Step 4-6: Style image Input SAR image classification deep neural network, get , , , Output value , , and ;

[0036] Step 4-7: Convert the content image Input SAR image classification deep neural network, get , , , The output value of , , and ;

[0037] Step 4-8: Clear the optimizer about the migration image The accumulated gradient value will migrate the image Input SAR image classification deep neural network, get , , , Output value , , and ;

[0038] Step 4-9: Calculate the content loss value:

[0039] ,

[0040] in, It means to find the mean of all elements;

[0041] Step 4-10: Calculate the style loss value:

[0042] ,

[0043] in, Calculate for the Gram matrix;

[0044] Step 4-11: Set weights , and the total migration loss is ;

[0045] Step 4-12: Use the total transfer loss to back propagate through the optimizer to transfer the image Make updates;

[0046] Step 4-13: Repeat steps 4-5 to 4-12 until the preset number of times is reached to obtain the final migration image. , which is the style-transferred SAR image.

[0047] The present invention has the following beneficial effects:

[0048] 1. Cross-modal image conversion capability. The present invention provides a cross-modal conversion technology from optical images to SAR images, which is not achievable in traditional image generation and conversion methods. By combining convolutional neural networks with style transfer, the visual features of optical images can be effectively mapped to SAR image features to generate highly realistic SAR simulation images. This conversion process provides a new technical path for SAR image generation, overcoming the limitation of the high complexity of existing SAR image acquisition.

[0049] 2. Data diversity expansion. The present invention effectively expands the scale and diversity of SAR data sets by converting optical images into SAR images, overcoming the shortcomings of high acquisition cost and limited data volume of traditional SAR images. This method can generate a large number of simulated SAR images.

[0050] 3. Low cost and high efficiency. The present invention generates SAR images based on existing optical images, without the need for complex and expensive hardware equipment for actual imaging, which greatly reduces the cost and time of SAR image acquisition and improves image acquisition efficiency.

[0051] 4. Multi-scenario application. The SAR images generated by the present invention are not only suitable for target detection, classification and geographic information processing in the field of remote sensing, but can also be widely used in scenes such as environmental monitoring and terrain analysis, and have wide application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of loss value.

[0053] Figure 2 This is a diagram showing the accuracy.

[0054] Figure 3 For style images.

[0055] Figure 4 For content images.

[0056] Figure 5 Schematic diagram of total migration loss value.

[0057] Figure 6 To migrate the image. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0059] A SAR image generation method based on style transfer is proposed. The method generates synthetic aperture radar images from optical images based on style transfer. First, a convolutional neural network is trained using a SAR image dataset so that the network can accurately extract the features of the SAR image. Then, the style transfer method is used in combination with the trained neural network. The optical image is used as the content image and the SAR image is used as the style image. The generation of the transfer image is achieved by optimizing the total transfer loss.

[0060] Specifically, the method comprises the following steps:

[0061] Step 1: Construct training and testing datasets based on SAR image data; the specific method is as follows:

[0062] Step 1-1: Construct SAR image data into a dataset.

[0063] The number of SAR image data is N=2000, with a total of M=10 categories. Each SAR image sample is , the dimension is , where L and K are both 128, and the corresponding category label is , with a dimension of 1. The dataset consists of a sample set and a label set. The sample set is constructed as , , the dimension is , the label set is , , the dimension is .

[0064] Step 1-2: Split the dataset into training dataset and test dataset.

[0065] The data set is divided into =0.8. The training data set includes the training sample data set And training label dataset , the dimensions are and The test data set includes the test sample set and the test label set , the dimensions are and .

[0066] Step 2: Build a deep neural network for SAR image classification based on graph neural network.

[0067] The classification deep neural network is composed of three convolutional layers, a pooling layer and a Relu activation function, and two fully connected layers and a Relu activation function in series. , , , and Indicates that the input is a sample variable , the output is the calculation results of each convolutional layer and the fully connected layer:

[0068]

[0069] The classification deep neural network parameters are .

[0070] The classification deep neural network structure is shown in the following table:

[0071]

[0072] Step 3: Set the loss function and optimizer, batch train the SAR image classification deep neural network using the training data set, and use the test data set for evaluation to obtain the trained SAR image classification deep neural network; the specific method is:

[0073] Step 3-1: The loss function uses the cross entropy loss function, and the optimizer uses the Adam optimizer to optimize the parameters of the classification deep neural network. Optimize

[0074] Step 3-2: Set the training rate =0.001;

[0075] Let's batch train the classification deep neural network:

[0076] Step 3-3: Set the classification deep neural network to training mode;

[0077] Step 3-4: From the training sample dataset And training label dataset Randomly extract B = sample data and label data to form a batch training sample data set and batch training labeled dataset ;

[0078] Step 3-5: Clear optimizer parameters for classification deep neural network The cumulative gradient value of

[0079] Step 3-6: Calculation Obtained through the network .Will and Input the loss function to obtain the loss value loss;

[0080] Step 3-7: Use the loss value to back propagate and use the optimizer to classify the deep neural network parameters Make updates;

[0081] Step 3-8: Repeat steps 3-4 to 3-7 until the data set is emptied.

[0082] Let’s evaluate the classification deep neural network:

[0083] Step 3-9: Set the classification deep neural network to evaluation mode;

[0084] Step 3-10: Calculate the test sample set Obtained through the network Get The position corresponding to the maximum value of the second dimension in the test label set The same probability, that is, accuracy If the probability Less than threshold =98%, then return to step 3-3; otherwise, save the network parameters of the classification deep neural network .

[0085] In this example, the loss value diagram is as follows Figure 1 As shown, the accuracy diagram is as follows Figure 2 shown.

[0086] Step 4: Obtain the style image and content image respectively, randomly generate an initial migration image, and optimize the initial migration image using an optimizer based on the style image, content image and trained SAR image classification deep neural network to obtain a style-migrated SAR image. The specific method is:

[0087] Step 4-1: Load the network parameters of the classification neural network .

[0088] Step 4-2: Randomly extract a SAR image sample from the dataset as the style image , the dimension is , style images such as Figure 3 As shown;

[0089] Step 4-3: Load the optical image and convert it into grayscale with the dimension , as the content image , content image such as Figure 4 shown.

[0090] Step 4-4: Initialize the migration image . Construct a random image with mean 0 and standard deviation 1 , the dimension is . Set weights , then the migration image is .

[0091] Step 4-5: The optimizer uses the Adam optimizer to optimize the migration image Optimize and set the training rate =0.001.

[0092] Step 4-6: Calculate the style image The output value of the classified neural network , , and ;

[0093] Step 4-7: Calculate the content image The output value of the classified neural network , , and ;

[0094] Let’s optimize the migration image:

[0095] Step 4-8: Clear the optimizer about the migration image The cumulative gradient value of the migration image The output value of the classified neural network , , and ;

[0096] Step 4-9: Calculate the content loss value:

[0097] ,

[0098] in, It means to find the mean of all elements.

[0099] Step 4-10: Calculate the style loss value:

[0100] ,

[0101] in, Compute the Gram matrix.

[0102] Step 4-11: Set weights =10, and the total migration loss is obtained .

[0103] Step 4-12: Back propagate the total transfer loss and use the optimizer to transfer the image The total migration loss value diagram is as follows Figure 5 shown.

[0104] Step 4-13: Repeat steps 4-5 to 4-12 10^6 times to obtain the final migration image. ,like Figure 6 shown.

[0105] In summary, the present invention utilizes convolutional neural networks and style transfer technology to extract visual features from optical images and convert them into features that conform to the style of SAR images, thereby realizing the automatic generation of optical images to SAR images. The SAR images generated by the present invention can be used to expand data sets and enhance data diversity, and have broad application prospects in the fields of remote sensing image analysis and data processing.

Claims

1. A SAR image generation method based on style transfer, characterized in that: The following steps are involved: Step 1: construct a training data set and a test data set based on SAR image data; Step 2: Build a deep neural network for SAR image classification based on graph neural network; Step 3, setting the loss function and optimizer, batch training the SAR image classification deep neural network through the training data set, and using the test data set for evaluation to obtain the trained SAR image classification deep neural network; Step 4: Obtain the style image and content image respectively, randomly generate an initial migration image, and optimize the initial migration image using an optimizer based on the style image, content image and trained SAR image classification deep neural network to obtain a style-migrated SAR image.

2. The SAR image generation method based on style transfer according to claim 1, characterized in that: The specific method of step 1 is: Obtain N SAR image data, which are divided into M categories, where each SAR image sample The dimension is , the corresponding category label is , ∈M; Construct a sample set based on SAR image data , build a label set . According to the proportion The sample set and label set are split to obtain a training data set and a test data set, where the training data set includes a training sample data set. And training label dataset , including data, the test data set includes the test sample set and the test label set , including data.

3. The SAR image generation method based on style transfer according to claim 2, characterized in that: In step 2, the deep neural network for SAR image classification based on graph neural network is composed of 3 convolutional layers, 1 pooling layer, the first Relu activation function, 2 fully connected layers, and the second Relu activation function in series. The five parts are respectively , , , , Indicates that the input of the deep neural network for SAR image classification is sample , , , , The outputs are: 。 4. The SAR image generation method based on style transfer according to claim 3, characterized in that: The specific method of step 3 is: Step 3-1, the loss function uses the cross entropy loss function, and the optimizer uses the Adam optimizer; Step 3-2, set the training rate ; Step 3-3, setting the SAR image classification deep neural network to training mode; Step 3-4, from the training sample data set And training label dataset Randomly extract B sample data and label data from the dataset to form a batch training sample data set and batch training labeled dataset ; Step 3-5: Clear the optimizer's network parameters for the SAR image classification deep neural network The cumulative gradient value of Steps 3-6, Input SAR image classification deep neural network, and get ,Will and Input the loss function and obtain the loss value loss; Step 3-7, use the loss value to back propagate and use the optimizer to adjust the network parameters Make updates; Step 3-8, repeat steps 3-4 to 3-7 until the data set is emptied; Step 3-9, setting the SAR image classification deep neural network to evaluation mode; Step 3-10: Set the test sample set Input SAR image classification deep neural network, and get , get The position corresponding to the maximum value of the second dimension in the test label set The same probability , if the probability Less than threshold , then return to step 3-3 to retrain, otherwise, save the network parameters , and obtain the trained SAR image classification deep neural network.

5. The SAR image generation method based on style transfer according to claim 4, characterized in that: The specific method of step 4 is: Step 4-1: Load network parameters ; Step 4-2: Randomly extract a SAR image sample from the SAR image data as the style image , the dimension is ; Step 4-3: Load the optical image to be converted and convert it into grayscale with the dimension , as the content image ; Step 4-4: Construct a random image with a mean of 0 and a standard deviation of 1 , the dimension is ; Set weights , get the initial migration image ; Step 4-5: Use Adam optimizer to optimize the migration image Optimize and set the training rate ; Step 4-6: Style image Input SAR image classification deep neural network, get , , , Output value , , and ; Step 4-7: Convert the content image Input SAR image classification deep neural network, get , , , The output value of , , and ; Step 4-8: Clear the optimizer about the migration image The accumulated gradient value will migrate the image Input SAR image classification deep neural network, get , , , Output value , , and ; Step 4-9: Calculate the content loss value: ， in, It means to find the mean of all elements; Step 4-10: Calculate the style loss value: ， in, Calculate for the Gram matrix; Step 4-11: Set weights , and the total migration loss is ; Step 4-12: Use the total transfer loss to back propagate through the optimizer to transfer the image Make updates; Step 4-13: Repeat steps 4-5 to 4-12 until the preset number of times is reached to obtain the final migration image. , which is the style-transferred SAR image.

Images