A UAV-borne high-fidelity SAR image reconstruction method for target detection

By introducing Wasserstein distance and KL divergence into the neural network, combined with multiple loss functions, the problem of low accuracy of SAR image recognition is solved, high-fidelity image reconstruction and data set expansion are achieved, and the accuracy of object detection is improved.

CN114549365BActive Publication Date: 2024-11-05远超航空科技(成都)有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210209912.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-03
Publication Date
2024-11-05
Estimated Expiration
2042-03-03

AI Technical Summary

Technical Problem

In the prior art, insufficient coherence noise and data sample size of SAR images lead to low recognition accuracy. Deep learning methods such as variational autoencoder and generative adversarial networks have problems with instability and gradient explosion during training, and poor generated image quality. .

Method used

Adopt a high-fidelity SAR image reconstruction method based on neural networks, including encoder, decoder, discriminator and classifier. Using Wasserstein distance and KL divergence, combining multiple loss functions, optimizes generator and discriminator to improve image reconstruction quality .

Benefits of technology

It effectively solves the problem of instability in network training, improves the quality of the generated SAR image, expands the data set, and improves the accuracy of target recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549365B_ABST
    Figure CN114549365B_ABST
Patent Text Reader

Abstract

The invention belongs to the field of radar image enhancement, and specifically relates to a method for reconstructing unmanned aerial vehicle high-fidelity SAR images for target detection, the method comprising: acquiring a SAR image to be reconstructed, preprocessing the SAR image, inputting the preprocessed image into a trained neural network, and obtaining a reconstructed SAR image; the neural network comprises an encoder, a decoder, a discriminator and a classifier; in the invention, Wasserstein distance is used for the discriminator, KL divergence is used for the middle layer of the generator, and multiple loss functions are used simultaneously, which can effectively solve the problem of network instability during training and improve the quality of generated images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of radar image enhancement, and in particular relates to an unmanned aerial vehicle-borne high-fidelity SAR image reconstruction method for target detection. Background Art

[0002] In the field of image processing, in order to accurately locate the target, it is necessary to identify the target in advance. UAV-mounted synthetic aperture radar (SAR) is often used to take photos. At the same time, in special occasions, the images taken are relatively few. However, there is a large amount of coherent speckle noise in SAR images, which has poor readability and is almost impossible to be recognized by the human eye. Generally, it is recognized through the network model in the machine. The model requires a large amount of data when training, so SAR images often need data expansion.

[0003] Research has shown that geometric methods such as rotation, inversion, scaling, and displacement transformation can expand SAR image data sets to a certain extent, and can reduce network overfitting and improve model training effects when used in network model training. However, from a macroscopic perspective, these operations can only change the position, angle, and other information represented by the image, and cannot increase the diversity of SAR images. Especially when the network does not understand the meaning of the image, the improvement in target recognition accuracy is limited.

[0004] In recent years, with the development of deep learning, researchers have proposed using deep learning-based methods to expand SAR images, which are mainly divided into two categories: variational autoencoders and generative adversarial networks. The variational autoencoder is an improvement on the autoencoder, which consists of an encoder and a decoder. The encoder compresses the data into the hidden layer z, and the decoder reconstructs the data based on the hidden layer z. The variational autoencoder can generate a variety of SAR images, but the generated SAR images are usually blurry. The generative adversarial network was proposed by Goodfellow et al., which is mainly based on the idea of ​​game theory. The generative adversarial network is mainly divided into two parts: the generator and the discriminator. The generator generates data, and the discriminator distinguishes between real data and generated data. The two are trained against each other, so that the capabilities of the generator and the discriminator are continuously improved, and finally the generated data and the real data are very similar, and the discriminator is difficult to distinguish. However, since the parameters of the generative adversarial network are difficult to set and the training is difficult, problems such as gradient explosion or gradient disappearance often occur, and the quality of the generated images is poor. Summary of the invention

[0005] In order to solve the problems existing in the above prior art, the present invention proposes a method for reconstructing high-fidelity SAR images onboard an unmanned aerial vehicle for target detection, the method comprising: obtaining a SAR image to be reconstructed and preprocessing the SAR image; inputting the preprocessed image into a trained neural network to obtain a reconstructed SAR image; the neural network comprises an encoder, a decoder, a discriminator and a classifier; in the present invention, Wasserstein distance is used for the discriminator, KL divergence is used for the middle layer of the generator, and multiple loss functions are used simultaneously, which can effectively solve the problem of network instability during training and improve the quality of the generated image;

[0006] The process of training a neural network includes:

[0007] S1: Obtain the original image data set, preprocess the original image data, and use the preprocessed data set as a training set;

[0008] S2: Initialize the model parameters, including the intermediate layer parameters, learning rate and training batch;

[0009] S3: Input the preprocessed image into two encoders of the neural network to obtain two convolution results, and use the two results as the mean and variance of the input image respectively; obtain the intermediate layer distribution of the input image according to the intermediate layer parameters, the mean and variance of the input image;

[0010] S4: randomly select a value on the middle layer distribution, and decode the selected value using a decoder to obtain a reconstructed image;

[0011] S5: input the reconstructed image into the classifier to obtain the category of the target in the reconstructed image;

[0012] S6: Input the reconstructed image and the image input to the neural network into the discriminator to judge whether the reconstructed image is true or false. If the image is true, the image is a reconstructed high-fidelity SAR image. If the reconstructed image is not true, the classified image is returned to step S3.

[0013] S7: Calculate the loss function of the model according to the reconstructed high-fidelity SAR image, and continuously adjust the parameters of the model. When the model training conditions are met, the model training is completed; the model training conditions include reaching the maximum training batch of the model or minimizing the loss function value.

[0014] Preferably, the process of preprocessing the SAR image includes: converting the format of the SAR image, and cropping the image after the format conversion, so that only one target is retained in the cropped image.

[0015] Preferably, the process of processing an image using an encoder includes: the encoder is composed of a convolution layer, a BN layer and a ReLU activation layer; the input image is converted into a tensor, and label information is added to the converted image and then input into the encoder. Each time the image passes through a convolution layer, the image size is reduced to 1 / 2 of the input image, and this is repeated three times; the convolved image is input into a fully connected layer to obtain an output result.

[0016] Preferably, the decoding process of the decoder includes: the decoder is composed of three deconvolution layers, the first deconvolution layer and the second deconvolution layer are respectively connected to a ReLU activation function, and the third deconvolution layer is connected to a Sigmoid activation function; the values ​​are taken on the distribution of the middle layer, input into the decoder, and after three deconvolution operations, an image equal to the size of the original image is finally generated.

[0017] Preferably, the process of the discriminator judging the authenticity of the reconstructed image includes: the process of the discriminator judging the authenticity of the reconstructed image includes: inputting the reconstructed image and the image input into the neural network into the discriminator, and outputting the probability of it being a true sample after convolution, pooling and activation operations, for a true sample, the output result is 1, and for a generated sample, the output result is 0; the formula for calculating the probability of a true sample is:

[0018]

[0019] Among them, D represents the discriminator, G represents the generator, m represents the batch size, x represents the real sample, and z represents the feature sampled from the intermediate layer.

[0020] Preferably, the loss function of the model includes the loss function of the encoder, the loss function of the decoder, the loss function of the classifier, and the loss function of the discriminator.

[0021] Furthermore, the loss function is expressed as:

[0022] L total =αL kl +βL rec +L dis +L C

[0023] L kl =KL(q φ (z|x (i) )||p θ (z|x (i) ))

[0024]

[0025]

[0026]

[0027] Among them, α and β are the weights of the corresponding network, L kl represents the loss function of the encoder, p θ (z|x (i) ) is the true posterior probability, q φ (z|x (i) ) is the posterior probability of the recognition model, z represents the distribution of the intermediate layer, and x (i) represents the real sample, L rec represents the loss function of the decoder, represents the distribution of the reconstructed image, represents the generated image, L dis represents the loss function of the discriminator, Represents the generated image Subject to the generated image P g Distribution, x~P r P represents the true image x that obeys the true image r distributed, represents the expected probability of judging the generated image as a real image, represents the probability of judging the generated image as a real image, represents the expected probability of judging the real sample as real, D(x) represents the probability of judging the real sample as real, L C represents the loss function of the classifier, P(c|x) represents the probability of classification as c, and c represents the category.

[0028] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements any of the above-mentioned unmanned aerial vehicle high-fidelity SAR image reconstruction methods for target detection.

[0029] To achieve the above-mentioned purpose, the present invention also provides an unmanned aerial vehicle (UAV)-mounted high-fidelity SAR image reconstruction device for target detection, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the unmanned aerial vehicle (UAV)-mounted high-fidelity SAR image reconstruction device for target detection performs any of the above-mentioned unmanned aerial vehicle (UAV)-mounted high-fidelity SAR image reconstruction methods for target detection.

[0030] Beneficial effects of the present invention:

[0031] The high-fidelity SAR image generation method based on variational generative adversarial network proposed in the present invention fully utilizes the advantages of variational autoencoder and generative adversarial network, and combined with the category information of the image, it can generate SAR images of specified categories, expand the SAR data set, and solve the problem of too small SAR image sample size. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a network structure diagram of the present invention;

[0033] Figure 2 is a partial scene graph of the data set used in the present invention;

[0034] Figure 3 It is a partial SAR image generated by the model in the present invention after training. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] A method for reconstructing high-fidelity SAR images carried by an unmanned aerial vehicle for target detection, the method comprising: obtaining a SAR image to be reconstructed, preprocessing the SAR image, inputting the preprocessed image into a trained neural network, and obtaining a reconstructed SAR image; the neural network comprises an encoder, a decoder, a discriminator, and a classifier; in the present invention, Wasserstein distance is used for the discriminator, KL divergence is used for the middle layer of the generator, and multiple loss functions are used simultaneously, which can effectively solve the problem of network instability during training and improve the quality of generated images;.

[0037] A specific implementation of a method for reconstructing high-fidelity SAR images onboard an unmanned aerial vehicle for target detection, wherein the process of training a neural network comprises:

[0038] S1: Obtain the original image data set and preprocess the original image data;

[0039] S2: Initialize the model parameters, including the intermediate layer parameters, learning rate and training batch;

[0040] S3: Input the preprocessed image into two encoders of the neural network respectively to obtain two convolution results, and use the two results as the mean and variance of the input image respectively; obtain the intermediate layer distribution of the input image according to the intermediate layer parameters, the mean and variance of the input image;

[0041] S4: randomly select a value on the middle layer distribution, and decode the selected value using a decoder to obtain a reconstructed image;

[0042] S5: input the reconstructed image into the classifier to obtain the category of the target in the reconstructed image;

[0043] S6: Input the reconstructed image and the image input to the neural network into the discriminator to judge whether the reconstructed image is true or false. If the image is true, the image is a reconstructed high-fidelity SAR image. If the reconstructed image is not true, the classified image is returned to step S3.

[0044] S7: Calculate the loss function of the model according to the reconstructed high-fidelity SAR image, and continuously adjust the parameters of the model. When the model training conditions are met, the model training is completed; the model training conditions include reaching the maximum training batch of the model or minimizing the loss function value.

[0045] like Figure 2 As shown, the original image dataset obtained is the MSTAR dataset. The data in the MSTAR dataset is a file type, which is first converted into a jpeg image format and then cropped. In the present invention, the images are uniformly cropped to a size of 128*128, and only one target object is retained in each image; in addition, after cropping, it is also necessary to ensure that the target object is located in the middle of the image for easy training.

[0046] During the model training process, the images in the training set are converted into tensors, and then the model parameters are set to θ, the training batch is 1000, the batch training size is 128, and in order to ensure the fairness of the training, the random seed is set to β.

[0047] like Figure 1 As shown, the neural network designed by the present invention consists of four parts: encoder, decoder, discriminator and classifier. The main function of the encoder is to extract the features of the picture, which is composed of a convolution layer, a BN layer and a ReLU activation layer. After each convolution layer, the size of the tensor is reduced to 1 / 2 of the original, and this is repeated three times, and then a fully connected layer is connected. Take two encoders, perform the same operation twice, and obtain two convolution results, which are used as the mean and variance respectively to obtain the distribution of the corresponding intermediate layer. The decoder is opposite to the encoder, and is mainly composed of three deconvolution layers. The first two deconvolution layers are followed by a ReLU activation function, and the last deconvolution layer is followed by a Sigmoid activation function to output a new tensor. The main function of the discriminator is to distinguish between the generated image and the real image. It is mainly composed of two convolution layers, a ReLU activation function layer, a maximum pooling layer, followed by a fully connected layer and a Sigmoid activation function layer. The classifier structure is similar to the discriminator structure, and its main function is to output the probability that the current generated image is a certain category. The encoder, decoder, discriminator and classifier are trained at the same time. When the training is completed, only the encoder and decoder are needed to generate images.

[0048] The process of using the encoder to process the input image includes: converting the image into a tensor, then connecting the label information, and sending it to the encoder, represented as x. After three convolutions, BN layers, and ReLU activation layers, and then passing through a fully connected layer, the mean and variance are obtained, and the distribution of the corresponding intermediate layer z is calculated. Assuming that the intermediate layer z follows the normal distribution z~N(0,I), since all data are independent and identically distributed, the distance between the two distributions is calculated using KL divergence:

[0049]

[0050] Among them, p θ (z|x (i) ) is the true posterior probability, q φ (z|x (i) ) is the posterior probability of the recognition model.

[0051] The process of using the decoder to decode the input data includes: randomly sampling from the middle layer z, then connecting the label information, and sending it to the decoder. Through a series of operations such as deconvolution, ReLU activation function, Sigmoid activation function, etc., a picture of a specific category is obtained, named Its size is 128*128. In order to improve the quality of the generated image, it is necessary to compare the generated image with the real image to calculate the reconstruction error, whose size is:

[0052]

[0053] L rec It can measure the difference between the generated image and the real image. If the generated image is similar to the real image, then L rec If the difference between the two is large, then L rec In addition, in order to generate better images, the Adam optimizer is selected as the optimization function, and the learning rate is set to 0.0001.

[0054] After generating the SAR image, the discriminator needs to be trained to enhance its ability to distinguish true from false images. The input data to the discriminator are two images, one is a real image without a label and the other is a generated image without a label. The discriminator scores them respectively, and the scoring range is [0,1]. The higher the score, the higher the confidence. The cross entropy loss function is used. The use of the cross entropy loss function can avoid the problem of gradient disappearance to a certain extent. Its loss function is:

[0055]

[0056] in Represents the generated image Subject to the generated image Pg Distribution, x~P r P represents the true image x that obeys the true image r Distribution, if the generated image is close to the real image, then L dis The value is small, otherwise it is large. For the optimization of the discriminator, the Adam optimizer is selected and the learning rate is set to 0.0001.

[0057] After the SAR image is generated, the classifier needs to be trained at the same time as the discriminator. The network structure of the classifier and the discriminator is the same. The output category is set to 10. It mainly determines which category the image input to the classifier belongs to. During training, the generated image is input into the selector, and then the cross entropy loss function is used to calculate the predicted category and the real category to obtain its loss, and then back propagation is performed to update the parameters. Adam is selected as the optimizer, and the learning rate is set to 0.0001. Its loss function is expressed as:

[0058]

[0059] Where x~P r P represents the true image x that obeys the true image r Distribution, if the predicted image category is close to the true category, then L C value is smaller, otherwise it is larger.

[0060] The loss function expression of the neural network in the present invention is:

[0061] L total =αL kl +βL rec +L dis +L C

[0062] Among them, L kl is KL(q φ (z|x (i) )||p θ (z|x (i) )), α and β are the weights of the corresponding network.

[0063] Figure 3 This is a partial SAR image generated using the model of the present invention.

[0064] In one embodiment of the present invention, the present invention also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned unmanned aerial vehicle high-fidelity SAR image reconstruction methods for target detection.

[0065] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0066] A high-fidelity SAR image reconstruction device for unmanned aerial vehicle for target detection comprises a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the high-fidelity SAR image reconstruction device for unmanned aerial vehicle for target detection performs any of the above-mentioned high-fidelity SAR image reconstruction methods for unmanned aerial vehicle for target detection.

[0067] Specifically, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk, etc., which can store program codes.

[0068] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0069] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for reconstructing high-fidelity SAR images onboard an unmanned aerial vehicle for target detection, characterized in that: The method comprises: obtaining a SAR image to be reconstructed and preprocessing the SAR image; inputting the preprocessed image into a trained neural network to obtain a reconstructed SAR image; the neural network comprises an encoder, a decoder, a discriminator and a classifier; The process of training a neural network includes: S1: Obtain an original image data set, where the images in the data set are real images, and preprocess the original image data; S2: Initialize the model parameters, including the intermediate layer parameters, learning rate and training batch; S3: Input the preprocessed real image into the two encoders of the neural network respectively to obtain two convolution results, and use the two results as the mean and variance of the input image respectively; obtain the intermediate layer distribution of the input image according to the intermediate layer parameters, the mean and variance of the input image; S4: randomly select a value on the middle layer distribution, and decode the selected value using a decoder to obtain a reconstructed image; S5: input the reconstructed image into the classifier to obtain the category of the target in the reconstructed image; S6: Input the reconstructed image and the image input to the neural network into the discriminator to judge whether the reconstructed image is true or false. If the image is true, then the image is a reconstructed high-fidelity SAR image. If it is false, the reconstructed image is not true, then the classified image is returned to step S3, and the reconstructed image fails. S7: Calculate the loss function of the model according to the reconstructed high-fidelity SAR image, and continuously adjust the parameters of the model. When the model training conditions are met, the model training is completed; the model training conditions include reaching the maximum training batch of the model or the minimum loss function value; the loss function of the model includes the loss function of the encoder, the loss function of the decoder, the loss function of the classifier, and the loss function of the discriminator; the expression of the loss function is: L total =αL kl +βL rec +L dis +L C L kl =KL(q φ (z|x (i) )||p θ (z|x (i) )) Among them, α and β are the weights of the corresponding network, L kl represents the loss function of the encoder, p θ (z|x (i) ) is the posterior probability of the real image, q φ (z|x (i) ) represents the approximate posterior distribution, z represents the intermediate layer distribution, and x (i) represents the real sample, L rec represents the loss function of the decoder, represents the probability of reconstructing the image, represents the generated image, L dis represents the loss function of the discriminator, Represents the generated image Subject to the generated image P g Distribution, x~P r P represents the true image x that follows the true image r distributed, represents the expected probability of judging the generated image as a real image, represents the probability of judging the generated image as a real image, represents the expected probability of judging the real sample as real, D(x) represents the probability of judging the real sample as real, L C represents the loss function of the classifier, P(c|x) represents the probability of classification as c, and c represents the category.

2. The method for reconstructing high-fidelity SAR images for target detection carried out by an unmanned aerial vehicle according to claim 1, characterized in that: The process of preprocessing the SAR image includes: converting the format of the SAR image, and cropping the converted image so that only one target is retained in the cropped image.

3. The method for reconstructing high-fidelity SAR images for target detection carried out by an unmanned aerial vehicle according to claim 1, characterized in that: The process of using the encoder to process the image includes: the encoder consists of a convolutional layer, a BN layer, and a ReLU activation layer; the input image is converted into a tensor, and the label information is added to the converted image and then input into the encoder. Each time the image passes through a convolutional layer, the image size is reduced to 1 / 2 of the input image, and this is repeated three times; the convolved image is input into the fully connected layer to obtain the output result.

4. The method for reconstructing high-fidelity SAR images for target detection carried out by an unmanned aerial vehicle according to claim 1, characterized in that: The decoding process of the decoder includes: the decoder is composed of three deconvolution layers, in which the first deconvolution layer and the second deconvolution layer are respectively connected to a ReLU activation function, and the third deconvolution layer is connected to a Sigmoid activation function; the values ​​are taken on the distribution of the middle layer and input into the decoder. After three deconvolution operations, an image of the same size as the original image is finally generated.

5. The method for reconstructing high-fidelity SAR images for target detection carried out by an unmanned aerial vehicle according to claim 1, characterized in that: The process of the discriminator judging the authenticity of the reconstructed image includes: inputting the reconstructed image and the image input to the neural network into the discriminator, and outputting the probability of it being a real sample after convolution, pooling and activation operations. For a real sample, the output result is 1, and for a generated sample, the output result is 0; the formula for calculating the probability of a real sample is: Among them, D represents the discriminator, G represents the generator, m represents the batch size, x represents the real sample, and z represents the feature sampled from the intermediate layer.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the unmanned aerial vehicle high-fidelity SAR image reconstruction method for target detection according to any one of claims 1 to 5.

7. A high-fidelity SAR image reconstruction device for target detection carried by an unmanned aerial vehicle, characterized in that: It comprises a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the unmanned aerial vehicle high-fidelity SAR image reconstruction device for target detection executes any one of claims 1 to 5 of the unmanned aerial vehicle high-fidelity SAR image reconstruction method for target detection.