A ground object classification method for single-channel high-resolution SAR images based on deep self-supervised generative adversarial model

Abstract

A single-channel high-resolution SAR image ground object classification method based on deep self-supervised generative adversarial analysis, wherein the upstream self-supervised task uses a cyclic generative adversarial network to perform a multi-domain image style conversion task, converting the style of real source domain images with different attributes into a target domain generated image with target attributes; and uses an adversarial network to distinguish whether the image comes from the source domain or the target domain, while extracting the attribute features of the image for classification and aligning the feature distribution; in the downstream ground object classification task, the upstream well-trained feature encoder is migrated as a pre-training model, and a small amount of labeled samples is used to fine-tune the classification model; the present invention provides a well-trained and versatile feature encoder for the downstream ground object classification task through the upstream self-supervised task, thereby alleviating the defects of lack of supervised training samples and insufficient feature generalization, and improving the classification performance of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of SAR image intelligent interpretation, and in particular relates to a single-channel high-resolution SAR image ground object classification method based on deep self-supervised generative adversarial technology. Background Art

[0002] Synthetic Aperture Radar (SAR) is an all-weather, all-day, active microwave imaging radar with a certain degree of penetration. These characteristics have enabled SAR imagery to be used in fields ranging from oceanography to archaeology. SAR image classification is a fundamental step in SAR interpretation and plays a vital role in numerous applications, including environmental protection and resource monitoring.

[0003] Effective and critical feature learning is crucial for object classification performance. Traditional feature extraction techniques include gray-level co-occurrence matrices, Gabor transforms, and sparse representations. Deep learning-based feature extraction uses a large number of training samples to train the network in an end-to-end manner, automatically learning features. Currently, deep learning-based feature extraction has achieved some success in SAR image object classification. However, existing deep learning methods that rely on supervised labeling are costly and face challenges such as cumbersome modeling steps, scarce labeled data, difficulty in image labeling, and limited versatility due to feature differences under different imaging parameters.

[0004] Existing methods usually use self-supervised learning to alleviate the above problems. Self-supervised learning generally consists of upstream auxiliary tasks and downstream target tasks. The upstream automatically mines the representational features of unlabeled data as supervision information by designing auxiliary tasks, and trains the model to improve feature learning capabilities; the downstream task transfers the feature encoder of the upstream task and trains the target task model with a small number of samples to obtain better classification performance. Self-supervised learning can be divided into generative self-supervised learning and discriminative self-supervised learning according to the upstream task. Generative adversarial network (GAN) is a mainstream method of generative self-supervised learning. It requires generating or reconstructing all or part of the input image. In this process, the model can learn the semantic features of the image and include some pixel-level detail information. Generative self-supervised learning based on reconstructing all or part of the input image is limited by the model and computational complexity, and the versatility of its feature extraction is still insufficient. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a single-channel high-resolution SAR image object classification method based on deep self-supervised generative adversarial learning, which adopts a multi-domain image style conversion task as an upstream auxiliary task to learn the feature representation of the image; a single model is used to realize style conversion between SAR images of multiple scenes and with different imaging parameters, which greatly simplifies the modeling steps, improves the model's feature learning of objects with different forms of expression, and reduces the demand for labeled data for downstream object classification tasks.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A single-channel high-resolution SAR image object classification method based on deep self-supervised generative adversarial learning includes the following steps:

[0008] Step 1: Input a high-resolution SAR image and randomly crop the same number of image patches for each scene and category of the input SAR image to form the training data for the upstream image style conversion task and the downstream ground object classification task;

[0009] Step 2: Preprocess the upstream style transfer task training data, including SAR image cropping, normalization, and pseudo-label generation.

[0010] Step 3: Input the source domain real image into the encoder network E of the upstream image style conversion task to extract image features; at the same time, send the target label into the mapping network M to generate the target domain style code; finally, send the image features and target domain style code into the generator network G to obtain the target domain generated image;

[0011] Step 4: Feed the source domain real image and the target domain generated image into the discriminator network D for training, and update the discriminator network parameters;

[0012] Step 5: Input the source domain real image and target label again to generate the target domain generated image. Then input the target domain generated image and source domain label into the encoder network, mapping network and generator network to obtain the source domain reconstructed image.

[0013] Step 6: Use the source domain reconstructed image and the source domain real image to calculate the reconstruction loss. At the same time, the source domain real image and the target domain generated image are fed into the discriminator network D for training. Then, the adversarial loss, classification loss, distribution feature similarity loss, and image feature similarity loss are calculated, and the encoder network, mapping network, and generator network parameters are updated.

[0014] Step 7: Repeat steps 3 to 6 until the maximum number of training times is reached;

[0015] Step 8: Evaluate the performance of the upstream image style transfer task.

[0016] Step 9: Migrate the encoder network E of the evaluated upstream image style conversion task as a feature extraction pre-training model to the downstream object classification network, and fine-tune the classification network;

[0017] Step 10: Use the downstream trained semantic segmentation model to classify high-resolution SAR images, obtain the prediction result map, and calculate the classification index.

[0018] Compared with existing SAR image classification technologies, the present invention has the following advantages:

[0019] As a self-supervised deep learning method, this technique can fully utilize a large amount of unsupervised data when supervised data samples are insufficient, alleviating the overfitting and low accuracy problems of supervised methods in the case of few samples. Furthermore, as a multi-domain image style transfer task, the upstream task generatively and adversarially learns the differences in the characteristics of different object categories under different imaging parameters and regions through image encoding and stylization, improving the model's feature extraction and generalization capabilities. In the upstream task, the image domain style information is controlled by the generator, while the content information features unrelated to the domain style are extracted by the feature encoder. Therefore, migrating to the downstream feature encoder can prevent the influence of different image style attributes on model classification performance. This form of multi-domain image style transfer uses the target label to control the target domain of the image style transfer, allowing a single model to achieve image style transfer tasks in multiple domains, significantly improving modeling efficiency. Finally, as a generative model, this model can effectively generate SAR images with target attributes, facilitating the expansion of certain datasets with sparse samples. The present invention adopts a typical generator-discriminator adversarial network structure, in which the generator part mainly adopts residual network connection, which can effectively expand the network depth and enhance the network's fitting ability; the adversarial structure promotes the generation of more detailed and realistic images, and improves the network's feature extraction ability; this method is also suitable for image detection and recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention.

[0021] Figure 2 It is a SAR image of the Napoli area in the upstream image style transfer task.

[0022] Figure 3 It is the SAR image of the PoDelta region in the upstream image style conversion task.

[0023] Figure 4 It is a SAR image of the Rosenheim area for the upstream image style transfer task.

[0024] Figure 5It is the SAR image of the Jiujiang area in the upstream image style conversion task.

[0025] Figure 6 This is the image style conversion task of the upstream image style conversion on four categories of objects to images in different regions, where (a), (b), (c) and (d) represent the generated images after the four categories of objects, namely water, forest, building and farmland, are converted to other domains; the images in each small picture are, from left to right, the real image of the source domain, the generated image of the target domain in the styles of Napoli, PoDelta, Rosenheim and JiuJiang.

[0026] Figure 7 It is the label map of the Jiujiang area in the downstream feature classification task.

[0027] Figure 8 It is the prediction map of the Jiujiang area using the upstream pre-training model in the downstream land feature classification task.

[0028] Figure 9 It is the prediction map of Jiujiang area in the downstream feature classification task without using the upstream pre-training model. DETAILED DESCRIPTION

[0029] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0030] Reference Figure 1 A single-channel high-resolution SAR image object classification method based on deep self-supervised generative adversarial learning includes the following steps:

[0031] Step 1: Input a high-resolution SAR image and randomly crop the same number of image patches for each scene and category of the input SAR image to form the training data for the upstream image style conversion task and the downstream ground object classification task;

[0032] This example inputs high-resolution single-channel SAR images of Napoli, PoDelta, Rosenheim, and Jiujiang, which are from different regions, have different imaging parameters, and are generated by different sensors. 400 samples are randomly cropped from each of the four categories: water, forest, building, and farmland. Each sample is a 200×200 image block, forming the upstream image style transfer task training dataset. 32 samples of each category of ground objects in the upstream image style transfer task training dataset are randomly selected to form the downstream ground object classification task training dataset.

[0033] Step 2: Preprocess the upstream style transfer task training data, including SAR image cropping, normalization, and pseudo-label generation.

[0034] This example uses a center cropping method for each sample in the training dataset to obtain a 128*128 image block. Each sample is divided by 255 to compress the dynamic range of the image to between 0 and 1. The image is then normalized by subtracting the mean of the entire dataset and dividing by the standard deviation. Each sample is labeled based on its image block category, region, resolution, polarization mode, satellite type, and band. This serves as a pseudo-label for the upstream image style transfer task.

[0035] Step 3: Input the source domain real image into the encoder network E of the upstream image style conversion task to extract image features; at the same time, send the target label into the mapping network M to generate the target domain style code; finally, send the image features and target domain style code into the generator network G to obtain the target domain generated image;

[0036] The encoder network E in the upstream image style conversion task of this embodiment consists of one convolutional layer, four downsampled residual blocks with average pooling, and two residual blocks without average pooling. Each residual block contains two convolutional layers, an instance normalization layer (IN), and a LeakyReLU activation function. The encoder network inputs the source domain real image s and outputs the image feature E(s) of size 512*8*8. The mapping network is an 8-layer MLP network that inputs the source domain real image to which the style conversion target label l is to be converted. t , output 64-dimensional target domain style code M(l t ); The generator network G consists of 1 convolutional layer, 4 residual blocks with adaptive instance normalization (AdaIN) and upsampling, and 2 adaptive instance normalization residual blocks without upsampling. It inputs the source domain real image feature E(s) and passes the target domain style code M(l t ) controls the adaptive instance normalization in the generator network to output the target domain generated image s'=G(E(s),M(l t ));

[0037] Step 4: Feed the source domain real image and the target domain generated image into the discriminator network D for training, and update the discriminator network parameters;

[0038] The discriminator network D in this embodiment consists of one convolutional layer and five residual blocks with downsampling and no normalization, and is connected to three convolutional layers as three output branches. The first branch uses the PatchGANs method to output a feature block to identify whether the input image is from the source domain or the target domain. The second branch outputs category scores for various attributes of the image. By classifying the various attributes of multi-domain images, more information is provided for generator optimization, which is conducive to improving the effect of generating images in the target domain and enabling a single discriminator to more flexibly control different image domains. The third branch outputs the distribution characteristics of the image, which facilitates the model to align the distribution on deep semantic features. Considering that GAN is different from traditional neural networks in optimizing in the direction of minimizing the objective function, GAN optimizes the generator and discriminator alternately and in opposite directions, forming an adversarial relationship, which is difficult to control in terms of training efficiency and stability. In order to stabilize the training process and generate higher quality images, this embodiment replaces the objective function of the traditional GAN ​​with the objective function of Wasserstein GAN with gradient penalty. Therefore, the loss function of the discriminator network consists of the following parts:

[0039] (1) Adversarial loss between source domain real images and target domain generated images:

[0040]

[0041] Where s represents the real image in the source domain, l t represents the target label, E, G and M represent the encoder network, generator network and mapping network respectively, and D src represents the adversarial branch of the discriminator network, EX represents the mathematical expectation;

[0042] For the discriminator network D, it is necessary to distinguish the real source domain images from the target domain generated images as much as possible, that is, to maximize the above loss; while the generator network G needs to make the discriminator network unable to distinguish the target domain generated images as much as possible, that is, to minimize the above loss;

[0043] (2) Classification loss of real images in the source domain:

[0044]

[0045] Among them, n represents the number of attribute categories of the image, l i represents the category of the i-th attribute of the image, s represents the real image in the source domain, and D cls represents the classification branch of the discriminator network, EX represents the mathematical expectation;

[0046] (3) Gradient penalty:

[0047]

[0048] in, represents uniform sampling along the straight line between the source domain real image and the target domain generated image, D src represents the adversarial branch of the discriminator network, is the derivative symbol, ||*||2 is the 2-norm, and EX represents the mathematical expectation;

[0049] (4) Objective function L of the discriminator network D D It is expressed as follows:

[0050]

[0051] Among them L adv , L r cls and L gp are adversarial loss, source domain real image classification loss and gradient penalty, respectively, cls and λ gp Represent the weight coefficients of the source domain real image classification loss term and the gradient penalty term respectively;

[0052] The final optimization goal is to minimize the above objective function and update the discriminator network D parameters;

[0053] Step 5: Input the source domain real image and target label again to generate the target domain generated image. Then input the target domain generated image and source domain label into the encoder network, mapping network and generator network to obtain the source domain reconstructed image.

[0054] Since the image samples used in the upstream are unpaired, and the image style conversion task requires the model to only convert the style-related information of the input sample while retaining the content feature information of the input sample, a method similar to CycleGAN is used to pass the target domain generated image s' generated by the source domain real image through the encoder network E again to obtain the target domain generated image feature E(s'), and use the source domain label l s Generate style code M(l s ), the target domain generates image features E(s') and style codes M(l s ) Input the generator network G and obtain the source domain reconstructed image G(E(s'),M(l s ));

[0055] Step 6: Use the source domain reconstructed image and the source domain real image to calculate the reconstruction loss. At the same time, the source domain real image and the target domain generated image are fed into the discriminator network D for training. Then, the adversarial loss, classification loss, distribution feature similarity loss, and image feature similarity loss are calculated, and the encoder network, mapping network, and generator network parameters are updated.

[0056] Compared with the discriminator network D, the generator network G needs to ensure that the content of the target domain generated image is similar to that of the source domain real image, and to ensure a one-to-one correspondence between multi-domain images. Therefore, the objective function of the generator network G adds image reconstruction loss and image feature similarity loss; the reconstruction loss is calculated by the source domain reconstructed image and the source domain real image in step 5; at the same time, the source domain real image and the target domain generated image in step 5 are sent to the discriminator network D for training, and then the adversarial loss, classification loss, distribution feature similarity loss and image feature similarity loss are calculated, and the encoder network, mapping network and generator network parameters are updated; the expressions of the objective functions of the encoder network E, mapping network M and generator network G are as follows:

[0057] (1) Image reconstruction loss:

[0058]

[0059] s′=G(E(s),M(l t ))

[0060] Where s represents the real image in the source domain, s' represents the generated image in the target domain, G(E(s'),M(l s )) represents the source domain reconstructed image generated by the target domain generated image, l t is the target label, l s is the source domain label, G, E, and M represent the generator network, encoder network, and mapping network respectively, EX represents the mathematical expectation, and ||*||1 represents the 1-norm;

[0061] (2) Image feature similarity loss L sim It is expressed as follows:

[0062]

[0063] Among them, MSE(*) represents the minimum mean square error, l t represents the target label, G, E, and M represent the generator network, encoder network, and mapping network respectively, and EX represents the mathematical expectation;

[0064] (3) Distribution feature similarity loss between source domain real images and target domain generated images:

[0065] L dis =EX s,s′ [kl(D dis (s),D dis (s′))]

[0066] Among them, s represents the source domain real image, s' represents the target domain generated image generated by the source domain real image, kl(*) represents the use of kl divergence to measure the similarity between the two distributions, Ddis is the distribution feature branch of the discriminator network, EX represents the mathematical expectation;

[0067] (4) Classification loss of generated images in the target domain:

[0068]

[0069] Among them, s′ represents the target domain generated image, D cls represents the classification branch of the discriminator network, l i represents the category of the i-th attribute of the image, n represents the number of attribute categories of the image, and EX represents the mathematical expectation;

[0070] (5) Objective functions of encoder network E, mapping network M and generator network G:

[0071]

[0072] Among them L adv 、 L dis 、L rec and L sim They represent adversarial loss, classification loss of target domain generated images, distribution feature similarity loss, reconstruction loss, and feature similarity loss respectively; λ cls ,λ dis ,λ rec and λ sim Represent the weight coefficients of the target domain generated image classification loss term, distribution feature similarity loss term, reconstruction loss term, and feature similarity loss term respectively;

[0073] The final optimization goal is to minimize the above objective function and update the network parameters of the encoder network E, mapping network M and generator network G; in the experiment, take λ cls =1,λ dis =10,λ rec =50,λ sim =50,λ gp = 10, Adam optimizer with a learning rate of 0.001, β1 of 0.5, and β2 of 0.999 for all networks;

[0074] Step 7: Repeat steps 3 to 6 until the maximum number of training times is reached;

[0075] Within the set maximum number of training times, observe the training situation until the requirements are met; the maximum number of training times set in this embodiment is 100,000 times;

[0076] Step 8: Evaluate the performance of the upstream image style transfer task.

[0077] In the upstream image style conversion task, the following metrics are used to measure the quality of the generated images in the target domain:

[0078] (1) Mean μ x It is expressed as follows:

[0079]

[0080] Where H and W represent the height and width of the image respectively, I i,j Represents the grayscale value of the image at point (i, j);

[0081] The mean of an image is used to describe the average brightness of the entire image. The backscatter intensity of the target varies in different regions and imaging parameters, resulting in differences in the mean of the image.

[0082] (2) Variance σ 2 x It is expressed as follows:

[0083]

[0084] Where H and W represent the height and width of the image respectively, I i,j Represents the grayscale value of the image at point (i, j), μ x Expressed as the mean of image x; the variance of the image describes the degree of deviation of the image pixel value from the mean and the unevenness of the style conversion image, which can effectively express the details of the image;

[0085] (3) Dynamic range DR is expressed as follows:

[0086]

[0087] Among them I max and I min Respectively represent the maximum and minimum grayscale values ​​in the image; dynamic range describes the degree of difference between the maximum and minimum values ​​of all pixel values ​​in the image; for images with rich features, such as trees and buildings, the dynamic range is large; for images with a single feature, such as water areas and farmland, the dynamic range is small;

[0088] (4) The equivalent coefficient ENL is expressed as follows:

[0089]

[0090] where μ x is the mean of image x, σ 2 x is the variance of image x; the equivalent coefficient can be used to reflect the intensity of speckle noise in the image, reflecting the contrast of noise intensity in the image; when the equivalent coefficient is relatively large, it means that its multiplicative noise is weak;

[0091] (5) The brightness similarity l(x, y) and contrast similarity c(x, y) are expressed as follows:

[0092]

[0093]

[0094] Among them, x, y represent the input image, μ x , μ y represent the mean of image x and y respectively, and Represents the variance of images x and y respectively; C1=(k1L) 2 and C2=(k2L) 2 are two factors used to stabilize the calculation, where L = 255, k1 = 0.01, k2 = 0.03;

[0095] (6) 1-NN classifier accuracy:

[0096] The basic idea is to calculate whether the distribution of real data and generated data is equal. If they are equal, it proves that the generated model is effective. If the gap is relatively large, it means that the model is relatively poor. Specifically, the real images in the source domain and the generated images in the target domain and their corresponding pseudo labels are combined into a new sample set. The leave-one-out cross-validation method is used to divide the entire sample into two parts, D1 and D2, where D2 has only one sample and D1 has all the remaining samples. D1 is used to train the 1-NN binary classifier, and the accuracy is calculated by verification in D2. Each time D2 selects a different sample, the above process is repeated, and the overall classification accuracy is finally calculated, and the accuracy is used as the evaluation indicator of the 1-NN classifier. For the 1-NN classifier, when its accuracy is 0.5, the image generation effect is the best; when the accuracy is 1, it means that the model is just a simple memory and does not really generate the desired image; when the accuracy is 0, it means that the model cannot complete the image generation task well.

[0097] Step 9: Migrate the encoder network E of the evaluated upstream image style conversion task as a feature extraction pre-training model to the downstream object classification network, and fine-tune the classification network;

[0098] According to the indicators in step 8, the encoder network E of the model with the best generation effect is selected as the pre-trained feature extraction model of the downstream segmentation network; in the downstream land object classification task of this embodiment, the pre-trained encoder network E is used to stack 4 convolution layers as the downstream land object classification model to perform SAR image land object classification; and the downstream land object classification task training data obtained in step 1 is normalized and sent to the downstream land object classification network for fine-tuning the classification network; 200*200 size image blocks are used for training in the downstream land object classification task; the downstream land object classification task experiment sets the number of training rounds to 500 rounds, the learning rate is 0.001, β1 is 0.5, and β2 is 0.999 Adam optimizer; the optimization goal is to minimize the multi-classification cross entropy loss between the category score output by the model and the true category;

[0099] Step 10: Use the downstream trained semantic segmentation model to classify the high-resolution SAR image, obtain the prediction result map, and calculate the classification index;

[0100] In this example, the high-resolution SAR image tested was overlaid and cropped to obtain a 200*200 image block with an overlap size of 86. The purpose of overlapping sampling is to remove the impact of the prediction boundary of the cropped small image on the performance of the final large image. The prediction results are input into the trained ground feature classification model to obtain the final prediction results, which are then spliced ​​into a complete large image.

[0101] In this invention, the indicators Precision, Recall, and F1ccore are calculated to evaluate the performance of a single category, and the indicators OA, kappa, MIoU, and FWIoU are calculated to evaluate the overall classification performance.

[0102] 1. Experimental conditions and methods:

[0103] The hardware platform is: GTX 2080Ti 12G, 256GB RAM;

[0104] The software platform is: Ubuntu 18.04.6LTS, PyTorch 1.7.1;

[0105] Experimental methods: supervised CNN and the method of the present invention

[0106] 2. Simulation content and results:

[0107] Figure 2-5 They are the SAR images of Napoli, PoDelta, Rosenheim and Jiujiang regions used in the upstream image style conversion task. Figure 6 It is the result of style conversion of each category of image blocks to images of different regions in the downstream object classification task; Figure 7It is the label map of Jiujiang area in the downstream feature classification task; Figure 8 This is the prediction map of the Jiujiang region using the upstream pre-trained model in the downstream feature classification task; Figure 9 It is the prediction map of Jiujiang area in the downstream feature classification task without using the upstream pre-training model.

[0108] The detailed data parameters used in the simulation experiment of the present invention are shown in Table 1 below.

[0109] Table 1. Detailed parameters of the data used in this invention

[0110] area size Resolution Polarization satellite Band Napoli 18332*16000 2.5m HH Cosmo-SkyMed X-band PoDelta 18308*16716 2.5m HH Cosmo-SkyMed X-band Rosenheim 7691*7224 1.75m HH TerraSAR-X X-band Jiujiang 8000*8000 3m DV GF3 C-band

[0111] We randomly cropped 400 images of each of the four categories of land features (water, forest, building, and farmland) in the four regions in Table 1, with a size of 200*200, to form the upstream image style transfer task dataset. For the downstream land feature classification task, we selected SAR images of the Jiujiang region for simulation experiments. The high-resolution SAR images were cropped into several small image blocks. Non-overlapping image blocks were used for training, and overlapping image blocks were used for testing.

[0112] The definitions of various evaluation indicators are as follows:

[0113] (1) Precision: The ratio of correctly predicted positive results for each category to all positive predictions for that category, defined as follows:

[0114]

[0115] Among them, p ij represents the number of samples whose true category is class j but predicted to be class i, and C represents the number of ground object categories;

[0116] (2) Recall: The ratio of correctly predicted positive samples in each class to all positive samples in that class, defined as follows:

[0117]

[0118] Among them, p ji represents the number of samples whose true category is category i but predicted to be category j, and C represents the number of ground object categories;

[0119] (3) F1 value: It is the harmonic mean of recall and precision and is defined as follows:

[0120]

[0121] Where P is the precision rate and R is the recall rate;

[0122] (4) Overall accuracy (OA): The ratio of correctly labeled pixels to the total pixels, defined as follows:

[0123]

[0124] Among them, p ij represents the number of samples whose true category is class j but predicted to be class i, and C represents the number of ground object categories;

[0125] (5) Kappa coefficient: Penalizes the "bias" of the model to obtain a more fair model, defined as follows:

[0126]

[0127]

[0128] Among them, OA represents the overall accuracy, a i represents the number of true samples of category i, b i Indicates the number of samples predicted to be class i, and N is the total number of samples;

[0129] (6) Mean Intersection over Union (MIoU): The average of the intersection over union ratios of each predicted result and the true value, defined as follows:

[0130]

[0131] Among them, p ij represents the number of samples whose true category is class j but predicted to be class i, and C represents the number of ground object categories;

[0132] (7) Frequency-weighted Intersection-over-Union (FWIoU): This is an improvement on MIoU, where weights are set based on the frequency of occurrence of categories. It is defined as follows:

[0133]

[0134] Among them, p ij represents the number of samples whose true category is class j but predicted to be class i, and C represents the number of ground object categories;

[0135] The simulation results of the upstream image style conversion task are as follows Figure 6 As shown in Figure 2, the mean, variance, dynamic range, equivalent coefficient and 1-NN accuracy of the source domain real images and the target domain generated images are used as evaluation indicators; the evaluation results are shown in Table 2, which shows the evaluation indicators of the source domain real images and target domain generated images of four types of land objects in the Napoli area. The first row of each type of land object in the table is the evaluation indicator of the source domain real image, and the second row is the evaluation indicator of the target domain generated image.

[0136] Table 2. Evaluation indicators of style transfer for four types of ground features in Napoli area

[0137]

[0138] The simulation results of the downstream image classification task are as follows Figure 8 、 9 As shown in the figure, Model_1 is the self-supervised method of the present invention, and Model_2 is the supervised method. Both models use 128 scene-level SAR images as training data. Compared with pixel-level training data, scene-level training data has less label information, which reflects the effect of the present invention in scenarios where label data is scarce.

[0139] Table 3. Comparison table of simulation results of image object classification in Jiujiang area

[0140] Evaluation indicators OA Kappa MIoU FWU Model_1 0.7941 0.6981 0.5734 0.7242 Model_2 0.6782 0.5407 0.4351 0.6103

[0141] 4. Analysis of experimental results:

[0142] For the upstream image style transfer task shown in Table 2, the target domain generated images are close to the source domain real images in terms of image mean, variance, and equivalent coefficient, resulting in high style similarity in image space. The accuracy of the 1-NN classifier demonstrates that the model performs well across the entire category of buildings and farmland. Therefore, the model for the upstream image style transfer task effectively learns the semantic information in SAR images and performs conversions between different image domains.

[0143] As for the indicators in Table 3, in the downstream land feature classification task, after using the upstream feature extraction pre-training model, the four evaluation indicators of the model, OA, kappa coefficient, MIoU and FWIoU, all have an accuracy improvement of more than 10% compared with the supervised model. For the Jiujiang area, the characteristics of its water area are relatively easy to learn. With a small number of samples, the method of the present invention and the supervised method with a small number of samples can both be well classified. However, the building and forest categories in the Jiujiang area are complex, and the classification performance of the supervised method with a small number of samples is greatly reduced. The method of the present invention has learned the feature representation of the building and forest categories in the upstream image style conversion task, and has a good starting point in the downstream land feature classification task. With a small number of samples, there is a large improvement in classification performance compared with the supervised method. More intuitive effects can be seen. Figure 8 、 9 .

[0144] In summary, the single-channel high-resolution SAR image object classification method based on deep self-supervised generative adversarial learning proposed in this invention provides a good pre-training model for the downstream object classification task by evaluating the upstream image style conversion task when sample labels are scarce or insufficient, which can improve the feature extraction ability of the model and thus improve the classification performance of SAR images.

[0145] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof; when implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.)) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0146] The above description is merely an embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A single-channel high-resolution SAR image object classification method based on deep self-supervised generative adversarial learning, characterized by: The following steps are involved: Step 1: Input a high-resolution SAR image and randomly crop the same number of image patches for each scene and category of the input SAR image to form the training data for the upstream image style conversion task and the downstream ground object classification task; Step 2: Preprocess the upstream style transfer task training data, including SAR image cropping, normalization, and pseudo-label generation. Step 3: Input the source domain real image into the encoder network E of the upstream image style conversion task to extract image features; at the same time, send the target label into the mapping network M to generate the target domain style code; finally, send the image features and target domain style code into the generator network G to obtain the target domain generated image; Step 4: Feed the source domain real image and the target domain generated image into the discriminator network D for training, and update the discriminator network parameters; Step 5: Input the source domain real image and target label again to generate the target domain generated image, and then input the target domain generated image and source domain label into the encoder, mapping network and generator to obtain the source domain reconstructed image; Step 6: Use the source domain reconstructed image and the source domain real image to calculate the reconstruction loss. At the same time, the source domain real image and the target domain generated image are fed into the discriminator network D for training. Then, the adversarial loss, classification loss, distribution feature similarity loss, and image feature similarity loss are calculated, and the encoder network, mapping network, and generator network parameters are updated. Step 7: Repeat steps 3 to 6 until the maximum number of training times is reached; Step 8: Evaluate the performance of the upstream image style transfer task. Step 9: Migrate the encoder network E of the evaluated upstream image style conversion task as a feature extraction pre-training model to the downstream object classification network, and fine-tune the classification network; Step 10: Use the downstream trained semantic segmentation model to classify high-resolution SAR images, obtain the prediction result map, and calculate the classification index.

2. The method according to claim 1, characterized in that Step 1 is as follows: High-resolution single-channel SAR images of different regions, different imaging parameters, and different sensors are input, and 400 samples are randomly cropped from each of the four categories of water, forest, building, and farmland. Each sample is an image block of 200×200, forming the upstream image style conversion task training dataset; 32 samples of each category of ground objects in the upstream image style conversion task training dataset are randomly selected to form the downstream ground object classification task training dataset.

3. The method according to claim 1, wherein: Step 2 is as follows: Each sample in the training dataset is center-cropped to obtain a 128*128 image block. The dynamic range of each sample is compressed to between 0 and 1 by dividing it by 255. The image is then normalized by subtracting the mean of the entire dataset and dividing it by the standard deviation. Each sample is labeled with attributes based on the category, region, resolution, polarization mode, satellite category, and band of its image block, which serve as pseudo-labels for the upstream image style transfer task.

4. The method according to claim 1, wherein: Step 3 is as follows: The encoder network E in the upstream image style conversion task consists of 1 convolutional layer, 4 downsampled residual blocks with average pooling, and 2 residual blocks without average pooling; each residual block contains two convolutional layers, an instance normalization layer (IN), and a LeakyReLU activation function; the encoder network inputs the source domain real image s and outputs an image feature E(s) of size 512*8*8; the mapping network is an 8-layer MLP network that inputs the source domain real image to which the style is to be converted. t , output 64-dimensional target domain style code M(l t ); The generator network G consists of 1 convolutional layer, 4 residual blocks with adaptive instance normalization (AdaIN) and upsampling, and 2 adaptive instance normalization residual blocks without upsampling. It inputs the source domain real image feature E(s) and passes the target domain style code M(l t ) controls the adaptive instance normalization in the generator network to output the target domain generated image s'=G(E(s),M(l t )).

5. The method according to claim 1, characterized in that Step 4 is as follows: The discriminator network D consists of one convolutional layer and five residual blocks with downsampling and no normalization, with three convolutional layers connected as three output branches. The first branch uses the PatchGANs method to output a feature block to identify whether the input image is from the source domain or the target domain. The second branch outputs the category scores of various image attributes. The third branch outputs the distribution characteristics of the image. The objective function of the Wasserstein GAN with gradient penalty is used. The loss function of the discriminator network consists of the following parts: (1) Adversarial loss between source domain real images and target domain generated images: Where s represents the real image in the source domain, l t represents the target label, E, G and M represent the encoder network, generator network and mapping network respectively, and D src represents the adversarial branch of the discriminator network, EX represents the mathematical expectation; For the discriminator network D, it is necessary to distinguish the real source domain images from the target domain generated images as much as possible, that is, to maximize the above loss; while the generator network G needs to make the discriminator unable to distinguish the target domain generated images as much as possible, that is, to minimize the above loss; (2) Classification loss of real images in the source domain: Among them, n represents the number of attribute categories of the image, l i represents the category of the i-th attribute of the image, s represents the real image in the source domain, and D cls represents the classification branch of the discriminator network, EX represents the mathematical expectation; (3) Gradient penalty: in, represents uniform sampling along the straight line between the source domain real image and the target domain generated image, D src represents the adversarial branch of the discriminator network, is the derivative symbol, ||*||2 is the 2-norm, and EX represents the mathematical expectation; (4) Objective function L of the discriminator network D D It is expressed as follows: Among them L adv , L r cls and L gp are adversarial loss, source domain real image classification loss and gradient penalty, respectively, cls and λ gp Represent the weight coefficients of the source domain real image classification loss term and the gradient penalty term respectively; The final optimization goal is to minimize the above objective function and update the parameters of the discriminator network D.

6. The method according to claim 1, wherein: Step 5 is as follows: The target domain generated image s' generated by the source domain real image is passed through the encoder network E again to obtain the target domain generated image feature E(s'), and the source domain label l is used s Generate style code M(l s ), the target domain generates image features E(s') and style codes M(l s ) Input the generator network G and obtain the source domain reconstructed image G(E(s'),M(l s )).

7. The method according to claim 1, characterized in that Step 6 is as follows: The reconstruction loss is calculated from the source domain reconstructed image and the source domain real image in step 5. At the same time, the source domain real image and the target domain generated image in step 5 are fed into the discriminator network D for training. Then, the adversarial loss, classification loss, distribution feature similarity loss, and image feature similarity loss are calculated, and the encoder network, mapping network, and generator network parameters are updated. The objective functions of the encoder network E, mapping network M, and generator network G are expressed as follows: (1) Image reconstruction loss: s′=G(E(s),M(l t )) Where s represents the real image in the source domain, s' represents the generated image in the target domain, G(E(s'),M(l s )) represents the source domain reconstructed image generated by the target domain generated image, l t is the target label, l s is the source domain label, G, E, and M represent the generator network, encoder network, and mapping network respectively, EX represents the mathematical expectation, and ||*||1 represents the 1-norm; (2) Image feature similarity loss L sim It is expressed as follows: Among them, MSE(*) represents the minimum mean square error, l t represents the target label, G, E, and M represent the generator network, encoder network, and mapping network respectively, and EX represents the mathematical expectation; (3) Distribution feature similarity loss between source domain real images and target domain generated images: L dis =EX s,s′ [kl(D dis (s),D dis (s′))] Among them, s represents the source domain real image, s' represents the target domain generated image generated by the source domain real image, kl(*) represents the use of kl divergence to measure the similarity between the two distributions, D dis is the distribution feature branch of the discriminator network, EX represents the mathematical expectation; (4) Classification loss of generated images in the target domain: Among them, s′ represents the target domain generated image, D cls represents the classification branch of the discriminator network, l i represents the category of the i-th attribute of the image, n represents the number of attribute categories of the image, and EX represents the mathematical expectation; (5) Objective functions of encoder network E, mapping network M and generator network G: Among them L adv 、 L dis 、L rec and L sic They represent adversarial loss, classification loss of target domain generated images, distribution feature similarity loss, reconstruction loss, and feature similarity loss respectively; λ cls ,λ dis ,λ rec and λ sim Represent the weight coefficients of the target domain generated image classification loss term, distribution feature similarity loss term, reconstruction loss term, and feature similarity loss term respectively; The final optimization goal is to minimize the above objective function and update the network parameters of the encoder network E, mapping network M and generator network G; take λ cls =1,λ dis =10,λ rec =50,λ sim =50,λ gp = 10, and the Adam optimizer with a learning rate of 0.001, β1 of 0.5, and β2 of 0.999 for all networks.

8. The method according to claim 1, wherein: Step 8 is as follows: In the upstream image style conversion task, the following metrics are used to measure the quality of the generated images in the target domain: (1) Mean μ x It is expressed as follows: Where H and W represent the height and width of the image respectively, I i,j Represents the grayscale value of the image at point (i, j); (2) Variance σ 2 x It is expressed as follows: Where H and W represent the height and width of the image respectively, I i,j Represents the grayscale value of the image at point (i, j), μ x Expressed as the mean of image x; (3) Dynamic range DR is expressed as follows: Among them I max and I min Represent the maximum and minimum grayscale values ​​in the image respectively; (4) The equivalent coefficient ENL is expressed as follows: where μ x is the mean of image x, σ 2 x is the variance of image x; (5) The brightness similarity l(x, y) and contrast similarity c(x, y) are expressed as follows: Among them, x, y represent the input image, μ x , μ y represent the mean of image x and y respectively, and Represents the variance of images x and y respectively; C1=(k1L) 2 and C2=(k2L) 2 These are two factors used to stabilize the calculation, where L = 255, k1 = 0.01, and k2 = 0.

03.

9. The method according to claim 1, wherein: Step 9 is as follows: According to the indicators in step 8, the encoder network E that generates the best model is selected as the pre-trained feature extraction model for the downstream segmentation network; In the downstream object classification task, the pre-trained encoder network E is used and then stacked with four convolutional layers as the downstream object classification model to perform SAR image object classification. The downstream object classification task training data obtained in step 1 is normalized and then sent to the downstream object classification network to fine-tune the classification network. The downstream object classification task uses 200*200 image blocks for training; The downstream feature classification task is trained with 500 rounds, an Adam optimizer with a learning rate of 0.001, β1 of 0.5, and β2 of 0.

999. The optimization goal is to minimize the multi-classification cross entropy loss between the category scores output by the model and the true categories.

10. A computer, characterized in that: Used to implement a single-channel high-resolution SAR image ground object classification method based on deep self-supervised generative adversarial technology as described in any one of claims 1-9.

Images