A method for extracting ground features from high-resolution SAR images based on deep unsupervised multi-step adversarial domain adaptation

Abstract

A high-resolution SAR image ground feature extraction method based on deep unsupervised multi-step adversarial domain adaptation first uses an image transformation model to transform the source domain image into a style similar to the target domain image, thereby reducing the distribution difference between the source and target domains; then the style-converted source domain image and the unlabeled target domain image are sent to the adversarial adaptive network. On the one hand, the deep network is trained to learn the features of the source and target domains and perform semantic segmentation; on the other hand, the discriminator is trained to distinguish whether its input comes from the source domain or the target domain, and feedback guides the deep network to align the distribution features of the source and target domains; finally, the trained model is used to predict the ground feature type of a large scene in the target domain, and pixel-level ground feature extraction of single-polarization high-resolution SAR images is completed; the present invention breaks through the bottleneck of poor model generalization caused by insufficient labeled samples and inconsistent distribution of source domain data and target domain data, and improves the accuracy and performance of ground feature classification in target domain SAR images.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent interpretation of radar remote sensing images, and specifically relates to a method for extracting ground feature elements from high-resolution SAR images based on deep unsupervised multi-step adversarial domain adaptation. Background Art

[0002] Synthetic Aperture Radar (SAR) is a high-resolution, active microwave remote sensing imaging radar with all-weather, all-day, short imaging cycles, and continuous monitoring capabilities. Its strong penetrating power is unaffected by weather conditions such as clouds, rain, and fog, and it can penetrate the ground, foliage, and other surface structures. It is widely used in military and civilian applications, such as environmental protection, disaster monitoring, ocean observation, resource conservation, land cover, precision agriculture, urban area monitoring, and geographic mapping. However, semantic segmentation of high-resolution, large-scale land cover imagery acquired by satellites is a challenging task, facing challenges such as a scarcity of labeled data and differences in data features caused by varying imaging parameters (sensor, frequency band, polarization, resolution, or angle of incidence). Therefore, leveraging existing labeled data for data transfer has become a popular solution. Domain adaptation can overcome the differences in SAR data and transfer knowledge from a source domain to a different but related target domain, addressing the limited number of training samples in the target domain.

[0003] Traditional domain adaptation methods are mainly based on features, instances, and models. The feature-based method aims to adjust the source domain samples and the target domain samples to the same feature space using a mapping Φ, so that the feature space samples can be "aligned"; the instance-based method takes into account that there are always some samples in the source domain that are very similar to the target domain samples, so the loss of all samples in the source domain is multiplied by a weight wi during training. The more similar the sample is to the target domain, the larger the weight; the model-based method aims to find a new parameter θ′, and through parameter migration, the model can work better in the target domain.

[0004] Deep learning domain adaptation mainly includes discrepancy-based, adversarial-based, and reconstruction-based methods. Adversarial domain adaptation methods have recently become popular. Because domain adaptation naturally involves source and target data with different distributions, the sample generation process can be eliminated, and the target domain can be used directly as the generated samples. In this case, the generator degenerates into a feature extractor, continuously learning domain features, making it difficult for the discriminator to distinguish between the two domains. However, unsupervised adversarial domain adaptation methods have many shortcomings. The long-tail distribution of data categories and the differences in category distributions across different data domains lead to poor robustness of adversarial domain adaptation. Adversarial domain adaptation has no theoretical guarantees for regression problems, where the distribution of features is discrete and diffuse throughout space. Even if the discriminator is successfully fooled, there is no guarantee that the features of the source and target domains will be brought together to share the same label. Furthermore, while adversarial domain adaptation brings the distribution of data from different domains closer in the feature space, improving the transferability of features from different domains, it may also reduce the discriminability of data features, making it more difficult to train a classifier using fixed adversarial domain adaptation features. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a high-resolution SAR image ground feature extraction method based on deep unsupervised multi-step adversarial domain adaptation, which improves the classification accuracy of the target domain through auxiliary tasks and solves the problems of insufficient training samples and inconsistent distribution of training and test samples.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for extracting ground features from high-resolution SAR images based on deep unsupervised multi-step adversarial domain adaptation uses the upstream task of style transfer to translate the source domain image into the style of the target domain, bringing the distributions of the source and target domains closer. The translated source domain and unlabeled target domain data are fed into an adversarial adaptive network, where a feature extractor is trained to extract and classify features from the source and target domains. A domain discriminant network is trained to distinguish whether the feature extractor's output comes from the source or target domain, while encouraging the feature extractor to align the output distributions of the target and source domains.

[0008] A method for extracting ground features from high-resolution SAR images based on deep unsupervised multi-step adversarial domain adaptation includes the following steps:

[0009] S1. Data preprocessing of source domain images and target domain images, including SAR image 16-bit to 8-bit conversion, truncation, cropping, partitioning, and data format conversion;

[0010] S2: Send the preprocessed source domain image S and target domain image T into the image translation network for style transfer to obtain the translation source data S′;

[0011] S3. Initialize the segmentation network M and its optimizer SGD and the domain discrimination network D and its optimizer Adam for the downstream task. Train the domain discrimination network D to distinguish whether the output of the feature extractor is from the source domain or the target domain. At the same time, encourage the feature extractor to align the output distribution of the target domain image and the source domain image, helping the feature extractor learn domain-invariant features.

[0012] S4, the translation source data S' and the corresponding label Ys and the target domain image T are sent to the segmentation network M to obtain the segmentation output M(S) and M(T), and the corresponding label Y S Calculate the segmentation loss of the source domain;

[0013] S5. Input the output M(T) of the segmentation network M to the target domain into the domain discriminant network D, calculate its adversarial loss and multiply it by the corresponding coefficient and add it to the target domain segmentation loss to update the segmentation network M and its optimizer SGD;

[0014] S6. Send the segmentation network outputs M(S) and M(T) to the domain discrimination network D to calculate the domain classification loss, and update the domain discrimination network D and its optimizer Adam;

[0015] S7, repeat S4 to S6 until the maximum number of training times is reached to obtain the model parameters of the segmentation network M;

[0016] S8. Send the target domain data to the trained segmentation network M for classification, then use TTA testing or train CRF for label optimization to obtain pixel-level classification results. Assign a color to each category to generate an RGB prediction result image, compare it with the true class label, and calculate the evaluation indicators Precision, Recall, F1ccore for each category, as well as the overall evaluation indicators OA, kappa, MIoU, and FWIoU.

[0017] The S1 is specifically:

[0018] (1a) Storage conversion plus truncation: The image is truncated / contrast stretched, grayscale levels with low probability of occurrence are discarded, and the grayscale range with high probability of occurrence is retained; the grayscale distribution of the 16-bit SAR image is counted, and the frequency of occurrence is counted according to the size of the grayscale value. When the current distribution accumulates to the threshold (Threshold) through the cumulative distribution function, the remaining pixels are discarded, and all pixels with grayscale levels exceeding the threshold are set to the grayscale level of the current threshold. The grayscale level is divided by the threshold and multiplied by 255 to convert to 8-bit SAR data storage;

[0019] The linear stretching formula is:

[0020]

[0021] Among them, gray represents the gray level; minin , max in Respectively represent the minimum grayscale and maximum grayscale at the truncation point in the input format; min out , max out Respectively represent the minimum and maximum values ​​of the grayscale level of the output format; for SAR data, the Threshold is generally set to 95%, min in Set to zero;

[0022] (1b) The test data is resampled by dilation sampling, and edge prediction is ignored during splicing. The predicted size of the actual cropped image is A*A, and the spliced ​​result is centered on a*a. The percentage of a*a in the area of ​​A*A is r, and the overlap ratio of adjacent cropped images is Set the expansion boundary slidesize to 100, which is the size of Aa;

[0023] (1c) Based on the AI ​​development platform ModelArts and its self-developed framework Mindspore, image data in jpg, png, and tif formats are converted into MindRecord format, and the data is further read through the MindDataset interface. The data format has the following features: unified data storage and access; data aggregation storage and efficient reading, making data easy to manage and move during training; efficient data encoding and decoding operations, making users unaware of data operations; flexible control of data partition size to achieve distributed data processing.

[0024] The S2 is specifically:

[0025] (2a) The translation network uses the classic CycleGAN to establish a bidirectional mapping relationship between the source domain image S and the target domain image T. Generators G and F are used, and two discriminators D are used. S and D T , to distinguish the source domain images S and F(T), and the target domain images T and G(S), respectively. The loss function contains two parts: adversarial loss and cycle consistency loss. In addition, the definition is the sample space of the source domain S, is the sample space of the target domain T;

[0026] (2b) Adversarial loss: The mapped data distribution is made close to the data distribution of the target domain. The generator G learns the mapping from the source domain image S to the target domain image T (G:S→T); the generator F learns the mapping from the target domain image T to the source domain image S (F:T→S).

[0027] The adversarial loss of S->T is:

[0028]

[0029] Where G(s) is a fake image generated by the generator G that is similar to the target domain Y, and D T Indicates that the input variable is the probability of sampling in the T space, aiming to distinguish the translation sample G(s) from the real sample t; the goal is to minimize G and maximize D T ;

[0030] The adversarial loss of T->S is:

[0031]

[0032] Where F(t) is a fake image generated by the generator F that is similar to the source domain S, and D S Indicates that the input variable is the probability of sampling in the S space, aiming to distinguish the translation sample F(t) from the real sample s; the goal is to minimize F and maximize D S ;

[0033] (2c) Cycle consistency loss: ensures that the two learned mapping generators G and F do not contradict each other. While learning the two mappings, generators G and F also hope that G(F(t)) is as similar as possible to t and F(G(s)) is as similar as possible to s, preventing generator G from over-learning samples in the target domain image T space and excessively changing samples in the source domain image S space. L1 loss is used;

[0034]

[0035] For each image s from the source domain S, G and F satisfy forward cycle consistency, and the image translation cycle brings s back to the original image after one cycle, that is, x→G(x)→F(G(x))≈x; similarly, for each image t in the target domain T, G and F should also satisfy reverse cycle consistency, that is, y→F(y)→G(F(y))≈y;

[0036] (2d) The final loss function:

[0037] l(G,F,D S ,D T )=l GAN (G,D T ,S,T)+l GAN (F,D S ,S,T)+λl(G,F)

[0038] The final overall loss is expressed as the adversarial loss of S->T, the adversarial loss of T->S, and the cycle consistency loss of generator G and generator F, where λ is a coefficient;

[0039] The ultimate goal is to optimize:

[0040]

[0041] The S3 is specifically:

[0042] (3a) The segmentation network M uses the DeepLabv2 model architecture based on ResNet101 to output spatial structured information and continuously learn domain features, making it impossible for the discriminator to distinguish between the two domains; the ASPP (Atrous spatial pyramid pooling) module uses multi-scale multi-scale to increase the receptive field, and the receptive field increases from k*k of ordinary convolution to (k+(k-1)(r-1))*(k+(k-1)(r-1));

[0043] (3b) The domain discrimination network D consists of an input layer, five convolutional layers, and an activation function layer. The convolutional layer uses 2d convolution, and the pooling layer uses LeakyReLU with an α coefficient of 0.2. LeakyReLU adjusts the zero gradient problem of negative values ​​by giving a very small linear component of x to the negative input αx. When x<0, it obtains a positive gradient of α, alleviating the Dead ReLU problem.

[0044] For the first input layer, set the number of feature maps to 5;

[0045] For the second convolutional layer, set the number of feature maps to 64, the filter size to 4, and the stride to 2;

[0046] For the third convolutional layer, set the number of feature maps to 128, the filter size to 4, and the stride to 2;

[0047] For the 4th convolutional layer, set the number of feature maps to 256, the filter size to 4, and the stride to 2;

[0048] For the fifth convolutional layer, set the number of feature maps to 512, the filter size to 4, and the stride to 2;

[0049] For the 6th convolutional layer, set the number of feature maps to 1, the filter size to 4, and the stride to 2;

[0050] For the 7th activation function layer, set the α coefficient to 0.2;

[0051] (3c) For the segmentation network M, set the maximum number of iterations to 56000iter and the initial learning rate lr to 2.5e -4 , weight decay is 5e -4 , use stochastic gradient descent method SGD to minimize the loss function of the segmentation network M;

[0052] (3d) For the domain discrimination network D, the initial learning rate is set to 1e -4 , adversarial loss coefficient λ adv is 0.001, and the adaptive moment estimation Adam is used to minimize the loss function of the domain discriminant network D.

[0053] The segmentation loss l in S4 seg Using cross entropy loss, the source domain segmentation loss is defined as follows:

[0054]

[0055] Among them, Y S is the label map of Is, C is the number of classes, H and W are the height and width of the output probability map, P S is the source domain probability of the segmentation adaptation model M, defined as P S =M(I' S ).

[0056] The adversarial loss in S5 is defined as follows:

[0057]

[0058] Among them, the definition is the sample space of the source domain S, is the sample space of the target domain T; I' s , I t Represent the input translation source domain and target domain samples respectively; the discriminator D of adversarial learning M Aims to reduce the difference between the source domain and target domain features extracted by the segmentation network M;

[0059] The total loss function for learning the segmentation network M is defined as follows:

[0060] lM=λ adv l adv (M(S′),M(T))+l seg (M(S′),Y S )

[0061] Among them, λ adv Represents the coefficient of the adversarial loss. The total loss of training the segmentation network M is the adversarial loss ladv and the segmentation loss l seg The harmony.

[0062] The domain discrimination network D in S6 uses BCE loss, which is defined as follows:

[0063]

[0064] Among them, S' represents the translated source domain data, T represents the target domain data, the segmentation network M maps the source domain and target domain to M(S'), M(T); the domain discrimination network D aims to distinguish whether the input data is the source domain or the target domain.

[0065] Test data augmentation (TTA) in S8: TTA augments the test input image by vertically and horizontally flipping and unflipping it to obtain an image copy, allowing the model to make predictions for each image, and then returns a set of these predictions. The final result of the image is obtained by averaging the predictions of the original image and the flipped image;

[0066] Conditional Random Field (CRF): An undirected graph model that models conditional distributions. When calculating the label of a pixel, it obtains information about the pixels in the neighborhood, making the segmentation result more accurate. The condition in the conditional random field refers to the conditional probability, which is the probability that the current pixel belongs to a certain category under the grayscale value of the current pixel and its surrounding area. The conditional probability distribution is specifically the Gibbs distribution, which refers to the conditional probability that the categories of all pixels in the image have been determined: P = exp(-E) / Z.

[0067] Among them, exp(-E) is the probability that the current pixel is the current segmentation result; Z represents a matrix of the same size as the image, and the grayscale value of each primary color just constitutes the probability of the current image.

[0068] Compared with the prior art, the present invention has at least the following beneficial effects:

[0069] The present invention is based on a high-resolution SAR image feature extraction method based on deep unsupervised multi-step adversarial domain adaptation. It uses labeled source domain data and unlabeled target domain data to train the network, and transfers knowledge from the source domain to a different but related target domain. This solves the problem that the training samples in the target domain are unlabeled or have limited labels that are insufficient to support network training due to the difficulty and high cost of labeling single-polarization high-resolution SAR data. With the help of the upstream style transfer task, the source domain image is translated into the style of the target domain, which brings the distribution of the source and target domains closer, reduces the domain gap, and makes downstream tasks easier to learn. The translated source domain and unlabeled target domain data are then fed into the adversarial adaptive network to train the feature extractor and extract and classify the features of the source and target domains. The domain discriminator is trained to distinguish whether the output of the feature extractor is from the source domain or the target domain. At the same time, the feature extractor is encouraged to align the output distributions of the target domain and the source domain, effectively helping the feature extractor learn more effective domain-invariant features. The present invention is based on deep unsupervised learning and uses a typical generator-discriminator structure, in which the encoder adopts the DeepLabv2 model architecture based on ResNet101 to output spatial structured information, which is returned to the generator as part of the adversarial loss to obtain effective domain-invariant features, prevent training deviation, and improve classification accuracy when training samples are insufficient. It can be used for classification and change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Flowchart of the present invention.

[0071] Figure 2 This is a data reading flow chart in the hardware of the present invention.

[0072] Figure 3 , which are source domain images and target domain images for the embodiment.

[0073] Figure 4 For the embodiment, a source domain (target domain style) image translated by a style transfer network and a target domain (source domain style) image translated are used.

[0074] Figure 5 This is the target domain label diagram of the embodiment.

[0075] Figure 6 This is a graph showing the target domain test results of the single-step adversarial domain adaptation method of the embodiment.

[0076] Figure 7 This is a test result diagram of the target domain of the embodiment. DETAILED DESCRIPTION

[0077] The present invention is described in further detail below with reference to the accompanying drawings.

[0078] Reference Figure 1A method for extracting ground features from high-resolution SAR images based on deep unsupervised multi-step adversarial domain adaptation is proposed, comprising the following steps:

[0079] S1. Data preprocessing of source domain images and target domain images, including SAR image 16-bit to 8-bit conversion, truncation, cropping, partitioning, and data format conversion;

[0080] (1a) Storage conversion and truncation: Since the acquired full-scene SAR images are all 16-bit unsigned integer data, and most of them are concentrated in the range of [0,500], directly using libraries such as cv2 and PIL to read the images and compress them from 65536 gray levels to 256 gray levels will compress the data into the first few very small gray levels, resulting in the inability to display the correct image and the inability of the neural network to learn. Therefore, the image must be truncated / contrast stretched to discard the gray levels with low probability of occurrence and retain the gray level range with high probability of occurrence; the gray level distribution of the 16-bit SAR image is counted, and the frequency of occurrence is counted according to the size of the gray value. When the current distribution accumulates to the threshold (Threshold) through the cumulative distribution function, the remaining pixels are discarded, and all pixels with gray levels exceeding the threshold are set to the gray level of the current threshold, and then divided by the threshold and multiplied by 255 to store 8-bit SAR data;

[0081] The linear stretching formula is:

[0082]

[0083] Among them, gray represents the gray level; min in , max in Respectively represent the minimum grayscale and maximum grayscale at the truncation point in the input format; min out , max out Respectively represent the minimum and maximum values ​​of the grayscale level of the output format; for SAR data, the Threshold is generally set to 95%, min in Set to zero;

[0084] (1b) As is known to all, in the prediction process of large scene semantic segmentation, if a large remote sensing image to be classified is directly input into the network model, memory overflow will occur. Therefore, the image to be classified is generally cropped into a series of smaller images and input into the network for prediction respectively, and then the prediction results are spliced ​​into a final result image in the order of cropping; if conventional regular grid cropping prediction splicing is adopted, there will inevitably be obvious splicing traces between blocks. This is because there is less information in the boundary part and the network cannot accurately estimate its category. To address this phenomenon, the present invention adopts the expansion sampling method to resample the test data and ignores the edge prediction during splicing; the actual cropped image prediction result size is A*A, the splicing result is centered at a*a, a*a accounts for the area of ​​A*A as a percentage r, and the overlapping ratio of adjacent cropped images is According to experimental experience, the expansion boundary slidesize is usually set to 100, which is the size of Aa;

[0085] (1c) Data is the foundation of deep learning. High-quality data input will play a positive role in the entire deep neural network. Since the experiment is based on Huawei's AI development platform ModelArts and its self-developed framework Mindspore, according to its characteristics, the present invention converts image data in formats such as jpg, png, and tif into MindRecord format, and further implements data reading through the MindDataset interface, such as Figure 2 As shown in the figure, the data format has the following features: unified data storage and access, making data reading easier during training; aggregated data storage and efficient reading, making data management and movement easier during training; efficient data encoding and decoding operations, making data operations imperceptible to users; and flexible control of the partition size of data segmentation to achieve distributed data processing.

[0086] S2: The preprocessed source domain image S and target domain image T are fed into the image translation network for style transfer to obtain the translation source data S′, which brings the data distribution of the source domain and the target domain closer, reduces the domain gap, and makes downstream tasks easier to learn.

[0087] (2a) The translation network uses the classic CycleGAN, which is a ring structure consisting of two generators and two discriminators. It establishes a bidirectional mapping relationship between the source domain image S and the target domain image T, using generators G and F, and two discriminators D. S and D T , to distinguish the source domain images S and F(T), and the target domain images T and G(S), respectively. The loss function contains two parts: adversarial loss and cycle consistency loss. In addition, the definition is the sample space of the source domain S, is the sample space of the target domain T;

[0088] (2b) Adversarial loss: The mapped data distribution is made close to the data distribution of the target domain. The generator G learns the mapping from the source domain image S to the target domain image T (G:S→T); the generator F learns the mapping from the target domain image T to the source domain image S (F:T→S).

[0089] The adversarial loss of S->T is:

[0090]

[0091] Where G(s) is a fake image generated by the generator G that is similar to the target domain Y, and D T Indicates that the input variable is the probability of sampling in the T space, aiming to distinguish the translation sample G(s) from the real sample t; the goal is to minimize G and maximize D T ;

[0092] The adversarial loss of T->S is:

[0093]

[0094] Where F(t) is a fake image generated by the generator F that is similar to the source domain S, and D S Indicates that the input variable is the probability of sampling in the S space, aiming to distinguish the translation sample F(t) from the real sample s; the goal is to minimize F and maximize D S ;

[0095] (2c) Cycle consistency loss: ensures that the two learned mapping generators G and F do not contradict each other. While learning the two mappings, generators G and F also hope that G(F(t)) is as similar as possible to t and F(G(s)) is as similar as possible to s. The purpose is to prevent generator G from over-learning samples in the target domain image T space and excessively changing samples in the source domain image S space. L1 loss is used;

[0096]

[0097] For each image s from the source domain S, G and F satisfy forward cycle consistency, and the image translation cycle brings s back to the original image after one cycle, that is, x→G(x)→F(G(x))≈x; similarly, for each image t in the target domain T, G and F should also satisfy reverse cycle consistency, that is, y→F(y)→G(F(y))≈y;

[0098] (2d) The final loss function:

[0099] l(G,F,D S ,D T )=l GAN (G,D T,S,T)+l GAN (F,D S ,S,T)+λl(G,F)

[0100] The final overall loss is expressed as the adversarial loss of S->T, the adversarial loss of T->S, and the cycle consistency loss of generator G and generator F, where λ is a coefficient;

[0101] The ultimate goal is to optimize:

[0102]

[0103] S3, initialize the segmentation network M and its optimizer SGD and the domain discrimination network D and its optimizer Adam of the downstream task;

[0104] (3a) The segmentation network M uses the DeepLabv2 model architecture based on ResNet101 to output spatial structured information and continuously learn domain features, making it impossible for the discriminator to distinguish between the two domains; the ASPP (Atrous spatial pyramid pooling) module uses multi-scale multi-scale to increase the receptive field, and the receptive field increases from k*k of ordinary convolution to (k+(k-1)(r-1))*(k+(k-1)(r-1));

[0105] (3b) The domain discrimination network D consists of an input layer, five convolutional layers, and an activation function layer. The convolutional layer uses 2d convolution, and the pooling layer uses LeakyReLU with an α coefficient of 0.2. LeakyReLU adjusts the zero gradient problem of negative values ​​by giving a very small linear component of x to the negative input αx. When x<0, it obtains a positive gradient of α, which alleviates the Dead ReLU problem to a certain extent. The domain discrimination network D is trained to distinguish whether the output of the feature extractor is from the source domain or the target domain, and at the same time encourages the feature extractor to align the output distribution of the target domain and the source domain, effectively helping the feature extractor learn more effective domain-invariant features.

[0106] For the first input layer, set the number of feature maps to 5;

[0107] For the second convolutional layer, set the number of feature maps to 64, the filter size to 4, and the stride to 2;

[0108] For the third convolutional layer, set the number of feature maps to 128, the filter size to 4, and the stride to 2;

[0109] For the 4th convolutional layer, set the number of feature maps to 256, the filter size to 4, and the stride to 2;

[0110] For the fifth convolutional layer, set the number of feature maps to 512, the filter size to 4, and the stride to 2;

[0111] For the 6th convolutional layer, set the number of feature maps to 1, the filter size to 4, and the stride to 2;

[0112] For the 7th activation function layer, set the α coefficient to 0.2;

[0113] (3c) For the segmentation network M, set the maximum number of iterations to 56000iter and the initial learning rate lr to 2.5e -4 , weight decay is 5e -4 , use stochastic gradient descent method SGD to minimize the loss function of the segmentation network M;

[0114] (3d) For the domain discrimination network D, the initial learning rate is set to 1e -4 , adversarial loss coefficient λ adv is 0.001, and the adaptive moment estimation Adam is used to minimize the loss function of the domain discriminant network D;

[0115] S4, the translation source data S' and the corresponding label Ys and the target domain image T are sent to the segmentation network M to obtain the segmentation output M(S) and M(T), and the corresponding label Y S Calculate the segmentation loss of the source domain;

[0116] The segmentation loss l seg Using cross entropy loss, the source domain segmentation loss is defined as follows:

[0117]

[0118] Among them, Y S is the label map of Is, C is the number of classes, H and W are the height and width of the output probability map, P S is the source domain probability of the segmentation adaptation model M, defined as P S =M(I' S );

[0119] S5. Input the output M(T) of the segmentation network M to the target domain into the domain discriminant network D, calculate its adversarial loss and multiply it by the corresponding coefficient and add it to the target domain segmentation loss to update the segmentation network M and its optimizer SGD;

[0120] The adversarial loss is defined as follows:

[0121]

[0122] Among them, the definition is the sample space of the source domain S, is the sample space of the target domain T; I' s , I t Represent the input translation source domain and target domain samples respectively; the discriminator D of adversarial learning M Aims to reduce the difference between the source domain and target domain features extracted by the segmentation network M;

[0123] Therefore, the total loss function for learning the segmentation network M can be defined as follows:

[0124] lM=λ adv l adv (M(S′),M(T))+l seg (M(S′),Y S )

[0125] Among them, λ adv Represents the coefficient of the adversarial loss. The total loss of training the segmentation network M is the adversarial loss ladv and the segmentation loss l seg of and;

[0126] S6. Send the segmentation network outputs M(S) and M(T) to the domain discrimination network D to calculate the domain classification loss, and update the domain discrimination network D and its optimizer Adam to further reduce the domain gap;

[0127] The domain discrimination network D uses BCEloss, which is defined as follows:

[0128]

[0129] Among them, S' represents the translated source domain data, T represents the target domain data, the segmentation network M maps the source domain and target domain to M(S'), M(T); the domain discrimination network D aims to distinguish whether the input data is the source domain or the target domain;

[0130] S7, repeat S4 to S6 until the maximum number of training times is reached to obtain the model parameters of the segmentation network M;

[0131] S8. Feed the target domain data into the trained segmentation network M for classification. Then use TTA testing or training CRF for label optimization to obtain pixel-level classification results. Assign a color to each category to generate an RGB prediction result image, compare it with the true class label, and calculate the evaluation indicators Precision, Recall, F1ccore for each category, as well as the overall evaluation indicators OA, kappa, MIoU, and FWIoU.

[0132] (8a) Test Data Augmentation (TTA): Use test data augmentation (TTA) to improve the prediction results. TTA and CRF are both means to improve the noise in the test results. TTA mainly amplifies the image by vertically and horizontally flipping and reverse flipping the test input image, allowing the model to make predictions for each image, and then returns a set of these predictions. The final result of the image is obtained by averaging the prediction results of the original image and the flipped image;

[0133] (8b) Conditional Random Field (CRF): An undirected graph model for modeling conditional distributions. The core idea is to obtain information about neighboring pixels when calculating the label of a pixel, making the segmentation result more accurate. The condition in the CRF refers to the conditional probability, which is the probability that the current pixel belongs to a certain category under the grayscale value of the current pixel and its surrounding area. The conditional probability distribution is specifically the Gibbs distribution, which refers to the conditional probability that the categories of all pixels in the image have been determined: P = exp(-E) / Z.

[0134] Among them, exp(-E) is the probability that the current pixel is the current segmentation result; Z represents a matrix of the same size as the image, and the grayscale value of each primary color just constitutes the probability of the current image.

[0135] The effect of the present invention can be further illustrated by the following simulation:

[0136] 1. Simulation conditions: The hardware test platform used in the simulation experiment is based on the ModelArts cloud AI development platform, with an Ascend-910 (32GB) graphics card | ARM: 24 cores and 96GB memory; the software platform is Python 3.7, MindSpore 1.6.0; and the operating system is the eulerosv2r8.aarch64r 64-bit operating system.

[0137] The single-polarization high-resolution SAR data used in the simulation experiment are two private datasets taken by the GF-3 satellite. The radar parameters are C wave, VV polarization, and a resolution of 1 meter. The private datasets are located in Dongying, China (shandong|10240*9216) and Pohang, South Korea (korea|9728*7680), and contain six types of land features: invalid class, building, water body, cultivated land, green space, and road.

[0138] During training, each dataset is seamlessly cropped into a 512*512 image. A set of labeled datasets is used as the source domain, and a set of unlabeled datasets is used as the target domain to be fed into the network for training. During testing, the target domain dataset is cropped using dilation sampling with a dilation size of 100. The image is still cropped to a size of 512*512, but only the central 312*312 image is taken for each test image. Finally, all the test images are spliced ​​together to obtain the final test results. Detailed data information is shown in Tables 1 and 2.

[0139] Table 1: Dataset information labels

[0140]

[0141] Table 2: Region category-RGB value comparison table

[0142] category other waters Trees / Greenery architecture farmland the way RGB values [0,0,0] [0,0,255] [0,255,0] [255,0,0] [255,255,0] [210,180,140]

[0143] The relevant evaluation indicators used in the simulation experiment and their definitions are as follows: i represents a positive example, j represents a negative example; p ii The total number of pixels whose true category is i and identified as category i is the true positive example (TP), p ij The total number of pixels whose true category is j and identified as category i is the false positive (FP), p ji The total number of false negatives (FN) p represents the number of pixels whose true category is i and whose category is j. jj It represents the total number of true negative examples (TN) of pixels whose true category is j and whose identification is category j.

[0144] Precision: The ratio of correctly predicted positive results to all positive results, defined as follows:

[0145]

[0146] Recall: The ratio of correctly predicted positive samples to all positive samples, defined as follows:

[0147]

[0148] F1 value: Based on the harmonic mean of recall and precision, it is defined as follows:

[0149]

[0150] Overall accuracy (OA): The ratio of correctly labeled pixels to the total pixels, defined as follows:

[0151]

[0152] Kappa coefficient: Penalizes the "bias" of the model to obtain a more fair model, defined as follows:

[0153]

[0154] Among them, p o represents the proportion of the number of correctly classified samples of each category to the total number of samples, equivalent to OA; a1, a2, ..., aC represent the number of real samples of each category; b1, b2, ..., bC represent the predicted number of samples of each category; the number of categories is C, and the total number of samples is n;

[0155] Mean Intersection over Union (MIoU): The average of the intersection and union ratios of each class of predictions and true values, defined as follows:

[0156]

[0157] Frequency-weighted intersection-over-union (FWIoU): This is an improvement on MIoU, where weights are set based on the frequency of occurrence of categories. It is defined as follows:

[0158]

[0159] 2. Simulation experiment content: The present invention and the classic single-step adversarial domain adaptive algorithm are used to complete the extraction of ground feature elements in single-polarization high-resolution SAR images with the same radar resolution and across regions (large intra-class differences) on the above private dataset, and calculate relevant evaluation indicators. Figure 3 The original SAR images of the source domain Korea and the target domain Shandong in the Korea->Shandong experiment are shown; Figure 4 The Korea->Shandong experiment shows the source domain Korea (Shandong style) SAR image and the target domain Shandong (Korea style) SAR image translated by comparing Figure 3 and Figure 4 From the two sets of images, it can be clearly seen that Figure 4 The translated picture (a) is closer to Figure 3 The brightness and texture of the middle image (b) are closer to the translated image (b). Figure 3 The brightness and texture of the middle image (a) bring the distributions of the two domains closer. The experimental results of each group are shown in Tables 3 and 4 below:

[0160] Table 3: Comparison of cross-region simulation results for the same radar and resolution in Shandong-Korea

[0161] Evaluation indicators OA Kappa MIoU FWU Model_1 0.6694 0.5372 0.3948 0.5385 Model_2 0.7355 0.6262 0.4553 0.6053

[0162] Table 4: Korea->Shandong cross-region simulation results comparison table for the same radar and resolution

[0163] Evaluation indicators OA Kappa MIoU FWU Model_1 0.5888 0.4430 0.3414 0.4347 Model_2 0.8137 0.7520 0.5626 0.7029

[0164] 3. Analysis of experimental results: As can be seen from Table 3, the overall accuracy OA of the domain adaptation of the present invention from shandong->korea on the same radar and resolution cross-regional dataset can reach 73.55%, and MIoU can reach 45.53%, both with a 5% accuracy improvement; the effect is more significant in the domain adaptation of the dataset korea->shandong, with the overall accuracy OA reaching 81.37% and MIoU reaching 56.26%, both with a 22% accuracy improvement. In shandong->korea, farmland and green plants are easily confused, and the method of the present invention has significantly improved green vegetation. In korea->shandong, shandong is a typical Chinese urban image, in which the water, farmland and building information contained therein are intertwined and more complex. The method of the present invention makes it easier to distinguish between greenery and buildings, with better regional consistency and clearer edge information. Figure 5 is the label graph of the domain, Figure 6 This is the prediction result diagram of Model_1. Figure 7 This is the prediction result of Model_2, it can be clearly seen Figure 7 Compare Figure 6 The closer it is to the label map, the more accurate the segmentation and the clearer the boundary.

[0165] Based on the analysis of the above simulation results, the high-resolution SAR image feature extraction method based on deep unsupervised multi-step adversarial domain adaptation in the present invention effectively solves the problem of poor prediction effect of the model on the test data set that does not meet the independent and identical distribution caused by the domain offset of cross-regional data. It transfers knowledge from the source domain to a different but related target domain, alleviates the problem that the existing SAR data is unlabeled and cannot support network training, and improves the classification accuracy of the classifier in the unlabeled target domain.

Claims

1. A method for extracting ground features from high-resolution SAR images based on deep unsupervised multi-step adversarial domain adaptation, characterized by: Leveraging the upstream style transfer task, we translate source domain images into the style of the target domain, narrowing the distributions of the source and target domains. We then feed the translated source domain and unlabeled target domain data into an adversarial adaptive network, train a feature extractor to extract and classify features from the source and target domains, and train a domain discriminant network to distinguish whether the feature extractor output is from the source or target domain. This network also encourages the feature extractor to align the output distributions of the target and source domains. The method for extracting ground feature elements from high-resolution SAR images based on deep unsupervised multi-step adversarial domain adaptation comprises the following steps: S1. Data preprocessing of source domain images and target domain images, including SAR image 16-bit to 8-bit conversion, truncation, cropping, partitioning, and data format conversion; S2: Send the preprocessed source domain image S and target domain image T into the image translation network for style transfer to obtain the translation source data S′; S3. Initialize the segmentation network M and its optimizer SGD and the domain discrimination network D and its optimizer Adam for the downstream task. Train the domain discrimination network D to distinguish whether the output of the feature extractor is from the source domain or the target domain. At the same time, encourage the feature extractor to align the output distribution of the target domain image and the source domain image, helping the feature extractor learn domain-invariant features. S4, the translation source data S' and the corresponding label Ys and the target domain image T are sent to the segmentation network M, and the segmentation network outputs M(S) and M(T) are obtained, and the corresponding label Y is used S Calculate the segmentation loss of the source domain; S5. Input the output M(T) of the segmentation network M to the target domain into the domain discriminant network D, calculate its adversarial loss and multiply it by the corresponding coefficient and add it to the target domain segmentation loss to update the segmentation network M and its optimizer SGD; S6. Send the segmentation network outputs M(S) and M(T) to the domain discrimination network D to calculate the domain classification loss, and update the domain discrimination network D and its optimizer Adam; S7, repeat S4 to S6 until the maximum number of training times is reached, and obtain the model parameters of the segmentation network M; S8. Send the target domain data to the trained segmentation network M for classification, and then use test data augmentation TTA or conditional random field CRF for label optimization to obtain pixel-level classification results. Assign a color to each category to generate an RGB prediction result image, compare it with the true class label, and calculate the evaluation indicators Precision, Recall, F1 Score for each category, as well as the overall evaluation indicators OA, kappa, MIoU, and FWIoU.

2. The method according to claim 1, characterized in that The S1 is specifically: (1a) Storage conversion plus truncation: The image is truncated / contrast stretched, gray levels with low probability of occurrence are discarded, and the gray level range with high probability of occurrence is retained; the gray level distribution of the 16-bit SAR image is counted, and the frequency of occurrence is counted according to the size of the gray value. When the current distribution accumulates to the threshold value, the remaining pixels are discarded and all pixels exceeding the threshold gray level are set to the gray level of the current threshold. The gray level is divided by the threshold value and multiplied by 255 to convert to 8-bit SAR data storage; The linear stretching formula is: Among them, gray represents the gray level; min in , max in Respectively represent the minimum grayscale and maximum grayscale at the truncation point in the input format; min out , max out They represent the minimum and maximum values ​​of the grayscale level in the output format respectively; for SAR data, the threshold is set to 95%, min in Set to zero; (1b) The test data is resampled by dilation sampling, and edge prediction is ignored during splicing. The predicted size of the actual cropped image is A*A, and the spliced ​​result is centered on a*a. The percentage of a*a in the area of ​​A*A is r, and the overlap ratio of adjacent cropped images is Set the expansion boundary slidesize to 100, which is the size of Aa; (1c) Based on the AI ​​development platform ModelArts and its self-developed framework Mindspore, image data in jpg, png, and tif formats are converted into MindRecord format, and the data is further read through the MindDataset interface. The data format has the following features: unified data storage and access; data aggregation storage and efficient reading, making data easy to manage and move during training; efficient data encoding and decoding operations, making users unaware of data operations; flexible control of data partition size to achieve distributed data processing.

3. The method according to claim 1, characterized in that The S2 is specifically: (2a) The image translation network uses the classic CycleGAN to establish a bidirectional mapping relationship between the source domain image S and the target domain image T. Generators G and F are used, and two discriminators D are used. S and D T , to distinguish the source domain images S and F(T), and the target domain images T and G(S), respectively. The loss function contains two parts: adversarial loss and cycle consistency loss. In addition, define E s～Pdata(s) is the sample space of the source domain image S, E t～Pdata(t) is the sample space of the target domain image T; (2b) Adversarial loss: The mapped data distribution is made close to the data distribution of the target domain. The generator G learns the mapping from the source domain image S to the target domain image T (G:S→T); the generator F learns the mapping from the target domain image T to the source domain image S (F:T→S). The adversarial loss of S->T is: Where G(s) is a fake image generated by the generator G that is similar to the target domain Y, and D T Indicates that the input variable is the probability of sampling in the T space, aiming to distinguish G(s) from the true sample t; our goal is to minimize G and maximize D T ; The adversarial loss of T->S is: Where F(t) is a fake image generated by the generator F that is similar to the source domain image S, and D S Indicates that the input variable is the probability of sampling in the S space, aiming to distinguish F(t) from the true sample s; The goal is to minimize F and maximize D S ; (2c) Cycle consistency loss: ensures that the two learned mapping generators G and F do not contradict each other. While learning the two mappings, generators G and F also hope that G(F(t)) is as similar as possible to t and F(G(s)) is as similar as possible to s, preventing generator G from over-learning samples in the target domain image T space and excessively changing samples in the source domain image S space. L1 loss is used; For each image s from the source domain image S, G and F satisfy forward cycle consistency, and the image translation cycle brings s back to the original image after one cycle, that is, x→G(x)→F(G(x))≈x; similarly, for each image t of the target domain image T, G and F should also satisfy reverse cycle consistency, that is, y→F(y)→G(F(y))≈y; (2d) The final loss function: l(G,F,D S ,D T )=l GAN (G,D T ,S,T)+l GAN (F,D S ,S,T)+λl(G,F) The final overall loss is expressed as the adversarial loss of S->T, the adversarial loss of T->S, and the cycle consistency loss of generator G and generator F, where λ is a coefficient; The ultimate goal is to optimize:

4. The method according to claim 1, wherein The S3 is specifically: (3a) The segmentation network M uses the DeepLabv2 model architecture based on ResNet101 to output spatial structured information and continuously learn domain features, making it impossible for the discriminator to distinguish between the two domains; the ASPP (Atrous spatial pyramid pooling) module uses multi-scale multi-scale to increase the receptive field, and the receptive field increases from k*k of ordinary convolution to (k+(k-1)(r-1))*(k+(k-1)(r-1)); (3b) The domain discrimination network D consists of an input layer, five convolutional layers, and an activation function layer. The convolutional layer uses 2d convolution, and the pooling layer uses LeakyReLU with an α coefficient of 0.

2. LeakyReLU adjusts the zero gradient problem of negative values ​​by giving a very small linear component of x to the negative input αx. When x<0, it obtains a positive gradient of α, alleviating the DeadReLU problem. For the first input layer, set the number of feature maps to 5; For the second convolutional layer, set the number of feature maps to 64, the filter size to 4, and the stride to 2; For the third convolutional layer, set the number of feature maps to 128, the filter size to 4, and the stride to 2; For the 4th convolutional layer, set the number of feature maps to 256, the filter size to 4, and the stride to 2; For the fifth convolutional layer, set the number of feature maps to 512, the filter size to 4, and the stride to 2; For the 6th convolutional layer, set the number of feature maps to 1, the filter size to 4, and the stride to 2; For the 7th activation function layer, set the α coefficient to 0.2; (3c) For the segmentation network M, the maximum number of iterations is set to 56,000 iterations and the initial learning rate lr is 2.5e -4 , weight decay is 5e -4 , using stochastic gradient descent SGD to minimize the loss function of the segmentation network M; (3d) For the domain discrimination network D, the initial learning rate is set to 1e -4 , adversarial loss coefficient λ adv is 0.001, and the adaptive moment estimation Adam is used to minimize the loss function of the domain discriminant network D.

5. The method according to claim 1, wherein The segmentation loss l in S4 seg Using cross entropy loss, the source domain segmentation loss is defined as follows: Among them, Y S is the label map of Is, C is the number of classes, H and W are the height and width of the output probability map, P S is the source domain probability of the segmentation adaptation model M, defined as P S =M(I' S ), where Is is the source domain image and I's is the result of translating Is by the image translation network CycleGAN.

6. The method according to claim 1, characterized in that The adversarial loss in S5 is defined as follows: Among them, the definition is the sample space of the source domain image S, is the sample space of the target domain image T; I′ s , I t Represent the input translation source domain and target domain samples respectively; the discriminator D of adversarial learning M Aims to reduce the difference between the source domain and target domain features extracted by the segmentation network M; The total loss function for learning the segmentation network M is defined as follows: lM=λ adv l adv (M(S′),M(T))+l seg (M(S′),Y S ) Among them, λ adv Represents the coefficient of the adversarial loss. The total loss of training the segmentation network M is the adversarial loss l adv and segmentation loss l seg The harmony.

7. The method according to claim 1, characterized in that The domain discrimination network D in S6 uses BCE loss, which is defined as follows: Among them, S′ represents the translated source domain data, T represents the target domain data, the segmentation network M maps the source domain and target domain to M(S′), M(T); the domain discrimination network D aims to distinguish whether the input data is the source domain or the target domain.

8. The method according to claim 1, characterized in that Test data augmentation (TTA) in S8: TTA augments the test input image by vertically and horizontally flipping and unflipping it to obtain an image copy, allowing the model to make predictions for each image, and then returns a set of these predictions. The final result of the image is obtained by averaging the predictions of the original image and the flipped image; Conditional Random Field (CRF): An undirected graph model that models conditional distributions. When calculating the label of a pixel, it obtains information about the pixels in the neighborhood, making the segmentation result more accurate. The condition in the conditional random field refers to the conditional probability, which is the probability that the current pixel belongs to a certain category under the grayscale value of the current pixel and its surrounding area. The conditional probability distribution is specifically the Gibbs distribution, which refers to the conditional probability that the categories of all pixels in the image have been determined: P = exp(-E) / Z. Among them, exp(-E) is the probability that the current pixel is the current segmentation result; Z represents a matrix of the same size as the image, and the grayscale value of each primary color just constitutes the probability of the current image.

Images