An intra-domain remote sensing image classification method based on adversarial learning

Abstract

The present application relates to a kind of same field remote sensing image classification method based on adversarial learning, belong to remote sensing image data processing technical field.The present application uses the classification model of generator and discriminator to classify remote sensing image data, so that classification model obtains as possible with true value label space similar data distribution, to improve the overall perception ability and classification precision of classification model to input image.In addition, generator uses the architecture of encoder and decoder, decoder uses the residual part, upsampling part and attention enhancement part set in multilayer convolutional neural network;The output of decoder is as the input of encoder, meanwhile, the corresponding layer between decoder and encoder is connected, to fuse low layer feature position information and high layer feature semantic information, improve the relevance between pixels, reduce the information loss in upsampling process, further improve the precision of classification.

Description

Technical Field

[0001] This invention relates to a co-domain remote sensing image classification method based on adversarial learning, belonging to the field of remote sensing image data processing technology. Background Technology

[0002] The superior backbone networks ResNet and ResNeSt failed to fully realize their performance advantages. Experimental results show that the improved methods using these two backbone networks as encoders still lag behind the VGG network. There are two main reasons for this: First, from a statistical perspective, traditional deep learning image classification methods assume that the training and test sets follow the same distribution, meaning their feature spaces are identical or similar. Theoretically, the model's classification accuracy on training data should be the same or similar to its accuracy on test data. However, in reality, the classification accuracy on test data is often lower than that on training data, indicating overfitting. Second, while complex deep learning models can learn most features in the training data well, they cannot effectively learn certain features that depend on spatial structure, tone, and texture. Especially in remote sensing imagery, there may be significant structural differences between the same ground features, while different ground features may have strong similarities in tone and texture. This reflects that there are certain semantic connections between pixels. However, traditional deep learning models use the cross-entropy loss function for optimization during training. The gradient used by this loss function during backpropagation is only related to the difference between a single pixel in the prediction result and its corresponding pixel in the ground truth label. It does not consider the correlation of neighboring pixels and ignores the interrelationships between pixels, resulting in low accuracy of classification results. It is easy to see situations where the edges of ground features are discontinuous or the classification result differs greatly from the ground truth label in terms of geometry. Summary of the Invention

[0003] The purpose of this invention is to provide a co-domain remote sensing image classification method based on adversarial learning to solve the problem of low classification accuracy in current remote sensing image classification methods.

[0004] To address the aforementioned technical problems, this invention provides a co-domain remote sensing image classification method based on adversarial learning. This classification method includes the following steps:

[0005] The remote sensing image data to be classified is acquired and input into a trained classification model for classification. The classification model includes a generator and a discriminator. The generator adopts an encoder-decoder structure to obtain pixel classification prediction results of the remote sensing image. The discriminator adopts a convolutional neural network to distinguish between the real labels and the prediction results generated by the generator by obtaining the high-order consistency between the real labels and the prediction results generated by the generator. The generator and discriminator are trained in an adversarial manner.

[0006] This invention employs a classification model consisting of a generator and a discriminator to classify remote sensing image data. By leveraging the powerful function fitting capabilities of generative adversarial networks, the classification model obtains a data distribution as similar as possible to the ground truth label space, thereby improving the overall perception capability and classification accuracy of the input image.

[0007] Furthermore, to improve the correlation between pixels and reduce information loss during the upsampling process, the encoder employs a feature extraction network to map the input remote sensing image data to a high-dimensional feature space; the decoder employs a multi-layer convolutional neural network, including multiple convolutional neural networks of different depths, each of which includes a residual part, an upsampling part, and an attention enhancement part; the output of the decoder serves as the input of the encoder, and corresponding layers in the decoder and encoder are connected to fuse low-level feature location information and high-level feature semantic information.

[0008] Furthermore, the residual part includes two convolutional modules. The input and output of the residual part are connected across layers. The input of the residual part is used to receive the features of the previous layer convolutional neural network and encoder after splicing. The input of the upsampling part is used to receive the output signal of the residual part, which is used to restore the feature map after processing by the residual part to the size of the corresponding high-order feature map.

[0009] Furthermore, the attention enhancement part includes a semantic information enhancement module and a location information enhancement module that process in parallel. The input of both is the feature map output by the upsampling part. The semantic information enhancement module is used to operate on the channel dimension of the input feature map and use the correlation between high and low order channels to complete the modeling of the relationship between specific semantic information between high and low order feature maps. The location information enhancement module is used to establish the location information correlation between the local features of the input feature map and other neighborhoods.

[0010] Furthermore, the processing procedure of the semantic information enhancement module is as follows:

[0011] Global average pooling is used to obtain statistical information of the input feature map in the channel dimension;

[0012] Based on the statistical information obtained in the channel dimension, the weights in each channel dimension are determined using linear transformation and activation functions. The formula for calculating the weights is as follows:

[0013]

[0014] g c This represents the feature vector obtained after global average pooling. and These represent the weights of a 1×1 convolutional layer, where relu represents the ReLU function, σ represents the Sigmoid operation, and C represents the total number of classes.

[0015] The enhanced feature map is determined based on the weights in each channel dimension.

[0016] Furthermore, the enhanced feature map obtained by the location information enhancement module is as follows:

[0017] v i，j =h i，j *x i，j

[0018]

[0019] q = W s *x

[0020] Where v = [v 1，1 v 1，2 , ..., v W，H ], v represents the enhanced feature map, x = [x 1，1 x 1，2 , ..., x W，H ], This represents a slice of the input feature map along the channel dimension, where (i, j) corresponds to the spatial coordinates of the feature map: i∈{1, 2, ..., W}, j∈{1, 2, ..., H}. This represents the mapping matrix after a 1×1 convolution operation. Indicates that x passes through W s Mapped weight graph; h i，j It is q i，j The result after being scaled to [0, 1] by Sigmoid represents the importance of positional information at position (i, j) in the feature map.

[0021] Furthermore, the generator employs the following loss function during training:

[0022]

[0023]

[0024] It is used to reduce the performance of the discriminator. θ is the focus loss function for multi-path fusion, used to generate the correct classification prediction for each pixel of the input image. G The parameters of generator G are represented. yes and A linear combination of λ, where λ is the linear combination of λ and ... The penalty coefficient, D(·), represents the discriminator's judgment that the input x is a prediction G(X) from the generator. (n) (or truth label Y) (n) .

[0025] Furthermore, the discriminator consists of eight convolutional layers connected in series. The kernel size of each convolutional layer is 4×4. Except for the stride of the last convolutional layer, which is 1, the stride of the first to seventh convolutional layers is 2. The first convolutional layer uses the ReLU activation function, and the remaining convolutional layers use the LeakyReLU activation function.

[0026] Furthermore, the loss function of the discriminator can be defined in the following form:

[0027]

[0028]

[0029] Where θ D This represents the parameters in discriminator D. D(·) represents the binary cross-entropy loss, where D(·) indicates that the discriminator determines the input x to be a predicted value G(X) from the generator. (n) (or truth label Y) (n) y represents the one-hot encoding of a certain class in the truth label, and x represents the prediction result of a certain class generated by the generator.

[0030] Furthermore, the discriminator's parameters θ D and the generator's parameters θ G A step-by-step update method is adopted, first fixing the generator parameter θ. G Update the discriminator parameters θ D This enables the discriminator to distinguish the predicted results; then the discriminator's parameter θ is fixed. D Update the generator's parameter θ G This causes the generator to produce predictions that the discriminator cannot distinguish between true and false. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the network structure of the classification model used in this invention;

[0032] Figure 2 This is a schematic diagram of the generator structure in the classification model of this invention;

[0033] Figure 3 This is a diagram of the VGG (VGG-19) network structure used by the encoder in the classification model of this invention.

[0034] Figure 4 This is a schematic diagram of the attention and residual structure used by the decoder in the classification model of this invention;

[0035] Figure 5-a This is a schematic diagram of an adversarial network using the cGAN model framework;

[0036] Figure 5-b This is a schematic diagram of an adversarial network using the pix2pix model framework;

[0037] Figure 6 This is a schematic diagram of the discriminator used in this invention;

[0038] Figure 7 This is a visualization of the focus loss function design for the multi-path fusion generator used in this invention;

[0039] Figure 8 This is a schematic diagram of some prediction results of different model methods on the Vaihingen dataset in the experimental examples of this invention;

[0040] Figure 9-a This is the 31st original image in the Vaihingen dataset in the experimental example of this invention;

[0041] Figure 9-b This is a schematic diagram of the ground truth label for image number 31 in the Vaihingen dataset in the experimental example of this invention;

[0042] Figure 9-c This is the classification result of image number 31 in the Vaihingen dataset in the experimental example of this invention using the SVL_3 method;

[0043] Figure 9-d This is the classification result of image number 31 in the Vaihingen dataset in the experimental example of this invention using the RIT_L7 method;

[0044] Figure 9-e This is the classification result of image number 31 in the Vaihingen dataset in the experimental example of this invention using the DLR_8 method;

[0045] Figure 9-f This is the classification result of image number 31 in the Vaihingen dataset in the experimental example of this invention using the CASIA method;

[0046] Figure 9-g This is the classification result of image number 31 in the Vaihingen dataset in the experimental example of this invention using the AREANS-VGG method;

[0047] Figure 9-h This is the classification result of image number 31 in the Vaihingen dataset in the experimental example of this invention using the AREANs-ResNet method;

[0048] Figure 9-i This is the classification result of image number 31 in the Vaihingen dataset in the experimental example of this invention using the AREANs-ResNeSt method;

[0049] Figure 10-a This is the original image number 6_13 in the Potsdam dataset in the experimental example of this invention;

[0050] Figure 10-b This is a schematic diagram of the ground truth labels for image number 6_13 in the Potsdam dataset in the experimental example of this invention;

[0051] Figure 10-c This is the classification result of image number 6_13 in the Potsdam dataset in the experimental example of this invention using the SVL_1 method;

[0052] Figure 10-d This is the classification result of image number 6_13 in the Potsdam dataset in the experimental example of this invention using the RIT_L7 method;

[0053] Figure 10-e This is the classification result of image number 6_13 in the Potsdam dataset in the experimental example of this invention using the UZ_1 method;

[0054] Figure 10-f This is the classification result of image number 6_13 in the Potsdam dataset in the experimental example of this invention using the DST_5 method;

[0055] Figure 10-g This is the classification result of image number 6_13 in the Potsdam dataset in the experimental example of this invention using the BKHZ_3 method;

[0056] Figure 10-h This is the classification result of image number 6-13 in the Potsdam dataset in the experimental example of this invention using the CASIA2 method;

[0057] Figure 10-i The classification result of image 6-13 in the Potsdam dataset in the experimental example of this invention is obtained using the BUCTY5 method;

[0058] Figure 10-j The classification results of images 6-13 in the Potsdam dataset in the experimental example of this invention are obtained using the AREANS-VGG method.

[0059] Figure 10-k The classification result of image 6-13 in the Potsdam dataset in the experimental example of this invention is obtained by using the AREANS-ResNet method;

[0060] Figure 10-1This is the classification result of image number 6_13 in the Potsdam dataset in the experimental example of this invention using the AREANS-ResNeSt method. Detailed Implementation

[0061] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0062] The classification model of the co-domain remote sensing image classification method based on adversarial learning in this invention includes a generator and a discriminator. The generator adopts an encoder-decoder structure with residual attention enhancement mechanism, and the discriminator adopts a convolutional neural network. The classification model is trained in two stages. In the first stage, it is trained under supervision alone, and optimized using a multi-path fusion focus loss function to give the model a certain classification ability, providing an initial training model for the adversarial training in the second stage. In the second stage, based on the first stage, a discriminator is added, and an adversarial training strategy is introduced. The entire architecture is optimized by combining adversarial loss and multi-path fusion focus loss function to improve the image classification accuracy of the generator.

[0063] I. Establishing a classification model

[0064] There are two types of conditional adversarial generative network architectures, such as Figure 5-a As shown (cGAN) and as Figure 5-b As shown in (pix2pix). The generator input in cGAN consists of two parts: random noise z and control conditions c. The generator G is affected by the control conditions and completes the mapping from z to G(z). The discriminator D learns and continuously identifies false G(z), as shown in formula (1):

[0065]

[0066] Here, "1" indicates that the input of D comes from the true value x, and "0" indicates that the input of D comes from the generator G(z). Its final objective function is:

[0067]

[0068] pix2pix aims to map from image space to ground truth label space. Therefore, its input to the generator is only the original image x, while random noise z is implemented through the Dropout layer in the network structure. The discriminator works in the same way as conditional generative adversarial networks. In order to obtain a result that is as similar as possible to the ground truth label y, pix2pix also adds an L1 distance constraint to the optimization objective, as shown in Equation (3):

[0069]

[0070] Therefore, combining the objective function of cGAN:

[0071]

[0072] The final objective function is:

[0073]

[0074] Where λ is the weight of the L1 distance constraint.

[0075] Based on the above, the classification model adopted in this invention is as follows: Figure 1 As shown, the network includes a generator network with a fusion attention mechanism and a residual module, and a discriminator network based on image fusion. The generator is mainly used to obtain pixel classification prediction results of remote sensing images. The discriminator is used to evaluate whether the prediction results generated by the generator are reliable. It judges the input fused with the original image and the ground truth label as true (here, "1"), and the input fused with the original image and the generator prediction result as false ("0").

[0076] The structure of the generator is as follows: Figure 2 As shown, the system comprises an encoder and a decoder. The encoder can be composed of various backbone networks, such as VGG, ResNet, and ResNeSt. The decoder employs a multi-layer convolutional neural network, including multiple convolutional neural networks of different depths. Each convolutional neural network of different depths includes a residual block, an upsampling block, and an attention-enhancing block. The "Residual block" represents the residual block, the "Upsampling block" represents the upsampling block, and the "Attention-Enforced block" represents the attention-enhancing block. The prediction result is output by a softmax layer. In this embodiment, the decoder uses a five-layer convolutional neural network, where Level-X (X = 2, 3, 4, 5) represents neural networks of different depths in the decoder, and the last neural network outputs through a softmax layer. In the diagram, A represents convolution + activation function, B represents downsampling operation, and C represents the module embedding attention and residual structures.

[0077] In this embodiment, the encoder uses a VGG network, such as Figure 3As shown, it consists of 5 blocks, each block being a combination of a convolutional layer, a rectified linear unit (RLU), and a max-pooling layer. Due to the human eye's abstraction of visual information at different levels, the number of channels increases block by block, while the size of the feature map decreases block by block. The features extracted by different blocks in VGG represent the target's expression at different levels; the higher the level, the higher the degree of abstraction. Therefore, the output at the end of the network is a high-order abstract representation of the input image. The given input dataset is X = [x1, x2, ..., x...]. N ],in This represents the i-th image in the dataset, which has a size of w×h and contains c channels. This represents the operation performed in the nth block of the encoder E (here referring to the VGG network), including a series of convolutional operations (conv), ReLU correction units (including Sigmoid, Tanh, or ReLU functions), and pooling operations. After a series of operations, the encoder's final output is:

[0078]

[0079] Encoders can learn hierarchical features and implicit local texture information containing visual representation information of the target. However, due to the lack of global contextual information for encoding the spatial relationships of the target, classification errors are easily caused. The high-order feature map of each channel obtained by the encoder can be regarded as a response to a specific category, which contains rich semantic information; however, high-order features often lack basic spatial information and cannot accurately describe the edge position of the target. While low-order features have more complete spatial information, they are limited by their receptive field and relatively lack semantic information.

[0080] The residual block consists of two 3×3 convolutions, such as... Figure 4 As shown, the input and output of the residual part are connected across layers. The input of the residual part consists of features from the previous layer network and the encoder, which have been spliced ​​together. This residual part not only increases the network depth and improves network performance, but also effectively solves the model degradation problem caused by increasing network depth and alleviates the difficulty of fitting identity mappings in multi-layer neural networks.

[0081] The feature map after residual processing enters the upsampling part, which restores the feature map to the size of the corresponding high-order feature map. This part is implemented by transposed convolution operation. Its advantage is that the parameters of interpolation methods such as nearest neighbor interpolation, bilinear interpolation or bicubic interpolation can be learned, so they do not need to be preset manually.

[0082] The attention enhancement part includes a semantic information enhancement module and a location information enhancement module. The feature map after upsampling is simultaneously fed into the two modules. The feature maps processed by the two modules are fused to obtain the enhanced result. Unlike the serial combination of CCNet, this invention adopts a parallel fusion method.

[0083] The semantic information enhancement module operates on the channel dimension of the input feature map, utilizing the correlation between high- and low-order channels to model the relationship of specific semantic information between high- and low-order feature maps. First, Global Average Pooling (GAP) is used to obtain the statistical information of the input feature x in the channel dimension:

[0084]

[0085] x c This represents the feature map of input x in the c-th channel. g c This represents the global statistical information obtained from the c-th channel. Subsequently, to enhance the correlation between feature maps across different channels, a combination of linear transformation and activation function is introduced:

[0086]

[0087] g c This represents the feature vector after gap processing. and σ represents the weights of a 1×1 convolutional layer, where ReLU represents the ReLU function and σ represents the Sigmoid operation. It should be noted that a fully connected layer is not used for linear transformation here, primarily to reduce computational cost and the number of model parameters. What we get are the different weights in each channel dimension, and the final feature map is:

[0088]

[0089] u = [u1, u2, ..., u] C [] represents the enhanced feature map, and × represents matrix multiplication in the channel dimension.

[0090] The location information enhancement module is used to establish the location information correlation between local features and other neighborhoods. Its structure is relatively simple and can be represented as follows:

[0091] q = W s *x (10)

[0092]

[0093] Where, x = [x 1，1 x 1，2, ..., x W，H ], This represents a slice of the input feature map along the channel dimension, where (i, j) corresponds to the spatial coordinates of the feature map: i∈{1, 2, ..., W}, j∈{1, 2, ..., H}. This represents the mapping matrix after a 1×1 convolution operation. Indicates that x passes through W s Mapped weight graph; h i，j It is q i，j The result after sigmoid scaling to the range [0, 1] represents the importance of positional information at location (i, j) in the feature map. Therefore, the final enhanced feature map result is:

[0094] v i，j =h i，j *x i，j (12)

[0095] v = [v 1，1 v 1，2 , ..., v W，H ], where v represents the enhanced feature map.

[0096] After parallel processing by two modules, the feature map simultaneously gains both location and semantic information enhancements, and the final result is as follows:

[0097] y = u + v (13)

[0098] The discriminator's role is to distinguish between the real labels and the predictions generated by the generator by obtaining high-order consistency between the two. The discriminator in this invention adopts a structure similar to a Markov discriminator (PatchGAN), such as... Figure 6 As shown. The Markov discriminator determines the authenticity of a specific-sized patch in an image without inputting the entire image. The final output is the average of the judgments for all patches. The aim is to reduce the dimensionality of the input data, decrease the number of parameters, and improve the network's computational speed while maintaining accuracy. In this embodiment, the discriminator consists of eight concatenated convolutional layers with a 4×4 kernel size. Except for the last convolutional layer, which has a stride of 1, all other convolutional layers have a stride of 2. The first convolutional layer uses the ReLU activation function, while the remaining layers use LeakyReLU. Here, the last layer of the discriminator is no longer a fully connected layer as in traditional discriminators, but a convolutional layer. This results in a matrix as the final output (e.g., ...). Figure 6As shown, where I represents the size of the discriminator's input image, it possesses a local receptive field on the input image, which is more conducive to the needs of semantic segmentation tasks. For the discriminator's input, we do not directly use ground truth labels or predicted values. Instead, we multiply the probability map of each class with the corresponding RGB (Red-Green-Blue) or IRRG (Infrared-Red-Green) image to obtain a new feature map as input. This feature map contains 3×C channels (where C represents the number of classes), and it is more conducive for the discriminator to utilize information from the original image to distinguish between predicted results and ground truth values.

[0099] II. Training the classification model

[0100] This invention employs adversarial training to train the generator and discriminator. The generator produces segmentation results, while the discriminator distinguishes between candidate samples and real samples. The two compete against each other within a zero-sum game framework according to the data distribution, which can be represented as follows:

[0101]

[0102] Where X = {X (1) X (2) , ..., X (N)} represents the input image set, Y = {Y (1) Y (2) , ..., Y (N)} represents the set of truth labels corresponding to Y, V represents the objective function of the minmax game, E represents the expected value of the distribution function; D(·) represents the discriminator, θ D θ represents the parameters in the discriminator D; G(·) represents the generator (referring to the classification network proposed in Chapter 3 in this chapter), θ G This represents the parameters in the generator G.

[0103] (1) Training the discriminator

[0104] In classification models based on generative adversarial networks, the loss function of the discriminator can be defined in the following form:

[0105]

[0106] θ D This represents the parameters in discriminator D. This represents the binary cross-entropy loss (i.e., adversarial loss). [40,42] D(·) indicates that the discriminator determines that the input x is the predicted value G(X) from the generator. (n) (or truth label Y) (n) y represents the one-hot encoding of a certain class in the truth labels, and x represents the prediction result of a certain class generated by the generator.

[0107] (2) Training generator

[0108] The generator is trained using a hybrid loss function. The generator produces samples that the discriminator struggles to distinguish between "real" and "fake". It includes two parts: and It is used to reduce the performance of the discriminator. This is the focus loss function for multi-path fusion, used to generate the correct classification prediction for each pixel of the input image. It is expressed as follows:

[0109]

[0110] θ G The parameters of generator G are represented. yes and A linear combination of λ, where λ is the linear combination of λ and ... The penalty coefficient.

[0111] The focus loss function for multipath fusion is defined as follows:

[0112]

[0113] This represents the one-hot encoded label of class c of the nth image in each batch. The output of the Softmax layer represents the total number of categories, and N represents the total number of images participating in training in each batch. γ is called the "focusing parameter," and its function is to make the network focus on samples that are difficult to classify; in this invention, γ = 2.

[0114] To further improve network performance, feature maps from different layers are extracted from the decoder and combined with FocalLoss to form a new loss function. For example... Figure 7 As shown, the outputs of Level-5 to Level-2 and the Softmax layer in the decoder are extracted respectively. Since the feature maps extracted from different layers have different channel dimensions (the feature channel dimensions extracted from Level-5 to Level-2 are 256, 256, 64, and 64 respectively, and the channel dimension of the Softmax layer output is the total number of classes), the feature maps extracted from Level-5 to Level-2 are passed through 1×1 convolutional layers respectively to make them consistent with the output of the Softmax layer in terms of channel dimension. Then, the Focal Loss is calculated separately and then summed. The specific form is as follows:

[0115] FL fusion =∑FL(p, q) i(18)

[0116] FL(p, q) i This represents the Focal Loss calculated after extracting feature maps from different layers; M represents the total number of layers extracted, where M = 5.

[0117] Adversarial training typically involves updating the discriminator's parameters θ in stages. D and the generator's parameters θ G Therefore, the AREAANs proposed in this invention consist of two steps: First, the generator parameter θ is fixed. G Update the discriminator parameters θ D This enables the discriminator to distinguish the predicted results; the second step is to fix the discriminator's parameter θ. D Update the generator's parameter θ G This makes the generator produce predictions that the discriminator struggles to distinguish between "true" and "false." Through adversarial training, a statistical relationship between the prediction results and their corresponding ground truth labels can be established at a high-level semantic level. Furthermore, in the adversarial process against the discriminator, each layer of the generator can play its role.

[0118] Experimental verification

[0119] To further verify the classification performance of the classification method of this invention, this experiment trained the method on infrared-red-blue (IR-RG) three-channel images from the Vaihingen and Potsdam datasets.

[0120] In the first training phase, the discriminator is not involved; the generator is trained using the proposed multi-path fusion Focal Loss method. The encoder weights are initialized using publicly available pre-trained models (VGG-19, ResNet-101, and ResNeSt-101 pre-trained models on ImageNet are used in this experiment), and the decoder is initialized using the Kaiming method. The initial learning rate is set to 10. -4 A total of 10 iterations 5 Second-rate.

[0121] In the second training phase, a discriminator is introduced for adversarial training. The generator uses the weights from the training model obtained in the first phase, and the discriminator is initialized using the Kaiming method. Regarding the learning rate setting, this invention employs the TTUR (Two Time-Scale Update Rule), setting different learning rates for both the generator and discriminator. This method can make GAN training more stable without increasing time costs. Here, the generator's initial learning rate is set to 10. -4 The initial learning rate of the discriminator is set to 5×10. -4 . The penalty coefficient λ is set to 0.05, and a total of 3 × 10 iterations are performed. 5 Second-rate.

[0122] There is no interruption between the two training phases, forming an end-to-end training method. The proposed AREAANs architecture is trained using Adam, with β1 = 0.5, β2 = 0.999, and weight decay of 10. -4 The learning rate is 10 per iteration 5 Each decay is halved, batch_size = 16.

[0123] This experiment evaluates the performance improvement by comparing the classification accuracy of the baseline method and methods with different improvement strategies on two datasets. "Baseline1", "Baseline2", and "Baseline3" represent the baseline methods using VGG-19, ResNet-101, and ResNeSt-101 encoders, respectively, before adding AR structures and adversarial learning; "Baseline1+AR", "Baseline2+AR", and "Baseline3+AR" represent the improved methods after adding AR structures; "Baseline1+GAN", "Baseline2+GAN", and "Baseline3+GAN" represent the improved methods after introducing adversarial learning; and "Baseline1+AR+GAN", "Baseline2+AR+GAN", and "Baseline3+AR+GAN" represent the improved methods after adding AR structures and adversarial learning, i.e., the improved methods of the ARENAs architecture proposed in this invention. Table 1 shows the performance comparison of the three backbone network encoders used in the Vaihingen dataset before and after introducing AR structures, adversarial learning, and the method proposed in this invention. The evaluation index for land cover categories is the F1 score, and the overall performance index of the method is the mean intersection-union ratio (mIoU) and overall accuracy (OA).

[0124] Table 1

[0125]

[0126]

[0127] Overall, the AREANs architecture proposed in this invention achieved the best test results on the Vaihingen dataset. The three improved methods of different backbone networks all showed significant performance improvements over the benchmark method: in terms of mIoU, the improvements were 4.57%, 7.22%, and 10.13%, respectively; and in terms of OA, the improvements were 1.09%, 2.53%, and 3.40%, respectively.

[0128] As can be seen from Table 1, apart from ARENAs proposed in this chapter, the benchmark methods after adopting the adversarial learning strategy have improved the performance of the three benchmark methods to a certain extent. This is mainly because GANs can learn the higher-order consistency between the ground truth label and the predicted label, so that the model's prediction results are as close as possible to the same manifold as the training data, thereby improving the accuracy of the prediction results.

[0129] Among the three improved methods based on the AREANs architecture, "Baseline3+AR+GAN" achieved the best experimental results across all metrics. The ResNeSt-101 network is a variant of the ResNet-101 network. While maintaining the latter's network depth, it adds a Split-Attention module and multi-path fusion improvements, resulting in enhanced basic network performance. The "Baseline3+AR+GAN" using ResNeSt-101 as the encoder showed significant performance improvements, with a 10.13% increase in mIoU and a 3.40% increase in OA.

[0130] The AREAANs method with three backbone networks improved classification accuracy by 4.23%, 9.27%, and 9.38% in mIoU, respectively, and by 1.58%, 2.66%, and 2.89% in OA, respectively. This verifies that adversarial learning can effectively improve classification accuracy without increasing the number of iterations.

[0131] In terms of classification performance for small targets (“vehicles”), all three AREAANs architectures showed significant improvements, with F1 scores increasing by 10.10%, 21.71%, and 23.18%, respectively. This demonstrates that the decoder improvement strategy combining AR architecture and adversarial learning makes the model more sensitive to small targets, especially on high-resolution remote sensing images with complex textures and structures.

[0132] The AREAANs proposed in this invention (AREANs-VGG, AREAANs-ResNet, and AREAANs-ResNeSt) are compared with other classic networks on the Vaihingen dataset. FCN, UNet, SegNet, PSPNet, and DeepLabv3+ are the five most typical deep learning networks. TreeUNet uses an adaptive Tree-CNN module approach, constructs a confusion matrix using Tree-CNN blocks, and combines tree pruning algorithms to reduce misclassification in prediction. HSN replaces different convolutional layers with inception modules, improving the network's multi-scale receptive field to rich texture information and enhancing its prediction ability. REMSNet improves the network's ability to perceive global texture information by constructing a parallel multi-kernel deconvolution module and adding an attention mechanism. SPNet proposes a lightweight network design by introducing striped pooling and hybrid pooling modules, combined with attention mechanisms and multi-path fusion ideas. All of the above networks adopt an encoder-decoder architecture. Specific experimental results are shown in Table 2.

[0133] Table 2

[0134]

[0135] Overall, the classification performance of different models is relatively similar, mainly because they all adopt an encoder-decoder architecture similar to FCN. Building upon this, various improvement methods, such as establishing high- and low-order feature connections (UNet, SegNet), constructing multi-scale feature fusion (PSPNet, HSN), and adding attention mechanisms (REMSNet), have improved the network performance to varying degrees. Thanks to Attention-Residual Blocks and adversarial training, the proposed AREANS achieves excellent results in overall accuracy and F1 score. AREANS-ResNeSt performs best, with OA and average F1 score improved by 1.73% and 3.78% respectively compared to SegNet, and by 3.11% and 4.91% respectively compared to UNet, demonstrating the effectiveness of the proposed method.

[0136] The AREAANs architecture used in this invention focuses on extracting useful features containing global context information and local location information. The adversarial training method implicitly learns high-order structural information during the training phase and refines the prediction results during the testing phase without requiring additional time costs (as shown in Table 3).

[0137] Table 3

[0138]

[0139] Partial prediction results of different model methods on the Vaihingen dataset are as follows Figure 8 As shown, where Figure 8 The first row shows six original images, the second row shows the ground truth labels for each original image, and the remaining rows show the classification results obtained by different algorithms. It can be seen that different colors, sizes, and textures of ground features lead to a general problem of large intra-class differences and small inter-class differences in the images, which greatly complicates image classification tasks. Specifically, different car colors create significant class differences; dark-colored cars and their shadows on the road are very similar, and objects with similar textures are also prone to misclassification, for example... Figure 8 The buildings and impermeable layers in the third column, Figure 8 Fifth column and Figure 8 The sixth column shows shrubs and trees. Furthermore, objects may fail to be extracted at the edges of the image due to a lack of relevance to the surrounding environment, as shown in the second column. FCN, UNet, and SegNet exhibited lower performance, primarily due to their inability to effectively reuse features, leading to a lack of useful contextual information. While DeepLabv3+ and PSPNet employed ASPP and pyramid pooling modules respectively, they failed to effectively perform pixel-level classification of high-resolution aerial imagery. Although SPNet introduced striped pooling and hybrid pooling modules with attention mechanisms, it did not adequately address misclassification caused by class imbalance. In contrast, the AREAANs architecture proposed in this chapter, through AR blocks and adversarial training strategies, outperforms other comparative methods. Moreover, the generator in the AREAANs architecture is optimized using a multi-path fusion focus loss function, mitigating the impact of class imbalance to some extent.

[0140] In addition, this experiment also compares the classification method of this invention with some convolutional neural network models published on the ISPRS benchmark (for details, please refer to: https: / / www2.isprs.org / commissions / comm2 / wg4 / results / ).

[0141] These models mainly include:

[0142] SVL_X: This method was provided by the ISPRS 2D Semantic Labeling Contest organizers as a benchmark for comparison among participants. It combines NVDI, nDSM, and SVL features, employing an Ada-boost classifier and a CRF model to obtain the final prediction results. In the following experiments, SVL_1 represents the result after CRF post-processing, and SVL_3 represents the prediction result without CRF post-processing.

[0143] RIT_L7: This model combines a random tree algorithm to extract structural information from images and labels. FCN uses this extracted structural information for pixel-level classification. IR-RG three-channel images and DSM data are used as training data.

[0144] DLR_8: This model utilizes ensemble learning to integrate feature information from FCN, SegNet, and VGG at different scales, introduces edge detection, and improves the accuracy of segmentation results; it uses IR-RG three-channel images and DSM data as training data.

[0145] UZ_1: This model adopts a "decoder-encoder" structure. The decoder learns the spatial information of the input data, and then the "deconvolution" structure in the decoder is used to restore the feature information. In addition, nDSM data is added as training data.

[0146] DST_X: It adopts a hybrid FCN structure, uses images and DSM as training data, and uses CRF for post-processing of the data; in the experiment, the generation trained with the Vaihingen dataset is called DST_2, and the generation trained with the Potsdam dataset is called DST_5.

[0147] BKHN_X: It adopts a hybrid structure of FCN and ResNet-101, and in addition to the input image, it also uses DSM and nDSM as training data. In the experiment, the code trained with the Vaihingen dataset is called BKHN_5, and the code trained with the Potsdam dataset is called BKHN_3.

[0148] GSN: It adopts an FCN structure with a threshold control mechanism and uses ResNet-101 as the encoder.

[0149] CASIA2: The model uses a single self-cascaded network structure, and the encoder is a variant of ResNet-101; the data used is only three-channel IRRG imagery, without using ISPRS 2D Semantic Labeling Contest elevation data (DSM and nDSM) or any post-processing methods, which is the same as the AREAANs mentioned in the paper.

[0150] ADL_3: This method combines CNN and handcrafted features to achieve pixel-level classification of dense image patches. A random forest classifier is trained using handcrafted features, and combined with a CNN to achieve initial image predictions; finally, a CRF is used to refine the prediction results.

[0151] ONE_7: This method fuses the prediction results of SegNets at two scales and uses IRRG imagery with NVDI (Normalized Difference Vegetation Index), DSM and nDSM data as training data.

[0152] BUCTY5: Uses a tree-like CNN structure, combined with pruning algorithms, and trains the network using both IRRGB and DSM data.

[0153] The quantitative comparison results of the present invention (AREANs) and the above-mentioned methods on the ISPRS Vaihingen test set are shown in Table 4, and the quantitative comparison results of the present invention and the above-mentioned methods on the ISPRS Potsdam test set are shown in Table 5.

[0154] Table 4

[0155]

[0156] Table 5

[0157]

[0158] Tables 4 and 5 present the quantitative comparison results between the AREANS proposed in this invention and other methods disclosed in the ISPRS 2D Semantic Labeling Contest. Overall, AREANS achieved excellent results on both datasets, with AREANS-ResNeSt achieving OA values ​​of 91.3% and 91.9% on the Vaihingen and Potsdam datasets, respectively. Furthermore, it significantly improved the correct recognition rate for small targets (“vehicle” class) compared to other methods on the benchmark, achieving F1 scores of 90.5% and 97.0% on the Vaihingen and Potsdam datasets, respectively.

[0159] Because high-resolution remote sensing images contain a wealth of complex texture and structural information, selecting deeper and more powerful models as feature extraction networks is one solution to improve the overall performance of the models. Following this approach, GSN, CASIA2, and BKHN employ the ResNet-101 pre-trained network, which performs exceptionally well on natural image datasets, as the encoder for feature extraction from high-resolution remote sensing images. This design is primarily based on the following reasons: fine-tuning methods based on pre-trained models can improve the network's generalization ability; while randomly initialized networks may focus more on the spectral information of image targets, neglecting the semantic information of the targets, resulting in a decrease in the network's generalization ability. Therefore, the classification accuracy of the above three methods is superior to other methods, especially CASIA2 and BKHN, which achieve slightly lower classification accuracy than this invention.

[0160] In the AREAANs architecture proposed in this invention, AREAANs-ResNeSt achieves optimal performance on all metrics across both datasets, while AREAANs-ResNet performs slightly worse. AREAANs-VGG is the weakest, but it still holds certain advantages over other methods on the benchmark. Thanks to the adversarial learning training strategy, compared to DST_X, ONE_7, and DLR_8, the proposed architecture effectively improves model classification accuracy without increasing time consumption or the number of parameters, even when only using the generator for testing. Furthermore, the pre-trained CNN network in the decoder avoids overfitting caused by the strong correlation between remote sensing image patches. Simultaneously, to improve the stability of generative adversarial network training, the proposed method uses the TTUR strategy, reducing the training difficulty. The method proposed in this invention does not use any additional data (such as DSM, nDSM, NDVI, manually designed features, etc.) for assistance, does not employ additional classifiers and post-processing steps (such as CRF in RIT_L7, DST_X and ADL_3), and does not use model ensemble methods to improve the classification accuracy of the network.

[0161] Figures 9-a to 9-i as well as Figures 10-a to 10-l The images show the classification results obtained by randomly selecting one image from each of the two datasets. It is evident that this invention embeds an Attention-Residual block, effectively enhancing the model's ability to perceive positional and semantic information. Combined with the pre-trained model, this results in better classification outcomes.

Claims

1. A co-domain remote sensing image classification method based on adversarial learning, characterized in that, This classification method includes the following steps: The remote sensing image data to be classified is acquired and input into a trained classification model for classification. The classification model includes a generator and a discriminator. The generator employs an encoder-decoder structure to obtain pixel classification probability maps for C categories of the remote sensing image. The decoder uses a multi-layer convolutional neural network, including multiple convolutional neural networks of different depths. Each convolutional neural network of different depths includes a residual part, an upsampling part, and an attention enhancement part. The residual part receives features concatenated from the features of the previous convolutional neural network layer and the corresponding features of the encoder layer. The attention enhancement part includes a semantic information enhancement module that processes information in parallel. The location information enhancement module; the discriminator adopts a convolutional neural network, and the last layer of the discriminator is a convolutional layer, which is used to judge the authenticity of each image patch in the input image. The output matrix represents the probability that each image patch in the input image is the true label. The input of the discriminator is a new feature map obtained by multiplying the probability map of each class with the corresponding image. The new feature map contains 3×C channels, which are used to distinguish between the two by obtaining the high-order consistency between the true label and the prediction result generated by the generator; the generator and the discriminator are trained in an adversarial training manner. The generator is jointly optimized by adversarial loss and multi-path fusion focus loss.

2. The co-domain remote sensing image classification method based on adversarial learning according to claim 1, characterized in that, The encoder employs a feature extraction network to map the input remote sensing image data into a high-dimensional feature space. Connections are made between corresponding layers in the decoder and encoder to fuse low-level feature location information and high-level feature semantic information.

3. The co-domain remote sensing image classification method based on adversarial learning according to claim 2, characterized in that, The residual part includes two convolutional modules, and the input and output of the residual part are connected across layers; the input of the upsampling part is used to receive the output signal of the residual part, and is used to restore the feature map after processing by the residual part to the size of the corresponding high-order feature map.

4. The co-domain remote sensing image classification method based on adversarial learning according to claim 2, characterized in that, The inputs to both the semantic information enhancement module and the location information enhancement module are feature maps output from the upsampling part. The semantic information enhancement module is used to operate on the channel dimension of the input feature map and use the correlation between high- and low-order channels to complete the modeling of the relationship between specific semantic information between high- and low-order feature maps. The location information enhancement module is used to establish the location information correlation between the local features of the input feature map and other neighborhoods.

5. The co-domain remote sensing image classification method based on adversarial learning according to claim 4, characterized in that, The processing procedure of the semantic information enhancement module is as follows: Global average pooling is used to obtain statistical information of the input feature map in the channel dimension; Based on the statistical information obtained in the channel dimension, the weights in each channel dimension are determined using linear transformation and activation functions. The formula for calculating the weights is as follows: This represents the feature vector obtained after global average pooling. and These represent the weights of a 1×1 convolutional layer. Represents the ReLU function. This represents the Sigmoid operation. Represents the total number of categories; The enhanced feature map is determined based on the weights in each channel dimension.

6. The co-domain remote sensing image classification method based on adversarial learning according to claim 4, characterized in that, The enhanced feature map obtained by the location information enhancement module is: in , This represents the enhanced feature map. , This represents a slice of the input feature map along the channel dimension. Spatial position coordinates corresponding to the feature map , ; This represents the mapping matrix after a 1×1 convolution operation. express go through Mapped weight graph; yes Scaled to Sigmoid The result between them represents the feature map. The importance of location information.

7. The co-domain remote sensing image classification method based on adversarial learning according to claim 2, characterized in that, The generator uses the following loss function during training: The binary cross-entropy loss is used to reduce the performance of the discriminator. This is the focus loss function for multi-path fusion, used to generate the correct classification prediction for each pixel of the input image. Represents generator The parameters, yes and linear combination, As The penalty coefficient, This indicates that the discriminator judges the input. These are predictions from the generator. or truth value tag , This represents the one-hot encoding of a certain class in the truth label. This represents a certain type of prediction result generated by the generator.

8. The co-domain remote sensing image classification method based on adversarial learning according to claim 1 or 2, characterized in that, The discriminator consists of eight convolutional layers connected in series. The kernel size of each convolutional layer is 4×4. Except for the stride of the last convolutional layer, which is 1, the stride of the first to seventh convolutional layers is 2. The first convolutional layer uses the ReLU activation function, and the remaining convolutional layers use the LeakyReLU activation function.

9. The co-domain remote sensing image classification method based on adversarial learning according to claim 7, characterized in that, The loss function of the discriminator can be defined in the following form: in Discriminator The parameters in This represents the binary cross-entropy loss. This indicates that the discriminator judges the input. These are predictions from the generator. or truth value tag , This represents the one-hot encoding of a certain class in the truth label. This represents a certain type of prediction result generated by the generator.

10. The co-domain remote sensing image classification method based on adversarial learning according to claim 9, characterized in that, Discriminator parameters and generator parameters A step-by-step update approach is adopted, first fixing the generator parameters. Update the discriminator parameters This enables the discriminator to distinguish the predicted results; then the parameters of the discriminator are fixed. Update the generator parameters This causes the generator to produce predictions that the discriminator cannot distinguish between true and false.

Images