Cross-domain remote sensing image classification method and system based on feature domain enhancement

Abstract

The invention discloses a cross-domain remote sensing image classification method and system based on feature domain enhancement, and the method comprises the steps: 1), extracting original features, and extracting remote sensing image features; 2) performing significance guide optimization on any two features in the extracted original application features; 3) performing feature domain enhancement on the two optimized features to create new features and corresponding labels; and 4) inputting the new features created in the step 3) and the corresponding labels into a classifier for classification. According to the method, new features and corresponding labels are generated by mixing saliency-guided features, the number and diversity of source domain data in a feature space are remarkably increased, the generalization ability of the model is improved, the model is excellent in performance when a target domain is not seen, depth features extracted by a baseline network are refined by using a saliency-guided optimization module, and the robustness of the model is improved. The feature domain enhancement is performed to mix the optimized features and their tags to produce new samples of the classifier to emphasize the important regions.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image classification and processing, relates to a cross-domain remote sensing image classification method based on feature domain enhancement, and also relates to a cross-domain remote sensing image classification system based on saliency-guided feature domain enhancement. Background Art

[0002] In the era of remote sensing big data, a large number of remote sensing images acquired by various aircraft and satellites have posed higher challenges to remote sensing image interpretation. Efficient and accurate automatic recognition of images is a basic and key component in remote sensing image interpretation, and scene-level image classification is a basic and key component. However, pixel-level and object-level image interpretation methods are limited due to the mixed pixel problem caused by the increase in spatial resolution. Therefore, scene-level image classification, which recognizes the entire image scene from a high semantic perspective, is widely used due to its high efficiency and stable performance, providing important support for further applications such as object detection, semantic segmentation, object tracking, natural disaster monitoring, and functional area classification.

[0003] At present, remote sensing image scene classification has made great progress due to the development of deep learning technology. At the same time, in order to meet the needs of developing and evaluating deep learning-based classification methods, a large number of datasets have been proposed continuously, such as AID, MlRSN, and NWPU45 datasets. However, deep learning-based methods can only achieve satisfactory performance when there are a large number of training samples, and collecting remote sensing images of each scene and annotating them is time-consuming and laborious. For remote sensing images, the differences between different datasets lie in many factors, such as differences in sensors, resolutions, lighting conditions, and standards for constructing datasets. Even images with the same labels in different datasets have visual differences. Therefore, even well-trained models have low generalization ability, which makes domain shift a very common problem and seriously hinders the development and application of scene classification methods.

[0004] In this context, cross-domain classification methods, also known as cross-scene classification methods, have been proposed to address the difficulties caused by the above-mentioned scene classification problems. These methods attempt to identify scene labels for remote sensing images by learning more generalizable models from existing datasets. Most cross-domain classification methods are implemented through domain adaptation (DA). As an important technique in transfer learning, DA can help the model learn domain-invariant features by exploring the correlation between the source domain and the target domain so as to better adapt to new domains. In the paper "D. Wittich and F. Rottensteiner, "Appearance based deep domain adaptation for the classification of aerial images," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 180, pp. 82–102, 2021", a joint training strategy of adversarial appearance adaptation (AAN) and classifiers is proposed to achieve pixel-level cross-domain classification. The paper "S. Zhu, B. Du, L. Zhang, and X. Li, "Attention-based multiscale residual adaptation network for cross-scene classification," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2021" proposes an attention-based multiscale residual adaptation network (AMRAN) to complete the cross-scene classification task.In addition, the paper "J. Zhang, J. Liu, B. Pan, Z. Chen, X. Xu, and Z. Shi, "An open set domain adaptation algorithm via exploring transferability and discriminability for remote sensing image scene classification," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–12, 2021" proposed an open set domain adaptation algorithm by exploring transferability and discriminability (OSDA-ETD) to expand the application scope of cross-domain adaptation. Although cross-domain classification methods based on domain adaptation (DA) have achieved excellent performance, they have a fatal flaw. In addition to labeled source domain data, they also require additional unlabeled data in the target domain for operations such as feature alignment, domain matching, and domain adversarial learning. At the same time, when DA-based models are adapted to new datasets, they also need to be retrained or fine-tuned based on new samples from unseen domains. However, the acquisition of target domain data is not always guaranteed, which limits the applicability of DA-based methods in practical applications. Domain generalization (DG), as a more challenging task than DA, requires only data from the source domain during training, and aims to learn a model that can achieve good performance on one or more different but related source domains to cope with unseen and inaccessible target domains. In recent years, various domain generalization methods have been proposed and have achieved remarkable developments in image classification, semantic segmentation, behavior recognition, video understanding, and other fields. Since DG is a relatively new research direction, there are relatively few studies on domain generalization in remote sensing.In the paper "Y. Zhang, M. Zhang, W. Li, S. Wang, and R. Tao, "Language-aware domain generalization network for cross-scene hyperspectral image classification," IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–12, 2023", a language-aware domain generalization network (LDGnet) was developed to learn cross-domain invariant representations from cross-domain shared prior knowledge for hyperspectral image classification. The paper "Y. Zhang, W. Li, W. Sun, R. Tao, and Q. Du, "Single-source domain expansion network for cross-scene hyperspectral image classification," IEEE Transactions on Image Processing, vol. 32, pp. 1498–1512, 2023" designs a single-source domain expansion network (SDEnet) to ensure domain reliability and effectiveness when using generative adversarial learning for hyperspectral image classification. The paper "S. Zhu, C. Wu, B. Du, and L. Zhang, "Style and content separation network for remote sensing image cross-scene generalization," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 201, pp. 1–11, 2023" proposes a style and content separation network (SCSN) to improve the generalization and discrimination capabilities of cross-scene generalization.

[0005] As we all know, deep learning has a huge demand for training data, and the generalization performance of the model often depends on the amount and diversity of training data. Therefore, especially when the training samples are limited, such as in the field of remote sensing, data manipulation is the most intuitive and economical way to enhance the generalization ability of the model. Domain generalization methods based on data manipulation are committed to using different algorithms to operate on existing training data to increase its diversity. Among them, Mixup and its variants such as ManifoldMixup, CutMix and SaliencyMix are the most popular and widely used types of data processing algorithms. Given two training samples X i and X j and its corresponding label Y i and Y j , Mixup generates a new instance and its tags Here λ∈[0,1] is a random mixing weight sampled from a Beta distribution. Mixup is able to generate new data without training a generative network, which helps the model improve its generalization ability. Mixup assumes that every pixel in the image is equal, so linear interpolation of feature vectors should lead to linear interpolation of the relevant target. On the other hand, CutMix randomly replaces a region of one image with a piece of another image and mixes their labels in proportion to the ratio of the mixed region. However, in real remote sensing images, the contribution of background pixels to label semantics is not always as important as that of pixels in salient regions. At the same time, simply cutting and mixing blocks of different images will inevitably destroy the continuity of generated samples.

[0006] As disclosed in the literature "Q. Wan, Z. Xiao, Y. Yu, Z. Liu, K. Wang, and D. Li, "A hyperparameter free attention module based on feature map mathematical calculation for remote-sensing image scene classiffcation," IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–18, 2023 (A hyperparameter free attention module based on feature map mathematical calculation for remote-sensing image scene classification method)", a hyperparameter free attention module based on feature map mathematical calculation is used in the remote sensing image scene classification method. When processing remote sensing images, the scenes are divided into two categories: scene categories determined by related targets or their relationships (category A) and semantic labels determined by large-scale image features such as color, texture, shape, and spatial structure (category B). Figure 1As shown, the school and bridge belong to category A, while the forest and bare land belong to category B. There are three cases when using the above data manipulation method to process remote sensing images: mixing two different samples in category A (as shown in the first row in the figure), mixing two different images in category B (as shown in the second row in the figure), and mixing two different instances from category A and category B respectively (as shown in the third row in the figure). When Mixup is implemented on two random samples, it treats pixels in the background and salient areas equally, resulting in insufficient attention to important areas when it comes to samples belonging to category A. At the same time, the random cutting operation in CutMix cannot guarantee that important relevant targets of the image can always be obtained, which may cause the generated image to lack information about key targets in the source image. Summary of the invention

[0007] The technical solution adopted by the present invention is: a cross-domain remote sensing image classification method and system based on feature domain enhancement, which can obtain important related targets of the image and effectively avoid causing the generated image to lack information of key targets in the source image.

[0008] The cross-domain remote sensing image classification method based on feature domain enhancement includes the following specific steps:

[0009] 1) Extract original features and remote sensing image features;

[0010] 2) Perform significance-guided optimization on any two features of the extracted original application features;

[0011] 3) The two optimized features are enhanced in feature domain to create new features and corresponding labels;

[0012] 4) Input the new features and corresponding labels created in step 3) into the classifier for classification.

[0013] Furthermore, the method of significance-guided optimization in step 2) above is:

[0014] Given any two extracted features x i ∈R H×W×C and x j ∈R H×W×C , and their corresponding labels y i and j , as the input of saliency-guided optimization, where H, W, and C refer to the height, width, and number of channels, respectively. In saliency-guided optimization, each feature is optimized for saliency individually without modifying their labels Y i and Y j ;

[0015] For the input feature x i ∈R H×W×C, calculate the average value through the spatial dimension to obtain a vector V∈R 1×1×C , that is, V=[v1, v2,...,v C ], and then obtain the mean value v by avg and variance v var :

[0016]

[0017] Then, for each element v in vector V, n With v avg The square of the difference is used to obtain a new vector V′∈R 1 ×1×C , where V′=[v′1,v′2,...,v′ C ], and v′ n =(v n -v avg ) 2 , v′ max and v′ min Represent the maximum and minimum values ​​in V′ respectively, and then let the vector P = [p1, p2, ..., p C ], P is a vector, p n refers to an element of P, computed as follows:

[0018]

[0019] Where n = [1, 2, ..., C], then the sigmoid function is executed to activate the vector P and obtain the weight P′∈R of each channel 1×1×C ;

[0020] At the same time, the input x is calculated i ∈R H×W×C The average value along the channel dimension produces a matrix U, where u nm is U∈R H×W×1 Similarly, the average value u is calculated by avg and variance u var :

[0021]

[0022] Construct another matrix U′∈R H×W×1 , u′ nm is an element of U′, and u′ is obtained by nm :

[0023] u′ nm =(u nm -u avg )2

[0024] Similarly, let u′ max and u′ min are the maximum and minimum values ​​in U′ respectively;

[0025] Then, we obtain a new matrix Q∈R H×W×1 , where q nm is Q∈R H×W×1 An element of is obtained by:

[0026]

[0027] Where n = [1, 2, ..., H] and m = [1, 2, ..., W], Q∈R H×W×1 After the sigmoid activation function, the spatial weight Q′∈R is obtained H×W×1 ;

[0028] Finally, the spatial weight Q′∈R H×W×1 and channel weight P′∈R 1×1×C Multiply them together to get the final weight of the same shape as the original input feature; then convert the original feature x into i ∈R H×W×C Multiply these final weights to obtain the final refined features x′ that emphasize saliency i , the calculation formula is expressed as:

[0029] x′ i =(P′×Q′)·x i (7)

[0030] In summary, the two original input features x i ∈R H×W×C and x j ∈R H×W×C Through SOM processing, two significant guided optimization features x′ are obtained. i ∈R H×W×C and x′ j ∈R H×W×C , which serve as the input of the subsequent feature domain enhancement module.

[0031] Furthermore, the feature domain enhancement method in the above step 3) is: obtain new features by mixing the optimized features, and realize enhancement on the feature domain, that is, generate a new feature and its label by two input optimized features. The formula is as follows:

[0032]

[0033]

[0034] where λ∈[0, 1] is a random mixing weight sampled from a Beta distribution.

[0035] Furthermore, the original feature extraction method in the above step 1) is performed using a ResNet-18 model.

[0036] A cross-domain remote sensing image classification system based on saliency-guided feature domain enhancement, including:

[0037] Feature extraction module, used to extract remote sensing image features;

[0038] A saliency guided optimization module, used for performing saliency guided optimization on any two features of the extracted remote sensing image features;

[0039] The feature domain enhancement module is used to enhance the feature domain of the two features of the saliency-guided optimization and create new features and corresponding labels;

[0040] The classification module is used to classify cross-domain remote sensing images based on new features and corresponding labels.

[0041] Beneficial effects of the present invention: Compared with the prior art, the simple and effective domain enhancement method of saliency-guided feature domain enhancement (SFDA) proposed in the present invention is used in cross-domain super-sensory image classification for cross-domain generalization. By mixing saliency-guided features to generate new features and corresponding labels, the number and diversity of source domain data in the feature space are significantly increased, thereby improving the generalization ability of the model and performing well on unseen target domains. Specifically, a saliency-guided optimization module (SOM) is first used to refine the deep features extracted by the baseline network to emphasize important areas, and then a feature domain enhancement module (FDAM) is implemented to mix the optimized features and their labels to generate new samples for the classifier. Extensive experiments have verified the effectiveness, stability and universality of the proposed SFDA. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The following are examples of mixing ratios for remote sensing images with λ=0.4 under three different conditions; in the first row, two different samples in class A are used as input, and in the second row, the mixing of two different images belonging to class B is simulated; the third row describes the mixing of two examples of class A and class B respectively;

[0043] Figure 2 This is the flow chart of SFDA;

[0044] Figure 3 Flowchart of the Significance Optimization Module (SOM);

[0045] Figure 4 Figure 1 is from the ACMR-7 dataset;

[0046] Figure 5 This is a graph of the AMNP-14 dataset. Similarly, images of different sizes are reduced to the same size for easier viewing.

[0047] Figure 6 The performance curve comparison diagram of the baseline ResNet-18 and the SFDA of this application on each category of different tasks on the ACMR-7 dataset, where the improvement and reduction of SFDA relative to the baseline are represented by the length of the red and green lines, respectively, and the blue circle indicates that the two methods have the same performance in this category;

[0048] Figure 7 A feature comparison diagram of the baseline ERM and the proposed SFDA on the multi-source generalization task C on the ACMR-7 dataset using t-SNE visualization;

[0049] Figure 8 A feature comparison diagram of the baseline ERM and the proposed SFDA on the single-source generalization task CA on the ACMR-7 dataset using t-SNE visualization;

[0050] Fig. 9 Under different training ratios, the baseline ERM and the SFDA proposed in this invention are compared, selecting tasks C and AC on the ACMR-7 dataset as example curves of multi-source and single-source cross-domain generalization. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with specific embodiments.

[0052] Compared with traditional classification tasks, cross-domain classification aims to alleviate the performance degradation caused by domain shift, which is more valuable and challenging in the field of remote sensing image interpretation and has become a hot topic in current research. Most of the previous cross-domain classification methods are based on domain adaptation (DA) technology, which inevitably requires access to samples in the target domain during training and cannot be applied to real-time processing and when the target domain is unavailable. Domain generalization (DG) is a more difficult and more suitable technique for cross-domain classification. It explores how to help the model better generalize to unseen target domains by alleviating domain shift when the target domain is inaccessible. Inspired by Mixup, a popular data manipulation method in DG, this paper develops a saliency-guided feature domain enhancement (SFDA) to increase the amount and diversity of source domain data in the feature space, thereby improving the generalization ability of the model. The saliency-guided optimization module (SOM) is used to optimize features to emphasize important salient regions. Subsequently, in the feature domain enhancement module (FDAM), new features and corresponding labels are generated by mixing any two optimized features in the feature space and then input into the classifier. The feature distribution of the enhanced source domain is broader and denser than the original distribution, which can significantly improve the generalization ability of the model when the target domain is inaccessible. Extensive experiments verify the effectiveness, stability and universality of the proposed SFDA in cross-domain classification.

[0053] Therefore, a saliency-guided feature domain enhancement (SFDA) method is proposed for cross-domain generalization of remote sensing scene classification to effectively improve the performance of the model in cross-domain generalization tasks. Mixing operations on the middle hidden layers of the network are proven to be more effective than mixing on the input data. Unlike Mixup and CutMix, which directly mix the original source images to generate new images, the method of the present invention mixes different samples on the feature map. The proposed method uses the features of the two images generated by the hidden layers of the network during training to generate new features and corresponding labels that focus on salient regions, effectively increasing the amount and diversity of training data and enabling the network to better generalize to unseen target domains. Specifically, two different features are optimized separately using the saliency-guided optimization module (SOM) to force them to focus on important salient regions in the image, and then new features and corresponding labels are generated by mixing the optimized features in the feature domain enhancement module (FDAM). The proposed SFDA is evaluated with some state-of-the-art methods in different DG settings, and extensive experiments demonstrate its superior performance.

[0054] Domain Generalization Introduction: Compared with domain adaptation (DA), domain generalization (DG) is a more difficult task, which focuses on learning a generalizable cross-domain model without using any target domain data in the training phase. Therefore, DG is more effective and general than DA in practical applications. When migrating to a new target domain, DA requires supplementary data from the new target domain for retraining, while DG can be applied directly. According to the number of source domains, domain generalization can be divided into two categories: single-source and multi-source domain generalization. The former means that there is only one source domain available, and the latter refers to there are multiple accessible source domains. Table I intuitively illustrates the difference between DA and the two types of DG, where Ds and Dt refer to the source domain and target domain, respectively. In order to cope with the cross-domain classification problem, many DG-based methods have been proposed, mainly including representation learning, learning strategies, and data manipulation.

[0055] Table 1 shows the differences between DA and DG

[0056]

[0057] The representation learning of DG can be divided into two main types according to the differences in learning principles: methods based on domain-invariant representation and methods based on feature decoupling. The former performs domain-invariant representation learning through kernel-based techniques, adversarial learning, explicit feature alignment, and invariant risk minimization to obtain a model with generalization ability for unseen target domains. The latter decomposes features into domain-shared and domain-private features through multi-component analysis or generative modeling to improve generalization ability.

[0058] Developing various learning strategies that can improve the generalization ability of the model is another feasible solution for DG, which can be divided into three different categories: ensemble learning-based methods, meta-learning-based methods, and other learning strategies such as self-supervised learning and self-challenge training algorithms.

[0059] Since improving the quality and diversity of training data can improve the generalization ability of the model, data manipulation methods have been developed to help the model learn more general feature representations from the perspective of manipulating training data. Data augmentation and data generation are the two most mainstream techniques for implementing data manipulation. Data augmentation is one of the most widely used techniques in machine learning, used to reduce overfitting and improve generalization ability, including typical operations such as flipping, rotation, cropping, etc., as well as atypical operations such as domain randomization and adversarial data augmentation. Data generation focuses on generating new data through different generation methods to improve the generalization ability of the model, such as variational autoencoders (VAE), generative adversarial networks (GANs), and Mixup algorithms.

[0060] Mixup algorithm and its variants: Mixup is a powerful and successful data manipulation method for generating augmented images between two examples through pixel-wise weighted interpolation to avoid overfitting on training data and encourage the model to generalize better. Inspired by this, many similar Mix-based data generation techniques have been proposed and applied in various tasks including domain adaptation (DA) and data generation (DG). As summarized in the paper "C.Cao, F.Zhou, Y.Dai, and J.Wang, "A survey of mix-based data augmentation: Taxonomy, methods, applications, and explainability," arXiv preprint arXiv:2212.10888, 2022", they can be divided into three types: global mixing, local mixing, and others. Global mixing methods such as Mixup, ManifoldMixup, and AlignMixup mix different images as a whole. Mixup directly linearly mixes the original input to obtain new augmented examples, while ManifoldMixup extends the mixing operation to the output of any layer of the network. AlignMixup further spatially aligns the two images in the feature space to allow consistent interpolation. On the other hand, local mixing methods such as CutMix, Puzzle-Mix, Attentive-CutMix, and SaliencyMix combine different images in regions. CutMix is ​​a combination of Cutout and Mixup, which replaces a region of one image with the corresponding patch of another image to obtain enhanced samples, and simultaneously interpolates and generates labels based on their area ratios. Unlike the random patch selection strategy adopted by CutMix, SaliencyMix selects representative patches of the image based on the saliency map and mixes them with other images to help the model learn more appropriate feature representations. Other Mix-based techniques adopt different strategies for data mixing than the above two types. For example, DJMix mixes each training example with its discretized one to obtain enhanced data.

[0061] Global mixing methods treat each pixel equally and assume that linear interpolation of feature vectors should lead to linear interpolation of related targets, which is inconsistent with the characteristics of remote sensing images. In fact, pixels in salient areas contribute more to semantic labels than background pixels. On the other hand, some local mixing methods such as Attentive-CutMix and PuzzleMix attempt to solve this problem by mixing salient areas only in the input image space. However, they completely discard the unselected background parts, resulting in the loss of continuity of the generated new image. At the same time, in order to obtain salient areas from the input image, they are usually implemented through complex operations, resulting in more parameters or training throughput. For example, AttentiveCutMix introduces an additional convolutional neural network with 24 million parameters, while the model using Puzzle-Mix propagates forward and backward twice in one iteration.

[0062] Different from the above methods, this application focuses on designing a global hybrid data operation method that also considers local saliency, generating new samples by mixing optimized features in feature space instead of enhanced image space, emphasizing saliency.

[0063] Since improving the quality and diversity of training data can improve the generalization ability of the model, data manipulation methods have been developed to help the model learn more general feature representations from the perspective of manipulating training data. Data augmentation and data generation are the two most mainstream techniques for implementing data manipulation. Data augmentation is one of the most widely used techniques in machine learning, used to reduce overfitting and improve generalization ability, including typical operations such as flipping, rotation, cropping, etc., as well as atypical operations such as domain randomization and adversarial data augmentation. Data generation focuses on generating new data through different generation methods to improve the generalization ability of the model, such as variational autoencoders (VAE), generative adversarial networks (GANs), and Mixup algorithms.

[0064] Example 1: Figure 1-9 As shown in FIG, the cross-domain remote sensing image classification method based on feature domain enhancement includes the following specific steps:

[0065] 1) Extract original features and remote sensing image features;

[0066] The original feature extraction method uses the ResNet-18 model for extraction;

[0067] 2) Perform significance-guided optimization on any two features of the extracted original application features;

[0068] For the two different types of remote sensing scenes mentioned above, category A refers to those semantic information mainly determined by relevant key objects or their relationships, while category B represents scene labels determined by a wide range of features such as color, texture, shape, and spatial structure. In the paper "Q. Wan, Z. Xiao, Y. Yu, Z. Liu, K. Wang, and D. Li, "A hyperparameterfree attention module based on feature map mathematical calculation for remote-sensing image scene classiffcation," IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-18, 2023", a lightweight plug-and-play attention module (HFAM) is designed to obtain refined features through mathematical calculation without any hyperparameters. It uses the difference between each pixel in the feature map and the background represented by the mean to quantitatively measure the importance of each pixel. At the same time, the range of the difference can also effectively distinguish similar features on a large scale. This simple and effective module greatly improves the effect of remote sensing scene recognition. Since it perfectly meets the need to emphasize the salient areas of the feature map, it is used to optimize any two feature maps simultaneously. According to the experimental demonstration in the literature "Q. Wan, Z. Xiao, Y. Yu, Z. Liu, K. Wang, and D. Li, "A hyperparameter free attention module based on feature map mathematical calculation for remote-sensing image scene classification," IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-18, 2023", when used alone, inserting HFAM into the deeper layers of the network works better, so in this work, it is performed after convolution block 4 of ResNet-18;

[0069] The method of significance-guided optimization is:

[0070] Given any two extracted features x i ∈R H×W×C and x j ∈R H×W×C , and their corresponding labels y i and j, as the input of saliency-guided optimization, where H, W, and C refer to the height, width, and number of channels, respectively. In saliency-guided optimization, each feature is optimized for saliency individually without modifying their labels Y i and Y j ;

[0071] Figure 3 The implementation overview of the saliency guided optimization module is shown. For the input feature x i ∈R H×W×C , calculate the average value through the spatial dimension to obtain a vector V∈R 1×1×C , that is, V=[v1, v2,...,v C ], and then obtain the mean value v by avg and variance v var :

[0072]

[0073] Then, for each element v in vector V, n With v avg The square of the difference is used to obtain a new vector V′∈R 1 ×1×C , where V′=[v′1,v′2,...,v′ C ], and v′ n =(v n -v avg ) 2 , v′ max and v′ min Represent the maximum and minimum values ​​in V′ respectively, and then let the vector P = [p1, p2, ..., p C ], p n Calculated by:

[0074]

[0075] Among them, n = [1, 2, .... C], then execute the sigmoid function to activate P, and obtain the weight of each channel P ′∈R 1×1×C ;

[0076] At the same time, the input x is calculated i ∈R H×W×C The average value along the channel dimension produces a matrix U, where u nm is U∈R H×W×1 Similarly, the average value u is calculated by avg and variance u var :

[0077]

[0078] Construct another matrix U′∈R H×W×1 ,u nm is an element of U′, and u′ is obtained by nm :

[0079] u′ nm =(u nm -u avg ) 2

[0080] Similarly, let u′ max and u′ min are the maximum and minimum values ​​in U′ respectively;

[0081] Then, we obtain a new matrix Q∈R H×W×1 , where q nm is Q∈R H×W×1 An element of is obtained by:

[0082]

[0083] Where n = [1, 2, ..., H] and m = [1, 2, ..., W], Q∈R H×W×1 After the sigmoid activation function, the spatial weight Q′∈R is obtained H×W×1 ;

[0084] Finally, the spatial weight Q′∈R H×W×1 and channel weight P′∈R 1×1×C Multiply them together to get the final weight of the same shape as the original input feature; then convert the original feature x into i ∈R H×W×C Multiply these final weights to obtain the final refined features x′ that emphasize saliency i , the calculation formula is expressed as:

[0085] x′ i =(P′×Q′)·x i (7)

[0086] In summary, the two original input features x i ∈R H×W×C and x j ∈R H×W×C Through SOM processing, two significant guided optimization features x′ are obtained. i ∈R H×W×C and x′ j ∈R H×W×C , which serve as the input of the subsequent feature domain enhancement module;

[0087] 3) The two optimized features are enhanced in feature domain to create new features and corresponding labels;

[0088] The Mixup described above is achieved by mixing two original images X i and X j (The corresponding label is Y i and Y j ) Generate new training samples. The generation formula is as follows:

[0089]

[0090] where λ∈[0,1] is a random mixing weight sampled from a Beta distribution;

[0091] The feature domain enhancement method in step 3) is: Different from Mixup, which generates samples by mixing the original image to enhance the image domain, the proposed Feature Domain Augmentation Module (FDAM) obtains new features by mixing the optimized features, thereby achieving enhancement in the feature domain. Since the SOM module only optimizes features without changing the labels, the input of FGM is the refined feature x′ i and x′ j and their labels y i and j ; New features are obtained by mixing optimized features and enhancing the feature domain, that is, a new feature and its label are generated by two input optimized features. The formula is as follows:

[0092]

[0093] where λ∈[0,1] is a random mixing weight sampled from a Beta distribution; for example, in Figure 2 The value of λ is 0.4. Obviously, the method of the present invention can effectively achieve feature domain enhancement in both single-source and multi-source cases. The difference is that when performing single-source domain generalization, the two inputs of each mixing operation come from the same domain, while when performing multi-source domain generalization, the mixing may occur in the same source domain or different source domains;

[0094] 4) Input the new features and corresponding labels created in step 3) into the classifier for classification.

[0095] This paper proposes a saliency-guided feature domain enhancement (SFDA) method, which can effectively increase the amount and diversity of training data, thereby improving the generalization ability of the model and achieving excellent cross-domain classification performance. Figure 2The overall framework of SFDA is presented in detail. ResNet-18 is used as the backbone network of SFDA to extract raw features. After convolution block 4, the required raw features are extracted to implement enhancement. First, a saliency guided optimization module (SOM) is performed to optimize any two features in the batch so that they can focus on important areas. Subsequently, the two optimized features are sent to the feature domain enhancement module (FDAM) to create new features and corresponding labels. Finally, the new features and labels are input into the classifier for classification.

[0096] The present invention has the following advantages:

[0097] 1) This paper proposes a saliency-guided feature domain enhancement (SFDA) method for cross-domain remote sensing scene classification. It can effectively increase the quantity and diversity of source domain data in the feature space, thereby improving the generalization performance of the model when generalizing to unseen target domains;

[0098] 2) Based on the characteristics of remote sensing images, the saliency guided optimization module (SOM) is used to adaptively refine the feature map to emphasize important salient regions. The optimized features are then mixed in the feature domain enhancement module (FDAM) to generate features and corresponding labels that focus on salient regions while maintaining the spatial continuity of the generated features;

[0099] 3) In addition to the common multi-source domain generalization, the present invention also conducts single-source domain generalization experiments as a supplement. Extensive comparative experiments with some state-of-the-art cross-domain methods verify the effectiveness, stability and universality of the proposed SFDA.

[0100] Example 2: Reference Figure 2 , a cross-domain remote sensing image classification system based on saliency-guided feature domain enhancement, including,

[0101] Feature extraction module, used to extract remote sensing image features;

[0102] A saliency guided optimization module, used for performing saliency guided optimization on any two features of the extracted remote sensing image features;

[0103] Flowchart of the Saliency Optimization Module (SOM) Figure 3 As shown, by performing simple calculations on the feature maps, optimized features that emphasize salient regions are adaptively generated without any hyperparameters.

[0104] The feature domain enhancement module is used to enhance the feature domain of the two features of the saliency-guided optimization and create new features and corresponding labels;

[0105] The classification module is used to classify cross-domain remote sensing images based on new features and corresponding labels.

[0106] Figure 2 The cross-domain remote sensing image classification system based on saliency-guided feature domain enhancement includes a saliency optimization module (SOM) and a feature domain enhancement module (FDAM). The former is used to optimize the deep features extracted by the baseline network, and the latter is used to mix the optimized features in the feature space to generate new samples. The feature distribution of the enhanced source domain is wider and denser than the original distribution.

[0107] In order to illustrate the effect of the present invention, the following simulation experiment is carried out:

[0108] In this simulation experiment, two datasets named ACMR-7 and AMNP-14 were constructed, which contain six public datasets including AID, CLRS, MlRSN, PatternNet and RSSCN7. In order to ensure fair comparison with some state-of-the-art methods, sufficient experiments involving multi-source and single-source cross-domain generalization tasks were re-implemented with consistent settings on the same workstation.

[0109] A) Dataset

[0110] Proceedings of the National Chemistry of Additives and Additives for Additive Aids“G.-S.Xia,J .Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, L. Zhang, and X.Lu,“Aid:Benchmark data set forperformance evaluation of aerial scene classiffcation,”IEEE Transactions onGeoscience and Remote Sensing,vol.55,no.7,pp.3965–3981,2017”; C.Tao,“Clrs:Continual learning benchmark for remote sensing image scene classiffcation,”Sensors,vol.20,no.4,p.1226,2020”、MlRSN“X.Qi,P.Zhu,Y.Wang,L.Zhang,J.Peng,M.Wu,J.Chen,X.Zhao,N.Zang,andP.T.Mathiopoulos,“Mlrsnet:A multi-label high spatial resolution remotesensing dataset for semantic scene understanding,”ISPRS Journal ofPhotogrammetry and Remote Sensing,vol.169,pp.337–350,2020”、NWPU45“G.Cheng,J.Han,and X.Lu,” IEEE,vol.105,no.10,pp.1865–1883,2017”;PatternNet“W.Zhou,S.Newsam,C.Li,and Z.Shao, "Patternnet: A benchmark dataset for performance evaluation of remote sensing image retrieval," ISPRS journal of photogrammetry and remote sensing, vol. 145, pp. 197–209, 2018" and RSSCN7 "Q. Zou, L. Ni, T. Zhang, and Q. Wang, "Deep learning based feature selection for remote sensing scene classification," IEEE Geoscience and remote sensing letters, vol. 12, no. 11, pp. 2321–2325, 2015". The details of these datasets are as follows:.

[0111] 1) AID: The AID dataset is a large-scale aerial image dataset for scene classification, collected from Google Earth images, containing 10,000 images of 30 different aerial scene categories. The number of images in each category ranges from 220 to 400, and the size of each image is 600×600 pixels. The spatial resolution of a pixel in the image ranges from 0.5 meters to 8 meters.

[0112] 2) CLRS: The CLRS dataset is another large-scale remote sensing scene dataset, containing 15,000 images divided into 25 categories. This is a multi-source dataset, and the image sources include Google Earth, Bing Maps, Google Maps, and Tiandi Maps. Each category has 600 images with a size of 256×256 pixels, and the pixel resolution ranges from 0.26 meters to 8.85 meters.

[0113] 3) MlRSN: The MLRSN dataset is a large-scale multi-label remote sensing image dataset consisting of 109,161 images from 46 different scene types. Each scene type contains about 1,500 to 3,000 images with an image size of 256×256 pixels. The resolution of the images ranges from 10 meters to 0.1 meters.

[0114] 4) NWPU45: The NWPU45 dataset is another challenging large-scale remote sensing scene classification dataset, containing 31,500 images of size 256 × 256 pixels. It is divided into 45 categories, each with 700 images. The spatial resolution ranges from 30 meters to 0.2 meters.

[0115] 5) PatternNet: The PatternNet dataset is a large-scale high-resolution remote sensing image dataset collected from Google Earth images or Google Maps. It contains 30,400 images with a fixed size of 256×256 pixels, divided into 38 categories, and each category has 800 images. The spatial resolution ranges from 0.062 meters to 4.693 meters.

[0116] 6)RSSCN7: The RSSCN7 dataset contains 2800 images with a size of 400×400 pixels, which are annotated into 7 remote sensing scene categories, with 400 images in each category. It comes from Google Earth and has multi-scale characteristics, with four different scales: 1:700, 1:1300, 1:2600, and 1:5200. In each category, there are one hundred images at each different scale.

[0117] For multi-source cross-domain generalization, two datasets named ACMR-7 and AMNP-14 were constructed respectively, using the above six public scene classification remote sensing datasets. For the former, the same construction method as in the literature "S. Zhu, C. Wu, B. Du, and L. Zhang, "Style and content separation network for remote sensing image cross-scene generalization," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 201, pp. 1–11, 2023" was adopted. Seven categories of images with common or similar semantic labels, including farmland, forest, industrial area, grassland, parking lot, residential area, and river, were extracted from four different datasets (AID, CLRS, MLRSN, and RSSCN7) for the cross-domain generalization task. According to the specific circumstances of different datasets, "residential area" or "dense residential area" was selected as the "residential area" category. Similarly, 20% samples of each category in MLRSN were randomly selected because its sample size is significantly larger than that of other datasets. In addition, AMNP-14 was constructed, which consists of 14 shared categories from four datasets (AID, MLRSN, NWPU45, and PatternNet): airport, anchorage, baseball field, beach, bridge, dense residential area, farmland, overpass, forest, game space, parking lot, river, sparse residential area, and oil storage tank. Compared with ACMR-7, it is a cross-scene generalization dataset of remote sensing scenes with a larger sample size and more categories. It should be noted that the same scene may have different labels in the four heterogeneous datasets.

[0118] Table 2 shows the details of the ACMR-7 dataset.

[0119]

[0120] Table 3 shows the details of the AMNP-14 dataset.

[0121]

[0122] As shown in Table 2 and Table 3. At the same time, Figure 4 and Figure 5 Some samples of ACMR-7 and AMNP-14 are shown respectively. Since the initials of the six datasets used are not repeated, the AID, CLRS, MLRSN, NWPU45, PatternNet, and RSSCN7 datasets can be abbreviated as A, C, M, N, P, and R, respectively. In the multi-source cross-scenario generalization task, three of the four datasets are selected as source domains, and the remaining one is the target domain. ACMR-7 and AMNP-14 generate four different tasks, corresponding to different target domains: A, C, M, and R for ACMR-7, and A, M, N, and P for AMNP-14. For example, task C in ACMR-7 refers to selecting A, M, and R as source domains and C as target domain. Similarly, task P in AMNP-14 means that A, M, and N are source domains, and P is the target domain.

[0123] For single-source cross-domain generalization, it is performed on the above-mentioned ACMR-7, which contains a total of 12 tasks, with two different datasets randomly selected as source and target domains. For convenience, the task with A as the source domain and C as the target domain is denoted as AC. Therefore, these 12 tasks are AC, AM, AR, CA, CM, CR, MA, MC, MR, RA, RC, and RM.

[0124] B) Experimental setup and evaluation metrics

[0125] Several state-of-the-art methods are selected for comparison, including empirical risk minimization (ERM), domain adversarial (DA) methods such as domain adversarial neural network (DANN) "Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky, "Domain-adversarial training of neural networks," Journal of machine learning research, vol. 17, no. 59, pp. 1–35, 2016" and deep CORAL "B. Sun and K. Saenko, "Deep coral: Correlation alignment for deep domain adaptation," in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III14.Springer,2016,pp.443–450”, and data augmentation (DG) methods, involving Mixup "H.Zhang,M.Cisse,YN Dauphin,and D.Lopez-Paz,“mixup:Beyond empirical risk minimization,”inInternational Conference on Learning Representations,2018”, Representation Self-Challenge (RSC) "Z.Huang,H.Wang,EP Xing,and D.Huang,“Self-challenging improves cross-domain generalization,”in Computer vision–ECCV 2020:16th European conference,Glasgow,UK,August 23–28,2020,proceedings,part II 16. Springer, 2020, pp. 124–140, “Variance Risk Extrapolation (VREx)” D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A.Courville, "Out-of-distribution generalization via risk extrapolation(rex)," in International Conference on Machine Learning.PMLR, 2021, pp.5815–5826", ANDmask "G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Scho¨lkopf, "Learning explanations that are hard to vary," in International Conference on Learning Representations(ICLR 2021).OpenReview,2021", DIFEX"W.Lu, J.Wang,H.Li,Y.Chen,and L.Zhang, “Style and content separation network for remote sensing image cross-scene generalization,"ISPRS Journal ofPhotogrammetry and Remote Sensing,vol.201,pp.1–11,2023". .

[0126] Table 4 compares different methods of multi-source cross-domain generalization on the ACMR-7 dataset

[0127]

[0128] Table 5 compares different methods of multi-source cross-domain generalization on the AMNP-14 dataset

[0129]

[0130] ResNet-18 was used as the backbone network for all methods and was fine-tuned using a model pre-trained on ImageNet. For fair comparison, all experiments were performed under the same training settings, while the specific parameters of some methods were set to the optimal values ​​in their proposed methods, and the training optimizer was a mini-batch SGD. Each model was trained for 60 cycles with a batch size of 32. The momentum and weight decay parameters were 0.9 and 0.0005, respectively. The initial learning rate was set to 0.001 and then reduced by 0.1 each time when the training progress reached 70% and 90%. All experiments were performed on a workstation equipped with an NVIDIA GeForce RTX 309024GB GPU.

[0131] What all methods have in common is that data from the target domain is not available during training. 80% of the data from each source domain is extracted to form the total training set, and the remaining 20% ​​is merged to obtain the total validation set. The target domain is used as a test set to measure the generalization ability of the method. The model with the best performance on the validation set is selected as the final model. Unlike DA, which can use data from the target domain during training, it is obvious that the selected model is not necessarily the one with the best performance on the test set. ERM of the original model without any generalization techniques is regarded as the baseline method. For multi-source cross-scene generalization, the baseline ERM aggregates all source domains to train the model, while the DA method implements adaptation between different source domains. For single-source cross-scene generalization, all methods are trained and validated on a single source domain.

[0132] The classification accuracy on the target domain is used to quantify the generalization performance of the method. In addition, the average accuracy of all tasks is calculated to more intuitively compare the performance of different methods.

[0133] C) Multi-source cross-domain generalization

[0134] Tables 4 and 5 provide the results of multi-source cross-domain generalization performance on the ACMR-7 and AMNP-14 datasets, respectively. Each joint dataset provides four different generalization tasks, and their accuracies are reported separately. Then, the average accuracy of each method is calculated for intuitive comparison. Based on the observations, the following findings can be drawn:

[0135] 1) Cross-domain generalization is definitely more challenging than the traditional single dataset training, validation, and testing classification tasks. In addition, cross-domain generalization is more difficult than cross-domain adaptation due to the lack of access to target domain data. Frustratingly, the performance of many methods is not necessarily better than the baseline. Fortunately, the proposed SFDA achieves the best average accuracy on both joint datasets and also performs well on most generalization tasks. Compared with the baseline ERM, the improvement in average accuracy is 2.07% on ACMR-7 and 0.93% on AMNP-14, respectively; it can be reasonably inferred that by performing domain enhancement operations on the feature space, SFDA can effectively improve the generalization ability of the model.

[0136] 2) The proposed SFDA performs saliency-guided domain enhancement in feature space and outperforms Mixup in all tasks. Mixup achieves domain enhancement by directly mixing each pixel evenly in the original image. The method of treating each pixel in the image equally is not suitable for remote sensing images. Obviously, SFDA emphasizes important salient regions when generating new features, which is more in line with the characteristics of remote sensing images and therefore performs better.

[0137] 3) The performance gap on different datasets reflects the difficulty of generalization on that dataset, i.e., the lower the accuracy value, the higher the generalization difficulty. Although AMNP-14 has a larger sample size and more diverse categories, ACMR-7 is more challenging in cross-domain generalization. In the same dataset, the performance difference between different tasks reveals the difficulty of generalizing to the corresponding domain. It can be easily observed that domain R is the most challenging, followed by domain M, while domain C and A are relatively less challenging in ACMR-7. Similarly, in AMNP-14, the task of generalizing to domains M and N is more difficult than that of domains A and P.

[0138] To better demonstrate the improvement effect of the proposed SFDA, the accuracy of each category of each generalization task in ACMR-7 is calculated and compared with the baseline ERM and the proposed SFDA. The accuracy gap of each category of the four tasks is also obtained by subtracting the former from the latter, and is Figure 6 It can be intuitively noticed that SFDA is able to improve the classification accuracy of most categories in different generalization tasks. Due to the unique characteristics of each domain, the improvement of SFDA in each task is expressed in different ways. Specifically, the increase or decrease of certain categories is not always fixed, but varies according to the target domain. For example, in task R, all categories except Parking and Residential categories have significant improvements, while in task C, the performance of all categories except Industrial and River categories has improved.

[0139] In addition, it can be seen from Table 4 that the overall accuracy of tasks A, C, M, and R is improved by 0.80%, 3.50%, 1.09%, and 2.89%, respectively. These values ​​are also consistent with Figure 6 Visually, SFDA achieves superior results on tasks C and R, significantly improving the accuracy of most categories with only a few categories declining, while tasks A and M achieve relatively small improvements.

[0140] D) Single-source cross-domain generalization

[0141] To further verify the universality and effectiveness of the proposed SFDA, comparative experiments were also conducted under a more difficult single-source domain generalization setting, involving 12 tasks on ACMR-7 and 5 comparison methods, including ERM, DANN, ANDMask, Mixup, and SCSN. As shown in Table 5, the overall accuracy of each task and the final average of each method are recorded. Combined with Table 4, it can be observed that:

[0142] 1) Consistent with the case of multi-source cross-domain generalization, the proposed SFDA achieves the best performance on most tasks and surpasses all other methods with the highest average accuracy, achieving the best performance for single-source cross-domain generalization. It improves the overall accuracy by 1.58% compared to the baseline ERM, and its results on all tasks are higher than ERM. This reaffirms that the proposed SFDA is a feasible and effective domain enhancement method that can generalize well to unseen target domains by increasing the amount and diversity of source domain data. It is simple and easy to implement, and is not affected by the number of source domains.

[0143] 2) In general, increasing the source domain can greatly improve the results of cross-domain generalization. For example, comparing task RM in Table 6 with task M in Table 4, all methods have significant improvements (e.g., 18.69% for ERM and 17.17% for SFDA). This huge improvement is mainly due to the significant increase in training data, which increases the involvement of domains A and C during training. However, increasing the source domain is not guaranteed to have a positive impact on the results. Comparing these two sets of tasks: AC and C and CA and A. It can be found that the operation of increasing the source domain has a very small improvement on the performance and may even degrade them. It is reasonable to assume that domains A and C are very similar, and adding training data from other domains that are very different from them may aggravate the degree of domain shift and therefore will not improve the generalization effect.

[0144] Table 6 compares different methods of single-source cross-domain generalization on the ACMR-7 dataset

[0145]

[0146] E) Ablation studies

[0147] A series of experiments are conducted on ACMR-7 to reveal the impact of each part of SFDA, and the results are provided in Table 7. Two possible approaches are considered, namely, performing the Saliency Guided Optimization Module (SOM) or the Feature Domain Enhancement Module (FDAM) alone. In addition, the results of the highly correlated ERM as the baseline ResNet-18 and Mixup are also listed. Obviously, Mixup directly mixes the original images to generate new samples, which is even worse than the baseline ERM in the cross-domain generalization task. Adding FDAM to the backbone network performs better than Mixup, which indicates that enhancing the source domain in the feature space is more effective than in the original image space. Inserting SOM into the backbone network alone can help it learn better feature representation guided by saliency, thereby improving the cross-domain generalization results. However, either SOM or FDAM alone can achieve slight improvements, while the proposed SFDA, which combines the two modules together, has a significant improvement in performance. The results demonstrate the superiority of SFDA, which optimizes features through saliency guidance and then enhances the source domain in the feature space.

[0148] Table 7 shows the comparison between SFDA and other methods on the ACMR-7 dataset

[0149]

[0150] F) Feature Visualization

[0151] To further demonstrate the effectiveness of SFDA, the features learned on the multi-source domain task C and the single-source domain task CA are extracted for visualization. As a powerful and popular visualization tool, t-distributed stochastic neighbor embedding (t-SNE) is used to extract and visualize features before the classifier; Figure 7 and Figure 8 The visualization results of tasks C and CA are shown respectively. In the first two sub-figures, the source domain and target domain are shown in blue and red, respectively, to compare the alignment ability of the methods. In the last two sub-figures, different categories of the target domain are shown in different colors to facilitate the analysis of the discrimination ability of the methods in the target domain.

[0152] It can be observed that SFDA achieves better alignment between the source and target domains than ERM in multi-source cross-domain generalization, which means that it has better generalization ability when generalizing to unseen target domains. At the same time, the distribution of target domain features extracted by SFDA makes it easier for samples to be accurately classified. Similarly, SFDA also performs consistently in single-source cross-domain generalization. It significantly helps align the target domain with the source domain, while also producing better feature distribution, which is reflected in more concentrated clusters and clearer distinguishing boundaries.

[0153] G) The impact of different training ratios

[0154] According to the common setting in domain generalization (DG), 80% of the source domain data is divided into a training set and the remaining 20% ​​is used as a validation set. Then the model with the best performance on the validation set is selected to be applied to the target domain. A series of comparative experiments of ERM (empirical risk minimization) and SFDA are conducted on the multi-source cross-domain generalization task C and the single-source cross-domain generalization task AC to explore the impact of this division ratio. The results are shown in Fig. 9 As shown in the figure, it can be found that for single-source cross-domain generalization, due to the small amount of data, increasing the training data can effectively improve the results of both methods. For multi-source cross-domain generalization, ERM is insensitive to the increase of training data, while SFDA has a huge improvement when the training ratio increases from 10% to 20%, and is relatively stable in the range of 20% to 90%. This shows that in cross-domain generalization, when the amount of data reaches a certain scale, simply increasing the training data does not always lead to performance improvements.

[0155] Table 8 Comparison of traditional classification tasks

[0156]

[0157] H) Performance on traditional classification tasks

[0158] As an effective domain enhancement technique, it is inferred that the proposed SFDA can also be well applied to traditional classification tasks, that is, the practice of dividing the training set and the test set on the same dataset. Therefore, SFDA is used to complete the traditional classification task on two widely used datasets UCM "Y.Yang and S.Newsam, "Bag-of-visual-words and spatial extensions for land-use classification," in Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, 2010, pp.270–279" and AID. For fair comparison, ResNet-50 is selected as the backbone network. According to convention, the training ratio of UCM is set to 50% and 80%, and the training ratio of AID is set to 20% and 50%. The overall accuracy performance on the two datasets is shown in Table 8. It is worth noting that the proposed SFDA can also achieve competitive results on traditional classification tasks, effectively improving the performance of the baseline network. Mixup is a popular and widely used data enhancement technique that generally improves the classification accuracy of the network. The proposed SFDA even outperforms Mixup on traditional classification tasks, which further verifies the universality and effectiveness of SFDA.

[0159] V) Conclusion

[0160] In this paper, a simple and effective domain enhancement technique named Saliency Guided Feature Domain Enhancement (SFDA) is proposed for cross-domain generalization. By mixing saliency-guided features to generate new features and corresponding labels, the number and diversity of source domain data in the feature space are significantly increased, thereby improving the generalization ability of the model and performing well on unseen target domains. Specifically, a Saliency Guided Optimization Module (SOM) is first used to refine the deep features extracted by the baseline network to emphasize important areas, and then a Feature Domain Enhancement Module (FDAM) is implemented to mix the optimized features and their labels to generate new samples for the classifier. Extensive experiments verify the effectiveness, stability and universality of the proposed SFDA.

Claims

1. A cross-domain remote sensing image classification method based on feature domain enhancement, characterized by: The method comprises the following steps: 1) Extract original features and remote sensing image features; 2) Perform significance-guided optimization on any two features of the extracted original application features; 3) The two optimized features are enhanced in feature domain to create new features and corresponding labels; 4) Input the new features and corresponding labels created in step 3) into the classifier for classification.

2. The cross-domain remote sensing image classification method based on feature domain enhancement according to claim 1 is characterized in that: The method of significance guided optimization in step 2) is: Given any two extracted features x i ∈R H×W×C and x j ∈R H×W×C , and their corresponding labels y i and j , as the input of saliency-guided optimization, where H, W, and C refer to the height, width, and number of channels, respectively. In saliency-guided optimization, each feature is optimized for saliency individually without modifying their labels Y i and Y j ; For the input feature x i ∈R H×W×C , calculate the average value through the spatial dimension to obtain a vector V∈R 1×1×C , that is, V=[v1, v2,...,v C ], and then obtain the mean value v by avg and variance v var : Then, for each element v in vector V, n With v avg The square of the difference is used to obtain a new vector V′∈R 1×1×C , where V′=[v′1,v′2,...,v′ C ], and v′ n =(v n -v avg ) 2 , v′ max and v′ min Represent the maximum and minimum values ​​in V′ respectively, and then let the vector P = [p1, p2, ..., p C ], P is a vector, p n refers to an element of P, computed as follows: Where n = [1, 2, ..., C], then the sigmoid function is executed to activate the vector P and obtain the weight P′∈R of each channel 1×1×C ; At the same time, the input x is calculated i ∈R H×W×C The average value along the channel dimension produces a matrix U, where u nm is U∈R H×W×1 Similarly, the average value u is calculated by avg and variance u var : Construct another matrix U′∈R H×W×1 , u′ nm is an element of U′, and u′ is obtained by nm ; in' nm =(in nm -in avg ) 2 Similarly, let u′ max and u′ min are the maximum and minimum values ​​in U′ respectively; Then, we obtain a new matrix Q∈R H×W×1 , where q nm is Q∈R H×W×1 An element of is obtained by: Where n = [1, 2, ..., H] and m = [1, 2, ..., W], the matrix Q ∈ R H×W×1 After the sigmoid activation function, the spatial weight Q′∈R is obtained H×W×1 ; Finally, the spatial weight Q′∈R H×W×1 and channel weight P′∈R 1×1×C Multiply them together to get the final weight of the same shape as the original input feature; then convert the original feature x into i ∈R H×W×C Multiply these final weights to obtain the final refined features x′ that emphasize saliency i , the calculation formula is expressed as: x′ i =(P′×Q′)·x i (7) Get two saliency-guided optimized features x′ i ∈R H×W×C and x′ j ∈R H×W×C , as the input of the subsequent feature domain enhancement module.

3. The cross-domain remote sensing image classification method based on feature domain enhancement according to claim 2 is characterized in that: The feature domain enhancement method in step 3) is: obtain new features by mixing the optimized features, and achieve enhancement on the feature domain, that is, generate a new feature and its label by two input optimized features. The formula is as follows: where λ∈[0, 1] is a random mixing weight sampled from a Beta distribution.

4. The cross-domain remote sensing image classification method based on feature domain enhancement according to claim 3 is characterized in that: The original features in step 1) are extracted using the ResNet-18 model.

5. A cross-domain remote sensing image classification system based on feature domain enhancement, characterized by: include, Feature extraction module, used to extract remote sensing image features; A saliency guided optimization module, used for performing saliency guided optimization on any two features of the extracted remote sensing image features; The feature domain enhancement module is used to enhance the feature domain of the two features of the saliency-guided optimization and create new features and corresponding labels; The classification module is used to classify cross-domain remote sensing images based on new features and corresponding labels.

Images