A remote sensing image scene classification method and system based on graph structure unsupervised domain adaptation

Abstract

The application provides a remote sensing image scene classification method and system based on graph structure unsupervised domain adaptation, and the method comprises the following steps: a network model comprising a graph structure feature extraction network GFE and an adversarial feature alignment framework AFA is constructed, wherein in the model training stage, a source domain image, a target domain weakly enhanced image and a target domain strongly enhanced image are jointly input into the network; multi-scale remote sensing scene features with enhanced graph structure are extracted from the source domain image, the target domain weakly enhanced image and the target domain strongly enhanced image through the GFE; the multi-scale remote sensing scene features are driven domain alignment and semantic learning through the AFA to generate corresponding mixed samples; the GFE and the AFA are optimized in an end-to-end manner based on the mixed samples to obtain a remote sensing scene classification model suitable for the target domain, and the GFE and the classifier are reserved in the inference stage to perform scene classification on the target domain remote sensing image. The application can greatly improve the remote sensing image scene classification efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a remote sensing image scene classification method and system based on graph structure unsupervised domain adaptation. Background Technology

[0002] Scene classification in remote sensing images is a fundamental task in intelligent interpretation of remote sensing images. Its goal is to automatically categorize remote sensing images into semantic scene types such as airports, forests, and ports. Existing deep learning methods typically rely on large-scale labeled training data, achieving good classification results within the same dataset. However, remote sensing images come from diverse sources, and different datasets exhibit significant differences in imaging sensors, spatial resolution, shooting areas, seasons, and lighting conditions. This leads to a noticeable performance drop when the model, trained in the source domain, is directly applied to a new target domain. Therefore, achieving stable and reliable remote sensing scene classification under conditions of distributional differences between the source and target domains is a primary problem that urgently needs to be solved.

[0003] Unsupervised domain adaptation technology offers an important path to solving the above problems. It utilizes labeled source domains and unlabeled target domains for cross-domain transfer. Existing methods mostly focus on two aspects: reducing domain differences and enhancing semantic learning in the target domain. They use deep neural networks as feature extractors, focusing on global feature distribution alignment and joint optimization. However, such general methods lack modeling of the characteristics of remote sensing images themselves. Remote sensing scene images contain multi-scale targets, complex spatial structures, and rich contextual information. There are large differences within the same type of sample, and different categories may share similar objects and background patterns. Simply reducing the overall distribution distance between domains can easily lead to blurred category boundaries or even misalignment. How to balance the multi-scale structural relationships and category semantic discriminativeness of remote sensing images has become a key challenge in the domain adaptation process.

[0004] While existing remote sensing scene classification models based on convolutional neural networks can extract hierarchical visual features, the convolutional operation is limited to local neighborhoods and cannot explicitly model cross-domain relationships between source and target domain samples, nor can it fully describe the dependencies between features at different network levels. Furthermore, for unsupervised domain adaptation tasks, the similarity between source and target domain samples and the complementarity of features at different levels both affect the quality of transferred features. Therefore, how to explicitly construct sample-level and hierarchical relationships during the feature extraction stage, enabling the model to obtain a structure-aware remote sensing scene representation, is another technical problem that needs to be solved to further improve cross-domain classification performance. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a remote sensing image scene classification method and system based on graph structure unsupervised domain adaptation, so as to solve the problem that the convolution operation of the prior art is limited to the local domain, cannot explicitly model the cross-domain relationship between source domain and target domain samples, and cannot fully describe the dependency relationship between features of different network levels.

[0006] The first aspect of the present invention proposes: A remote sensing image scene classification method based on graph-structured unsupervised domain adaptation is characterized by its applicability to closed-set unsupervised domain adaptation scenes where the source and target domain class spaces are consistent, wherein the source domain samples contain class labels, and the target domain samples do not contain class labels. The method includes: A network model is constructed that includes a graph structure feature extraction network (GFE) and an adversarial feature alignment framework (AFA). During the model training phase, the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image are input into the network. The GFE extracts multi-scale remote sensing scene features enhanced by graph structure from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image. The GFE uses ResNet50 as the backbone network and introduces graph structure relationships on the multi-layer features of ResNet50 for modeling. The AFA is used to perform driving domain alignment and semantic learning on the multi-scale remote sensing scene features to generate corresponding hybrid samples; Based on the mixed samples, the GFE and the AFA are jointly optimized end-to-end to obtain a remote sensing scene classification model suitable for the target domain. The GFE and classifier are retained during the inference phase to classify the remote sensing images of the target domain.

[0007] The beneficial effects of this invention are as follows: This technical solution effectively solves the problems of insufficient explicit modeling of cross-domain relationships and inadequate description of hierarchical feature dependencies caused by the locality limitations of existing convolutional operations. By using GFE with ResNet50 as the backbone and introducing graph structure modeling in multi-layer features, it is possible to extract multi-scale remote sensing scene features enhanced by graph structure, fully capturing the dependencies between features at different network levels; AFA performs driving domain alignment and semantic learning on multi-scale features, which can explicitly mine cross-domain associations between source and target domain samples. The generated mixed samples help GFE and AFA to jointly optimize end-to-end, improving the model's generalization ability. The GFE and classifier are retained in the inference stage, which can efficiently and accurately classify target domain remote sensing images, significantly improving the accuracy and stability of cross-domain remote sensing scene classification and adapting to the classification needs of complex remote sensing scenes.

[0008] Furthermore, the step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE includes: The source domain samples and target domain samples are concatenated into a mixed batch, and feature maps are output from five levels (Conv1, Conv2_x, Conv3_x, Conv4_x and Conv5_x) through ResNet50. An intra-layer graph is constructed within each level of the feature map, and global average pooling is performed on each intra-layer graph to obtain a set of sample nodes. Within each feature level, all sample nodes in the batch other than themselves are treated as neighbor nodes to construct a fully connected layer inner graph. Based on the fully connected layer inner graph, multi-head aggregation is used to obtain graph augmentation nodes. The corresponding graph augmentation information is detected based on the graph augmentation nodes, and the multi-scale remote sensing scene features are generated based on the graph augmentation information.

[0009] Furthermore, the step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE further includes: The residual update method is used to re-inject the graph enhancement information into the convolutional feature map to enhance the original features. The expression for the residual update algorithm is as follows:

[0010] in, Represents the set of sample nodes. Let g( represent the graph within the i-th layer) ) represents a linear transformation function used to map node features to the corresponding convolutional feature channel dimension.

[0011] Furthermore, the step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE further includes: A corresponding cross-scale graph is constructed based on the enhanced original features to aggregate structural information between different levels. The node set of the cross-scale graph is composed of enhanced nodes from five levels after being modeled by the intra-scale graph. Attention aggregation processing is performed on the cross-scale map to fuse structural information between different levels and samples. The aggregated cross-scale features are then linearly projected and residually fused with deep features to obtain the final graph augmentation features. Each graph augmentation feature is then integrated to generate the corresponding multi-scale remote sensing scene features.

[0012] Furthermore, the step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples includes: Domain adversarial learning is introduced for global alignment, and the multi-scale remote sensing scene features are input into the domain discriminator. The feature extractor learns the domain and performs indistinguishable representation through a gradient inversion layer to identify several domain categories. Fine-grained alignment is performed on several domain categories using category-level constraints. For each domain category, the mean difference between the category features of the source domain and the category pseudo-label features of the target domain in the regenerated kernel Hilbert space is calculated to generate source domain samples and target domain samples accordingly. The source domain sample and the target domain sample are mixed to generate the mixed sample.

[0013] Furthermore, the step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples also includes: For the source domain samples, supervised training is performed using standard cross-entropy loss: For the target domain samples, weak and strong enhancement consistency constraints are adopted, and weak and strong enhancements are performed on the same target domain image respectively. The prediction probabilities of the two enhanced views are required to be consistent in order to complete the alignment and semantic learning processing.

[0014] Furthermore, the step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples also includes: The cross-domain MixUp algorithm is used to mix the source domain samples and the target domain samples to generate the mixed samples.

[0015] The second aspect of the present invention proposes: A remote sensing image scene classification system based on graph-structured unsupervised domain adaptation is characterized by its applicability to closed-set unsupervised domain adaptation scenarios where the source and target domain class spaces are consistent, the source domain samples contain class labels, and the target domain samples do not contain class labels; the system includes: The building module is used to construct a network model that includes a graph structure feature extraction network (GFE) and an adversarial feature alignment framework (AFA). During the model training phase, the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image are input into the network. The alignment module is used to extract multi-scale remote sensing scene features enhanced by graph structure from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE. The GFE is based on ResNet50 as the backbone network and introduces graph structure relationships for modeling on the multi-layer features of ResNet50. The optimization module is used to perform driving domain alignment and semantic learning on the multi-scale remote sensing scene features through the AFA to generate corresponding hybrid samples; The classification module is used to perform end-to-end joint optimization of the GFE and the AFA based on the mixed samples to obtain a remote sensing scene classification model suitable for the target domain, and retain the GFE and classifier during the inference phase to perform scene classification on remote sensing images of the target domain.

[0016] Furthermore, the alignment module is specifically used for: The source domain samples and target domain samples are concatenated into a mixed batch, and feature maps are output from five levels (Conv1, Conv2_x, Conv3_x, Conv4_x and Conv5_x) through ResNet50. An intra-layer graph is constructed within each level of the feature map, and global average pooling is performed on each intra-layer graph to obtain a set of sample nodes. Within each feature level, all sample nodes in the batch other than themselves are treated as neighbor nodes to construct a fully connected layer inner graph. Based on the fully connected layer inner graph, multi-head aggregation is used to obtain graph augmentation nodes. The corresponding graph augmentation information is detected based on the graph augmentation nodes, and the multi-scale remote sensing scene features are generated based on the graph augmentation information.

[0017] Furthermore, the alignment module is specifically used for: The residual update method is used to re-inject the graph enhancement information into the convolutional feature map to enhance the original features. The expression for the residual update algorithm is as follows:

[0018] in, Represents the set of sample nodes. Let g( represent the graph within the i-th layer) ) represents a linear transformation function used to map node features to the corresponding convolutional feature channel dimension.

[0019] Furthermore, the alignment module is specifically used for: A corresponding cross-scale graph is constructed based on the enhanced original features to aggregate structural information between different levels. The node set of the cross-scale graph is composed of enhanced nodes from five levels after being modeled by the intra-scale graph. Attention aggregation processing is performed on the cross-scale map to fuse structural information between different levels and samples. The aggregated cross-scale features are then linearly projected and residually fused with deep features to obtain the final graph augmentation features. Each graph augmentation feature is then integrated to generate the corresponding multi-scale remote sensing scene features.

[0020] Furthermore, the optimization module is specifically used for: Domain adversarial learning is introduced for global alignment, and the multi-scale remote sensing scene features are input into the domain discriminator. The feature extractor learns the domain and performs indistinguishable representation through a gradient inversion layer to identify several domain categories. Fine-grained alignment is performed on several domain categories using category-level constraints. For each domain category, the mean difference between the category features of the source domain and the category pseudo-label features of the target domain in the regenerated kernel Hilbert space is calculated to generate source domain samples and target domain samples accordingly. The source domain sample and the target domain sample are mixed to generate the mixed sample.

[0021] Furthermore, the optimization module is specifically used for: For the source domain samples, supervised training is performed using standard cross-entropy loss: For the target domain samples, weak and strong enhancement consistency constraints are adopted, and weak and strong enhancements are performed on the same target domain image respectively. The prediction probabilities of the two enhanced views are required to be consistent in order to complete the alignment and semantic learning processing.

[0022] Furthermore, the optimization module is specifically used for: The cross-domain MixUp algorithm is used to mix the source domain samples and the target domain samples to generate the mixed samples.

[0023] The third aspect of the present invention proposes: A computer includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the remote sensing image scene classification method based on graph structure unsupervised domain adaptation as described above.

[0024] The fourth aspect of the present invention proposes: A readable storage medium having a computer program stored thereon, characterized in that, when executed by a processor, the program implements the remote sensing image scene classification method based on graph structure unsupervised domain adaptation as described above.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall framework of the remote sensing image scene classification method based on graph structure unsupervised domain adaptation provided in the second embodiment of the present invention; Figure 2This is a schematic diagram of the remote sensing scene classification model in the graph-structure-based unsupervised domain adaptation remote sensing image scene classification method provided in the second embodiment of the present invention.

[0027] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0028] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0029] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0031] Please see Figure 1 The diagram illustrates a graph-structured unsupervised domain adaptation-based remote sensing image scene classification method provided in the first embodiment of this invention. This method generates mixed samples that facilitate end-to-end joint optimization of GFE and AFA, enhancing the model's generalization ability. The GFE and classifier are retained during the inference phase, enabling efficient and accurate scene classification of target domain remote sensing images. This significantly improves the accuracy and stability of cross-domain remote sensing scene classification, adapting to the classification needs of complex remote sensing scenes.

[0032] Specifically, the first embodiment of the present invention provides: A remote sensing image scene classification method based on graph-structured unsupervised domain adaptation is characterized by its applicability to closed-set unsupervised domain adaptation scenes where the source and target domain class spaces are consistent, wherein the source domain samples contain class labels, and the target domain samples do not contain class labels. The method includes: Step S10: Construct a network model that includes a graph structure feature extraction network (GFE) and an adversarial feature alignment framework (AFA). During the model training phase, the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image are input into the network. It's important to note that the two core modules of the network model, GFE and AFA, each have their own distinct functions and complement each other. GFE is responsible for extracting discriminative remote sensing scene features, while AFA is responsible for aligning features between the source and target domains. Together, they constitute the core architecture of unsupervised domain adaptive classification. The absence of either module prevents the achievement of the goal. Specifically, GFE alone cannot address the difference in feature distribution between the source and target domains, and AFA alone lacks effective feature input, making semantic classification impossible. The design of three types of input images during the training phase is to fully leverage the information in the target domain data: the source domain image provides labeled supervision signals to guide the model in learning category semantics; the weakly enhanced target domain image retains the original scene features of the target domain and can be used to learn the basic distribution of the target domain; the strongly enhanced target domain image increases data diversity through data augmentation (such as rotation, flipping, and brightness adjustment), which not only improves the model's generalization ability but also helps the model capture the essential features of the target domain scene and reduces noise interference. The combination of these three provides comprehensive and high-quality input data for subsequent feature extraction and domain alignment, and none of them can be omitted.

[0033] Step S20: The GFE extracts multi-scale remote sensing scene features enhanced by graph structure from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image. The GFE uses ResNet50 as the backbone network and introduces graph structure relationships on the multi-layer features of ResNet50 for modeling. It is worth noting that choosing ResNet50 as the basic backbone network is absolutely necessary. As a mature deep convolutional neural network, ResNet50 can effectively extract deep semantic features from images, adapting to the complex and detailed characteristics of remote sensing imagery scenes, and avoiding the problem of insufficient feature extraction by shallow networks. Introducing graph structure relationship modeling on the multi-layer features of ResNet50 is one of the core innovations of this method and a key to feature enhancement. In remote sensing imagery scenes, there are complex spatial relationships between different targets (such as buildings, roads, vegetation, and water bodies). Traditional convolutional networks can only capture local features and cannot effectively model these global relationships. The introduction of graph structure relationships allows pixels and regions in multi-layer features to be treated as nodes, and their spatial and semantic relationships can be modeled through edges between nodes, thereby enhancing features and making the extracted features more discriminative. Meanwhile, multi-scale remote sensing scene features are extracted because the scale differences of different scene targets in remote sensing images are large (such as large areas of farmland and small buildings). Multi-scale features can cover target information at different scales, ensuring the accuracy of subsequent domain alignment and classification. This design is in line with the scene characteristics of remote sensing images and is a necessary guarantee for achieving accurate classification.

[0034] Step S30: The multi-scale remote sensing scene features are aligned with the driving domain and semantically learned by the AFA to generate corresponding hybrid samples; It's important to note that the core challenge of unsupervised domain adaptation lies in the feature distribution shift between the source and target domains. This means that scenes of the same category exhibit different feature representations in the source and target domains, leading to a decline in the model's classification performance in the target domain. The core function of the AFA framework is to drive domain alignment through adversarial learning, making the multi-scale feature distributions of the source and target domains as consistent as possible, thus eliminating inter-domain differences. Simultaneously, AFA also undertakes semantic learning tasks. During domain alignment, it combines the annotation information from the source domain to guide the model in learning category semantics, avoiding semantic distortion during alignment and ensuring that the aligned features still possess clear category discriminative power. Generating hybrid samples is a crucial output form for domain alignment and semantic learning. Hybrid samples combine the advantages of features from both the source and target domains, retaining the category semantic information of the source domain while incorporating the distribution features of the target domain. This provides high-quality training samples for subsequent joint optimization of the model, enabling it to better learn common features across domains and laying the foundation for further optimization.

[0035] Step S40: Based on the mixed samples, the GFE and the AFA are jointly optimized end-to-end to obtain a remote sensing scene classification model suitable for the target domain. The GFE and classifier are retained during the inference phase to classify the remote sensing images of the target domain.

[0036] It is worth noting that the end-to-end joint optimization of GFE and AFA is essential: the feature extraction quality of GFE directly affects the domain alignment effect of AFA, while the domain alignment result of AFA can in turn guide the feature learning of GFE. The joint optimization of the two can achieve mutual promotion and synergistic improvement, avoiding the model performance bottleneck caused by optimizing a single module, and ensuring that the final model has both excellent feature extraction ability and domain adaptability, and can adapt to the feature distribution of the target domain. The reason for retaining GFE and the classifier while discarding AFA in the inference stage is that the core role of AFA is to achieve domain alignment and semantic learning during the training stage, providing support for model optimization. Domain alignment is not needed in the inference stage; only multi-scale features of the graph structure enhancement of the remote sensing image in the target domain need to be extracted by GFE, and then the classifier is used to determine the category of the features to complete the scene classification. This design ensures both inference efficiency and makes full use of the feature extraction and classification capabilities learned in the training stage, ensuring the accuracy of the classification results and realizing the practical application value of the method.

[0037] Second Embodiment Furthermore, the step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE includes: The source domain samples and target domain samples are concatenated into a mixed batch, and feature maps are output from five levels (Conv1, Conv2_x, Conv3_x, Conv4_x and Conv5_x) through ResNet50. An intra-layer graph is constructed within each level of the feature map, and global average pooling is performed on each intra-layer graph to obtain a set of sample nodes. Within each feature level, all sample nodes in the batch other than themselves are treated as neighbor nodes to construct a fully connected layer inner graph. Based on the fully connected layer inner graph, multi-head aggregation is used to obtain graph augmentation nodes. The corresponding graph augmentation information is detected based on the graph augmentation nodes, and the multi-scale remote sensing scene features are generated based on the graph augmentation information.

[0038] Furthermore, the step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE further includes: The residual update method is used to re-inject the graph enhancement information into the convolutional feature map to enhance the original features. The expression for the residual update algorithm is as follows:

[0039] in, Represents the set of sample nodes. Let g( represent the graph within the i-th layer) ) represents a linear transformation function used to map node features to the corresponding convolutional feature channel dimension.

[0040] Furthermore, the step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE further includes: A corresponding cross-scale graph is constructed based on the enhanced original features to aggregate structural information between different levels. The node set of the cross-scale graph is composed of enhanced nodes from five levels after being modeled by the intra-scale graph. Attention aggregation processing is performed on the cross-scale map to fuse structural information between different levels and samples. The aggregated cross-scale features are then linearly projected and residually fused with deep features to obtain the final graph augmentation features. Each graph augmentation feature is then integrated to generate the corresponding multi-scale remote sensing scene features.

[0041] Furthermore, the step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples includes: Domain adversarial learning is introduced for global alignment, and the multi-scale remote sensing scene features are input into the domain discriminator. The feature extractor learns the domain and performs indistinguishable representation through a gradient inversion layer to identify several domain categories. Fine-grained alignment is performed on several domain categories using category-level constraints. For each domain category, the mean difference between the category features of the source domain and the category pseudo-label features of the target domain in the regenerated kernel Hilbert space is calculated to generate source domain samples and target domain samples accordingly. The source domain sample and the target domain sample are mixed to generate the mixed sample.

[0042] Furthermore, the step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples also includes: For the source domain samples, supervised training is performed using standard cross-entropy loss: For the target domain samples, weak and strong enhancement consistency constraints are adopted, and weak and strong enhancements are performed on the same target domain image respectively. The prediction probabilities of the two enhanced views are required to be consistent in order to complete the alignment and semantic learning processing.

[0043] Furthermore, the step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples also includes: The cross-domain MixUp algorithm is used to mix the source domain samples and the target domain samples to generate the mixed samples.

[0044] Additionally, in this embodiment, it should be noted that, please refer to... Figures 1 to 2 The remote sensing image scene classification method based on graph structure unsupervised domain adaptation provided by this invention is specifically implemented by the following: This invention proposes a remote sensing image scene classification method based on graph-structured unsupervised domain adaptation, abbreviated as GUDA. This method is applicable to closed-set unsupervised domain adaptation scenarios where the source and target domains have consistent class spaces. In this scenario, source domain samples have class labels, while target domain samples do not. Let the source domain be... The target domain is ,in and These represent remote sensing image samples from the source and target domains, respectively. This represents the category label corresponding to the source domain sample. and These represent the number of samples in the source and target domains, respectively. The goal of this invention is to learn a network model capable of accurately classifying scenes in remote sensing images of the target domain using only source domain annotation information and unlabeled samples in the target domain.

[0045] like Figure 1 As shown, the overall framework of this invention includes GFE and AFA. During the training phase, the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image are all input into the network. GFE acts as a feature extractor, used to extract multi-scale remote sensing scene features after graph structure enhancement; AFA is used to drive domain alignment and semantic learning, including a domain discriminator, classifier, domain adversarial alignment, category-level alignment, and semantic learning loss. GFE and AFA are jointly optimized end-to-end to finally obtain a remote sensing scene classification model suitable for the target domain. Figure 2 The detailed structure of GFE is shown.

[0046] GFE uses ResNet50 as its backbone network, but ResNet50 is only used to extract hierarchical convolutional features. The core of this invention lies in introducing graph structure relationship modeling on its multi-layer features. For each training batch, this invention concatenates source domain samples and target domain samples into a mixed batch. ,in And usually set This hybrid input strategy enables the graph structure to simultaneously contain nodes from both the source and target domains, thereby establishing cross-domain sample relationships during the feature extraction stage.

[0047] ResNet50 outputs feature maps from five layers: Conv1, Conv2_x, Conv3_x, Conv4_x, and Conv5_x, denoted as . ,in, Indicates the first The sample at the th Feature maps at each level.

[0048] First, this invention constructs an in-layer graph within each layer. For the first... Layer feature map Perform global average pooling to obtain sample-level node representations:

[0049] in, This represents the global average pooling function, which transforms a two-dimensional feature map into a one-dimensional feature map to reduce computational overhead. At this point, we can obtain the... Layer node set Since this set of nodes contains samples from both the source and target domains, the in-layer graph can describe the relationships between samples in the source domain, between samples in the target domain, and between samples in the source and target domains. Furthermore, in the... Within each feature level, this invention treats all sample nodes in a batch, excluding itself, as neighbor nodes to construct a fully connected layer graph. Its adjacency relationship is defined as follows:

[0050] in, Indicates the first Layer nodes With nodes Does a connection exist? This adjacency matrix is ​​only used to determine the connection relationships between nodes; the specific information and aggregation weights are adaptively learned by the subsequent graph attention mechanism based on node features. Afterwards, nodes... Sourced from nodes Influence is defined as:

[0051] in, This represents the learnable projection matrix. This indicates a learnable scoring function. Indicates attention. Represents a node The set of neighbors. Then, multi-head aggregation is used to obtain the graph augmentation node representation as:

[0052] in, Indicates the number of heads of attention. This indicates a concatenation operation. This process allows each sample to obtain information from structurally related cross-domain samples at the same level, rather than performing indiscriminate fusion of all samples within a batch. Afterward, graph enhancement information is re-injected into the convolutional feature map to enhance the representation of the original features; this invention employs residual updates.

[0053] in, This represents a linear transformation function used to map node features to the corresponding convolutional feature channel dimension. This one-dimensional feature is then added to the feature map obtained from that layer. It's worth noting that the one-dimensional feature here is expanded to the same dimension as the feature map through duplication; that is, the residual between each pixel in the feature map and this feature is calculated. After completing the intra-scale map modeling, this invention further constructs a cross-scale map to aggregate structural information between different levels. The node set of the cross-scale map consists of enhanced nodes from five levels after intra-scale map modeling, i.e. Cross-scale graphs only establish connections between nodes at different feature levels, ignoring connections within the same feature level. Their adjacency relationships are defined as follows:

[0054] in Represents a node The corresponding layer number is then determined. Attention aggregation is then performed on the cross-scale map to fuse structural information from different layers and samples. The aggregated cross-scale features are then linearly projected and residually fused with the deep features to obtain the final map-enhanced features. This feature simultaneously incorporates visual semantics extracted by the convolutional network, intra-scale cross-domain sample relationships, and cross-scale hierarchical dependencies, which can be used for subsequent classification and domain alignment.

[0055] Adversarial Graph Analysis (AFA) is used to perform domain alignment and semantic learning based on graph augmentation features. Its goal is not simply to reduce the overall difference between the source and target domains, but to maintain class discriminability while reducing distributional bias. First, AFA introduces domain adversarial learning for global alignment. Features from both the source and target domains are then input into the domain discriminator. The gradient reversal layer enables the feature extractor to learn indistinguishable domain representations. Domain adversarial loss is defined as:

[0056] in, and Let represent the features extracted from the source and target domains, respectively. This loss promotes a convergence in the overall feature distributions of the source and target domains. After this, AFA uses class-level constraints for fine-grained alignment. For each class, the mean difference between the features of that class in the source domain and the pseudo-label features of that class in the target domain is calculated in the regenerating kernel Hilbert space:

[0057] in, Indicates the number of categories. Represents the Hilbert space mapping function, Let represent the square norm of the Hilbert space. Class-level alignment reduces false matches between different classes, ensuring that target domain samples retain clear class boundaries after alignment. The total loss for domain alignment is:

[0058] At the semantic learning level, different learning strategies are employed. For source domain samples, this invention uses standard cross-entropy loss for supervised training.

[0059] in, This represents the predicted probability of a sample from the source domain. This represents the true label of the source domain. For the target domain samples, this invention employs a consistency constraint between weak and strong enhancement. Weak and strong enhancements are applied to the same target domain image respectively, requiring that the predicted probabilities of the two enhanced views remain consistent:

[0060] in, and These represent the predicted probabilities of weakly and strongly enhanced samples in the target domain, respectively. This constraint can improve the prediction stability of the target domain without relying on manual labels.

[0061] Furthermore, this invention employs cross-domain MixUp technology to generate mixed samples between the source and target domains:

[0062]

[0063] in, The pseudo-labels are for the target domain and are obtained directly through network prediction. To reduce pseudo-label noise, only those with a prediction confidence greater than a threshold are selected. The target domain samples are used in the MixUp computation, and cross-entropy loss is employed for optimization. This strategy can construct smooth transition samples between the source and target domains, improving the stability of the decision boundary. The final total semantic learning loss is:

[0064] Ultimately, the objective function of the entire network is:

[0065] in, These are weighting coefficients used to balance the domain alignment loss and semantic learning loss. After training, the GFE and classifier are retained during the inference phase to obtain the target domain scene category prediction.

[0066] The third embodiment of the present invention provides: A remote sensing image scene classification system based on graph-structured unsupervised domain adaptation is characterized by its applicability to closed-set unsupervised domain adaptation scenarios where the source and target domain class spaces are consistent, the source domain samples contain class labels, and the target domain samples do not contain class labels; the system includes: The building module is used to construct a network model that includes a graph structure feature extraction network (GFE) and an adversarial feature alignment framework (AFA). During the model training phase, the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image are input into the network. The alignment module is used to extract multi-scale remote sensing scene features enhanced by graph structure from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE. The GFE is based on ResNet50 as the backbone network and introduces graph structure relationships for modeling on the multi-layer features of ResNet50. The optimization module is used to perform driving domain alignment and semantic learning on the multi-scale remote sensing scene features through the AFA to generate corresponding hybrid samples; The classification module is used to perform end-to-end joint optimization of the GFE and the AFA based on the mixed samples to obtain a remote sensing scene classification model suitable for the target domain, and retain the GFE and classifier during the inference phase to perform scene classification on remote sensing images of the target domain.

[0067] Furthermore, the alignment module is specifically used for: The source domain samples and target domain samples are concatenated into a mixed batch, and feature maps are output from five levels (Conv1, Conv2_x, Conv3_x, Conv4_x and Conv5_x) through ResNet50. An intra-layer graph is constructed within each level of the feature map, and global average pooling is performed on each intra-layer graph to obtain a set of sample nodes. Within each feature level, all sample nodes in the batch other than themselves are treated as neighbor nodes to construct a fully connected layer inner graph. Based on the fully connected layer inner graph, multi-head aggregation is used to obtain graph augmentation nodes. The corresponding graph augmentation information is detected based on the graph augmentation nodes, and the multi-scale remote sensing scene features are generated based on the graph augmentation information.

[0068] Furthermore, the alignment module is specifically used for: The residual update method is used to re-inject the graph enhancement information into the convolutional feature map to enhance the original features. The expression for the residual update algorithm is as follows:

[0069] in, Represents the set of sample nodes. Let g( represent the graph within the i-th layer) ) represents a linear transformation function used to map node features to the corresponding convolutional feature channel dimension.

[0070] Furthermore, the alignment module is specifically used for: A corresponding cross-scale graph is constructed based on the enhanced original features to aggregate structural information between different levels. The node set of the cross-scale graph is composed of enhanced nodes from five levels after being modeled by the intra-scale graph. Attention aggregation processing is performed on the cross-scale map to fuse structural information between different levels and samples. The aggregated cross-scale features are then linearly projected and residually fused with deep features to obtain the final graph augmentation features. Each graph augmentation feature is then integrated to generate the corresponding multi-scale remote sensing scene features.

[0071] Furthermore, the optimization module is specifically used for: Domain adversarial learning is introduced for global alignment, and the multi-scale remote sensing scene features are input into the domain discriminator. The feature extractor learns the domain and performs indistinguishable representation through a gradient inversion layer to identify several domain categories. Fine-grained alignment is performed on several domain categories using category-level constraints. For each domain category, the mean difference between the category features of the source domain and the category pseudo-label features of the target domain in the regenerated kernel Hilbert space is calculated to generate source domain samples and target domain samples accordingly. The source domain sample and the target domain sample are mixed to generate the mixed sample.

[0072] Furthermore, the optimization module is specifically used for: For the source domain samples, supervised training is performed using standard cross-entropy loss: For the target domain samples, weak and strong enhancement consistency constraints are adopted, and weak and strong enhancements are performed on the same target domain image respectively. The prediction probabilities of the two enhanced views are required to be consistent in order to complete the alignment and semantic learning processing.

[0073] Furthermore, the optimization module is specifically used for: The cross-domain MixUp algorithm is used to mix the source domain samples and the target domain samples to generate the mixed samples.

[0074] The fourth embodiment of the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the remote sensing image scene classification method based on graph structure unsupervised domain adaptation as described above.

[0075] The fifth embodiment of the present invention provides a readable storage medium storing a computer program, characterized in that the program, when executed by a processor, implements the remote sensing image scene classification method based on graph structure unsupervised domain adaptation as described above.

[0076] In summary, the graph-structured unsupervised domain adaptation-based remote sensing image scene classification method and system provided in the above embodiments of the present invention can generate mixed samples to facilitate end-to-end joint optimization of GFE and AFA, thereby improving the model's generalization ability. By retaining GFE and the classifier during the inference phase, scene classification of remote sensing images in the target domain can be performed efficiently and accurately, significantly improving the accuracy and stability of cross-domain remote sensing scene classification and adapting to the classification needs of complex remote sensing scenes.

[0077] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0078] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0079] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0080] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0081] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0082] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A remote sensing image scene classification method based on graph structure unsupervised domain adaptation, characterized in that, A method applicable to unsupervised domain adaptation scenarios involving closed sets where the source and target domain category spaces are identical, wherein the source domain samples contain category labels and the target domain samples do not, the method includes: A network model is constructed that includes a graph structure feature extraction network (GFE) and an adversarial feature alignment framework (AFA). During the model training phase, the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image are input into the network. The GFE extracts multi-scale remote sensing scene features enhanced by graph structure from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image. The GFE uses ResNet50 as the backbone network and introduces graph structure relationships on the multi-layer features of ResNet50 for modeling. The AFA is used to perform driving domain alignment and semantic learning on the multi-scale remote sensing scene features to generate corresponding hybrid samples; Based on the mixed samples, the GFE and the AFA are jointly optimized end-to-end to obtain a remote sensing scene classification model suitable for the target domain. The GFE and classifier are retained during the inference phase to classify the remote sensing images of the target domain.

2. The remote sensing image scene classification method based on graph structure unsupervised domain adaptation according to claim 1, characterized in that, The step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE includes: The source domain samples and target domain samples are concatenated into a mixed batch, and feature maps are output from five levels (Conv1, Conv2_x, Conv3_x, Conv4_x and Conv5_x) through ResNet50. An intra-layer graph is constructed within each level of the feature map, and global average pooling is performed on each intra-layer graph to obtain a set of sample nodes. Within each feature level, all sample nodes in the batch other than themselves are treated as neighbor nodes to construct a fully connected layer inner graph. Based on the fully connected layer inner graph, multi-head aggregation is used to obtain graph augmentation nodes. The corresponding graph augmentation information is detected based on the graph augmentation nodes, and the multi-scale remote sensing scene features are generated based on the graph augmentation information.

3. The remote sensing image scene classification method based on graph structure unsupervised domain adaptation according to claim 2, characterized in that, The step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE further includes: The residual update method is used to re-inject the graph enhancement information into the convolutional feature map to enhance the original features. The expression for the residual update algorithm is as follows: in, Represents the set of sample nodes. Let g( represent the graph within the i-th layer) ) represents a linear transformation function used to map node features to the corresponding convolutional feature channel dimension.

4. The remote sensing image scene classification method based on graph structure unsupervised domain adaptation according to claim 3, characterized in that, The step of extracting graph-structure-enhanced multi-scale remote sensing scene features from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE further includes: A corresponding cross-scale graph is constructed based on the enhanced original features to aggregate structural information between different levels. The node set of the cross-scale graph is composed of enhanced nodes from five levels after being modeled by the intra-scale graph. Attention aggregation processing is performed on the cross-scale map to fuse structural information between different levels and samples. The aggregated cross-scale features are then linearly projected and residually fused with deep features to obtain the final graph augmentation features. Each graph augmentation feature is then integrated to generate the corresponding multi-scale remote sensing scene features.

5. The remote sensing image scene classification method based on graph structure unsupervised domain adaptation according to claim 1, characterized in that, The step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples includes: Domain adversarial learning is introduced for global alignment, and the multi-scale remote sensing scene features are input into the domain discriminator. The feature extractor learns the domain and performs indistinguishable representation through a gradient inversion layer to identify several domain categories. Fine-grained alignment is performed on several domain categories using category-level constraints. For each domain category, the mean difference between the category features of the source domain and the category pseudo-label features of the target domain in the regenerated kernel Hilbert space is calculated to generate source domain samples and target domain samples accordingly. The source domain samples and the target domain samples are mixed to generate the corresponding mixed samples.

6. The remote sensing image scene classification method based on graph structure unsupervised domain adaptation according to claim 5, characterized in that, The step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples further includes: For the source domain samples, supervised training is performed using standard cross-entropy loss: For the target domain samples, weak and strong enhancement consistency constraints are adopted, and weak and strong enhancements are performed on the same target domain image respectively. The prediction probabilities of the two enhanced views are required to be consistent in order to complete the alignment and semantic learning processing.

7. The remote sensing image scene classification method based on graph structure unsupervised domain adaptation according to claim 6, characterized in that, The step of performing driving domain alignment and semantic learning on the multi-scale remote sensing scene features using the AFA to generate corresponding hybrid samples further includes: The cross-domain MixUp algorithm is used to mix the source domain samples and the target domain samples to generate the mixed samples.

8. A remote sensing image scene classification system based on graph structure unsupervised domain adaptation, characterized in that, This system is applicable to unsupervised domain adaptation scenarios involving closed sets where the source and target domains have identical category spaces, with source domain samples containing category labels and target domain samples without category labels. The system includes: The building module is used to construct a network model that includes a graph structure feature extraction network (GFE) and an adversarial feature alignment framework (AFA). During the model training phase, the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image are input into the network. The alignment module is used to extract multi-scale remote sensing scene features enhanced by graph structure from the source domain image, the weakly enhanced target domain image, and the strongly enhanced target domain image using the GFE. The GFE is based on ResNet50 as the backbone network and introduces graph structure relationships for modeling on the multi-layer features of ResNet50. The optimization module is used to perform driving domain alignment and semantic learning on the multi-scale remote sensing scene features through the AFA to generate corresponding hybrid samples; The classification module is used to perform end-to-end joint optimization of the GFE and the AFA based on the mixed samples to obtain a remote sensing scene classification model suitable for the target domain, and retain the GFE and classifier during the inference phase to perform scene classification on remote sensing images of the target domain.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the remote sensing image scene classification method based on graph structure unsupervised domain adaptation as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the remote sensing image scene classification method based on graph structure unsupervised domain adaptation as described in any one of claims 1 to 7.

Images