Probabilistic alignment unsupervised domain adaptation method and system for cross-scene classification of remote sensing images

Abstract

The application belongs to the technical field of remote sensing image processing, and is a kind of probability alignment unsupervised domain adaptation remote sensing image cross-scene classification method and system, comprising: inputting source domain remote sensing image dataset and target domain remote sensing image dataset into prompt visual encoder to extract image features, constructing initial text description including categories according to the categories of source domain remote sensing images, and inputting into prompt text encoder to obtain text features, calculating the cosine similarity between each image feature and the text features of all categories, and converting into predicted category probability; constructing a category probability matrix according to the predicted category probability, calculating the kernel norm of the category probability matrix, and calculating the difference between the expectation of the kernel norm of the category probability matrix on the source domain and the expectation of the kernel norm of the category probability matrix on the target domain to obtain the domain difference, and minimizing the domain difference by adjusting the model parameters, overcoming the problem of insufficient cross-modal fusion, and more fully integrating multi-modal information.

Description

Technical Field

[0001] This application belongs to the field of remote sensing image processing technology, specifically a probabilistic alignment unsupervised domain adaptive remote sensing image cross-scene classification method and system. Background Technology

[0002] Unsupervised domain adaptation has gained widespread attention in tasks such as cross-scene classification of remote sensing images because it avoids the time-consuming and costly process of acquiring large amounts of manually labeled data. Its core objective is to transfer knowledge from the source domain (labeled) to the target domain (unlabeled). However, the domain shift problem often leads to a decline in model performance, and it is necessary to solve the problem of inconsistent distributions between the source and target domains across different scenes.

[0003] To address domain offset, existing methods focus on distribution alignment and domain-invariant feature learning. Traditional unsupervised domain adaptation methods are divided into two categories: implicit domain adversarial methods and explicit domain alignment methods. Implicit domain adversarial methods introduce a domain discriminator, constructing an adversarial framework of "feature generation and domain discrimination," forcing the model to generate features that confuse domain discrimination, hoping to overcome the limitations of domain differences in feature learning. However, this approach focuses on "deceiving" the discriminator to achieve domain alignment at the feature level, without delving into the semantic association of categories. In remote sensing cross-scene classification, the feature differences of the same category under different scenes are complex (e.g., the "building" category under different lighting conditions). Relying solely on feature adversarial methods can easily lead the model to learn "general features" without class discrimination, thus blurring category boundaries and causing misclassification of the target domain. Explicit domain alignment methods directly constrain the distribution differences between domains by minimizing the distance between the feature distributions of the source and target domains. However, these methods are mostly based on global feature distribution optimization, without considering the distribution consistency within categories. In remote sensing image cross-scene tasks, some categories in the target domain may have "local distribution offsets" from the categories in the source domain. Global metric optimization cannot accurately align these differences, resulting in poor category alignment performance. Furthermore, traditional unsupervised domain adaptation methods utilize only unimodal visual features for domain adaptation, completely ignoring textual information. This unimodal strategy limits the model's ability to utilize complementary semantic cues and hinders its generalization ability across different scenarios.

[0004] With the development of large-scale pre-trained visual language models (such as CLIP), the semantic information they introduce provides strong zero-shot reasoning capabilities. CLIP-based cue fine-tuning methods have been used for tasks such as few-shot learning and domain adaptation. Some methods focus solely on designing cues within text branches, typically using only textual cues to distinguish domain-specific context and category labels. However, this ignores domain-related information present in visual features. These methods struggle to achieve robust domain adaptation in the diverse and complex environments encountered in remote sensing images. While other methods attempt to connect visual and textual modalities through cue interaction, cross-modal interaction itself remains limited. Using cues with fixed linear mappings results in weak cross-modal fusion and insufficient hierarchical connections between layers. This unidirectional cue interaction fails to fully utilize complementary information across modalities and layers. Furthermore, domain adaptation emphasizes feature-level adaptation, guiding visual features to align with the source domain through textual cues, but neglects precise matching of category probability distributions. Probability distributions reflect the global statistical regularity of class assignment. If the class probability matrices of similar samples in the source and target domains differ, simply aligning features can easily lead to the problem of "similar features but incorrect class classification," resulting in a high risk of class confusion and directly affecting cross-scene classification accuracy. During cross-domain processing, the target domain image is affected by noise and scene differences. Existing methods do not specifically constrain the consistency and robustness of cross-domain features. When the model generalizes to the target domain, feature fluctuations can cause significant fluctuations in classification results, especially in scenarios with varying data quality, such as remote sensing images, making it difficult to guarantee adaptation. Summary of the Invention

[0005] The first aspect of this application provides a probability-aligned unsupervised domain adaptation method for cross-scene classification of remote sensing images, which solves the problem that cross-scene classification of remote sensing images faces large domain differences and diverse imaging conditions. Existing unsupervised domain adaptation methods are difficult to effectively achieve knowledge transfer due to insufficient cross-modal fusion, lack of probability distribution alignment, and poor feature stability.

[0006] A second aspect of this application provides a probabilistically aligned unsupervised domain-adaptive remote sensing image cross-scene classification system.

[0007] The first aspect of this application provides a probabilistically aligned unsupervised domain-adaptive remote sensing image cross-scene classification method, the method comprising:

[0008] Obtain the labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset;

[0009] The source domain remote sensing image dataset and the target domain remote sensing image dataset are input into the cue visual encoder to extract image features. The visual cue encoding of the first layer in the cue visual encoder is randomly generated from a normal distribution, and the visual cue encoding of each subsequent layer is derived from the fusion of the visual cue encoding and the text cue encoding of the previous layer based on cross attention.

[0010] An initial text description including the category is constructed based on the category of the source domain remote sensing image and input into the cue text encoder to obtain text features. The text cue encoding of the first layer in the cue text encoder adopts the initial text description, and the text cue encoding of each subsequent layer is derived from the fusion of the visual cue encoding and the text cue encoding of the previous layer based on cross attention.

[0011] Calculate the cosine similarity between each image feature and text features of all categories, and convert the resulting cosine similarities into predicted category probabilities;

[0012] Construct a class probability matrix based on the predicted class probabilities, calculate the nuclear norm of the class probability matrix, and calculate the difference between the expected nuclear norm of the class probability matrix in the source domain and the expected nuclear norm of the class probability matrix in the target domain to obtain the domain difference. Minimize the domain difference by adjusting the model parameters.

[0013] Furthermore, the visual cue encoding of each layer originates from the fusion of the visual cue encoding and the Chinese text cue encoding of the previous layer based on cross-attention, including: using a fusion function with consistent parameters, taking the visual cue encoding of the previous layer as the key and the Chinese text cue encoding of the previous layer as the query, and calculating the visual cue encoding of each layer;

[0014] The text prompt encoding of each layer originates from the fusion of the visual prompt encoding and the text prompt encoding of the previous layer based on cross-attention. This includes: using a fusion function with consistent parameters, taking the text prompt encoding of the previous layer as the key and the visual prompt encoding of the previous layer as the query, and calculating the text prompt encoding of each layer.

[0015] Furthermore, the labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset are input into the pre-trained visual encoder to obtain pre-trained visual features. The cosine similarity between the pre-trained visual features and the image features output by the cue visual encoder is used to calculate the image feature stability constraint loss. By minimizing the image feature stability constraint loss, the image features are placed in the semantic space of the pre-trained visual encoder.

[0016] Furthermore, the image feature stability constraint loss is: ,in, These are labeled source domain remote sensing images. It is an unlabeled target domain remote sensing image. These are source domain pre-trained visual features extracted by a pre-trained visual encoder. These are pre-trained visual features of the target domain extracted by a pre-trained visual encoder. It prompts the visual encoder to obtain source domain image features. It prompts the visual encoder to obtain the target domain image features. Calculate the cosine similarity between the pre-trained visual features output by the pre-trained visual encoder in the source domain and the image features output by the cue visual encoder. Calculate the cosine similarity between the pre-trained visual features output by the pre-trained visual encoder and the image features output by the cue visual encoder in the target domain.

[0017] Furthermore, the initial text description is generated by embedding the template "a [category] photo" through a pre-trained text encoder and performing initialization. The cosine similarity between the pre-trained text features extracted by the pre-trained text encoder and the text features output by the prompt text encoder is used to calculate the text feature stability constraint loss. The consistency of text features is ensured by minimizing the text feature stability constraint loss.

[0018] Furthermore, the text feature stability constraint loss is: ,in, It is the initial text description of the source domain. It is the initial text description of the target domain. These are the source domain pre-trained text features extracted by the pre-trained text encoder. These are the target domain pre-trained text features extracted by the pre-trained text encoder. These are the source domain text features obtained by the prompt text encoder. These are the target domain text features obtained by the prompt text encoder. Calculate the cosine similarity between the pre-trained text features output by the pre-trained text encoder and the text features output by the cue text encoder in the source domain. Calculate the cosine similarity between the pre-trained text features output by the pre-trained text encoder and the text features output by the cue text encoder in the target domain.

[0019] Furthermore, after converting the obtained multiple cosine similarities into predicted class probabilities, the predicted class probabilities obtained in the source domain are used to calculate the source domain classification loss, which is used to evaluate the degree of matching between the predicted probability of the source domain sample and the true label of the source domain.

[0020] In the target domain, target samples with the maximum predicted class probability exceeding a preset confidence threshold are assigned pseudo-labels; otherwise, the target samples are excluded from the target domain classification loss calculation. The target domain classification loss is calculated to evaluate the consistency between the predicted probability of the target domain samples and the high-confidence pseudo-labels in the target domain. Further, a probability difference loss is calculated based on the class probability matrices of the source and target domains. The gradient formed by the partial derivatives of the probability difference loss with respect to the model parameters is then backpropagated to the cue visual encoder through a gradient inversion layer to update the parameters of the cue visual encoder.

[0021] A second aspect of this application provides a probabilistically aligned unsupervised domain-adaptive remote sensing image cross-scene classification system, comprising:

[0022] The data acquisition module is used to acquire labeled source domain remote sensing image datasets and corresponding unlabeled target domain remote sensing image datasets.

[0023] A cue visual encoder is used to extract image features from a source domain remote sensing image dataset and a target domain remote sensing image dataset. It includes a multi-layer image encoder. The visual cue codes of the first layer in the cue visual encoder are randomly generated from a normal distribution, and the visual cue codes of each subsequent layer are derived from the fusion of the visual cue codes and text cue codes of the previous layer based on cross-attention.

[0024] The cue text encoder includes a multi-layer text encoder for extracting text features. The text cue encoding of the first layer in the cue text encoder uses the initial text description, and the text cue encoding of each subsequent layer is derived from the fusion of visual cue encoding and text cue encoding in the previous layer based on cross attention.

[0025] The progressive attention interaction prompt module is based on the fusion of cross attention and calculates the text prompt code or visual prompt code of the current layer according to the visual prompt code and text prompt code in the previous layer.

[0026] The cosine similarity calculation module calculates the cosine similarity between each image feature and text features of all categories, and converts the obtained cosine similarities into predicted category probabilities.

[0027] The category probability difference measurement module constructs a category probability matrix based on the predicted category probabilities, calculates the nuclear norm of the category probability matrix, and calculates the difference between the expected nuclear norm of the category probability matrix in the source domain and the expected nuclear norm of the category probability matrix in the target domain to obtain the domain difference. By adjusting the model parameters, the domain difference is minimized.

[0028] Furthermore, it also includes a first feature stability constraint module, which inputs the labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset into the pre-trained visual encoder to obtain pre-trained visual features. The cosine similarity between the pre-trained visual features and the image features output by the cue visual encoder is used to calculate the image feature stability constraint loss. By minimizing the image feature stability constraint loss, the image features are placed in the semantic space of the pre-trained visual encoder.

[0029] The second feature stability constraint module uses the cosine similarity between the pre-trained text features extracted by the pre-trained text encoder and the text features output by the prompt text encoder to calculate the text feature stability constraint loss. By minimizing the text feature stability constraint loss, the consistency of text features is ensured.

[0030] Compared with existing technologies, the advantages of this application are as follows: The progressive attention-based interactive cues proposed in this application achieve bidirectional updating and deep fusion of visual and textual cues at each layer, overcoming the problem of insufficient cross-modal fusion caused by existing methods that only design cues for the text modality or unidirectionally map text cues to visual cues, thus enabling more comprehensive integration of multimodal information. By aligning the probability matrices of the source and target domains through the nuclear norm in adversarial training, both domain invariance and class discriminability are promoted, without the need for complex additional modules, improving computational efficiency and robustness. The feature stability constraint module ensures the consistency between learnable cue features and pre-trained model features, reducing semantic shift and enhancing the model's generalization ability in complex scenarios. Attached Figure Description

[0031] Figure 1 A flowchart of a probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method provided in this application embodiment;

[0032] Figure 2 This is a block diagram of a probabilistic alignment unsupervised domain adaptive remote sensing image cross-scene classification system provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] Cross-scene classification of remote sensing images faces challenges such as large domain differences and diverse imaging conditions. Existing unsupervised domain adaptation methods are difficult to effectively achieve knowledge transfer due to insufficient cross-modal fusion, lack of probability distribution alignment, and poor feature stability.

[0035] Regarding the background mentioned above, see [link to relevant documentation]. Figure 1 The flowchart shown is a probabilistic alignment unsupervised domain adaptive remote sensing image cross-scene classification method provided in this application embodiment. The provided probabilistic alignment unsupervised domain adaptive remote sensing image cross-scene classification method includes the following steps:

[0036] S1 obtains the labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset;

[0037] In S1, a labeled source domain remote sensing image dataset is collected. ,in For the first A labeled remote sensing scene image, For the first Category labels, The number of remote sensing scene images. Target domain remote sensing image dataset. This is an unlabeled target domain remote sensing image dataset, where Indicates the first An unlabeled remote sensing image of the target domain. The number of unlabeled remote sensing images is given. The data in the source domain and the target domain exhibit distributional discrepancies due to imaging conditions (such as illumination, resolution, and sensor differences). Here, the source domain refers to the dataset used during model training, while the target domain refers to the dataset that needs to be processed in practical applications.

[0038] S2 inputs the source domain remote sensing image dataset and the target domain remote sensing image dataset into the cue visual encoder to extract image features. The visual cue encoding of the first layer in the cue visual encoder is randomly generated from a normal distribution, and the visual cue encoding of each subsequent layer is derived from the fusion of the visual cue encoding and the text cue encoding of the previous layer based on cross attention.

[0039] In the cue visual encoder: the cue visual encoder consists of multiple layers of visual encoders, each of which employs a Transformer layer. The source domain remote sensing image dataset and the target domain remote sensing image dataset are processed through all Transformer layers of the cue visual encoder. The visual cue codes inserted into the first layer are randomly generated from a normal distribution. Subsequent visual cue codes are derived from the fusion of the visual cue codes and text cue codes from the previous layer based on cross-attention. Here, the previous layer refers to the layer preceding both the cue visual encoder and the cue text encoder. The final output is the category label. ,pass The projection layer obtains the final image features. This can be expressed by the formula:

[0040] ,

[0041] ,

[0042] ,

[0043] in, Indicates the first Layer category tags, This indicates the visual encoder's first... Visual embedding encoding of layers, Indicates the first Layer category tags, This indicates the visual encoder's first... Visual cue encoding of layers, This indicates the visual encoder's first... Visual embedding encoding of layers, The text encoder indicates the prompt text. Layer text hint encoding, For the visual encoder Visual cue encoding of layers, This indicates that the image has been processed by the Transformer layer of the visual encoder.

[0044] S3 constructs an initial text description including the category based on the category of the source domain remote sensing image, and inputs it into the cue text encoder to obtain text features. The text cue encoding of the first layer in the cue text encoder adopts the initial text description, and the text cue encoding of each subsequent layer is derived from the fusion of the visual cue encoding and the text cue encoding of the previous layer based on cross attention.

[0045] The cue text encoder consists of multiple text encoder layers, each employing a Transformer layer. The text is processed through all Transformer layers of the text encoder. The first layer's cue encoding uses a pre-trained text encoder embedding template of "a photo of [category]" for initialization. Subsequent layers' cue encodings are derived from the fusion of visual cue encodings and text cue encodings from the previous layer based on cross-attention. The final output is the text embedding encoding. ,pass The projection layer obtains the final text features. This can be expressed by the formula:

[0046] ,

[0047] ,

[0048] ,

[0049] in, The text encoder indicates the prompt text. Layer text embedding encoding, The text encoder indicates the prompt text. Layer text embedding encoding, The text encoder indicates the prompt text. Layer text hint encoding, The text encoder indicates the prompt text. Layer text embedding encoding, This indicates that the text has been processed by the Transformer layer of the text encoder.

[0050] in , The depth of the progressive attention interaction cues is less than the length of the cue text encoder and the cue visual encoder.

[0051] The template for constructing the initial text description is "a photo of [category]", where [category] is replaced with the source domain category name. The corresponding initial text description and remote sensing image description are generated into initial text embedding codes and initial image embedding codes respectively through a pre-trained text encoder and a pre-trained visual encoder.

[0052] S4 calculates the cosine similarity between each image feature and text features of all categories, and converts the resulting cosine similarities into predicted category probabilities.

[0053] In one embodiment of this application, the outputs of the cue text encoder and the cue visual encoder undergo cosine similarity processing as follows: First, the cue text encoder obtains text features, and the cue visual encoder obtains image features; then, the cosine similarity between each image feature and all categories of text features is calculated to measure the degree of association between the two in the shared embedding space; subsequently, combined with the temperature parameter, these cosine similarities are converted into predicted category probabilities using the softmax function. This probability value is not only used for calculating the classification loss between the source and target domains, but also provides a basis for measuring the difference in category probability distributions between the source and target domains, thereby promoting probability alignment in the cross-modal matching and domain adaptation process.

[0054] S5 constructs a class probability matrix based on the predicted class probabilities, calculates the nuclear norm of the class probability matrix, and calculates the difference between the expected nuclear norm of the class probability matrix in the source domain and the expected nuclear norm of the class probability matrix in the target domain to obtain the domain difference. By adjusting the model parameters, the domain difference is minimized.

[0055] To significantly enhance the interaction between different levels and modalities of cue information, this application proposes a progressive attention interaction cue, namely: the visual cue encoding of each layer originates from the fusion of the visual cue encoding and the Chinese text cue encoding of the previous layer based on cross attention, including: using a fusion function with consistent parameters, taking the visual cue encoding of the previous layer as the key and the Chinese text cue encoding of the previous layer as the query, and calculating the visual cue encoding of each layer.

[0056] The text prompt encoding of each layer originates from the fusion of the visual prompt encoding and the text prompt encoding of the previous layer based on cross-attention. This includes: using a fusion function with consistent parameters, taking the text prompt encoding of the previous layer as the key and the visual prompt encoding of the previous layer as the query, and calculating the text prompt encoding of each layer.

[0057] Progressive attentional interactive cues are a method that combines progressive cue design with attention mechanisms, aiming to improve the model's performance in multimodal tasks by gradually guiding the model's attention. Specifically, this stage borrows from the CLIP dual-encoder model architecture (contrastive language-image pre-training), introducing cue information into each branch and using a cross-attention mechanism to promote the fusion of visual and linguistic information during training. This can be represented as:

[0058] ,

[0059] It is a custom function based on a cross-attention mechanism; in this application, it refers to the fusion function, where... Represents the softmax function. This refers to querying. , This refers to the dimension of the key vector. Transpose.

[0060] Unlike traditional methods, the visual cue encoder's first layer of visual cue encoding is randomly generated from a normal distribution. Subsequent layers' visual cue encodings originate from the fusion of the visual cue encoding and the text cue encoding from the previous layer based on cross-attention. In the text cue encoder, the first layer's text cue encoding uses the initial text description. Therefore, in the visual cue encoder, only the first layer of cues is randomly generated; the generation of subsequent cues depends on the previous layer's cues in both the visual and text modal dimensions. From the perspective of text and visual cue fusion, efforts are made in both hierarchical and modal dimensions. In the hierarchical dimension, the association between different layers' cues is achieved through "cross-layer cue dependency + parameter sharing," forming a progressive dependency relationship where "previous layer cue forms current layer cue." Therefore, this cueing method is called progressive attention interactive cueing. Simultaneously, a parameter sharing mechanism can be used between layers, meaning that the fusion parameters used in the fusion function across all layers remain consistent, ensuring unified processing logic between layers and strengthening the association between cues from different layers. In the modal dimension, "multi-head cross-modal attention + bidirectional fusion" is used. Strengthening the interaction between visual and textual modalities: A multi-head cross-modal attention mechanism is introduced at each layer. Visual cues and textual cues serve as inputs for cross-attention. In the cue-text encoder, visual cues act as queries and textual cues as keys; similarly, in the cue-visual encoder, textual cues act as queries and visual cues as keys. By calculating attention weights, the two modalities' information permeates each other. Visual cues absorb semantic information from the text modality, while textual cues integrate spatial features from the visual modality. This fusion is bidirectional, not unidirectional. For example, updates to visual cues depend on textual cues, and updates to textual cues also depend on visual cues, enabling deep information exchange between the two modalities at each layer. Progressive attention interaction cues employ a parameter-sharing mechanism across different layers. This measure reduces the model training burden and improves training efficiency. Furthermore, because very few new trainable parameters are added, it helps enhance the model's generalization ability and avoids the risk of overfitting.

[0061] Bidirectional progressive update refers to a progressive attention-based interactive cueing approach where visual and textual cues collaborate and update in two directions. This update method achieves deep fusion of multimodal information through cross-layer information interaction, improving the model's performance in cross-scene classification tasks.

[0062] Cross-layer bidirectional interaction: At each layer of the model, visual cues and text cues are updated based on the results of the previous layer. In the visual cue encoder, the visual cues of the current layer are obtained by fusing the visual and text cues of the previous layer; similarly, in the text cue encoder, the text cues of the current layer are obtained by fusing the visual and text cues of the previous layer. This cross-layer bidirectional interaction runs throughout the entire encoder process, enabling information exchange and fusion between the visual and text modalities at each layer.

[0063] Progressive information fusion: Starting from the first layer, visual and textual cues begin to interact and merge. As the number of layers increases, this fusion deepens, with each layer further integrating and strengthening multimodal information based on the previous one. Through this progressive information fusion, the model can gradually learn richer and more discriminative cross-modal features, thereby improving its understanding and classification capabilities for complex scenes.

[0064] Bidirectional Promotion and Enhancement: The updates of visual cues and textual cues are mutually reinforcing. Updating visual cues influences the generation of textual cues, and vice versa. This bidirectional promotion mechanism allows the two branches to learn from and complement each other, enhancing the model's understanding and representation of multimodal information. In remote sensing image classification tasks, visual cues provide spatial and textural information, while textual cues provide semantic information. Bidirectional updates of both can better integrate this information, improving classification accuracy.

[0065] The parameter sharing mechanism adopted between different layers means that in progressive attention interaction prompts, some parameters are shared between different layers during the fusion of visual and text prompts and feature calculation.

[0066] Shared parameters: The parameters involved in the key computational processes for fusing visual and textual cues are shared. Specifically, the cross-attention-based fusion function used by the visual and textual cues encoders at each layer when computing new cues is used. The relevant parameters remain consistent across all layers. Taking the cue visual encoder as an example, from the first layer to each subsequent layer, the calculation... At that time, the fusion function The internal parameters (such as those related to calculating attention weights) do not change with the number of layers.

[0067] This parameter sharing mechanism offers several advantages. In terms of computational cost, it avoids the need for each layer to relearn a complete set of fusion parameters, significantly reducing the computational load and memory consumption during model training, enabling efficient model operation even with limited resources. Regarding feature consistency, it ensures that different layers have a unified standard and logic when fusing visual and textual cues, promoting cross-layer feature consistency, making the multimodal features learned by the model more stable and coherent, improving the model's understanding and adaptability to complex scenes, and enhancing its performance in cross-scene classification tasks.

[0068] In one embodiment, a labeled source domain remote sensing image dataset and a corresponding unlabeled target domain remote sensing image dataset are input into a pre-trained visual encoder to obtain pre-trained visual features. The cosine similarity between the pre-trained visual features and the image features output by the cue visual encoder is used to calculate the image feature stability constraint loss. By minimizing the image feature stability constraint loss, the image features are placed in the semantic space of the pre-trained visual encoder.

[0069] The initial text description is generated by embedding the template "a photo of [category]" through a pre-trained text encoder. The cosine similarity between the pre-trained text features extracted by the pre-trained text encoder and the text features output by the prompt text encoder is used to calculate the text feature stability constraint loss. The consistency of text features is ensured by minimizing the text feature stability constraint loss.

[0070] In cross-scene classification of remote sensing images, feature stability constraints aim to address the semantic deviation between learnable cue features and pre-trained model features caused by interference from sensor noise and imaging blur in the target domain data. By explicitly constraining, the constraints ensure that the features generated by cue learning are consistent in direction with the features of the pre-trained CLIP (Contrastive Language-Image Pre-training) model, avoid the loss of semantic information during cross-domain adaptation, and improve the generalization robustness of the model in the target domain. In this embodiment, the pre-trained CLIP model includes a pre-trained visual encoder and a pre-trained text encoder. Both use cosine similarity instead of Euclidean distance to measure feature consistency. Specifically, the cosine similarity is calculated as the cosine of the angle between the pre-trained visual features output by the pre-trained CLIP model and the feature vectors output by the visual encoder or text encoder. For example, the cosine of the angle between the pre-trained visual features output by the pre-trained visual encoder and the image features output by the cue visual encoder is used to obtain the cosine similarity, as is the cosine of the angle between the pre-trained text features output by the pre-trained text encoder and the text features output by the cue text encoder. This measures the directional consistency between the two in the semantic space, avoiding feature deviation from the semantic space of the pre-trained CLIP model due to cue optimization.

[0071] In one embodiment, the feature stability constraint loss includes:

[0072] The image feature stability constraint loss is: ,in, These are labeled source domain remote sensing images. It is an unlabeled target domain remote sensing image. These are source domain pre-trained visual features extracted by a pre-trained visual encoder. These are pre-trained visual features of the target domain extracted by a pre-trained visual encoder. It prompts the visual encoder to obtain source domain image features. It prompts the visual encoder to obtain the target domain image features. Calculate the cosine similarity between the pre-trained visual features output by the pre-trained visual encoder in the source domain and the image features output by the cue visual encoder. Calculate the cosine similarity between the pre-trained visual features output by the pre-trained visual encoder and the image features output by the cue visual encoder in the target domain.

[0073] Image feature stability constraint loss is used to align image features between the source and target domains.

[0074] Similarly, the text feature stability constraint loss is ,in, It is the initial text description of the source domain. It is the initial text description of the target domain. These are the source domain pre-trained text features extracted by the pre-trained text encoder. These are the target domain pre-trained text features extracted by the pre-trained text encoder. These are the source domain text features obtained by the prompt text encoder. These are the target domain text features obtained by the prompt text encoder. Calculate the cosine similarity between the pre-trained text features output by the pre-trained text encoder and the text features output by the cue text encoder in the source domain. Calculate the cosine similarity between the pre-trained text features output by the pre-trained text encoder and the text features output by the cue text encoder in the target domain.

[0075] Text feature stability constraint loss ensures the consistency of text features.

[0076] The image feature stability constraint loss and the text feature stability constraint loss are combined to form the total stability constraint loss: This measure assesses the alignment between images and text using pre-trained features and cue-based features. By implementing this feature stability constraint loss, it is possible to maintain semantic consistency at the category level and effectively mitigate semantic drift and information loss during cross-domain adaptation.

[0077] In one embodiment, the cosine similarity between each image feature and text features of all categories is calculated, and the resulting multiple cosine similarities are converted into predicted category probabilities;

[0078] After converting the obtained cosine similarities into predicted class probabilities, the predicted class probabilities obtained in the source domain are used to calculate the source domain classification loss, which is used to evaluate the degree of matching between the predicted probabilities of the source domain samples and the true labels of the source domain.

[0079] In the target domain, target samples whose maximum predicted class probability exceeds a preset confidence threshold are assigned pseudo-labels; otherwise, the target samples are excluded from the target domain classification loss calculation. The target domain classification loss is then calculated to evaluate the consistency between the predicted probabilities of target domain samples and the high-confidence pseudo-labels in the target domain. This is called the classification loss stage. The classification loss is used to assess the matching degree between the predicted probabilities of source domain samples and the true labels of the source domain, and to evaluate the consistency between the predicted probabilities of target domain samples and the high-confidence pseudo-labels in the target domain. The overall classification loss is expressed as... Defined as: .

[0080] In the source domain, the data, after undergoing progressive attention-based interactive cues, is categorized as follows: The source domain classification loss is calculated using the predicted class probabilities:

[0081] ,

[0082] ,

[0083] in, , It is the first The true label of the source domain data corresponding to each sample It is the number of categories. These are text features obtained after progressive attention-based interactive cues. These are image features obtained after progressive attentional interactive cues. It is the first in the source domain data The predicted probability of a sample belonging to a certain category. It is calculated by analyzing text features after progressive attention interaction cues. and image features The cosine similarity is obtained by normalizing it using the softmax function, where... It is a temperature parameter used to adjust the smoothness of the probability distribution. The predicted class probabilities obtained from the source domain. The loss is the source domain classification loss.

[0084] In the target domain, the maximum predicted probability exceeds the preset confidence threshold. The target samples are assigned pseudo-labels; otherwise, the samples are excluded from the loss calculation. The classification loss for category f in the target domain is:

[0085] ,

[0086] ,

[0087] in, It is an indicator function. These are pseudo-tags corresponding to the target domain. , It is the number of categories. These are text features obtained after progressive attention-based interactive cues. These are the target domain text features obtained after progressive attention interaction cues. These are image features obtained after progressive attentional interactive cues. It is the first in the target domain data The predicted probability of each sample belonging to a certain category, where It is a temperature parameter.

[0088] To address the generalization problem of the source domain to the target domain, a class probability difference measurement strategy is proposed. Its core lies in using the kernel-Wasestein difference to measure the distance between the class probability distributions of the source and target domains. Specifically, by introducing the kernel norm, class probability matrices of the source and target domains are constructed, i.e., the class probability matrices of the source and target domains are built, and the kernel norm is calculated to quantify the difference in distribution between the two domains, measuring the degree of difference in their class probability distributions. By quantifying the kernel norm difference, the class probability distributions of the source and target domains are encouraged to converge, thereby helping the model learn more consistent and more generalizable feature representations across different domains. The calculation is as follows: Therefore, the domain difference can be expressed as:

[0089] ,

[0090] in, Represents the nuclear norm number. It is the symbol for supremacy, indicating that... Taking the maximum value under certain conditions makes it easier to transfer knowledge from the source domain to the target domain. For the category probability matrix The nuclear norm is used to constrain [the population]. It is a set threshold that limits the category probability matrix. The range of values ​​for the nuclear norm is continuously adjusted during model training to make the value of this supremum decrease. Representing image features The probability of class f. Indicates in the source domain Top Expectations Indicates in the target domain Top The expectation. Using the nuclear norm. The core purpose of measuring the characteristics of the probability distributions of the source domain and the target domain is to achieve domain adaptation. Specifically, firstly, the nuclear norm is used to accurately measure the difference in probability distributions between the source and target domains, clarifying the distribution gap between the two and providing a basis for subsequent adaptation work; secondly, domain alignment is promoted based on the measurement results, by minimizing the distribution difference and exploring suitable feature transformation methods to make the feature distributions of the source and target domains more similar; the ultimate goal is to improve the model's generalization ability in the target domain.

[0091] In one embodiment, a probability difference loss is calculated based on the class probability matrix of the source domain and the class probability matrix of the target domain, and the gradient formed by the partial derivative of the probability difference loss with respect to the model parameters is backpropagated to the cue visual encoder through a gradient inversion layer to update the parameters of the cue visual encoder.

[0092] Probability difference loss The calculation formula is:

[0093] ,

[0094] in, It is the number of data samples in the source domain. It is the number of data samples in the target domain. Indicates the first in the source domain The class probability matrix of each sample Indicates the first in the target domain The class probability matrix of each sample. To further optimize the domain adaptation effect, this application performs a kernel norm maximization operation on the class probability matrix of the target domain, aiming to enhance the discriminability and diversity of the target prediction results, thereby enabling the model to more accurately capture the feature information of the target domain. At the same time, a kernel norm minimization operation is performed on the class probability matrix of the source domain to promote more efficient transfer of source domain knowledge to the target domain. By adopting this differentiated processing method for the kernel norm, the synergistic optimization of domain alignment and class discrimination is achieved, improving the adaptability and generalization ability of the model across different domains. In the transfer learning process, the gradient inversion layer plays a key role in adversarial optimization. During training, the gradient inversion layer inverts the gradient and backpropagates it during backpropagation. The purpose of this operation is to allow the feature extractor (i.e., the cue visual encoder and the cue visual encoder) to learn domain-independent features. Because the gradient inversion layer inverts the gradient during backpropagation, the feature extractor will adjust in the direction of reducing the feature differences between the source and target domains when updating parameters. Through this adversarial approach, the feature extractor can generate more generalized features that are no longer limited to a specific domain (whether source or target domain), thereby helping to improve the model's performance in the target domain and achieve better domain adaptation. Unlike traditional adversarial methods (which require explicit domain discriminators), the class probability difference metric uses the nuclear norm difference as an implicit alignment target, avoiding additional modules and improving computational efficiency.

[0095] Based on the above, the total loss consists of three parts: classification loss, feature stability constraint loss, and probability difference loss.

[0096] ,

[0097] in, and These are the balance weights of the feature stability constraint loss and the probability difference loss, respectively.

[0098] On the other hand, see Figure 2 As shown, this application provides a block diagram of a probabilistically aligned unsupervised domain adaptive remote sensing image cross-scene classification system. This system, corresponding to the aforementioned probabilistically aligned unsupervised domain adaptive remote sensing image cross-scene classification method, includes:

[0099] The data acquisition module is used to acquire labeled source domain remote sensing image datasets and corresponding unlabeled target domain remote sensing image datasets.

[0100] A cue visual encoder is used to extract image features from a source domain remote sensing image dataset and a target domain remote sensing image dataset. It includes a multi-layer image encoder. The visual cue codes of the first layer in the cue visual encoder are randomly generated from a normal distribution, and the visual cue codes of each subsequent layer are derived from the fusion of the visual cue codes and text cue codes of the previous layer based on cross-attention.

[0101] The cue text encoder includes a multi-layer text encoder for extracting text features. The text cue encoding of the first layer in the cue text encoder uses the initial text description, and the text cue encoding of each subsequent layer is derived from the fusion of visual cue encoding and text cue encoding in the previous layer based on cross attention.

[0102] The progressive attention interaction prompt module is based on the fusion of cross attention and calculates the text prompt code or visual prompt code of the current layer according to the visual prompt code and text prompt code in the previous layer.

[0103] The cosine similarity calculation module calculates the cosine similarity between each image feature and text features of all categories, and converts the obtained cosine similarities into predicted category probabilities.

[0104] The category probability difference measurement module constructs a category probability matrix based on the predicted category probabilities, calculates the nuclear norm of the category probability matrix, and calculates the difference between the expected nuclear norm of the category probability matrix in the source domain and the expected nuclear norm of the category probability matrix in the target domain to obtain the domain difference. By adjusting the model parameters, the domain difference is minimized.

[0105] It also includes a first feature stability constraint module, which inputs the labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset into the pre-trained visual encoder to obtain pre-trained visual features. The cosine similarity between the pre-trained visual features and the image features output by the cue visual encoder is used to calculate the image feature stability constraint loss. By minimizing the image feature stability constraint loss, the image features are placed in the semantic space of the pre-trained visual encoder.

[0106] The second feature stability constraint module uses the cosine similarity between the pre-trained text features extracted by the pre-trained text encoder and the text features output by the prompt text encoder to calculate the text feature stability constraint loss. By minimizing the text feature stability constraint loss, the consistency of text features is ensured.

[0107] This application first utilizes a progressive attention interaction cue module at each encoder layer. This module, employing a multi-head cross-modal attention mechanism, fuses the visual and textual cue encodings from the previous layer to generate new cues for the current layer. This allows for deep and progressive information exchange between the visual and textual modalities at each layer, resulting in fused features. These features are projected into a shared embedding space for subsequent class probability prediction. Regarding domain adaptation, a class probability difference measurement module, implemented in adversarial training mode (through a gradient inversion layer), uses the kernel norm to calculate and minimize the difference between the class probability matrices of the source and target domains. This promotes knowledge transfer from the source to the target domain and enhances the discriminativeness and diversity of the target domain prediction results. Simultaneously, a feature stability constraint module uses cosine similarity to measure and reduce the gap between the features generated by the learnable cue and those extracted by the pre-trained CLIP model, ensuring feature stability and preventing feature embedding shifts due to cue tuning. The entire framework jointly optimizes three objectives: first, classification loss, used to supervise source domain samples and target domain samples with high-confidence pseudo-labels; second, probability difference loss, used to align the predicted distributions of the source and target domains; and third, feature stability constraint loss, used to regularize the cue embeddings during training, ultimately achieving accurate cross-domain adaptation and robust cross-modal feature consistency.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A probabilistically aligned unsupervised domain-adaptive remote sensing image cross-scene classification method, characterized in that, The method includes: Obtain the labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset; The source domain remote sensing image dataset and the target domain remote sensing image dataset are input into the cue visual encoder to extract image features. The visual cue encoding of the first layer in the cue visual encoder is randomly generated from a normal distribution, and the visual cue encoding of each subsequent layer is derived from the fusion of the visual cue encoding and the text cue encoding of the previous layer based on cross attention. An initial text description including the category is constructed based on the category of the source domain remote sensing image and input into the cue text encoder to obtain text features. The text cue encoding of the first layer in the cue text encoder adopts the initial text description, and the text cue encoding of each subsequent layer is derived from the fusion of the visual cue encoding and the text cue encoding of the previous layer based on cross attention. Calculate the cosine similarity between each image feature and text features of all categories, and convert the resulting cosine similarities into predicted category probabilities; Construct a class probability matrix based on the predicted class probabilities, calculate the nuclear norm of the class probability matrix, and calculate the difference between the expected nuclear norm of the class probability matrix in the source domain and the expected nuclear norm of the class probability matrix in the target domain to obtain the domain difference. Minimize the domain difference by adjusting the model parameters.

2. The probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method according to claim 1, characterized in that, The visual cue encoding of each layer originates from the fusion of the visual cue encoding and the Chinese text cue encoding of the previous layer based on cross-attention, including: using a fusion function with consistent parameters, taking the visual cue encoding of the previous layer as the key and the Chinese text cue encoding of the previous layer as the query, and calculating the visual cue encoding of each layer; The text prompt encoding of each layer originates from the fusion of the visual prompt encoding and the text prompt encoding of the previous layer based on cross-attention. This includes: using a fusion function with consistent parameters, taking the text prompt encoding of the previous layer as the key and the visual prompt encoding of the previous layer as the query, and calculating the text prompt encoding of each layer.

3. The probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method according to claim 1, characterized in that, The labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset are input into the pre-trained visual encoder to obtain pre-trained visual features. The cosine similarity between the pre-trained visual features and the image features output by the cue visual encoder is used to calculate the image feature stability constraint loss. By minimizing the image feature stability constraint loss, the image features are placed in the semantic space of the pre-trained visual encoder.

4. The probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method according to claim 3, characterized in that, The image feature stability constraint loss is: ,in, These are labeled source domain remote sensing images. It is an unlabeled target domain remote sensing image. These are source domain pre-trained visual features extracted by a pre-trained visual encoder. These are pre-trained visual features of the target domain extracted by a pre-trained visual encoder. It prompts the visual encoder to obtain source domain image features. It prompts the visual encoder to obtain the target domain image features. Calculate the cosine similarity between the pre-trained visual features output by the pre-trained visual encoder in the source domain and the image features output by the cue visual encoder. Calculate the cosine similarity between the pre-trained visual features output by the pre-trained visual encoder and the image features output by the cue visual encoder in the target domain.

5. The probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method according to claim 1, characterized in that, The initial text description is generated by embedding the template "a photo of [category]" through a pre-trained text encoder. The cosine similarity between the pre-trained text features extracted by the pre-trained text encoder and the text features output by the prompt text encoder is used to calculate the text feature stability constraint loss. The consistency of text features is ensured by minimizing the text feature stability constraint loss.

6. The probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method according to claim 5, characterized in that, Text feature stability constraint loss is ,in, It is the initial text description of the source domain. It is the initial text description of the target domain. These are the source domain pre-trained text features extracted by the pre-trained text encoder. These are the target domain pre-trained text features extracted by the pre-trained text encoder. These are the source domain text features obtained by the prompt text encoder. These are the target domain text features obtained by the prompt text encoder. Calculate the cosine similarity between the pre-trained text features output by the pre-trained text encoder and the text features output by the cue text encoder in the source domain. Calculate the cosine similarity between the pre-trained text features output by the pre-trained text encoder and the text features output by the cue text encoder in the target domain.

7. The probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method according to claim 1, characterized in that, After converting the obtained cosine similarities into predicted class probabilities, the predicted class probabilities obtained in the source domain are used to calculate the source domain classification loss, which is used to evaluate the degree of matching between the predicted probabilities of the source domain samples and the true labels of the source domain. In the target domain, target samples whose maximum predicted class probability exceeds a preset confidence threshold are assigned pseudo-labels; otherwise, the target samples are excluded from the target domain classification loss calculation. The target domain classification loss is calculated to evaluate the consistency between the predicted probability of the target domain samples and the high-confidence pseudo-labels of the target domain.

8. The probabilistic alignment unsupervised domain-adaptive remote sensing image cross-scene classification method according to claim 1, characterized in that, The probability difference loss is calculated based on the class probability matrix of the source domain and the class probability matrix of the target domain. The gradient formed by the partial derivative of the probability difference loss with respect to the model parameters is then passed back to the cue visual encoder through the gradient inversion layer to update the parameters of the cue visual encoder.

9. A probabilistically aligned unsupervised domain-adaptive remote sensing image cross-scene classification system, characterized in that, include: The data acquisition module is used to acquire labeled source domain remote sensing image datasets and corresponding unlabeled target domain remote sensing image datasets. A cue visual encoder is used to extract image features from a source domain remote sensing image dataset and a target domain remote sensing image dataset. It includes a multi-layer image encoder. The visual cue codes of the first layer in the cue visual encoder are randomly generated from a normal distribution, and the visual cue codes of each subsequent layer are derived from the fusion of the visual cue codes and text cue codes of the previous layer based on cross-attention. The cue text encoder includes a multi-layer text encoder for extracting text features. The text cue encoding of the first layer in the cue text encoder uses the initial text description, and the text cue encoding of each subsequent layer is derived from the fusion of visual cue encoding and text cue encoding in the previous layer based on cross attention. The progressive attention interaction prompt module is based on the fusion of cross attention and calculates the text prompt code or visual prompt code of the current layer according to the visual prompt code and text prompt code in the previous layer. The cosine similarity calculation module calculates the cosine similarity between each image feature and text features of all categories, and converts the obtained cosine similarities into predicted category probabilities. The category probability difference measurement module constructs a category probability matrix based on the predicted category probabilities, calculates the nuclear norm of the category probability matrix, and calculates the difference between the expected nuclear norm of the category probability matrix in the source domain and the expected nuclear norm of the category probability matrix in the target domain to obtain the domain difference. By adjusting the model parameters, the domain difference is minimized.

10. The probabilistically aligned unsupervised domain-adaptive remote sensing image cross-scene classification system according to claim 9, characterized in that, It also includes a first feature stability constraint module, which inputs the labeled source domain remote sensing image dataset and the corresponding unlabeled target domain remote sensing image dataset into the pre-trained visual encoder to obtain pre-trained visual features. The cosine similarity between the pre-trained visual features and the image features output by the cue visual encoder is used to calculate the image feature stability constraint loss. By minimizing the image feature stability constraint loss, the image features are placed in the semantic space of the pre-trained visual encoder. The second feature stability constraint module uses the cosine similarity between the pre-trained text features extracted by the pre-trained text encoder and the text features output by the prompt text encoder to calculate the text feature stability constraint loss. By minimizing the text feature stability constraint loss, the consistency of text features is ensured.

Images