Discriminative text prompt learning based cross-domain few-shot remote sensing target classification method

Abstract

The application belongs to the technical field of image information processing, and discloses a cross-domain few-shot remote sensing target classification method based on discriminative text prompt learning. First, the support set and the query set are sent into an image encoder to obtain encoded features. Through a semantic prompt generation module, a support set prototype feature guides a prompt generator to generate discriminative semantic prompts S that adapt to the few-shot task, and replace the fixed template to accurately represent the category attributes. Then, the text encoder processes the discriminative semantic prompts S, the intra-class contrast regularization is used to construct the same-class multi-view prompt pair and the different-class negative sample pair, the temperature modulation contrast loss is used to suppress instance noise and improve cross-domain robustness. Finally, the application proposes a bimodal decision hybrid prediction, which overcomes the problem that the previous method cannot balance the domain invariance and the detail discrimination under few-shot.

Description

Technical Field

[0001] This invention belongs to the field of image information processing and pattern recognition technology, and relates to a cross-domain few-shot remote sensing target classification method based on discriminative textual prompt learning. Background Technology

[0002] Deep learning-based few-shot learning algorithms can be broadly categorized into three types: One type, optimization-based methods, utilize meta-learning to optimize the network and obtain a set of parameters that can be quickly generalized to the target domain. Finn et al., in their paper "Model-agnostic meta-learning for fast adaptation of deep networks," proposed a method for learning universal initial parameters that require only minor adjustments to adapt to new tasks. Rusu et al., in their paper "Meta-learning with latent embedding optimization," proposed a method for optimizing low-dimensional latent embedding spaces, addressing the challenge of learning high-dimensional parameters in MAML with a limited number of samples. Baik et al., in their paper "Meta-learning with task-adaptive loss function for few-shot learning," introduced a powerful auxiliary loss function to facilitate inner and outer loop optimization. Another type, metric-based methods, uses a self-designed distance function to compare the similarity between samples. Snell et al., in their paper "Prototypical networks for few-shot learning," proposed a prototype network that performs classification by measuring the distance between the query sample embedding and the class prototype. In their paper "Few-shot learning with graph neural networks," Garcia et al. proposed graph neural networks, which measure the similarity between different samples by the weights of edges in the feature graph. Another type of transfer-based method transfers prior knowledge learned in the source domain to the target domain. Shen et al., in their paper "Partialis better than all: revisiting fine-tuning strategy for few-shot learning," proposed selectively transferring knowledge from specific layers of the model. Zhang et al., in their paper "Tip-adapter: Training-free adaptation of clip for few-shot classification," utilized strong prior knowledge from large visual language models (such as CLIP) and generalized it to the target domain through fine-tuning.

[0003] There has been considerable research in recent years on cross-domain few-shot object classification algorithms based on deep learning. One approach involves simulating multi-domain training methods by enriching data at different levels to mimic multi-domain training. Specifically, Tseng et al., in their paper "Cross-domain few-shot classification via learned feature-wise transformation," enhanced features by incorporating a feature-level affine transformation layer into the feature extractor. Meanwhile, Liang et al., in their paper "Boosting the generalization capability in cross-domain few-shot learning via noise-enhanced supervised auto-encoder," used a noisy autoencoder to enhance image-level input data. Recent studies, such as Wang et al.'s "Cross-domain few-shot classification via adversarial task augmentation" (ATA), Fu et al.'s "Styleadv: Meta-style adversarial training for cross-domain few-shot learning" (StyleAdv) through meta-style adversarial training, and Hu et al.'s "Adversarial feature augmentation for cross-domain few-shot classification" (AFA) through adversarial feature augmentation, generate pseudo-domain data at the task, style, and feature levels to expand the effective data distribution. On the other hand, domain-invariant feature learning is achieved using feature constraints or alignment. For example, Tian et al.'s "Revisiting prototypical network for cross-domain few-shot learning" (LDP) improves generalization ability by regularizing the relationships between samples. Some studies introduce additional parameter adapters (such as task-aware adversarial networks) or utilize cross-modal information for constraints, such as mapping text prototypes to visual prototypes or introducing diverse text descriptions to enhance visual feature diversity. Feature alignment techniques aim to minimize the difference between the feature distributions of the source and target domains.

[0004] Cue learning techniques based on large-scale vision-language models have become a recent research hotspot. These methods leverage strong semantic priors from pre-trained models (such as CLIP) to enhance classification by dynamically generating textual cues.

[0005] In their paper "Learning to prompt for vision-language models," Zhou et al. proposed a context optimization method to improve zero-shot performance by learning static text prompt templates. To enhance adaptability to new classes, Zhou et al., in their paper "Conditional prompt learning for vision-language models," further generated instance-level adaptive prompts. Liu et al., in their paper "Stylip: Prompt tuning with optimal style for zero-shot compositional generalization," focused on decoupling content and style prompts, utilizing domain-independent prompts to meet domain generalization requirements.

[0006] In practical applications, acquiring large amounts of labeled remote sensing datasets to supervise network training is often expensive and time-consuming. Furthermore, due to factors such as imaging equipment, lighting conditions, and weather, there is significant domain shift between the training dataset (source domain) and the test dataset (target domain). To address these challenges, existing methods struggle to overcome the inherent high intra-class visual blur and low inter-class discriminability of remote sensing images. Additionally, feature alignment or constraint methods are prone to distorting discriminative information under complex domain shifts. Static cueing techniques cannot dynamically adapt to tasks with few samples, and noise in the support set can impair cross-domain generalization ability.

[0007] Therefore, there is an urgent need for a new method that can dynamically generate discriminative semantic prompts, accurately represent category attributes, and effectively eliminate instance-specific noise interference. The discriminative text prompt learning method of this invention addresses these problems. Summary of the Invention

[0008] To address the problem of remote sensing target classification under conditions of limited cross-domain samples, this invention proposes a method to enhance feature representation through discriminative textual prompt learning. For example... Figure 1 As shown, this method leverages the domain invariance of text modalities to accurately represent category attributes by dynamically generating semantic cues, and combines intra-class contrastive learning to eliminate instance noise interference.

[0009] The technical solution of this invention:

[0010] A cross-domain few-shot remote sensing target classification method based on discriminative text prompt learning, comprising the following steps:

[0011] (1) Semantic prompt generation

[0012] First, one of the following datasets—DIOR remote sensing dataset, DOTA dataset, HRRSD remote sensing image dataset, NWPU VHR-10 remote sensing image dataset, and xView dataset—is used as the original training dataset, and the remaining four are used as test datasets.

[0013] The original training dataset was then divided into several pseudo-support sets. and pseudo query sets The CLIP model is trained and updated in batches based on groups; for any given pseudo-support set... and pseudo query sets , will pseudo support set Input frozen CLIP image encoder (e.g., ResNet50) to extract features , will pseudo query set Input frozen CLIP image encoder extracts features Then calculate the features. The average features of all samples under each category are used as the prototype of that category to obtain a pseudo support set category prototype containing all categories. For pseudo-support set category prototypes For any category in the dataset, a learnable cue generator G maps the category prototype to M dynamic cue word vectors. Where k=1,2,3,...,M:

[0014] (1)

[0015] The learnable cue generator G contains two linear layers and a ReLU activation function;

[0016] Then concatenate the dynamic prompt word vectors With pseudo-support set The [CLASS] category label is used for all categories, where CLASS is the English name of all categories, forming a discriminative semantic cues:

[0017] (2)

[0018] Since the average features obtained from multiple samples can summarize the salient characteristics of a category, the text prompts obtained through the prompt generator can effectively describe the semantic information of the category. Therefore, this discriminative semantic prompt S accurately encodes the essential attributes of the category in the current task. This discriminative semantic prompt S is processed by the CLIP text encoder to obtain text embeddings t containing all categories. Subsequently, the image-text matching similarity is calculated using the CLIP model, outputting the true category prediction probability.

[0019] (3)

[0020] in, For the text embedded in t, the first Text embeddings in each category and For the text embedding of the j-th category contained in the text embedding t; Using the cosine function, calculate text embeddings and features. Cosine similarity; For exponentiation with base e;

[0021] Under cross-entropy CE, the true class label Y in the original training dataset guides the learning of the true class prediction probability P, thereby updating the cue generator parameters and ensuring that the discriminative semantic cue can correctly guide the classification task, as shown below:

[0022] (4)

[0023] (2) Intra-class comparative learning;

[0024] To address instance-specific bias introduced by noise in the support set samples, we force similar prompts to remain compact in the feature space. Specifically, we use multiple text prompt embeddings generated from different samples in different batches within the same category as positive sample pairs. This balances the accuracy of descriptions with multiple prompts and ensures intra-class consistency through contrastive learning, suppressing instance noise. Text prompt embeddings from other categories are used as negative sample pairs. Contrastive loss is employed. Constraints are applied to keep the distance between positive and negative sample pairs within the hyperparameter margin, thereby obtaining class-specific discriminative cues and filtering out instance-specific noise.

[0025] (5)

[0026] in, For any sample, for Relevant positive samples, for The relevant negative samples, where ReLU is the activation function, dist is the L2 distance function, and margin is the control distance between positive and negative sample pairs;

[0027] (3) Optimization and classification decision-making;

[0028] The overall loss function includes classification loss. Compared with loss To update the discriminative hints and hint generator parameters:

[0029] (6)

[0030] When making classification decisions, calculate the features extracted from the pseudo-query set separately. With text embedding t, pseudo-support set category prototype The cosine similarity score between the two values ​​is used as the final classification result.

[0031] The beneficial effects of this invention are as follows: For cross-domain, few-sample remote sensing target classification tasks, the discriminative text prompt learning method proposed in this invention achieves significant performance improvements through three technological innovations: 1. First, it overcomes the limitations of single-modal learning, achieving enhanced cross-domain invariance. Traditional methods (such as StyleAdv and ATA) rely solely on visual features and are easily affected by domain bias. This invention utilizes a visual-text dual-modal collaborative mechanism, and the semantic prompts generated by the CLIP text encoder possess inherent domain invariance, significantly outperforming mainstream single-modal methods, proving that text semantics effectively compensate for the sensitivity defects of the visual domain. 2. Next, it dynamically generates discriminative semantic prompts, accurately representing category attributes and adapting to the characteristics of remote sensing categories. This invention designs a semantic prompt generator (SPG), which dynamically generates prompts through support set prototypes. Prompts of the same category are highly aggregated in the feature space, while prompts of different categories are significantly separated. 3. Finally, intra-class contrastive learning suppresses instance noise and improves cross-domain robustness. To address instance bias introduced by support set sample noise, this invention proposes intra-class invariance constraints. By constructing contrastive learning for multiple samples of the same category, instance bias is filtered out, improving the model's domain generalization ability. Attached Figure Description

[0032] Figure 1 This is a diagram of the overall network structure. Detailed Implementation

[0033] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0034] This method uses the DIOR remote sensing dataset as the original training dataset and validates its classification performance on four target domain datasets—DOTA, HRRSD, NWPU VHR-10, and xView—in a cross-domain scenario. The implementation process comprises five key stages: First, the DIOR remote sensing dataset is divided into a training set (15 classes), a validation set (5 classes), and a test set (5 classes) in a 3:1:1 ratio to ensure strict segregation of the class space. In each training task, five classes are randomly selected (5-way), and 1 or 5 samples are sampled from each class (1-shot / 5-shot) to construct a pseudo-support set. Simultaneously, one sample is drawn from each class as a pseudo-query set. This partitioning method simulates the data distribution characteristics in real-world scenarios with few samples.

[0035] The samples from the pseudo-support set are then input into a frozen CLIP image encoder (ResNet50 backbone) to extract features. The mean feature value of each class is calculated as the class prototype. A cue generator consisting of two fully connected layers maps the class prototype to four dynamic word vectors. These dynamic word vectors are concatenated with the [CLASS] classification labels of all classes in the pseudo-support set to form discriminative semantic cues, accurately representing the core attributes of the class. This process abandons fixed templates and adaptively captures the discriminative semantics of the support set samples. To address the instance bias problem caused by noise in the support set samples, multiple sets of heterogeneous cues are generated for the same class. Positive sample pairs (same class cues) and negative sample pairs (different class cues) are constructed in the projected feature space. A temperature-modulated contrastive loss function is used to optimize positive samples, i.e., to bring the same class cues closer together, compress the intra-class feature manifold and the constraints of negative samples, and push away different class cues, thus expanding the decision boundary. Specifically, this mechanism makes the "airport" class cues tightly aggregated in the feature space, while significantly separating them from the "train station" cues, effectively filtering instance noise. Specifically, Table 1 verifies the role of semantic cue generation and intra-class contrastive learning.

[0036] Table 1 shows the recognition accuracy of the 5-way 5-shot task using different modules trained on the DOTA dataset in this invention.

[0037]

[0038] Finally, a weighted fusion of image-to-image similarity and image-to-text similarity is used to form the final score, achieving 100% confidence in the helipad category of the xView dataset. This design considers both semantic priors and visual details, showing a significant improvement over single-modal methods. In cross-domain few-shot performance validation, in cross-domain tests (5-way 5-shot) trained on the DIOR remote sensing dataset: the average classification accuracy reached 77.38%, showing a significant improvement over state-of-the-art methods, as shown in Table 2.

[0039] Table 2 Comparison of 5-way 5-shot task recognition accuracy of the present invention trained on the DIOR remote sensing dataset

[0040] .

Claims

1. A cross-domain few-shot remote sensing target classification method based on discriminative text prompt learning, characterized in that, The steps are as follows: (1) Semantic prompt generation; First, one of the following datasets—DIOR remote sensing dataset, DOTA dataset, HRRSD remote sensing image dataset, NWPU VHR-10 remote sensing image dataset, and xView dataset—is used as the original training dataset, and the remaining four are used as test datasets. Then, the original training dataset is divided into several pseudo-support sets. and pseudo query sets The CLIP model is trained and updated in batches based on groups; for any given pseudo-support set... and pseudo query sets , will pseudo support set Input frozen CLIP image encoder extracts features , will pseudo query set Input frozen CLIP image encoder extracts features Then calculate the features. The average features of all samples under each category are used as the prototype of that category to obtain a pseudo support set category prototype containing all categories. ; For pseudo-support set category prototypes For any category in the dataset, a learnable cue generator G maps the category prototype to M dynamic cue word vectors. Where k=1,2,3,...,M: （1） The learnable cue generator G contains two linear layers and a ReLU activation function; Then concatenate the dynamic prompt word vectors With pseudo-support set The [CLASS] category label is used for all categories, where CLASS is the English name of all categories, forming a discriminative semantic cues: （2） The discriminative semantic cue S is processed by the CLIP text encoder to obtain a text embedding t containing all categories. Then, the image-text matching similarity is calculated based on the CLIP model, and the true category prediction probability is output. （3） in, The first text embedded in t contains the first... Text embeddings in each category and For the text embedding of the j-th category contained in the text embedding t; Using the cosine function, calculate text embeddings and features. Cosine similarity; For exponentiation with base e; Under cross-entropy CE, the true class label Y in the original training dataset guides the learning of the true class prediction probability P, thereby updating the cue generator parameters and ensuring that the discriminative semantic cue can correctly guide the classification task, as shown below: （4） (2) Intra-class comparative learning; Multiple text prompt embeddings generated from different samples in different batches within the same category are used as positive sample pairs, while text prompt embeddings from other different categories are used as negative sample pairs; a contrastive loss is employed. Constraints are applied to keep the distance between positive and negative sample pairs within the hyperparameter margin, thereby obtaining class-specific discriminative cues and filtering out instance-specific noise. （5） in, For any sample, for Relevant positive samples, for The relevant negative samples, where ReLU is the activation function, dist is the L2 distance function, and margin is the control distance between positive and negative sample pairs; (3) Optimization and classification decision-making; The overall loss function includes classification loss. Compared with loss To update the discriminative hints and hint generator parameters: （6） When making classification decisions, the features extracted from the pseudo-query set are calculated separately. With text embedding t, pseudo-support set category prototype The cosine similarity score between the two values ​​is used as the final classification result.

Images