A zero-shot open-set remote sensing scene classification method based on multi-modal prototypes

Abstract

This invention discloses a zero-shot open-set remote sensing scene classification method based on multimodal prototypes. It primarily overcomes the limitations of traditional methods in complex remote sensing scenes, which suffer from low classification accuracy and poor unknown class rejection capabilities due to alignment differences between visual and linguistic modalities. This invention expands the semantic space from broad and singular to comprehensive and refined by introducing surrogate unknown class labels and fine-grained textual prompts. Simultaneously, it utilizes high-confidence samples to construct visual prototypes reflecting the true visual distribution, and dynamically reconstructs textual prototypes based on these prototypes. During inference, bidirectional alignment between visual and textual data effectively balances accurate classification of known remote sensing scenes with robust rejection of unknown interference items, facilitating efficient and secure model deployment in remote sensing applications with unlabeled data and dynamically changing environments.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision and remote sensing image processing technology, and specifically relates to a method for zero-sample open set remote sensing scene classification using a test-time adaptive strategy without training samples. Background Technology

[0002] Zero-shot open-set remote sensing scene classification plays a crucial role in practical applications such as urban planning, environmental monitoring, disaster assessment, and military reconnaissance. In these fields, constructing a large-scale, high-quality labeled dataset covering all potential scene categories presents significant challenges. Remote sensing data acquisition is costly, and pixel-level or image-level professional annotation of images requires substantial manpower and expert knowledge. The core value of zero-shot open-set remote sensing scene classification lies primarily in its ability to significantly alleviate the reliance on large-scale labeled data. By utilizing the semantic information of category labels to replace visual samples for reasoning, this technique enables models to achieve classification even in the complete absence of training samples for the target category. This greatly reduces the data cost and barrier to model deployment, solving the problem of ineffective identification due to data scarcity or labeling difficulties in practical applications.

[0003] Furthermore, the actual remote sensing observation environment is a highly complex and dynamic open space. During the inference process, the image stream to be classified will inevitably contain unknown samples that do not belong to any predefined known category. Traditional closed-set models, lacking awareness of unknown categories, often force these interfering items to be classified into a known category with high confidence, leading to serious misjudgments and affecting the safety of decision-making. Another core value of zero-shot open-set remote sensing scene classification tasks lies in their ability to reject the unknown. That is, while accurately identifying known categories, they can determine abnormal samples that do not belong to the known distribution as unknown, thereby ensuring the robustness and reliability of the system in open environments.

[0004] However, existing recognition techniques are mainly designed for natural images. For example, Ming et al. in “Y.Ming, Z.Cai, J.Gu, Y.Sun, W.Li, and Y.Li, “Delving into out-of-distributiondetection with vision-language representations,”Advances in NeuralInformationProcessing Systems, vol.35, pp.35087-35102, 2022.” and Liu et al. in “X.Liu and C.Zach, “Tag:Text prompt augmentation for zero-shot out-of-distribution detection,” in Proceedings of the European Conference on ComputerVision, pp.364-380, 2024” respectively utilize the maximization of concept matching strategy and the text description random recombination augmentation method to improve the model’s ability to distinguish known classes in natural image datasets. Jiang et al., in their paper "X.Jiang,F.Liu,Z.Fang,H.Chen,T.Liu,F.Zheng, andB.Han, “Negative label guided OODdetection with pretrained vision-language models,” in Proceedings of the International Conference on Learning Representations, 2024," defined the unknown space by selecting labels from WordNet that were semantically furthest from natural image categories as negative samples.

[0005] Directly applying these methods to the field of remote sensing faces significant challenges. Due to the unique aerial perspective, complex textures of ground features, and diverse background interference in remote sensing images, there is a significant modal alignment gap between their visual features and the general textual semantics upon which pre-trained models rely. Most existing methods rely on predefined static textual prototypes for feature matching; this static, singular semantic description is insufficient to fully represent the vast intra-class differences in remote sensing scenes. This lack of alignment between static text and dynamic vision makes it difficult for existing models to strike a balance between accurate classification of known classes and effective rejection of unknown classes in complex remote sensing scenarios, limiting their performance in practical applications. Summary of the Invention

[0006] This invention discloses a zero-shot open-set remote sensing scene classification method based on multimodal prototypes, mainly addressing the problems of low classification accuracy and poor unknown class rejection ability in existing technologies due to alignment differences between visual and linguistic modalities in remote sensing scenes. Specifically, the objective of this invention is: 1. By constructing a text space that includes proxy unknown classes and fine-grained attributes, the problem of standard category label information being too broad and unable to identify unknown classes is solved.

[0007] 2. Abandoning the single text modality discrimination, we propose a test-time adaptive mechanism that uses high-confidence samples to construct visual prototypes online and dynamically reweights text prototypes using visual feedback to achieve bidirectional alignment between visual and text, thereby significantly improving recognition and rejection performance in zero-sample open set environments.

[0008] This invention proposes a zero-shot open-set remote sensing scene classification method based on multimodal prototypes, comprising the following steps: Generate a proxy set of unknown categories based on the text labels of a known set of categories; Calculate the cosine similarity between the image feature vector and the static text prototype to obtain the initial probability of the input image on the union of the known category set and the proxy unknown category set; For high-confidence image samples, for each category, the visual prototype feature vector of the category is calculated to generate a dynamic text prototype; for low-high-confidence image samples, the graph-graph similarity is calculated using the visual prototype feature vector. The final similarity score is obtained by fusing cosine similarity and graph-to-graph similarity. Calculate the Softmax probability of the low-confidence image sample pair for each class based on the final similarity score, and calculate the open set score. If the open set score is greater than or equal to the open set threshold, the sample is classified into the corresponding known class; otherwise, it is rejected as an unknown class.

[0009] Furthermore, by utilizing a large language model, through all known category sets... The text labels are used to generate a set of candidate agents for unknown categories, and the maximum cosine similarity between these candidates and the text features of known categories is calculated. Candidates with similarity scores higher than a threshold are then filtered out. The category is used to obtain the agent's unknown category set. .

[0010] Furthermore, a large language model is used for each category. Generate a set of prompts ,in Given a set of known categories, To proxy an unknown category set, the cue set is input into a CLIP pre-trained text encoder based on the Transformer architecture, which maps the input text into semantic feature vectors and averages them to obtain a static text prototype. For input image samples By using a CLIP pre-trained image encoder based on the Transformer architecture, input image samples are mapped into image feature vectors. The cosine similarity between the image feature vector and the static text prototype is calculated using the following formula:

[0011] Furthermore, image samples are then calculated using cosine similarity. In category collection The initial probability is: in, Representative image sample Category The probability of.

[0012] Furthermore, if the maximum probability in the initial probability of the image sample is greater than the threshold... If a sample is identified as a high-confidence sample, it will be added to the image sample pool of the corresponding category. Select the sample with the highest similarity to the original text from the sample pool. Image samples, for each category Construct its representative image sample set Subsequently, the mean value of image features is calculated using a representative image sample set to obtain the visual prototype feature vector of the category. The calculation formula is as follows: , in, This represents the selected image sample; Image feature vectors are image samples The input is obtained by CLIP pre-trained image encoder based on the Transformer architecture.

[0013] Furthermore, utilizing visual prototype feature vectors The aforementioned set of prompts generated by the large language model Perform reweighting; calculate the visual prototype and the semantic feature vector obtained through the aforementioned text encoder. cosine similarity The weights are obtained through Softmax normalization. ,in, Indicates the first in the prompt set Tip: Utilize weights The prompt features in the prompt set are weighted and summed to generate a dynamic text prototype. .

[0014] Furthermore, if the maximum probability in the initial probability of the image sample is lower than the threshold... If it is a low-confidence sample, then it is judged as a low-confidence sample. Calculate graph-graph similarity using visual prototype feature vectors The calculation formula is as follows:

[0015] Calculating dynamic image-text similarity using dynamic text prototypes The calculation formula is as follows:

[0016] in This represents the calculation of cosine similarity.

[0017] Furthermore, calculate low-confidence image samples. Category Softmax probability The formula is as follows:

[0018] in, This represents the final similarity score.

[0019] Furthermore, open set fractions Low-confidence image samples Given the highest probability in the category, the calculation formula is as follows: .

[0020] The present invention also provides a computer-readable storage medium, including a readable storage medium on which a computer program is stored, wherein when the computer program is executed, it implements the zero-sample open set remote sensing scene classification method based on multimodal prototypes as described in the above technical solution.

[0021] This invention has the following significant advantages over existing zero-sample open set identification methods: Significantly improved recognition accuracy and robustness: This invention bridges the modal gap between remote sensing images and general text through a dynamic modal alignment mechanism. Extensive experiments on the large-scale remote sensing image scene classification dataset (RESISC45) and the aerial image dataset (AID) demonstrate that the performance of this invention is improved by 11.10% and 7.30% respectively in overall accuracy compared to the current state-of-the-art comparative methods. Attached Figure Description

[0022] To more clearly illustrate the technical solution of the present invention, the various modules involved in the embodiment description are briefly introduced below. Obviously, the accompanying drawings in the following description are merely flowcharts of the present invention. For those skilled in the art, these drawings can be extended to generate other related drawings without any creative effort.

[0023] Figure 1 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0025] This invention proposes a zero-shot open-set remote sensing scene classification method based on multimodal prototypes. This method, without requiring any remote sensing images for training, constructs an open-set text space, adaptively builds visual prototypes, reweights the category text prototypes, and finally achieves accurate classification of known classes and effective rejection of unknown classes through maximum Softmax probability scoring. The specific technical solution is as follows: 1. Constructing an Open-Set Text Space. To address the lack of prior knowledge for unknown classes and the broad semantics of standard labels, a large language model is used to generate proxy unknown class labels that are related to known class concepts but semantically distant, providing the model with prior knowledge about the unknown. Simultaneously, the large language model is used to generate fine-grained text prompts for each category, constructing an information-rich initial text space.

[0026] 2. Adaptive Visual Prototype Construction. To address the issue of static text failing to adapt to dynamic visual features, preliminary predictions are first made on the test samples, and high-confidence samples are selected to construct a sample pool. By aggregating the image features with the highest similarity in the sample pool, a visual prototype for each category is calculated. Using this visual prototype, the classification criteria for low-confidence samples are expanded to a weighted fusion of image-to-image similarity and dynamic image-to-text similarity, enhancing the robustness of the judgment.

[0027] 3. Reweighting Category Text Descriptions. To address the issue of inconsistent description quality generated by large models, a feedback loop from visual to textual representation is constructed. Using the visual prototype of a category as an anchor, its similarity to each text prompt is calculated, and the text prompts are dynamically reweighted accordingly. The generated dynamic text prototypes adaptively highlight semantic descriptions that better align with the current visual data distribution, thereby achieving accurate alignment of cross-modal features.

[0028] 4. Open Set Score Calculation and Classification Decision. For low-confidence samples, image-image similarity and image-text similarity are calculated using visual prototypes and dynamic text prototypes, respectively. The maximum Softmax probability of the known class is used as the open set score. If the open set score of a sample is higher than a preset open set threshold, it is classified into the corresponding known class; otherwise, it is rejected as an unknown class.

[0029] In summary, this invention is a zero-shot open-set remote sensing scene classification method based on multimodal prototypes. It primarily overcomes the limitations of traditional methods in complex remote sensing scenes, which suffer from low classification accuracy and poor unknown class rejection capabilities due to alignment differences between visual and linguistic modalities. This invention expands the semantic space from broad and singular to comprehensive and refined by introducing surrogate unknown class labels and fine-grained textual prompts. Simultaneously, it utilizes high-confidence samples to construct visual prototypes reflecting the true visual distribution, and dynamically reconstructs textual prototypes based on these prototypes. During inference, bidirectional alignment between visual and textual data effectively balances accurate classification of known remote sensing scenes with robust rejection of unknown interference items, facilitating efficient and secure model deployment in remote sensing applications with unlabeled data and dynamically changing environments.

[0030] Reference Figure 1 The specific implementation steps of this invention are as follows: Step 1: First, utilize the large language model, through all known classes The text labels are used to generate a set of candidate agents for unknown categories, and the maximum cosine similarity between these candidates and the text features of known categories is calculated. Candidates with similarity scores higher than a threshold are then filtered out. The category is used to obtain the agent's unknown category set. .

[0031] Step 2: Utilize a large language model for each category Generate a set of prompts The cue set is input into a CLIP pre-trained text encoder based on the Transformer architecture, which maps the input text into semantic feature vectors and averages them to obtain a static text prototype. For the input image By using a CLIP pre-trained image encoder based on the Transformer architecture, the input image is mapped into an image feature vector. The cosine similarity between the image feature vector and the static text prototype is calculated using the following formula: .

[0032] Then, the image was calculated using cosine similarity. In category collection The initial probability is: in, Representative image Category The probability of.

[0033] Step 3: If the highest probability in the initial probability of the image sample is greater than or equal to the threshold If a sample is identified as a high-confidence sample, it will be added to the image sample pool of the corresponding category. Select the sample with the highest similarity to the text prototype from the sample pool. Image samples, for each category Construct its representative image sample set Subsequently, the mean value of image features was calculated using a representative image sample set to obtain the visual prototype feature vector of the category. The calculation formula is as follows: , in, This represents the selected image sample; Step 4: Utilize visual prototype feature vectors The aforementioned set of prompts generated by the large language model Reweighting is then performed. The computational visual prototype is compared with the semantic feature vector obtained through the aforementioned text encoder. (This refers to the cosine similarity obtained by inputting the i-th prompt into the text encoder) The weights are obtained through Softmax normalization. ,in, Indicates the first in the prompt set Tip. Utilize weights. The feature vectors of each prompt are weighted and summed to generate a dynamic text prototype. .

[0034] Step 5: In step 3, if the highest probability in the initial probability of the image sample is lower than the threshold... If the sample is not found to be of low confidence, it is considered a low-confidence sample. Calculate graph-graph similarity using visual prototype feature vectors The calculation formula is as follows: .

[0035] Calculating dynamic image-text similarity using dynamic text prototypes The calculation formula is as follows:

[0036] in This represents the calculation of cosine similarity.

[0037] Through hyperparameters The final similarity score is obtained by fusing the two similarity scores. The calculation formula is as follows: .

[0038] Step 6: Calculate low-confidence image samples Category The Softmax probability is given by the following formula:

[0039] Then calculate the open set fraction. Its score is the image sample The formula for calculating the highest probability among known categories is as follows: .

[0040] like Greater than or equal to the open set threshold If the condition is met, the sample is classified into the corresponding known class; otherwise, it is rejected as an unknown class.

[0041] The effects of this invention can be further illustrated by the following experiments.

[0042] 1. Experimental conditions This invention was simulated using a Gen Intel® Core™ i9-10920X CPU, an NVIDIA RTX 3080 GPU, 32GB of RAM, and a Linux operating system, with a model built using the PyTorch deep learning framework. To comprehensively evaluate the performance of this invention, the performance of the proposed method was verified on the large-scale remote sensing image scene classification dataset (RESISC45) and the aerial image dataset (AID). RESISC45 contains remote sensing image data of 45 categories; to simulate a zero-shot open set scenario, this invention adopted a 2:1 category splitting strategy, where 30 categories were designated as known classes for constructing the semantic space and distribution prototype, and the remaining 15 categories were designated as unknown classes to test the model's ability to reject unseen categories. AID includes 30 scene categories; similarly, a 2:1 splitting principle was adopted, selecting 20 categories as known classes and the remaining 10 as unknown classes. The experiments strictly adhered to the zero-shot setting, using no images for training and relying solely on the textual semantic information of the known classes for inference testing.

[0043] 2. Experiment Content Considering the popularity of zero-shot open set algorithms, five deep learning-based zero-shot open set methods were selected from RESISC35 and AID for performance comparison to verify the performance of the probability distribution matching mechanism of this invention. This invention uses overall accuracy to quantify the performance of each method. The overall accuracy of the method of this invention and the comparison algorithms are shown in Tables 1 and 2 below: Table 1 Overall accuracy of each method in RESISC45

[0044] Table 2 Overall accuracy of each method in AID

[0045] As shown in Tables 1 and 2, the zero-shot open-set remote sensing scene classification method based on multimodal prototypes of this invention exhibits the best classification performance. This is because the proposed multimodal prototype framework first utilizes a large language model to generate surrogate unknown class labels and fine-grained textual prompts, constructing an information-rich open-set text space. This effectively overcomes the problems of broad semantics and lack of unknown prior knowledge in traditional methods. Simultaneously, it abandons the single-modal discrimination paradigm that relies solely on static text prototypes, instead using high-confidence samples to adaptively construct visual prototypes, and dynamically reweighting text prototypes based on these prototypes, achieving precise bidirectional alignment of visual and textual features. Therefore, the method of this invention can more accurately identify known remote sensing scenes and robustly reject unknown interference terms, significantly outperforming comparative methods. Comparative experiments on public datasets further demonstrate the advanced effectiveness of the method.

[0046] The comparison algorithms are derived from the following literature: [1] "Y.Ming, Z.Cai, J.Gu, Y.Sun, W.Li, and Y.Li, "Delving into out-of-distribution detection with vision-language representations," Advances in Neural Information Processing Systems, vol.35, pp.35087-35102, 2022." [2] "X.Liu and C.Zach, "Tag:Text prompt augmentation for zero-shotout-of-distribution detection," in Proceedings of the European Conference on Computer Vision, pp.364-380, 2024." [3] "C.Cao, Z.Zhong, Z.Zhou, Y.Liu, T.Liu, and B.Han, "Envisioningoutlier exposure by large language models for out-of-distribution detec-tion," in Proceedings of the 41th International Conference on MachineLearning, pp.5629-5659, 2024." [4] "X. Jiang, F. Liu, Z. Fang, H. Chen, T. Liu, F. Zheng, and B. Han, "Negative label guided OOD detection with pretrained vision-language models," in Proceedings of the International Conference on Learning Representations, 2024." [5] "A.Miyai, Q.Yu, G.Irie, and K.Aizawa, "Gl-mcm: Global and localmaximum concept matching for zero-shot out-of-distribution detection," International Journal of Computer Vision, vol.133, no.6, pp.3586- 3596, 2025." On the other hand, embodiments of the present invention also provide a computer-readable storage medium, including a readable storage medium on which a computer program is stored, wherein when the computer program is executed, it implements the zero-sample open set remote sensing scene classification method based on multimodal prototypes as described in the above technical solution.

[0047] The above description is merely a specific embodiment of the present invention, used to illustrate the technical solution of the present invention and not to limit it. Those skilled in the art can make various modifications, equivalent substitutions, or additions to the above embodiments without departing from the spirit and scope of the claims, all of which should fall within the protection scope of the present invention.

Claims

1. A zero-shot open-set remote sensing scene classification method based on multimodal prototypes, characterized in that, Includes the following steps: Generate a proxy set of unknown categories based on the text labels of a known set of categories; Calculate the cosine similarity between the image feature vector and the static text prototype to obtain the initial probability of the input image on the union of the known category set and the proxy unknown category set; For high-confidence image samples, for each category, the visual prototype feature vector of the category is calculated to generate a dynamic text prototype; for low-high-confidence image samples, the graph-graph similarity is calculated using the visual prototype feature vector. The final similarity score is obtained by fusing cosine similarity and graph-to-graph similarity. Calculate the Softmax probability of the low-confidence image sample pair for each class based on the final similarity score, and calculate the open set score. If the open set score is greater than or equal to the open set threshold, the sample is classified into the corresponding known class; otherwise, it is rejected as an unknown class.

2. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 1, characterized in that: Using a large language model, through all known category sets The text labels are used to generate a set of candidate agents for unknown categories, and the maximum cosine similarity between these candidates and the text features of known categories is calculated. Candidates with similarity scores higher than a threshold are then filtered out. The category is used to obtain the agent's unknown category set. .

3. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 1, characterized in that: Utilize large language models for each category Generate a set of prompts ,in Given a set of known categories, To proxy an unknown category set, the cue set is input into a CLIP pre-trained text encoder based on the Transformer architecture, which maps the input text into semantic feature vectors and averages them to obtain a static text prototype. For input image samples By using a CLIP pre-trained image encoder based on the Transformer architecture, input image samples are mapped into image feature vectors. The cosine similarity between the image feature vector and the static text prototype is calculated using the following formula: .

4. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 3, characterized in that: Subsequently, image samples were calculated using cosine similarity. In category collection The initial probability is: in, Representative image sample Category The probability of.

5. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 1, characterized in that: If the highest probability in the initial probability of an image sample is greater than a threshold If a sample is identified as a high-confidence sample, it will be added to the image sample pool of the corresponding category. Select the sample with the highest similarity to the original text from the sample pool. Image samples, for each category Construct its representative image sample set Subsequently, the mean value of image features is calculated using a representative image sample set to obtain the visual prototype feature vector of the category. The calculation formula is as follows: ， in, This represents the selected image sample; Image feature vectors are image samples The input is obtained by CLIP pre-trained image encoder based on the Transformer architecture.

6. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 3, characterized in that: Using visual prototype feature vectors The aforementioned set of prompts generated by the large language model Perform reweighting; calculate the visual prototype and the semantic feature vector obtained through the aforementioned text encoder. cosine similarity The weights are obtained through Softmax normalization. ,in, Indicates the first in the prompt set Tip: Utilize weights The prompt features in the prompt set are weighted and summed to generate a dynamic text prototype. .

7. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 1, characterized in that: If the highest probability in the initial probability of an image sample is lower than a threshold If it is a low-confidence sample, then it is judged as a low-confidence sample. Calculate graph-graph similarity using visual prototype feature vectors The calculation formula is as follows: Calculating dynamic image-text similarity using dynamic text prototypes The calculation formula is as follows: in This represents the calculation of cosine similarity.

8. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 1, characterized in that: Calculate low-confidence image samples Category The Softmax probability is given by the following formula: in, This represents the final similarity score.

9. The zero-shot open-set remote sensing scene classification method based on multimodal prototypes as described in claim 1, characterized in that: Open set fractions Low-confidence image samples The formula for calculating the highest probability among known categories is as follows: 。 10. A computer-readable storage medium, characterized in that, It includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements the zero-sample open-set remote sensing scene classification method based on multimodal prototypes as described in any one of claims 1-8.

Images