Remote sensing image classification method based on unsupervised cache model and semantic regular weight

Abstract

The invention discloses a remote sensing image classification method based on an unsupervised cache model and semantic regular weight. The method comprises the following steps of: 1, constructing an unsupervised cache model by using a high-confidence pseudo tag sample acquired by a pre-training model; 2, based on an unsupervised cache model, performing semantic similarity calculation on a cache sample and a test sample, and converting the cache sample and the test sample into a semantic regular weight; and step 3, normalizing similarity calculation among test cache samples by using the semantic normalizing weight obtained by calculation in the step 2, making up for the deficiency of diversity of an unsupervised cache model, realizing deep fusion of semantic information and image information, and then optimizing an original model prediction probability through residual connection. And step 4, using a strong and weak enhancement strategy, obtaining a pseudo label based on the optimized prediction probability in the step 3, and optimizing an extra adapter by using supervision information obtained by the strong prediction capability of the pre-training model and the cache model while keeping the generalization performance of the original model.

Description

Technical Field

[0001] This invention relates to unsupervised caching models, semantic regularization weights, strong and weak enhancement strategies, and unsupervised remote sensing image classification. Background Technology

[0002] Remote sensing image classification refers to the technical process of assigning semantic labels to remote sensing images based on their visual content. This technology has wide-ranging applications, primarily in urban monitoring and management, disaster detection and prevention, and applications in the military and agricultural fields. However, traditional image recognition models heavily rely on supervised learning and are facing the challenge of a scarcity of high-quality labeled data. The large intra-class differences and small inter-class differences in remote sensing images make this challenge particularly pronounced in the field of remote sensing. To overcome this fundamental bottleneck of supervised learning, unsupervised learning has become an important research direction.

[0003] As a promising solution, researchers are increasingly focusing on the application of large-scale visual language models in unsupervised learning scenarios. These models, trained on massive image-text pairs with a contrastive learning objective, possess exceptional representation learning capabilities and exhibit strong generalization performance across diverse downstream tasks. A representative achievement is the Contrastive Language Image Pre-trained Model (CLIP), which achieves remarkable zero-shot classification performance directly using hand-designed cue templates. CLIP's powerful zero-shot capability is a key driving force behind recent breakthroughs in unsupervised learning.

[0004] Recent unsupervised prompting learning optimization frameworks leverage CLIP's zero-shot capability to improve the efficiency of visual language models in adapting to different downstream tasks while eliminating the need for data labeling. These methods typically fine-tune the model or learnable prompts directly on unlabeled data: the UPL method uses pseudo-labels generated by a pre-trained visual language model to select the K samples with the highest confidence in each category for full model tuning; the POUF method treats the text prompt representations of specific categories as category prototypes and further aligns them with the features of the target image in the latent space.

[0005] These methods often underestimate the importance of the generalization ability of pre-trained CLIP models, which is crucial for achieving robust performance across downstream datasets. By focusing excessively on optimizing CLIP performance on specific datasets, they may overlook the importance of learning general features for handling unknown data. Furthermore, directly using text prompts as parameters for optimization severely limits the scalability of generating prompts for CLIPs using large language models. In contrast, LaFTer uses a large language model to generate a set of prompts for each category as labeled data to train a text classifier. It then retrains the model using image pseudo-labels obtained from this classifier, thus introducing additional semantic information. However, this method fails to achieve a deep fusion of semantic and image information, making its effectiveness highly dependent on the quality of the generated text prompts.

[0006] Based on the observed limitations, we propose a remote sensing image classification method based on an unsupervised caching model and semantic regularization weights. First, we propose an unsupervised caching model that uses high-confidence pseudo-labeled samples as substitutes for labeled samples to construct a key-value caching model, acquiring knowledge from the unsupervised dataset without training. Second, we propose a semantic regularization weight based on a large language model to generate prompts. This weight can regularize the similarity between cached and test samples from a semantic perspective, compensating for the lack of diversity in the unsupervised caching model and achieving deep fusion of semantic and image information. Third, we propose an adapter optimization algorithm based on a strong-weak enhancement strategy. We obtain weakly enhanced view pseudo-labels through the unsupervised caching model and threshold filtering, optimizing the additional adapter module to further optimize the model while preserving the generalization ability of the visual language model and the knowledge of the caching model. Summary of the Invention:

[0007] To address the shortcomings of current unsupervised methods in generalizing pre-trained models and deeply integrating semantic and image information, this paper proposes a remote sensing image classification method based on an unsupervised caching model and semantic regularization weights. The invention mainly includes: an unsupervised caching model that extracts knowledge from the unsupervised training set without training by utilizing high-confidence pseudo-label samples; semantic regularization weights that introduce additional semantic information to compensate for the lack of sample diversity and regularize the similarity between samples; and an adapter optimization algorithm based on strong and weak enhancements that preserves the generalization ability of both the pre-trained and caching models and further optimizes the model.

[0008] The remote sensing image classification method proposed in this paper, based on an unsupervised caching model and semantic regularization weights, has the following main steps:

[0009] Step 1: Using an unsupervised remote sensing dataset as input, construct a caching model that can extract knowledge from the unsupervised dataset without training. First, use a large language model to generate corresponding class text descriptions for all N categories. Then, input the class text descriptions and unsupervised dataset samples into the text encoder and image encoder of the pre-trained CLIP model, respectively, to obtain the normalized text features W for all categories. l Normalized image features of all samples are used, and then the cosine similarity between text features and image features is calculated to obtain predicted probabilities and corresponding pseudo-labels. Confidence filtering is applied to the predicted probabilities, i.e., the predicted probabilities of all samples for each category are sorted, and the top 2K samples with higher predicted probabilities for each category are filtered out. Next, the mean KL divergence of the predicted probabilities of each sample with other samples of the same category is calculated, and the top K samples with smaller KL divergence mean are obtained. The image features of the filtered samples are cached to obtain F. cache ; Construct a One-Hot vector for each cached sample based on the pseudo-label, and cache it as L cache .

[0010] Step 2: Based on the unsupervised caching model built in Step 1, calculate the semantic normalization weights used to semantically normalize the cosine similarity between the image features of the test samples and cached samples. First, input the test samples into the image encoder of the pre-trained CLIP model to obtain image features f. test Calculate the features F of the cached sample images respectively. cache Test sample image features f test Text features W l The cosine similarity between samples yields the corresponding predicted probability. Then, the KL divergence D between the predicted probabilities of the test sample and each cached sample is calculated, normalized, and inverted to obtain the semantic similarity between the test sample and the cached sample. Multiplying this by an auxiliary normalization parameter γ greater than 1 gives the semantic normalization weight W based on semantic information. s Its range is [0, γ].

[0011] Step 3: Based on the semantic regularization weights obtained using semantic information in Step 2, calculate the prediction probability of the cache model for the test sample and optimize the prediction probability of CLIP. First, calculate the image features f of the test sample. test With cached sample image features F cache The cosine similarity between them is then compared with the semantic regularization weight W. s Element-wise multiplication assigns higher similarity scores to test cache sample pairs with high semantic similarity, while suppressing similarity scores between test cache samples with low semantic similarity, thus obtaining a normalized similarity. This normalized similarity is then reduced by 1 and multiplied by the sharpness adjustment parameter μ. Finally, an exponential parameter is used to transform it into a non-negative affinity A. Affinity A is then correlated with the label matrix L. cacheMatrix multiplication is performed to obtain the predicted probability of the cached model for the test sample. Finally, the predicted probabilities of the cached model and the CLIP model are joined by residuals to obtain the final predicted probability.

[0012] Step 4: Based on the optimized prediction probabilities from Step 3, optimize the additional learnable adapter using a strong-weak augmentation strategy. First, freeze the CLIP model and cache model to preserve their generalization ability and prevent the negative impact of incorrect pseudo-labels. Then, add an additional learnable adapter after the CLIP text encoder and image encoder. Perform strong and weak augmentation transformations on the training samples to obtain strong and weak augmentation views, respectively. The strong augmentation view obtains its prediction probability through the CLIP model and the learnable adapter; the weak augmentation view obtains its optimized prediction probability through the CLIP model and cache model. A threshold is used to filter these prediction probabilities, retaining only samples with high confidence and obtaining their pseudo-labels as supervision information. The prediction probability of the strong augmentation view serves as input to both the pseudo-label cross-entropy loss function and the marginal distribution loss function. The marginal distribution loss function minimizes the entropy of the mean prediction probabilities of all training samples to ensure training stability.

[0013] Beneficial effects:

[0014] Compared with existing unsupervised learning methods applied to remote sensing datasets, the design scheme proposed in this paper can achieve the following technical effects:

[0015] 1. The unsupervised caching model is constructed from high-confidence pseudo-label samples obtained by double filtering. By calculating the similarity between the image features of the test sample and the cached sample and combining the corresponding pseudo-label vectors, the prediction probability of the pre-trained CLIP model can be effectively optimized without labels and training.

[0016] 2. Define semantic regularization weights. The calculation of semantic regularization weights incorporates knowledge from a large language model, regularizing the similarity between test samples and cached samples from a semantic perspective. This compensates for the general lack of diversity caused by using high-confidence pseudo-label samples to build a cached model, and successfully achieves deep fusion of semantic and image information.

[0017] 3. In unsupervised training, erroneous pseudo-labels inevitably exist, negatively impacting model learning and further affecting subsequent learning, significantly reducing prediction accuracy. Our proposed method mitigates the accumulation of errors caused by erroneous pseudo-labels by freezing the pre-trained CLIP model and caching the model, and introduces an additional adapter as a learnable module, thus preserving the generalization ability of the original model.

[0018] 4. This remote sensing image classification method based on unsupervised caching model and semantic regularization weights, while retaining the scalability of text prompts, fully explores the knowledge of CLIP model, unsupervised dataset, and large language model, and fully integrates them to improve the classification accuracy of test samples. Attached image description:

[0019] Figure 1 Method framework flowchart

[0020] Figure 2 Unsupervised caching model construction flowchart

[0021] Figure 3 Semantic regularization weight calculation flowchart Detailed implementation method:

[0022] Step 1: Using unsupervised remote sensing datasets as input, construct a caching model that can extract knowledge from unsupervised datasets without training.

[0023] (1) First, a large language model is used to generate corresponding class text descriptions T for all N categories. Then, the class text descriptions and unsupervised dataset samples are input into the text encoder G and image encoder I of the pre-trained CLIP model to obtain the normalized text features W for all categories. l And normalized image features for each unlabeled sample:

[0024] W l =G(T)

[0025] f i =I(x) i )

[0026] (2) Then calculate the cosine similarity between the text features and the image features to obtain the predicted probability and the corresponding pseudo-label:

[0027]

[0028] label i =arg max(softmax(logits) i ))

[0029] (3) Confidence filtering is applied to the predicted probabilities, i.e., the predicted probabilities of all samples for each class are sorted, and the top 2K samples with the highest predicted probabilities for each class are filtered out. Then, to further remove noisy samples from other classes from these 2K samples, the mean of the KL divergence between the predicted probabilities of each sample and other samples of the same class is calculated:

[0030]

[0031] Sort and select the top K samples with the smallest mean KL divergence (i.e., the smallest intra-class distance), and cache the image features of the filtered samples to obtain F. cache ; Construct a One-Hot vector for each cached sample based on the pseudo-label, and cache it as L cache .

[0032] Step 2: Based on the unsupervised caching model built in Step 1, calculate the semantic normalization weights used to normalize the cosine similarity between the image features of the test samples and cached samples from a semantic perspective.

[0033] (1) First, the test sample is input into the image d encoder of the pre-trained CLIP model to obtain the image features f. test Calculate the features F of the cached sample images respectively. cache Test sample image features f test Text features W l The cosine similarity between the pairs yields the corresponding prediction probability, as shown in the following formula:

[0034] f test =I(x) test )

[0035]

[0036] (2) Then calculate the KL divergence D between the test sample and the predicted probability of each cached sample, as shown in the following formula:

[0037] D = KL(logits) test ||logits cache )

[0038] The numerical distribution of D is improved by Z-score normalization and min-max normalization, and its range is projected to [0, 1]:

[0039]

[0040] Through 1-D norm Converting it to a similarity metric and then multiplying it by an auxiliary normalization parameter γ greater than 1 yields the semantic normalization weight W based on semantic information. s Its range is [0, γ], and the formula is as follows:

[0041] W s =γ(1-D norm )

[0042] Where γ is generally set to 2, i.e., W s The range is [0, 2].

[0043] Step 3: Based on the semantic regularization weights obtained using semantic information in Step 2, calculate the prediction probability of the cache model for the test sample and optimize the prediction probability of CLIP.

[0044] (1) First, calculate the feature f of the test sample image. test With cached sample image features F cache The cosine similarity between the test samples and the cached samples yields the image-level similarity.

[0045]

[0046] (2) However, the high-confidence cached samples selected from the unsupervised dataset exhibit limited diversity. Considering only image-level similarity may lead the model to misclassify instances near the class decision boundary. Therefore, we combine image-level similarity with semantic regularization weights W. s Element-wise multiplication assigns higher similarity scores to test cache sample pairs with high semantic similarity, while suppressing similarity scores between test cache samples with low semantic similarity, thus obtaining the normalized similarity score:

[0047]

[0048] W s If the value range is [0, 2], then test cache sample pairs with a semantic similarity greater than 0.5 are assigned a higher similarity score, and test cache sample pairs with a semantic similarity less than 0.5 are assigned a lower similarity score.

[0049] (3) After normalization, the similarity is reduced by 1 and multiplied by the sharpness adjustment parameter μ. Then, the exponential parameter is used to convert it into a non-negative affinity A, as shown in the following formula:

[0050] A = exp(μ(Sim) c -1))

[0051] Affinity A and Tag Matrix L cache Matrix multiplication is performed to obtain the predicted probability of the cached model for the test sample. Then, the predicted probabilities of the cached model and the CLIP model are joined by residuals to obtain the final predicted probability, as shown in the formula below:

[0052]

[0053] Where λ is the residual ratio. This represents CLIP's predicted probability for the test sample.

[0054] Step 4: Based on the optimized prediction probabilities in Step 3, use a strong-weak enhancement strategy to optimize the additional learnable adapter.

[0055] (1) First, in order to preserve the powerful generalization ability of the CLIP model itself and to prevent the negative impact of erroneous pseudo-label samples on the cache model during training, we freeze both the CLIP model and the cache model, and then add an additional adapter as an alternative learnable parameter after the CLIP text encoder and image encoder.

[0056] (2) Perform strong enhancement transformation and weak enhancement transformation on the training samples respectively to obtain strong enhancement view and weak enhancement view:

[0057] x s =T s (x)

[0058] x w =T w (x)

[0059] Where T s T w These represent strong enhancement transform and weak enhancement transform, respectively.

[0060] (3) The enhanced view obtains adapted image features and text features W through the CLIP image encoder and its subsequent learnable adapter. l The adapted text features are obtained through the learnable adapter following the CLIP text encoder, and multiplied to obtain the predicted probability of the strongly augmented view, as shown in the following formula:

[0061] f s =I(x) s )

[0062] f adapted =αAdapter I (f s )+(1-α)f s

[0063] W adapted =βAdapter G (W l )+(1-β)W l

[0064] logits adapted =f adapted W adapted

[0065] Where α and β are the residual proportions, which are 0.2 and 0.5 respectively.

[0066] (4) The weakly augmented view obtains weakly augmented image features through the CLIP image encoder, and calculates the optimized prediction probability logits corresponding to the weakly augmented view based on the caching model and our proposed semantic regularization weights according to the first three steps. w And obtain its pseudo-tags:

[0067] label w =arg max(softmax(logits) w ))

[0068] (5) For the obtained strong enhancement prediction probabilities and weak enhancement prediction probabilities and their corresponding pseudo-labels, we use a pseudo-label cross-entropy loss with a threshold to optimize the two adapters. The pseudo-label cross-entropy loss function with a threshold is shown below:

[0069] L CE =1(max(logits) w )>τ)CE(logits adapted label w )

[0070] Here, τ is set to 0.9, and CE represents the cross-entropy loss function.

[0071] (6) In addition, error accumulation is inevitable in unsupervised learning. Therefore, we use an additional marginal distribution loss function to constrain the model from a global perspective. The marginal distribution loss function is shown below:

[0072]

[0073] The marginal distribution loss function minimizes the entropy of the mean predicted probabilities of all training samples to ensure training stability. n Let n represent the marginal distribution of the predicted probability of an image over class index n, where n∈{0,1,...,N-1}.

[0074] Based on the above steps, the global optimization objective for unsupervised remote sensing image classification is determined as follows:

[0075] L Total =L CE +L MD

[0076] The unsupervised remote sensing image classification model is obtained by iteratively training two learnable adapters using this loss. For M different unsupervised remote sensing image classification tasks, only 2M lightweight adapters and one CLIP pre-trained model need to be stored. The trained model retains the strong generalization ability of the CLIP pre-trained model, while fully combining the knowledge of high-confidence cached samples with the knowledge of the large language model. Its performance outperforms existing remote sensing image classification methods, and the image classification accuracy is significantly improved.

Claims

1.A method for remote sensing image classification based on unsupervised cache model and semantic regularization weight, characterized in that: At least comprising the following steps: Step one: using an unsupervised remote sensing dataset as input, a cache model is constructed to extract knowledge from the unsupervised dataset without training. First, a large language model is used to generate corresponding class text description for each class. Then, the class text description and the unsupervised dataset samples are input into the pre-trained CLIP model to obtain the text features of the classes and the image features of all samples. Then, the cosine similarity between the text features and the image features is calculated to obtain the prediction probability and the corresponding pseudo label. The prediction probability is filtered using confidence filtering and prototype filtering based on KL divergence to filter out the most representative samples for each class. The image features of the filtered samples and their corresponding pseudo label vectors are cached to obtain the unsupervised cache model. Step two: based on the unsupervised cache model constructed in step one, the semantic regularization weight used to regularize the cosine similarity between the image features of the test samples and the cache samples from the semantic perspective is calculated. First, the test samples are input into the image encoder of the pre-trained CLIP model to obtain the test sample image features. The cosine similarity between the cache sample image features and the test sample image features is calculated to obtain the corresponding prediction probability. Then, the KL divergence between the test sample and each cache sample prediction probability is calculated and converted into a similarity measure. Then, multiply the auxiliary regularization parameter γ to obtain the semantic regularization weight based on the semantic information. Step three: based on the semantic regularization weight obtained in step two using semantic information, the prediction probability of the cache model for the test sample is calculated and the prediction probability of CLIP is optimized. First, the cosine similarity between the test sample image features and the cache sample image features is calculated, then multiplied by the semantic regularization weight element by element to dynamically regularize the similarity. Then, using the sharpness adjustment parameter μ and the exponential parameter, it is converted into a non-negative affinity A. The affinity A is multiplied by the cache label matrix to obtain the prediction probability of the cache model for the test sample. Finally, the prediction probability of the cache model and the prediction probability of the CLIP model are connected to obtain the final prediction probability. Step four: based on the optimized prediction probability in step three, the strong and weak enhancement strategy is used to optimize the additional learnable adapter. First, freeze the CLIP model and the cache model, then add an additional learnable adapter after the CLIP text encoder and image encoder. Perform strong and weak enhancement transformations on the training samples to obtain strong and weak enhancement views. The strong enhancement view passes through the CLIP model and the learnable adapter to obtain its prediction probability. The weak enhancement view passes through the CLIP model and the cache model to obtain the optimized prediction probability. Use the threshold to filter the prediction probability and only keep the samples with high confidence to obtain their pseudo labels as supervision information. The prediction probability of the strong enhancement view is used as the input of the pseudo label cross entropy loss function and the marginal distribution loss function, where the marginal distribution loss function minimizes the entropy of the average of all training sample prediction probabilities to ensure the stability of the training. 2.The method of claim 1, wherein: In step one, an unsupervised cache model is constructed using an unsupervised remote sensing dataset, and knowledge can be obtained from unsupervised vision without training. First, a large language model is used to generate corresponding class text descriptions for all N categories. Then, the class text descriptions and unsupervised dataset samples are respectively input into the text encoder and image encoder of the pre-trained CLIP model to obtain normalized text features W l and normalized image features of all samples. Then, the cosine similarity of the text features and image features is calculated to obtain the prediction probability and the corresponding pseudo label. The prediction probability is filtered using confidence, that is, the prediction probability of each class for all samples is sorted, and the top 2K samples with larger prediction probability are filtered out for each class. Then, the KL divergence mean of each sample and other samples of the same class is calculated to obtain the top K samples with smaller KL divergence mean. The image features F cache of the filtered samples are cached. A One Hot vector is constructed for each cached sample according to the pseudo label, and the cached L cache . 3.The method of claim 1, wherein: The cosine similarity between the test sample and the cached sample image features in step two is normalized from a semantic perspective to calculate the semantic normalization weight. First, input the test sample into the image encoder of the pre-trained CLIP model to obtain the image feature f test , respectively, and calculate the cosine similarity between the cached sample image feature F cache , the test sample image feature f test and the text feature W l to obtain the corresponding prediction probability, then calculate the KL divergence D between the test sample and each cached sample prediction probability, normalize it and reverse it to obtain the similarity between the test sample and the cached sample based on semantic information, and then multiply it by an auxiliary normalization parameter γ greater than 1 to obtain the semantic normalization weight W s based on semantic information, whose value range is [0, γ]. 4.The method of claim 1, wherein: In step four, the pre-trained model and the cached model are frozen, and an extra learnable adapter is optimized using a strong-weak augmentation strategy. First, the CLIP model and the cached model are frozen to preserve the generalization ability of the model itself and prevent the negative impact of false pseudo-labels. Then, an extra learnable adapter is added after the CLIP text encoder and image encoder as a substitute for the feasibility parameter. Strong and weak augmented views are obtained by performing strong and weak augmentations on the training samples, respectively. The strong augmented view obtains its prediction probability through the CLIP model and the learnable adapter; The weak augmented view obtains the prediction probability through the CLIP model and the cached model according to step three, and uses a threshold to filter the prediction probability, keeping only the samples with high confidence and obtaining their pseudo-labels as supervision information. The prediction probability of the strong augmented view and the pseudo-labels of the weak augmented view are used as the input of the pseudo-label cross-entropy loss function. In addition, the prediction probability of the strong augmented view is also used as the input of the marginal distribution loss function, which minimizes the entropy of the average prediction probability of all training samples to ensure the stability of the training.

Images