Remote sensing small sample image classification method based on two-stage fine-tuning multimodal model

Abstract

The present invention discloses a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model, relating to the technical field of small sample image classification. The method comprises the following steps: constructing a remote sensing small sample image classification dataset; constructing a multimodal model fine-tuning framework, and adopting a multimodal image-text comparison pre-training model as the fine-tuning framework of the multimodal model; in the first stage, introducing structural prior knowledge, using symmetric cross-entropy loss to perform comparative fine-tuning on the image encoder and text encoder of the multimodal model; in the second stage, modifying the multimodal model structure, adding a fully connected layer linear classification head and fine-tuning the classification head parameters; and applying the fine-tuned multimodal model to perform remote sensing image classification tasks. The present invention utilizes the advantages of the multimodal model and combines the method of introducing structural prior knowledge in model fine-tuning to enhance the global information understanding ability and generalization of the multimodal model, making it more suitable for small sample classification tasks in the remote sensing field.

Description

Technical Field

[0001] The present invention belongs to the technical field of small sample image classification, and specifically relates to a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model. Background Art

[0002] Remote sensing image classification, as one of the important research areas in remote sensing technology, is widely used in practical scenarios such as land cover monitoring, environmental change detection, urban expansion monitoring, agricultural production management, and natural disaster assessment. Remote sensing image data mainly comes from aerospace equipment, including optical, radar, multispectral, and hyperspectral sensors. It has the characteristics of wide coverage, long time span, and rich information. However, despite the huge amount of data, the acquisition of labeled data is extremely challenging. Especially in hyperspectral imagery, high-precision pixel-by-pixel labeling is time-consuming and labor-intensive, resulting in a scarcity of high-quality, large-scale labeled data. Therefore, in this scenario of scarce labeled data, how to improve the accuracy and generalization ability of remote sensing image classification has become a problem that researchers urgently need to solve.

[0003] Traditional remote sensing image classification methods mostly rely on supervised learning, using convolutional neural networks (CNNs) to model and extract features from images. Convolutional neural networks (CNNs) perform well in natural image classification tasks, effectively extracting local features from images. However, in remote sensing image classification, CNNs are limited in their performance, especially when the number of samples is limited. This is primarily due to the following factors: First, CNNs are highly dependent on the amount of sample data, requiring a large number of labeled samples to train the model to avoid overfitting; second, remote sensing images are complex, characterized by heterogeneity, spectral diversity, and spatiotemporal heterogeneity. CNNs often perform less well than expected when extracting these complex features. Furthermore, remote sensing images require not only analysis of local features but also, to a certain extent, consideration of the image's global information structure. CNNs are limited in their ability to process global structural information, further limiting their application in remote sensing.

[0004] To address the problem of learning under data-scarce conditions, few-shot learning has been proposed and has garnered considerable attention in the field of computer vision. Few-shot learning involves teaching models to address problems with limited samples, enabling them to adapt to diverse tasks even with minimal data. Currently, mainstream few-shot methods include those based on meta-learning, transfer learning, metric learning, and generative adversarial networks. In the field of image classification, few-shot learning methods have made significant progress and are widely applied to various visual tasks.

[0005] Meta-learning methods train a general model on a wide range of classification tasks, enabling it to rapidly adjust its weights and perform image classification on new tasks using a small number of examples. Representative examples of these methods include Model-Agnostic Meta-Learning (MAML) and Prototypical Networks (ProtoNet). The former uses gradient optimization to rapidly adapt to new tasks, while the latter achieves classification by calculating the distance between image features and prototypes based on category prototypes. Metric learning methods, on the other hand, learn the similarity between image features by designing effective distance metrics, mapping similar examples to adjacent feature spaces. Transfer learning transfers knowledge from pre-trained models on large-scale datasets to small-shot tasks, enabling the pre-trained feature extractor to function effectively even with limited data. While these small-shot learning methods have achieved significant progress in computer vision, they only train models using data from one modality, images, and fail to fully utilize the closely related textual information. In addition, existing methods still face some challenges in the field of remote sensing image classification, including the inability to fully capture complex features such as multi-scale, spatial structure and multi-spectral information in remote sensing data. The training process mostly relies on the training and migration of natural image data, and cannot fully adapt to the inherent complexity of remote sensing image classification tasks.

[0006] Among existing small-sample image classification models, a relatively new method demonstrates good generalization performance with a small number of samples. This method is based on a large visual language model for small-sample image classification. The core idea of ​​this method is to improve the model's generalization ability when the number of samples is small by fusing information from both visual and language modalities. In this method, a large visual language model (such as CLIP) relies not only on the visual information of the image but also on information from the text modality to assist in understanding and classifying image features. This multimodal fusion enables the model to use the rich semantic information of language as a bridge to establish a close association between visual content and text descriptions, thereby enhancing the ability to understand images, which is especially effective in small-sample scenarios.

[0007] Although the large visual language pre-trained model has good generalization, the accuracy of the model in specific downstream tasks needs to be improved. In order to further improve the accuracy of small-sample remote sensing image classification, more and more studies have begun to explore how to use prior knowledge to enhance the performance of the model. Prior knowledge can come from the experience of domain experts, such as terrain features, land use patterns, climate conditions, etc. This knowledge can provide additional contextual information for the model and help the model better understand the characteristics of remote sensing images. Traditional methods of introducing prior knowledge usually rely on static knowledge in a specific field. However, these methods are often difficult to extend to large, complex datasets or dynamic environments, and cannot adapt well to diverse tasks.

[0008] To this end, this paper proposes a remote sensing small-sample image classification method based on a two-stage fine-tuning multimodal model. By utilizing a specific dataset to perform a two-stage fine-tuning of the multimodal model, this paper aims to effectively improve the model's accuracy and robustness in remote sensing image classification tasks. Summary of the Invention

[0009] The purpose of the present invention is to provide a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model to solve the problems that the prior knowledge introduction method in the existing technology usually relies on static knowledge in a specific field, is difficult to expand to large and complex data sets or dynamic environments, and cannot adapt well to diverse tasks.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] The present invention proposes a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model, comprising the following steps:

[0012] S1. Construct a remote sensing small sample image classification dataset;

[0013] S2. Construct a multimodal model fine-tuning framework; use a multimodal image-text comparison pre-trained model as the fine-tuning framework of the multimodal model, wherein the fine-tuning framework of the multimodal model includes an image-text feature extractor, a global pooling layer, a flattening layer, and a linear classification head; the image-text feature extractor includes an image encoder and a text encoder; the multimodal image-text comparison pre-trained model can simultaneously process the visual information and the corresponding text description in the remote sensing image, realize multimodal information fusion, and fully utilize the specific features of the remote sensing data;

[0014] S3: In the first stage, we introduce structural prior knowledge to fine-tune the image encoder and text encoder of the multimodal model. We then construct an image-text bidirectional matching classification task based on the Tangram small-sample image classification dataset. We find the corresponding samples based on the maximum similarity, calculate the matching loss and take the average, and adjust the multimodal model parameters through backpropagation and optimization algorithms.

[0015] S4, the second stage modifies the multimodal model structure, adds a fully connected layer linear classification head and fine-tunes the classification head parameters; trains the model based on a small sample remote sensing image dataset, and expresses the optimal weight parameters of the model as a linear combination of the training samples in the feature space according to the Representer Theorem. The model outputs the prediction of the input, calculates the cross-modal loss, and updates the weight parameters of the fully connected layer linear classification head;

[0016] S5. Apply the fine-tuned multimodal model to perform remote sensing image classification tasks.

[0017] Preferably, the S1 is as follows:

[0018] Collect remote sensing datasets, preprocess them, segment them according to labels, extract some images and corresponding text indexes from the remote sensing data of each label category, segment the data according to the specified ratio to construct remote sensing small sample image classification training and validation sets, and support ignoring specific categories and renaming category names.

[0019] During segmentation, statistical functions are used to determine the number of categories in the dataset, obtain a mapping from labels to category names, and then segment the dataset by label. Finally, a specified number of samples are extracted from each of the segmented training and validation sets, with repeated extraction supported.

[0020] The preprocessing of remote sensing datasets is to unify the image resolution and retain the spectral and spatial information in the image to facilitate model processing and feature extraction; adjust the image size to adapt the input data to the model requirements, and ensure the consistency of the image format through white filling.

[0021] Preferably, the S3 is as follows:

[0022] Advanced image encoders (ViT-B / 32) and text encoders (Transformer) are used to encode remote sensing images and text. By introducing structural prior knowledge and combining the topographic, spectral, and spatiotemporal characteristics of remote sensing images, the model can capture complex remote sensing data features even with small sample sizes, improving classification accuracy.

[0023] The multimodal model CLIP uses contrastive learning pre-training to tightly integrate visual and textual semantics, resulting in powerful zero-shot recognition capabilities. For any given image, it calculates its cosine similarity with all text features and uses the category corresponding to the text with the highest similarity as the true category of the image, completing the zero-shot classification task.

[0024] The Kilogram dataset is used as the first stage to set up the image-text bidirectional matching classification task. The method randomly selects k tangram images and their corresponding text annotations, shuffles the order of the images and texts, and constructs a k×k text-image feature matrix.

[0025] Set up two classification tasks, including image to text and text to image; specifically: given text features and a set of image features I={I1,...,I k}, the goal is to find image samples I that match the text features m ∈I, and calculate the image-to-text matching loss L i2t ; Accordingly, the image features are given and a set of text features T = {T1,...,T k}, the goal is to find a text sample T that matches the image features n ∈T, and calculate the text-to-image matching loss L t2i ;

[0026] The image and text are matched by calculating the cosine similarity S(.,.), which is calculated as follows:

[0027]

[0028] Where, I * and T * Represents the image and text with the greatest similarity. The goal of each task is to match the corresponding image and text samples. Translated into mathematical form, it is to find the image and text pair with the greatest similarity;

[0029] Count the number of correct predictions for text and image orientation by comparing the found feature matching samples with the true labels;

[0030] Image-text matching loss L i2t and L t2i The calculation formula is as follows:

[0031]

[0032]

[0033] After obtaining the losses in two directions, calculate the average loss of the two. The formula is as follows:

[0034] L=(L i2t +L t2i ) / 2

[0035] Where L is L i2t and L t2i This process increases the number of tasks, prevents overfitting due to limited data, and enhances the symmetry of text and image processing, improving the robustness of the model.

[0036] Specifically, the Kilogram dataset was used to construct a small-sample tangram image classification dataset for the first stage of fine-tuning. This dataset consists of tangram images, each representing an abstract concept. This dataset was used to fine-tune the model because tangram images omit much of the background and detail found in traditional images, emphasizing the overall structure of the target category. The fine-tuned model incorporates structural prior knowledge, improving its reasoning, cognition, and generalization capabilities.

[0037] The image samples in the Kilogram dataset include two forms: each part of the tangram is the same color, and different parts of the tangram are different colors;

[0038] The text samples in the Kilogram dataset include two types: annotations for the whole and annotations for the parts;

[0039] The two forms of image samples and two types of text samples were arranged and combined to construct four different forms of tangram small sample datasets. The four datasets were used to perform the first stage fine-tuning of the multimodal model, and four fine-tuned multimodal models were obtained.

[0040] To determine the best-performing model weights for the classification task, four fine-tuned multimodal models were tested on the ImageNet dataset, which contains common real-world object categories and is more challenging, effectively evaluating the model's generalization capabilities. The generalization performance of the four fine-tuned multimodal models was evaluated, and the best-performing multimodal model was selected as the multimodal model for the first stage of fine-tuning.

[0041] Preferably, the S4 is specifically as follows:

[0042] S41, all the data of the input image and text modalities are used as training samples x j , the corresponding label is y j , specifically:

[0043]

[0044] Wherein, the modal set is defined as M = {image, text}, belongs to mode m j The set of all data points;

[0045] S42. Establish loss functions L for images and texts respectively cross-modal , whose expression is:

[0046]

[0047] Where, It is a feature extractor for two input modalities and uses the extracted features to calculate the softmax loss

[0048] The premise for calculating the loss function is to map the features of different modalities into the same feature space. The multimodal model meets this requirement by mapping the extracted image and text features into the same N-dimensional space, thereby allowing operations on data from different modalities. The mapping formula is as follows:

[0049]

[0050] According to the Representer theorem, the optimal weight parameters of the model are expressed as a linear combination of training samples in the feature space. The formula is as follows:

[0051]

[0052] Where w y Represents the weight parameter of the linear classification head, by setting the weight parameter α iy , the cross-modal sample features The linear combination of is represented as the optimal weight parameters of the linear classification head of the fully connected layer; the multimodal model is further fine-tuned to adapt to the downstream classification task by minimizing the cross-modal loss.

[0053] Specifically, the Eurosat remote sensing dataset is used to construct a remote sensing small sample image classification dataset for the second stage of fine-tuning;

[0054] The multimodal model is trained by using 4, 8, and 16 training samples K from each label category of the Eurosat remote sensing dataset as different remote sensing small sample image classification datasets.

[0055] After training the multimodal model using each remote sensing small sample image classification dataset, the cross-modal loss is calculated and the weight parameters of the linear classification head of the fully connected layer are updated; the classification accuracy of the multimodal model trained with different numbers of samples and with updated parameters is tested separately.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) By introducing prior knowledge and multimodal fusion technology, the present invention effectively improves the model's ability to perceive global information of remote sensing images, enabling it to show higher classification accuracy in complex scenes, and is particularly suitable for classification tasks of multispectral and hyperspectral data.

[0058] (2) The two-stage fine-tuning strategy in the present invention not only improves the robustness of the model in remote sensing image classification tasks, but also increases the number of tasks through the image-text bidirectional matching module, prevents overfitting, and enables the model to achieve good generalization performance in small sample conditions, and is suitable for different types of remote sensing data.

[0059] (3) By reducing the number of fine-tuning parameters of the model and adopting a reasonable optimization strategy, the present invention significantly reduces the computational cost while ensuring the classification accuracy, thereby improving the processing efficiency of the model in remote sensing data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the multimodal model in the present invention;

[0061] Figure 2 A schematic diagram of constructing an image-text bidirectional matching classification task in the present invention;

[0062] Figure 3 This is a test classification result diagram of the four multimodal models trained using four tangram datasets in the present invention. DETAILED DESCRIPTION

[0063] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0064] Example 1:

[0065] A remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model covers the following steps:

[0066] Step S1: For a given remote sensing dataset, divide the dataset into training set, validation set and test set according to the category label.

[0067] To ensure that the model can still fully utilize the spatial, spectral, and temporal information of remote sensing data under small sample conditions, this step determines the number of categories in the dataset through statistical functions and obtains the mapping relationship between labels and category names. The divided dataset supports extracting a specified number of samples based on the labels to form a small sample training set and validation set. For each extracted sample, the original features of the remote sensing image, such as spectral information and spatial resolution, are maintained to ensure data integrity. The present invention uses the Kilogram Tangram dataset and the remote sensing image dataset for experiments.

[0068] The Kilogram dataset contains two types of image samples: one in which every part of the tangram is black, and another in which different parts are colored differently. Furthermore, the text annotations are divided into two types: one for the entire puzzle and the other for its parts. Therefore, this dataset can be permuted and combined into four different datasets.

[0069] The Eurosat remote sensing dataset was used to fine-tune and test the model. The dataset consists of 27,000 remote sensing images, covering 10 different categories including industrial buildings, residential buildings, farmland, etc. The resolution of the input images was uniformly adjusted to 224×224 pixels, and white padding was added to keep them in a square format for easy model processing and training. Although the Eurosat dataset has a large number of samples, the present invention did not use all samples for training during training. Instead, according to the definition of small sample learning, a small sample dataset was constructed by sampling from the dataset. In each category, only a small number of samples with the number of training samples K of 4, 8, and 16 were used to train the model, which meets the requirements of small sample learning. The method proposed in the present invention is also applicable to other remote sensing small sample datasets.

[0070] Step S2: Construct a multimodal model fine-tuning framework.

[0071] like Figure 1 As shown in the figure, a multimodal model fine-tuning framework is constructed using the Contrastive Lang guage–Image Pretraining (CLIP) model; the framework consists of a graphic feature extractor, a global pooling layer, a flattening layer, and a linear classification head, and uses an advanced image encoder (ViT-B / 32) and a text encoder (Transformer) to encode remote sensing images and text.

[0072] The model fine-tuning process is divided into two stages. The first stage focuses on optimizing the model's image and text encoders so that it can process the spatial structure information contained in remote sensing images. The second stage optimizes the parameters of the linear classification head to enable the model to effectively handle actual remote sensing classification tasks.

[0073] Specifically, we used the Tangram dataset to fine-tune the encoder of the multimodal model in a first phase, introducing structural prior knowledge. We also used the remote sensing dataset to fine-tune the classification head of the multimodal model in a second phase. We set the Adam optimizer's learning rate to 5e-8 and weight decay to 1e-6, and trained the model for 20 epochs.

[0074] Step S3: The first stage of fine-tuning introduces structural prior knowledge to fine-tune the image encoder and text encoder.

[0075] In the first stage of fine-tuning, image-text pair samples were randomly selected from the Kilogram dataset to construct an image-text matching task. To optimize the model's performance in remote sensing image classification, this paper introduces structural prior knowledge to help the model better understand the global and local structure of remote sensing images, thereby improving its classification accuracy. Specifically:

[0076] S31: Use the Kilogram dataset to set up the image-text bidirectional matching classification task for the first stage.

[0077] Reference Figure 2 , the present invention randomly selects k tangram images and their corresponding text annotations, and shuffles the order of images and texts to construct a text-image feature matrix of size k×k. Two classification tasks are set, including image to text and text to image: given the text features and a set of image features I={I1,...,I k}, the goal is to find image samples I that match the text features m ∈I, and calculate the image-to-text matching loss L i2t Similarly, given the image features and a set of text features T = {T1,...,T k}, the goal is to find a text sample T that matches the image features n ∈T, and calculate the text-to-image matching loss L t2i Since the size of the text image feature matrix is ​​k×k, a total of 2k classification tasks are constructed.

[0078] The CLIP model is used to encode image and text samples. Its principle is to match images and text by calculating the cosine similarity S(.,.). The calculation formula is as follows:

[0079]

[0080] Where, I * and T * Represents the image and text with the greatest similarity. The goal of each task is to match the corresponding image and text samples. Converted into mathematical form, it means finding the image and text pair with the greatest similarity.

[0081] S32: Count the number of correct predictions for text and image orientation by comparing the predicted values ​​with the true labels.

[0082] Image-text matching loss L i2t and L t2i The calculation formula is as follows:

[0083]

[0084] After obtaining the losses in two directions, calculate the average loss of the two. The formula is as follows:

[0085] L=(L i2t +L t2i ) / 2

[0086] Where L is L i2t and Lt2i This process increases the number of tasks, prevents overfitting due to limited data, and enhances the symmetry of text and image processing, improving the robustness of the model.

[0087] S33: Test the classification accuracy of the multimodal model and determine the model with the best performance.

[0088] In the one-stage fine-tuning process, the encoder of the multimodal model is fine-tuned in one stage using the four datasets in step S1, thereby obtaining four fine-tuning models. In order to determine the best performance model for the classification task, these four fine-tuning models need to be tested. In order to test the generalization ability of the model, the present invention uses the ImageNet dataset for testing. The ImageNet dataset contains common object categories in the real world, is highly challenging, and can effectively evaluate the generalization performance of the model. Figure 3 , the model fine-tuned on the partially annotated all-black dataset has better performance.

[0089] Step S4: In the second stage, the multimodal model structure is modified, and a fully connected layer linear classification head is added; the classification head parameters are fine-tuned based on the remote sensing small sample image classification dataset.

[0090] According to the definition of small sample learning, the model is trained using a small number of samples (K) of 4, 8, or 16 from each category. For each sample, the optimal weight parameters of the model are expressed as a linear combination of the training samples in the feature space according to the Representer Theorem. The model's prediction of the input is output, the cross-modal loss is calculated, and the weight parameters of the fully connected layer are updated. Specifically:

[0091] S41: All input remote sensing image and text modality data are regarded as training samples x j , the corresponding label is y j , specifically:

[0092]

[0093] Wherein, the modal set is defined as M = {image, text}, belongs to mode m j The set of all data points.

[0094] S42: Establish remote sensing image and text loss functions respectively, and their expressions are:

[0095]

[0096] Where, It is a feature extractor for two input modalities and uses the extracted features to calculate the softmax loss

[0097] The premise for calculating the loss function is to map the features of different modalities to the same feature space; the CLIP model satisfies this requirement by mapping the extracted remote sensing image and text features into the same N-dimensional space, thereby allowing operations on data from different modalities. The mapping formula is as follows:

[0098]

[0099] According to the Representer theorem, the optimal weight parameters of the model are expressed as a linear combination of training samples in the feature space. The formula is as follows:

[0100]

[0101] Where w y Represents the weight parameter of the linear head, by setting the weight parameter α iy , the cross-modal sample features The linear combination of is represented as the optimal weight parameter of the fully connected layer. The model is further fine-tuned to adapt to the downstream classification task by minimizing the cross-modal loss.

[0102] Step S5: Apply the fine-tuned multimodal model to perform specific image classification tasks.

[0103] Experimental verification:

[0104] This paper conducts classification experiments on a small-sample remote sensing dataset. The dataset used is the Eurosat remote sensing dataset, as shown in Table 2. This dataset contains 10 categories: industrial buildings, residential buildings, crops, permanent crops, rivers, oceans and lakes, herbaceous vegetation, highways, pastures, and forests. Each category contains 2,000-3,000 sample images. Table 1 compares the small-sample image classification results of the two-stage fine-tuning method and other methods on the Eurosat dataset, where K = 4, K = 8, and K = 16 represent the number of training samples for each category, i.e., different small-sample scenarios.

[0105] As shown in Table 1, the two-stage fine-tuning method proposed in this paper outperforms other methods in all small sample cases, especially in the settings of K=8 and K=16, where significant improvements are achieved.

[0106] Table 1 Comparison of small sample image classification experimental results of the method of the present invention and other methods

[0107]

[0108] Table 2 Eurosat remote sensing dataset

[0109] category Number of image samples Industrial buildings 2500 residential buildings 3000 crops 3000 permanent crops 2500 river 2500 Oceans and lakes 3000 herb 3000 highway 2500 pasture 2000 forest 3000

[0110] In summary, the two-stage fine-tuning multimodal model method proposed in this paper can improve the accuracy of small sample image classification, especially in the field of remote sensing images.

[0111] The above description is only used to help understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, equivalent replacements or modifications based on the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention should be included in the scope of protection of the present invention. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model, characterized by: The following steps are involved: S1. Construct a remote sensing small sample image classification dataset; S2. Construct a multimodal model fine-tuning framework; use a multimodal image-text comparison pre-trained model as the fine-tuning framework of the multimodal model, the fine-tuning framework of the multimodal model including an image-text feature extractor, a global pooling layer, a flattening layer, and a linear classification head, the image-text feature extractor including an image encoder and a text encoder; S3: In the first stage, structural prior knowledge is introduced and the image encoder and text encoder of the multimodal model are compared and fine-tuned using symmetric cross-entropy loss. An image-text bidirectional matching classification task is constructed based on a small-sample remote sensing image classification dataset. The corresponding samples are found based on the maximum similarity, the matching loss is calculated and averaged, and the multimodal model parameters are adjusted through backpropagation and optimization algorithms. S4, the second stage modifies the multimodal model structure, adds a fully connected layer linear classification head and fine-tunes the classification head parameters; trains the model based on a remote sensing small sample image dataset using a fully supervised training method, calculates the cross-modal loss, and updates the weight parameters of the fully connected layer linear classification head; S5. Apply the fine-tuned multimodal model to perform remote sensing image classification tasks.

2. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 1 is characterized in that: The S1 is specifically as follows: Collect remote sensing data sets, preprocess them, segment them according to labels, extract some images and corresponding text indexes from the remote sensing data of each label category, and construct remote sensing small sample image classification training sets and validation sets.

3. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 2 is characterized in that: The remote sensing dataset is preprocessed, specifically, the image resolution is unified and the spectral and spatial information in the image is retained; the image size is adjusted and white filling is used to ensure the consistency of the image format.

4. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 3 is characterized in that: The S3 is as follows: The image-text bidirectional matching classification task includes image-to-text and text-to-image tasks; specifically, it is to construct a k×k text-image feature matrix, given the text features and a set of image features I={I1,...,I k }, the goal is to find image samples I that match the text features m ∈I, and calculate the image-to-text matching loss L i2t ; Accordingly, the image features are given and a set of text features T = {T1,...,T k }, the goal is to find a text sample T that matches the image features n ∈T, and calculate the text-to-image matching loss L t2i ; The image and text are matched by calculating the cosine similarity S(.,.), which is calculated as follows: Where, I * and T * Indicates the image and text with the greatest similarity; Count the number of correct predictions for text and image orientation by comparing the found feature matching samples with the true labels; Image-text matching loss L i2t and L t2i The calculation formula is as follows: After obtaining the losses in two directions, calculate the average loss of the two. The formula is as follows: L=(L i2t +L t2i ) / 2 Where L is L i2t and L t2i The average loss.

5. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 4 is characterized in that: The Kilogram dataset is used to construct a tangram small sample image classification dataset for the first stage of fine-tuning; The image samples in the Kilogram dataset include two forms: each part of the tangram is the same color, and different parts of the tangram are different colors; The text samples in the Kilogram dataset include two types: annotations for the whole and annotations for the parts; By combining two forms of image samples and two types of text samples, we constructed four different forms of tangram small sample image classification datasets. We used the four datasets to perform the first stage fine-tuning on the multimodal model, and obtained four fine-tuned multimodal models. The four fine-tuned multimodal models are tested and their generalization performance is evaluated, and the multimodal model with the best performance is selected as the multimodal model after fine-tuning in the first stage.

6. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to any one of claims 1 to 5, characterized in that: The S4 is specifically as follows: S41, all the data of the input image and text modalities are used as training samples x j , the corresponding label is y j , specifically: Wherein, the modal set is defined as M = {image, text}, belongs to mode m j The set of all data points; S42. Establish loss functions L for images and texts respectively cross-modal , whose expression is: Where, It is a feature extractor for two input modalities and uses the extracted features to calculate the softmax loss The multimodal model maps the extracted image and text features into the same N-dimensional space. The mapping formula is as follows: According to the Representer theorem, the optimal weight parameters of the model are expressed as a linear combination of training samples in the feature space. The formula is as follows: Where w y Represents the weight parameter of the linear classification head, by setting the weight parameter α iy , the cross-modal sample features The linear combination of is represented as the optimal weight parameter of the linear classification head of the fully connected layer.

7. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 6, characterized in that: The Eurosat remote sensing dataset is used to construct a remote sensing small sample image classification dataset for the second stage of fine-tuning; The multimodal model is trained by using 4, 8, and 16 training samples K from each label category of the Eurosat remote sensing dataset as different remote sensing small sample image classification datasets. After training the multimodal model using each remote sensing small sample image classification dataset, the cross-modal loss is calculated and the weight parameters of the linear classification head of the fully connected layer are updated; The classification accuracy of multimodal models trained with different numbers of samples and updated parameters was tested separately.

Images