Remote sensing small sample image classification method based on two-stage fine tuning multi-modal model

Abstract

The invention discloses a remote sensing small sample image classification method based on a two-stage fine tuning multi-modal model, and relates to the technical field of small sample image classification. The method comprises the following steps: constructing a remote sensing small sample image classification data set; constructing a multi-modal model fine tuning framework, and adopting the multi-modal image-text comparison pre-training model as the multi-modal model fine tuning framework; in the first stage, structure priori knowledge is introduced, and symmetric cross entropy loss is utilized to carry out contrast fine tuning on an image encoder and a text encoder of a multi-modal model; in the second stage, a multi-modal model structure is modified, a full-connection layer linear classification head is added, and classification head parameters are finely adjusted; and performing a remote sensing image classification task by using the finely-adjusted multi-modal model. According to the method, the advantages of the multi-modal model are utilized, the method that structure priori knowledge is introduced through model fine adjustment is combined, the global information understanding ability and generalization of the multi-modal model are enhanced, and the method is better suitable for small sample classification tasks in the remote sensing field.

Description

Technical Field

[0001] The present invention belongs to the technical field of small sample image classification, and specifically relates to a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model. Background Art

[0002] As one of the important research fields in remote sensing technology, remote sensing image classification is widely used in practical scenarios such as land cover monitoring, environmental change detection, urban expansion monitoring, agricultural production management, and natural disaster assessment. Remote sensing image data mainly comes from aerospace equipment, including optical, radar, multispectral, hyperspectral and other sensors, with the characteristics of wide coverage area, long time span, and rich information. However, despite the huge amount of data, the acquisition of labeled data is extremely challenging, especially in hyperspectral images. High-precision pixel-by-pixel labeling is time-consuming and labor-intensive, resulting in a scarcity of high-quality large-scale labeled data. Therefore, in this scenario of scarce labeled data, how to improve the accuracy and generalization ability of remote sensing image classification has become a topic that researchers need to solve urgently.

[0003] Most traditional remote sensing image classification methods rely on supervised learning, using convolutional neural networks (CNNs) to model and extract features from images. Convolutional neural networks perform well in natural image classification tasks and can extract local features of images well. However, in remote sensing image classification, especially when the number of samples is limited, the performance of CNNs is limited. This is mainly reflected in the following aspects: First, CNNs are highly dependent on the amount of sample data, and a large number of labeled samples are required to avoid overfitting when training the model; second, remote sensing images are highly complex, with heterogeneity, spectral dimension diversity, and spatiotemporal heterogeneity. CNNs often perform less well than expected when extracting these complex features. In addition, remote sensing images not only require analysis of local features, but also need to consider the global information structure of the image to a certain extent. CNNs are not capable of processing global structural information, which further limits their application in the field of remote sensing.

[0004] In order to solve the learning problem under the condition of scarce data, small sample learning was proposed and has received much attention in the field of computer vision. Small sample learning refers to teaching the model to solve the problem of limited number of samples so that it can adapt to different tasks even with a small amount of data. The current mainstream small sample methods include technologies based on meta-learning, transfer learning, metric learning, and generative adversarial networks. In the field of image classification, small sample learning methods have made significant progress and are widely used in various visual tasks.

[0005] Meta-learning methods train a general model on a large number of different classification tasks, so that it can quickly adjust weights and complete image classification in new tasks with a small number of samples. Typical representatives of this type of method include Model-Agnostic Meta-Learning (MAML) and Prototypical Networks (ProtoNet). The former quickly adapts to new tasks through gradient optimization, while the latter achieves classification by calculating the distance between image features and prototypes based on category prototypes. Metric learning methods learn the similarity between image features by designing effective distance measurement mechanisms and map similar samples to adjacent feature spaces. Transfer learning transfers knowledge to small sample tasks by pre-training models on large-scale datasets, so that pre-trained feature extractors can still work with limited data. Although the above small sample learning methods have made significant progress in the field of computer vision, they only use data from one modality, images, to train models, and do not fully utilize the text modality information closely related to images. In addition, existing methods still face some challenges in the field of remote sensing image classification, including the inability to fully capture complex features such as multi-scale, spatial structure, and multi-spectral information in remote sensing data. The training process mostly relies on the training and migration of natural image data and cannot fully adapt to the inherent complexity of remote sensing image classification tasks.

[0006] Among the existing small-sample image classification models, there is a relatively new method that shows good generalization performance when the number of samples is small. This method is a small-sample image classification method based on a large visual language model. The core idea of ​​this method is to improve the generalization ability of the model when the number of samples is small by fusing information from the two modalities of vision and language. In this method, the large visual language model (such as CLIP) not only relies on the visual information of the image, but also uses information from the text modality to assist in understanding and classifying image features. This multimodal fusion enables the model to use the rich semantic information of the language as a bridge to establish a close association between the visual content and the text description, thereby enhancing the ability to understand the image, which is especially effective in small-sample scenarios.

[0007] Although the visual language pre-trained large model has good generalization, the accuracy of the model in specific downstream tasks needs to be improved. In order to further improve the accuracy of small sample remote sensing image classification, more and more studies have begun to explore how to use prior knowledge to enhance the performance of the model. Prior knowledge can come from the experience of domain experts, such as terrain features, land use patterns, climate conditions, etc. This knowledge can provide additional contextual information for the model to help the model better understand the characteristics of remote sensing images. Traditional methods of introducing prior knowledge usually rely on static knowledge in a specific field. However, these methods are often difficult to extend to large, complex data sets or dynamic environments, and cannot adapt well to diverse tasks.

[0008] To this end, the present invention proposes a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model. The present invention uses a two-stage fine-tuning method of a multimodal model using a specific data set to effectively improve the accuracy and robustness of the model in the classification task of remote sensing image data sets. Summary of the invention

[0009] The purpose of the present invention is to provide a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model to solve the problems that the prior art methods for introducing prior knowledge usually rely on static knowledge in a specific field, are difficult to expand to large and complex data sets or dynamic environments, and cannot adapt well to diverse tasks.

[0010] To achieve the above purpose, the technical solution adopted by the present invention is:

[0011] The present invention proposes a remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model, comprising the following steps:

[0012] S1. Construct a remote sensing small sample image classification dataset;

[0013] S2. Construct a multimodal model fine-tuning framework; adopt a multimodal image-text comparison pre-trained model as the fine-tuning framework of the multimodal model, wherein the fine-tuning framework of the multimodal model includes an image-text feature extractor, a global pooling layer, a flattening layer, and a linear classification head, and the image-text feature extractor includes an image encoder and a text encoder; the multimodal image-text comparison pre-trained model can simultaneously process the visual information and the corresponding text description in the remote sensing image, realize multimodal information fusion, and make full use of the specific features of the remote sensing data;

[0014] S3. In the first stage, structural prior knowledge is introduced to fine-tune the image encoder and text encoder of the multimodal model; an image-text bidirectional matching classification task is constructed based on the tangram small sample image classification dataset, corresponding samples are found according to the maximum similarity, matching loss is calculated and averaged, and multimodal model parameters are adjusted through back propagation and optimization algorithms;

[0015] S4, the second stage modifies the multimodal model structure, adds a fully connected layer linear classification head and fine-tunes the classification head parameters; trains the model based on a remote sensing small sample image dataset, represents the optimal weight parameters of the model as a linear combination of the training samples in the feature space according to the Representer theorem, outputs the model's prediction of the input, calculates the cross-modal loss, and updates the weight parameters of the fully connected layer linear classification head;

[0016] S5. Apply the fine-tuned multimodal model to perform remote sensing image classification tasks.

[0017] Preferably, the S1 is as follows:

[0018] Collect remote sensing data sets, preprocess remote sensing data sets, segment remote sensing data sets according to labels, extract some images and corresponding text indexes from remote sensing data of each label category, segment data according to specified ratios to construct remote sensing small sample image classification training sets and validation sets, and support ignoring specific categories and renaming category names.

[0019] When segmenting, the number of categories in the data set can be determined through statistical functions, the mapping from labels to category names can be obtained, and the data set can be segmented by labels. Finally, a specified number of samples are extracted from the segmented training set and validation set, respectively, and repeated extraction is supported.

[0020] The preprocessing of remote sensing datasets is to unify the image resolution and retain the spectral and spatial information in the image to facilitate model processing and feature extraction; adjust the image size to adapt the input data to the model requirements, and ensure the consistency of the image format through white filling.

[0021] Preferably, S3 is specifically as follows:

[0022] Advanced image encoder (ViT-B / 32) and text encoder (Transformer) are used to encode remote sensing images and text. By introducing structural prior knowledge and combining the topographic, spectral and spatiotemporal characteristics of remote sensing images, the model can still capture complex remote sensing data features under small sample conditions and improve classification accuracy.

[0023] The multimodal model CLIP uses a contrastive learning pre-training method to tightly integrate visual and textual semantics, and has strong zero-shot recognition capabilities. For any given image, its cosine similarity with all text features is calculated, and the category corresponding to the text with the highest similarity is taken as the true category of the image to complete the zero-shot classification task.

[0024] The Kilogram dataset is used to set the image-text bidirectional matching classification task for the first stage. The present invention randomly selects k tangram images and their corresponding text annotations, shuffles the order of images and texts, and constructs a k×k text-image feature matrix.

[0025] Set up two classification tasks, including image to text and text to image; specifically: given text features and a set of image features I = {I1,...,I k}, the goal is to find image samples I that match the text features m ∈I, and calculate the image-to-text matching loss L i2t ; Accordingly, the image features are given and a set of text features T = {T1,...,T k}, the goal is to find a text sample T that matches the image features n ∈T, and calculate the text-to-image matching loss L t2i ;

[0026] The image and text are matched by calculating the cosine similarity S(.,.), and the calculation formula is as follows:

[0027]

[0028] In the formula, I * and T * Represents the image and text with the greatest similarity. The goal of each task is to match the corresponding image and text samples. Translated into mathematical form, it is to find the image and text pair with the greatest similarity.

[0029] Count the number of correct predictions for text and image orientation by comparing the found feature matching samples with the true labels;

[0030] Image-text matching loss L i2t and L t2i The calculation formula is as follows:

[0031]

[0032]

[0033] After obtaining the losses in two directions, calculate the average loss of the two. The formula is as follows:

[0034] L=(L i2t +L t2i ) / 2

[0035] Where L is L i2t and L t2i This process increases the number of tasks, prevents overfitting due to limited data, and enhances the symmetry of text and image processing, improving the robustness of the model.

[0036] Specifically, the Kilogram dataset is used to construct a tangram small sample image classification dataset for the first stage of fine-tuning; it consists of tangram graphics, and each sample represents an abstract concept. The reason for using this dataset to fine-tune the model is that the tangram images omit most of the background and detail information found in traditional images, emphasizing the overall structure of the target category. The fine-tuned model introduces structural prior knowledge, and the reasoning cognition and generalization capabilities are improved.

[0037] The image samples in the Kilogram dataset include two forms: each part of the tangram is the same color, and different parts of the tangram are different colors;

[0038] The text samples in the Kilogram dataset include two types: annotations for the whole and annotations for parts;

[0039] The two forms of image samples and the two types of text samples are arranged and combined to construct four different forms of tangram small sample datasets; the four datasets are used to perform the first stage fine-tuning of the multimodal model respectively, and four fine-tuned multimodal models are obtained;

[0040] In order to determine the best performance model weights for the classification task, four fine-tuned multimodal models were tested; the dataset used for testing is the ImageNet dataset, which contains common object categories in the real world, is more challenging, and can effectively evaluate the generalization ability of the model. The generalization performance of the four fine-tuned multimodal models was evaluated, and the multimodal model with the best performance was selected as the multimodal model after fine-tuning in the first stage.

[0041] Preferably, the S4 is specifically as follows:

[0042] S41, all the data of the input image and text modalities are used as training samples x j , the corresponding label is y j , specifically:

[0043]

[0044] In the formula, the modal set is defined as M = {image, text}, belongs to mode m j The set of all data points of ;

[0045] S42. Establish the loss function L for image and text respectively cross-modal , whose expression is:

[0046]

[0047] In the formula, It is a feature extractor for two input modalities, and the softmax loss is calculated using the extracted features

[0048] The premise for calculating the loss function is to map the features of different modalities to the same feature space. The multimodal model meets this requirement by mapping the extracted image and text features into the same N-dimensional space, thereby allowing operations on data from different modalities. The mapping formula is as follows:

[0049]

[0050] According to the Representer theorem, the optimal weight parameters of the model are expressed as a linear combination of training samples in the feature space. The formula is as follows:

[0051]

[0052] In the formula, w y Represents the weight parameter of the linear classification head, by setting the weight parameter α iy , the cross-modal sample features The linear combination of is represented as the optimal weight parameters of the fully connected layer linear classification head; the multimodal model is further fine-tuned to adapt to the downstream classification task by minimizing the cross-modal loss.

[0053] Specifically, the Eurosat remote sensing dataset is used to construct a remote sensing small sample image classification dataset for the second stage of fine-tuning;

[0054] From each label category of the Eurosat remote sensing dataset, samples with the number of training samples K of 4, 8, and 16 are used as different remote sensing small sample image classification datasets to train the multimodal model;

[0055] After training the multimodal model with each remote sensing small sample image classification dataset, the cross-modal loss is calculated and the weight parameters of the linear classification head of the fully connected layer are updated; the classification accuracy of the multimodal model trained with different numbers of samples and with updated parameters is tested respectively.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) The present invention effectively improves the model's ability to perceive the global information of remote sensing images by introducing prior knowledge and multimodal fusion technology, enabling it to show higher classification accuracy in complex scenes, and is particularly suitable for classification tasks of multispectral and hyperspectral data.

[0058] (2) The two-stage fine-tuning strategy in the present invention not only improves the robustness of the model in remote sensing image classification tasks, but also increases the number of tasks through the image-text bidirectional matching module to prevent overfitting, so that the model can still achieve good generalization performance in small sample conditions and is suitable for different types of remote sensing data.

[0059] (3) The present invention significantly reduces the computational cost and improves the processing efficiency of the model in remote sensing data scenarios by reducing the amount of fine-tuning parameters of the model and adopting a reasonable optimization strategy while ensuring classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the multimodal model in the present invention;

[0061] Figure 2 A schematic diagram of constructing an image-text bidirectional matching classification task in the present invention;

[0062] Figure 3 This is a test classification result diagram of four multimodal models trained using four tangram data sets in the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Embodiment 1:

[0065] A remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model covers the following steps:

[0066] Step S1: For a given remote sensing dataset, divide the dataset into training set, validation set and test set according to the category label.

[0067] To ensure that the model can still make full use of the spatial, spectral and temporal information of remote sensing data under small sample conditions, this step determines the number of categories in the data set through statistical functions and obtains the mapping relationship between labels and category names. The divided data set supports extracting a specified number of samples according to the labels to form a small sample training set and a validation set. For each extracted sample, the original features of the remote sensing image, such as spectral information and spatial resolution, are maintained to ensure the integrity of the data. The present invention uses the Kilogram Tangram data set and the remote sensing image data set for experiments.

[0068] There are two forms of image samples in the Kilogram dataset: one is that each part of the tangram is black, and the other is that different parts of the tangram use different colors. In addition, text annotations are also divided into two types: one is for the whole annotation, and the other is for the part annotation. Therefore, this dataset can produce four permutations and combinations, forming four different forms of datasets.

[0069] The Eurosat remote sensing dataset was used to fine-tune and test the model. The dataset consists of 27,000 remote sensing images, covering 10 different categories including industrial buildings, residential buildings, farmland, etc. The resolution of the input image is uniformly adjusted to 224×224 pixels, and white padding is added to keep it in a square format for easy model processing and training. Although the Eurosat dataset has a large number of samples, the present invention does not use all samples for training during training. Instead, according to the definition of small sample learning, a small sample dataset is constructed by sampling from the dataset. In each category, only a small number of samples with a training sample number K of 4, 8, and 16 are used to train the model, which meets the requirements of small sample learning. The method proposed in the present invention is also applicable to other remote sensing small sample datasets.

[0070] Step S2: Construct a multimodal model fine-tuning framework.

[0071] like Figure 1 As shown in the figure, a multimodal model fine-tuning framework is constructed using a multimodal image-text contrast pre-training model (Contrastive Lang guage–Image Pretraining, CLIP); the framework consists of an image-text feature extractor, a global pooling layer, a flattening layer, and a linear classification head, and uses an advanced image encoder (ViT-B / 32) and text encoder (Transformer) to encode remote sensing images and text.

[0072] The model fine-tuning process is divided into two stages. The first stage fine-tuning focuses on optimizing the model's image and text encoders so that it can process the spatial structure information contained in remote sensing images. The second stage fine-tuning optimizes the linear classification head parameters so that the model can effectively handle actual remote sensing classification tasks.

[0073] Specifically, the encoder of the multimodal model is fine-tuned in one stage using the Tangram dataset to introduce structural prior knowledge; the classification head of the multimodal model is fine-tuned in two stages using the remote sensing dataset. The learning rate of the Adam optimizer is set to 5e-8, the weight decay is set to 1e-6, and the model is trained for up to 20 epochs.

[0074] Step S3: The first stage of fine-tuning introduces structural prior knowledge to fine-tune the image encoder and text encoder.

[0075] In the first stage of fine-tuning, image-text pair samples are randomly selected from the Kilogram dataset to construct the image-text matching task. In order to optimize the performance of the model in remote sensing image classification, the present invention introduces structural prior knowledge to help the model better understand the global and local structure of remote sensing images, thereby improving its classification accuracy. Specifically:

[0076] S31: Use the Kilogram dataset to set up the image-text bidirectional matching classification task for the first stage.

[0077] Reference Figure 2 , the present invention randomly selects k tangram images and their corresponding text annotations, and shuffles the order of images and texts to construct a text-image feature matrix of size k×k. Two classification tasks are set, including image to text and text to image: given a text feature and a set of image features I = {I1,...,I k}, the goal is to find image samples I that match the text features m ∈I, and calculate the image-to-text matching loss L i2t Similarly, given the image features and a set of text features T = {T1,...,T k}, the goal is to find a text sample T that matches the image features n ∈T, and calculate the text-to-image matching loss L t2i Since the size of the text image feature matrix is ​​k×k, a total of 2k classification tasks are constructed.

[0078] The CLIP model is used to encode image and text samples. Its principle is to match images and texts by calculating the cosine similarity S(.,.). The calculation formula is as follows:

[0079]

[0080] In the formula, I * and T * Represents the most similar images and texts. The goal of each task is to match the corresponding image and text samples. Converted into mathematical form, it means finding the most similar image and text pair.

[0081] S32: Count the number of correct predictions for text and image directions by comparing the predicted values ​​with the true labels.

[0082] Image-text matching loss L i2t and L t2i The calculation formula is as follows:

[0083]

[0084] After obtaining the losses in two directions, calculate the average loss of the two. The formula is as follows:

[0085] L=(L i2t +L t2i ) / 2

[0086] Where L is L i2t and Lt2i This process increases the number of tasks, prevents overfitting due to limited data, and enhances the symmetry of text and image processing, improving the robustness of the model.

[0087] S33: Test the classification accuracy of the multimodal model and determine the model with the best performance.

[0088] In the one-stage fine-tuning process, the encoder of the multimodal model is fine-tuned in one stage using the four data sets in step S1, thereby obtaining four fine-tuned models. In order to determine the best performance model for the classification task, these four fine-tuned models need to be tested. In order to test the generalization ability of the model, the present invention uses the ImageNet data set for testing. The ImageNet data set contains common object categories in the real world, is highly challenging, and can effectively evaluate the generalization performance of the model. Figure 3 ,The model fine-tuned on the partially annotated all-black dataset has better performance.

[0089] Step S4: In the second stage, the multimodal model structure is modified, and a fully connected layer linear classification head is added; the classification head parameters are fine-tuned based on the remote sensing small sample image classification dataset.

[0090] According to the definition of small sample learning, a small number of samples K of 4, 8, and 16 are used from each category to train the model. For each sample, the optimal weight parameters of the model are expressed as a linear combination of the training samples in the feature space according to the Representer theorem, the model's prediction of the input is output, the cross-modal loss is calculated, and the weight parameters of the fully connected layer are updated. Specifically:

[0091] S41: All data of input remote sensing images and text modalities are regarded as training samples x j , the corresponding label is y j , specifically:

[0092]

[0093] In the formula, the modal set is defined as M = {image, text}, belongs to mode m j The set of all data points.

[0094] S42: Establish remote sensing image and text loss functions respectively, and their expressions are:

[0095]

[0096] In the formula, It is a feature extractor for two input modalities, and the softmax loss is calculated using the extracted features

[0097] The premise for calculating the loss function is to map the features of different modalities to the same feature space; the CLIP model meets this requirement by mapping the extracted remote sensing image and text features into the same N-dimensional space, allowing operations on data from different modalities. The mapping formula is as follows:

[0098]

[0099] According to the Representer theorem, the optimal weight parameters of the model are expressed as a linear combination of training samples in the feature space. The formula is as follows:

[0100]

[0101] In the formula, w y Represents the weight parameter of the linear head, by setting the weight parameter α iy , the cross-modal sample features The linear combination of is represented as the optimal weight parameter of the fully connected layer. The model is further fine-tuned to adapt to the downstream classification task by minimizing the cross-modal loss.

[0102] Step S5: Apply the fine-tuned multimodal model to perform a specific image classification task.

[0103] Experimental verification:

[0104] The present invention conducts a classification experiment of a remote sensing small sample data set: the data set used is the Eurosat remote sensing data set, as shown in Table 2, the data set includes 10 categories of industrial buildings, residential buildings, crops, permanent crops, rivers, oceans and lakes, herbaceous vegetation, highways, pastures and forests, and each category scene contains 2000-3000 sample images. Table 1 shows the comparison of the small sample image classification experimental results of the two-stage fine-tuning method and other methods on the EuroSAT data set, where K=4, K=8 and K=16 represent the number of training samples in each category, that is, different small sample scenes.

[0105] As shown in Table 1, the two-stage fine-tuning method proposed in the present invention performs better than other methods in all small sample cases, especially in the settings of K=8 and K=16, achieving significant improvements.

[0106] Table 1 Comparison of small sample image classification experimental results of the method of the present invention and other methods

[0107]

[0108] Table 2 Eurosat remote sensing dataset

[0109] category Number of image samples Industrial buildings 2500 Residential Building 3000 Crops 3000 Permanent crops 2500 river 2500 Ocean Lakes 3000 herb 3000 highway 2500 pasture 2000 forest 3000

[0110] In summary, the two-stage fine-tuning multimodal model method proposed in the present invention can improve the accuracy of small sample image classification, especially in the field of remote sensing images.

[0111] The above description is only used to help understand the method of the present invention and its core essence, but the protection scope of the present invention is not limited thereto. For those skilled in the art in the art, equivalent replacement or change according to the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model, characterized in that: The following steps are involved: S1. Construct a remote sensing small sample image classification dataset; S2. Construct a multimodal model fine-tuning framework; adopt a multimodal image-text comparison pre-trained model as a fine-tuning framework of the multimodal model, wherein the fine-tuning framework of the multimodal model includes an image-text feature extractor, a global pooling layer, a flattening layer, and a linear classification head, and the image-text feature extractor includes an image encoder and a text encoder; S3. In the first stage, structural prior knowledge is introduced, and the image encoder and text encoder of the multimodal model are compared and fine-tuned using symmetric cross entropy loss. An image-text bidirectional matching classification task is constructed based on a remote sensing small sample image classification dataset, and corresponding samples are found according to the maximum similarity. The matching loss is calculated and the average value is obtained, and the multimodal model parameters are adjusted through back propagation and optimization algorithms. S4, the second stage modifies the multimodal model structure, adds a fully connected layer linear classification head and fine-tunes the classification head parameters; trains the model based on the remote sensing small sample image dataset in a fully supervised training manner, calculates the cross-modal loss, and updates the weight parameters of the fully connected layer linear classification head; S5. Apply the fine-tuned multimodal model to perform remote sensing image classification tasks.

2. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 1 is characterized in that: The S1 is specifically as follows: Collect remote sensing data sets, preprocess them, segment them according to labels, extract some images and corresponding text indexes from the remote sensing data of each label category, and construct remote sensing small sample image classification training sets and verification sets.

3. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 2 is characterized in that: The remote sensing dataset is preprocessed, specifically, the image resolution is unified and the spectral and spatial information in the image is retained; the image size is adjusted and the consistency of the image format is ensured by white filling.

4. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 3 is characterized in that: The S3 is as follows: The image-text bidirectional matching classification task includes image-to-text and text-to-image tasks; specifically, it constructs a k×k text-image feature matrix and gives the text features and a set of image features I = {I1,...,I k }, the goal is to find image samples I that match the text features m ∈I, and calculate the image-to-text matching loss L i2t ; Accordingly, the image features are given and a set of text features T = {T1,...,T k }, the goal is to find a text sample T that matches the image features n ∈T, and calculate the text-to-image matching loss L t2i ; The image and text are matched by calculating the cosine similarity S(.,.), and the calculation formula is as follows: In the formula, I * and T * Indicates the image and text with the greatest similarity; By comparing the found feature matching samples with the true labels, the number of correct predictions for text and image orientation is calculated; Image-text matching loss L i2t and L t2i The calculation formula is as follows: After obtaining the losses in two directions, calculate the average loss of the two. The formula is as follows: L=(L i2t +L t2i ) / 2 Where L is L i2t and L t2i The average loss.

5. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 4 is characterized in that: The Kilogram dataset is used to construct a tangram small sample image classification dataset for the first stage of fine-tuning; The image samples in the Kilogram dataset include two forms: each part of the tangram is the same color, and different parts of the tangram are different colors; The text samples in the Kilogram dataset include two types: annotations for the whole and annotations for parts; The two forms of image samples and two types of text samples are arranged and combined to construct four different forms of tangram small sample image classification datasets; the four datasets are used to perform the first stage fine-tuning of the multimodal model respectively, and four fine-tuned multimodal models are obtained; The four fine-tuned multimodal models were tested and their generalization performance was evaluated, and the multimodal model with the best performance was selected as the multimodal model after fine-tuning in the first stage.

6. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to any one of claims 1 to 5, characterized in that: The S4 is specifically as follows: S41, all the data of the input image and text modalities are used as training samples x j , the corresponding label is y j , specifically: In the formula, the modal set is defined as M = {image, text}, belongs to mode m j The set of all data points of ; S42. Establish the loss function L for image and text respectively cross-modal , whose expression is: In the formula, It is a feature extractor for two input modalities, and the softmax loss is calculated using the extracted features The multimodal model maps the extracted image and text features to the same N-dimensional space. The mapping formula is as follows: According to the Representer theorem, the optimal weight parameters of the model are expressed as a linear combination of training samples in the feature space. The formula is as follows: In the formula, w y Represents the weight parameter of the linear classification head, by setting the weight parameter α iy , the cross-modal sample features The linear combination of is represented as the optimal weight parameter of the linear classification head of the fully connected layer.

7. The remote sensing small sample image classification method based on a two-stage fine-tuning multimodal model according to claim 6 is characterized in that: The Eurosat remote sensing dataset is used to construct a remote sensing small sample image classification dataset for the second stage of fine-tuning; From each label category of the Eurosat remote sensing dataset, samples with the number of training samples K of 4, 8, and 16 are used as different remote sensing small sample image classification datasets to train the multimodal model; After training the multimodal model with each remote sensing small sample image classification dataset, the cross-modal loss is calculated and the weight parameters of the linear classification head of the fully connected layer are updated; The classification accuracy of the multimodal models trained with different numbers of samples and updated parameters was tested separately.

Images