Remote sensing image classification method based on prompt learning

Abstract

The invention relates to a remote sensing image classification method based on prompt learning, and the method comprises the steps: obtaining a remote sensing image sample and a real class label, and carrying out the preprocessing, and forming a remote sensing image data set; obtaining an image feature vector and a text feature vector of the remote sensing image sample through an image encoder and a text encoder according to the remote sensing image data set; calculating the similarity between the image feature vector and the text feature vector of the remote sensing image sample to obtain a prediction category label of the remote sensing image sample; calculating classification loss and comparison loss, repeatedly executing the steps, and training and outputting a classification model; and performing remote sensing image classification by using the classification model. According to the method, the problem of insufficient semantic information in a remote sensing image classification method is relieved, and the remote sensing image classification accuracy is improved.

Description

Technical Field

[0001] The invention relates to the field of intelligent interpretation of remote sensing images, and in particular to a remote sensing image classification method based on prompt learning. Background Art

[0002] Remote sensing image classification is a key task in the application of remote sensing technology. It is of great significance to the development of agriculture and environmental monitoring, urban planning, disaster relief and military reconnaissance. Due to the rapid development of remote sensing imaging technology and the increase in remote sensing satellites, the quantity and quality of remote sensing image data have been continuously improved. Based on this, researchers have developed a large number of remote sensing image classification methods with outstanding performance. Among them, deep learning methods based on convolutional neural networks and attention have achieved amazing performance. The remote sensing image classification method based on convolutional neural networks uses local and global spatial features to learn the feature representation of remote sensing images and improve the classification performance of remote sensing images. The convolutional neural network consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer. At the end of the convolutional neural network, the abstract features of the image are obtained through a fully connected layer, and then input into the classifier for classification. Such as VGG, ResNet, etc. The remote sensing image classification method based on the attention mechanism enables the model to focus on specific areas of the image to improve the classification performance of remote sensing images, such as ViT, etc. Although these methods have achieved performance improvements, they only focus on visual representation and ignore the understanding of image semantics, thus hindering the performance improvement of remote sensing image classification methods. Summary of the invention

[0003] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a remote sensing image classification method based on prompt learning, comprising the following steps:

[0004] Step S1, obtaining remote sensing image samples and their true category labels, and performing preprocessing to form a remote sensing image dataset;

[0005] Step S2, obtaining an image feature vector and a text feature vector of a remote sensing image sample through an image encoder and a text encoder according to the remote sensing image data set;

[0006] Step S3, calculating the similarity between the image feature vector and the text feature vector of the remote sensing image sample to obtain the predicted category label of the remote sensing image sample;

[0007] Step S4, calculating the classification loss and the contrast loss, repeating the steps S1 to S4, training and outputting the classification model;

[0008] Step S5: classify the remote sensing image using the classification model.

[0009] According to a technical solution of the present invention, in step S1, the data preprocessing includes:

[0010] Step S11, determining the category information description and the number of categories included in the remote sensing image dataset according to the real category labels;

[0011] Step S12, setting a prompt template, wherein the structure of the prompt template is "This is a photo of the {pro} {cls} used for the classification task", where pro is the attribute information description of the remote sensing image sample, and cls is the category information description of the remote sensing image sample;

[0012] Step S13: generating a prompt for each remote sensing image sample in the remote sensing image dataset according to the true category label and the prompt template.

[0013] According to a technical solution of the present invention, in the step S13, it specifically includes:

[0014] Step S131, defining a category information description list and an attribute information description list, wherein the elements of the category information description list are category information descriptions corresponding to the categories contained in the remote sensing image dataset, and the elements of the attribute information description list are attribute information descriptions corresponding to the category information descriptions one by one, and the lengths of the category information description list and the attribute information description list are equal to the number of categories in the remote sensing image dataset;

[0015] Step S122: according to the real category label, search the category information description list and the attribute information description list, obtain the category information description and the attribute information description corresponding to the real category label, and put them into the pro and cls positions of the prompt template respectively to form a prompt of the remote sensing image sample.

[0016] According to a technical solution of the present invention, in the step S2, it specifically includes:

[0017] Step S21, obtaining an image feature vector of a remote sensing image through the image encoder, wherein the network structure of the image encoder is a Vision Transformer structure of a CLIP image branch;

[0018] Step S22: obtaining a prompt text feature vector of the remote sensing image sample through the text encoder, wherein the network structure of the text encoder is the Transformer structure of the CLIP text branch.

[0019] According to a technical solution of the present invention, in step S3, the similarity between the image feature vector and the text feature vector is calculated to obtain the probability that the remote sensing image sample belongs to each category, and the category with the largest probability value is taken as the predicted category label. The probability that the remote sensing image sample belongs to category k is The calculation formula is as follows:

[0020]

[0021] Among them, <·> is the cosine similarity, f(·) is the image encoder, g(·) is the text encoder, f(x i ) represents the label x of the i-th remote sensing image i The image feature vector, g(t j ) represents the corresponding prompt t of label j j The text feature vector of , the range of k and j is [1, K], K is the number of categories of the remote sensing image dataset, and τ is the temperature hyperparameter.

[0022] According to a technical solution of the present invention, in step S4, it specifically includes:

[0023] Step S41: Calculate the classification loss L according to the category probability obtained in step S3 cls ;

[0024] Step S42: Calculate the contrast loss L according to the image feature vector and text feature vector obtained in step S2. con ;

[0025] Step S43: construct the total loss L total =L cls +λL con , where λ is the balance coefficient, which is used to adjust the weight between losses, L cls is the classification loss, L con is the comparison loss.

[0026] According to a technical solution of the present invention, in step S41, the real category label y of the remote sensing image sample is used. i and the predicted category label obtained in step S3 is the true category label y i The probability of calculating the classification loss L cls , expressed as:

[0027]

[0028] Among them, B is the size of the batch.

[0029] According to a technical solution of the present invention, in the step S422, it specifically includes:

[0030] Step S4221: construct a supervision matrix gt according to the real category labels of the remote sensing image samples. The shape of gt is B×B, where B is the size of the batch and its elements are:

[0031]

[0032] Among them, y p and q Respectively represent the labels of images with serial numbers p and q in the batch of images, and the range of p and q is [0, B-1];

[0033] Step S4222: Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on the image feature vector and text feature vector obtained in step S2, and calculate the contrast loss L using the supervision matrix gt obtained in step S4221. con =(L i2t +L t2i ) / 2, where L i2t and L t2i They are all BCEWithLogitsLoss loss functions, expressed as:

[0034] BCEWithLogitsLoss={l1,...,l N},l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))]

[0035] Where N represents the number of batches, n represents the number of labels predicted for each batch, and σ(·) represents the Sigmoid function, which is expressed as:

[0036]

[0037] According to a technical solution of the present invention, in the step S422, it specifically includes:

[0038] Step S4221, calculate the image text similarity matrix i2t_logit of the remote sensing image sample, expressed as:

[0039] i2t_logit=I·T Τ

[0040] Where, represents matrix multiplication, I is the image feature matrix of a batch of remote sensing image samples, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing image samples;

[0041] Step S4222, calculate the text image similarity matrix t2i_logit:

[0042] t2i_logit=T·I Τ ;

[0043] Step S4223: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the image-text similarity matrix i2t_logit obtained in step S4221. i2t :

[0044] L i2t =BCEWithLogitsLoss(i2t_logit,gt);

[0045] Step S4224: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the text image similarity matrix t2i_logit obtained in step S4222. t2i :

[0046] L t2i =BCEWithLogitsLoss(t2i_logit,gt).

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention proposes a remote sensing image classification method based on prompt learning, a remote sensing image classification method based on prompt learning, converts the classification task into an image-text matching task based on the CLIP model framework, designs a prompt template that reflects the remote sensing image attributes and context information to convert the classification label into a text prompt, and uses contrast loss to solve the problem of too few positive sample pairs. The present invention uses prompt learning and two modal information, image and text, to provide the model with richer semantic and context information, alleviates the problem of low classification accuracy caused by the remote sensing image classification method only focusing on visual representation and ignoring semantic information mining, and improves the accuracy of remote sensing image classification. The present invention makes full use of the prior knowledge of the visual language model, and through contrast learning, converts the classification task into an image and text matching task, further improving the classification accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A model training flow chart of a remote sensing image classification method based on prompt learning according to an embodiment of the present invention is schematically shown;

[0051] Figure 2 A flowchart schematically showing a remote sensing image classification method based on prompt-based learning according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram schematically illustrates the structure of a remote sensing image classification model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The description of the embodiments of this specification should be combined with the corresponding drawings, which should be considered as part of the complete specification. In the drawings, the shape or thickness of the embodiments may be enlarged and indicated for simplification or convenience. Furthermore, the parts of each structure in the drawings will be described separately. It is worth noting that the elements not shown in the drawings or not described in words are in a form known to ordinary technicians in the relevant technical field.

[0054] The description of the embodiments herein and any reference to directions and orientations are only for the convenience of description and are not to be construed as any limitation on the scope of protection of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination, and the present invention is not particularly limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0055] The remote sensing image classification method based on prompt learning provided by the present invention integrates prompt learning into visual tasks, so that the model learns rich image semantic information and improves the classification performance of the model. Prompt learning originates from the field of natural language processing. It processes the input text information according to a specific template and reconstructs the task into a form that fully utilizes the processing of the pre-trained language model to train the model. The prompt is the input context. The visual language model constructs a two-branch network consisting of a visual encoder and a text encoder by drawing on the idea of ​​prompt learning. The text information is used as a prompt for the image, so that the model can fully learn the semantic information of the image. CLIP is one of the first works to introduce prompt learning into the visual field. It reconstructs the classification task into an image and text matching task through contrastive learning to complete the model training. CoOp, based on CLIP, uses a learnable vector instead of a manually pre-designed prompt template. However, during their training, each image in a batch only constitutes a positive sample pair with a text prompt. The imbalance between the positive and negative sample pairs may hinder the improvement of the model classification performance. At the same time, their text prompt templates are designed for natural images, without considering the characteristics of remote sensing images, and cannot be directly used for remote sensing image classification. The present invention can effectively solve the above problems.

[0056] like Figures 1 to 3 As shown, a remote sensing image classification method based on prompt learning of the present invention comprises the following steps:

[0057] Step S1: obtain remote sensing image samples and their true category labels, and perform preprocessing to form a remote sensing image dataset.

[0058] In step S1, data preprocessing includes:

[0059] Step S11: Determine the category information description and the number of categories included in the remote sensing image dataset according to the real category labels.

[0060] By accurately labeling the category of each pixel in the remote sensing image dataset, a more accurate classification model can be trained, thereby improving the accuracy of remote sensing image interpretation.

[0061] Step S12, setting a prompt template, the structure of the prompt template is "This is a photo of the {pro} {cls} used for the classification task", pro is the attribute information description of the remote sensing image sample, and cls is the category information description of the remote sensing image sample.

[0062] Step S13: Generate a prompt for each remote sensing image sample in the remote sensing image dataset according to the true category label and the prompt template.

[0063] In step S13, it specifically includes:

[0064] Step S131, define a category information description list and an attribute information description list, the elements of the category information description list are category information descriptions corresponding to the categories contained in the remote sensing image dataset, the elements of the attribute information description list are attribute information descriptions corresponding one-to-one to the category information descriptions, and the length of the category information description list and the attribute information description list are equal to the number of categories in the remote sensing image dataset.

[0065] By defining an explicit category information description for each category, we can ensure that consistent and standardized category labels are used throughout the data processing and model training process; the attribute information description list provides detailed attribute information related to each category, which helps the model better understand the characteristics and context of each category; in addition, by combining category information and attribute information, the model can more comprehensively learn the semantic content of remote sensing images and improve the in-depth understanding of image content.

[0066] Step S132: search the category information description list and the attribute information description list according to the real category label, obtain the category information description and the attribute information description corresponding to the real category label, and put them into the pro and cls positions of the prompt template respectively to form a prompt of the remote sensing image sample.

[0067] Each image sample generates a prompt containing its specific category and attribute information, allowing the model to receive more personalized and targeted information; by embedding the correct category and attribute information into the prompt, the model's ability to semantically match image content with text descriptions can be enhanced; it also helps the model better understand the context of the image, thereby making more accurate predictions in classification tasks.

[0068] In some embodiments of the present invention, after acquiring the remote sensing image and the label, a prompt template is designed, and the template structure is "This is a photo of the {pro} {cls} used for the classification task", pro is the attribute and semantic information description of the remote sensing image, and cls is the category information description of the remote sensing image.

[0069] By combining category information and attribute information into the prompt template, the model not only learns the visual features of the image, but also learns the semantic information of the image, enhancing the model's in-depth understanding of the image content; rich semantic information can help the model better distinguish different categories of remote sensing images, especially when the visual features between categories are similar, thereby improving the accuracy of classification.

[0070] Step S2: According to the remote sensing image data set, an image feature vector and a text feature vector of the remote sensing image sample are obtained through an image encoder and a text encoder.

[0071] In step S2, it specifically includes:

[0072] Step S21, obtaining an image feature vector of the remote sensing image through an image encoder, wherein the network structure of the image encoder is a Vision Transformer structure of a CLIP image branch;

[0073] Step S22: obtaining a prompt text feature vector of the remote sensing image sample through a text encoder, wherein the network structure of the text encoder is the Transformer structure of the CLIP text branch.

[0074] Through the image encoder and text encoder, cross-modal features can be extracted from remote sensing images and related texts, enabling the model to understand the visual content of the image and its semantic information; the CLIP model as an encoder can utilize pre-trained knowledge to improve the ability of feature extraction; the Vision Transformer structure can effectively capture the local and global features of the image, and the Transformer structure performs well in processing sequence data (such as text). The combination of the two can provide rich feature representation.

[0075] Step S3: Calculate the similarity between the image feature vector and the text feature vector of the remote sensing image sample to obtain the predicted category label of the remote sensing image sample.

[0076] In step S3, the similarity between the image feature vector and the text feature vector is calculated to obtain the probability that the remote sensing image sample belongs to each category. The category with the largest probability value is taken as the predicted category label. The probability that the remote sensing image sample belongs to category k is The calculation formula is as follows:

[0077]

[0078] Among them, <·> is the cosine similarity, f(·) is the image encoder, g(·) is the text encoder, f(x i ) represents the i-th remote sensing image sample x i The image feature vector, g(t j ) represents the corresponding prompt t of label j j The text feature vector of , the range of k and j is [1, K], K is the number of categories of the remote sensing image dataset, and τ is the temperature hyperparameter.

[0079] The model can comprehensively consider the visual features of the image and the semantic features of the text, and predict the category of the remote sensing image by calculating the similarity between them, thereby improving the accuracy and reliability of classification.

[0080] Step S4, calculating the classification loss and contrast loss, repeating steps S1 to S4, training and outputting the classification model;

[0081] In step S4, it specifically includes:

[0082] Step S41: Calculate the classification loss L according to the category probability obtained in step S3 cls ;

[0083] In step S41, the real category label y of the remote sensing image sample is used i and the predicted class label obtained in step S3 is the true category label y i The probability of calculating the classification loss Lcls , expressed as:

[0084]

[0085] Among them, B is the size of the batch.

[0086] Step S42: Calculate the contrast loss L according to the image feature vector and text feature vector obtained in step S2. con ;

[0087] In step S42, it specifically includes:

[0088] Step S421: construct a supervision matrix gt according to the real category labels of the remote sensing image samples. The shape of gt is B×B, where B is the size of the batch and its elements are:

[0089]

[0090] Among them, y p and q Respectively represent the labels of images with serial numbers p and q in the batch of images, and the range of p and q is [0, B-1];

[0091] Step S422: Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on the image feature vector and text feature vector obtained in step S2, and calculate the contrast loss L using the supervision matrix gt obtained in step S421. con =(L i2t +L t2i ) / 2, where L i2t and L t2i They are all BCEWithLogitsLoss loss functions, expressed as:

[0092] BCEWithLogitsLoss={l1,...,l N},l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))]

[0093] Where N represents the number of batches, n represents the number of labels predicted for each batch, and σ(·) represents the Sigmoid function, which is expressed as:

[0094]

[0095] In step S422, it specifically includes:

[0096] Step S4221, calculate the image text similarity matrix i2t_logit of the remote sensing image sample, expressed as:

[0097] i2t_logit=I·T Τ

[0098] Where, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing image samples, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing image samples;

[0099] Step S4222, calculate the text image similarity matrix t2i_logit:

[0100] t2i_logit=T·I Τ ;

[0101] Step S4223: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the image-text similarity matrix i2t_logit obtained in step S4221 i2t :

[0102] L i2t =BCEWithLogitsLoss(i2t_logit,gt);

[0103] Step S4224: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the text image similarity matrix t2i_logit obtained in step S4222 t2i :

[0104] L t2i =BCEWithLogitsLoss(t2i_logit,gt)

[0105] Step S43: construct the total loss L total =L cls +λL con , where λ is the balance coefficient, which is used to adjust the weight between losses, L cls is the classification loss, L con is the comparison loss.

[0106] Considering classification loss and contrast loss at the same time can enable the model to learn richer and more discriminative feature representations while accurately classifying; contrast loss helps the model learn more robust features, which not only perform well on training data, but also maintain stable performance on new and unseen data; through contrast loss, the model is encouraged to cluster similar samples together and push dissimilar samples apart, thereby enhancing the discrimination in the feature space; in remote sensing image classification, some categories may have a small number of samples, and contrastive learning can help the model make better use of limited samples and improve model performance by learning discriminative features; by introducing the balance coefficient λ, the contribution of classification loss and contrast loss to the total loss can be flexibly adjusted, making model training more adaptable to different data distributions and task requirements.

[0107] Step S5: classify the remote sensing image using the classification model.

[0108] Based on the advanced techniques and strategies adopted in the previous steps, step S5 can achieve high-accuracy classification.

[0109] In summary, the present invention proposes a remote sensing image classification method based on prompt learning. In training, first, according to the label of the remote sensing image and the designed prompt template, a corresponding text prompt is generated for each remote sensing image, an image encoder is constructed based on the VisionTransformer network structure of the CLIP image branch, and the feature representation of the remote sensing image is obtained through the image encoder. A text encoder is constructed based on the Transformer network structure of the CLIP text branch, and the text encoder is used to obtain the text features of the prompt; then, the similarity between the image and the text is calculated to obtain the category probability and the similarity matrix, and the classification task is converted into an image-text matching task; the classification loss is calculated according to the real category label and the contrast loss is calculated by the constructed supervision matrix, and the total loss is obtained by weighting the two, and the model training is completed by back propagation. The present invention enables the model to learn the semantics and context information of the remote sensing image through prompt learning and the designed prompt template suitable for remote sensing images, thereby alleviating the problem that the remote sensing image classification method only focuses on the visual representation and ignores the semantic information. At the same time, by constructing a supervision matrix for calculating the contrast loss to avoid the problem of too few positive sample pairs in the training process, the classification accuracy of the remote sensing image is further improved, which is of great significance to the classification of remote sensing images.

[0110] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0111] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] It should also be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0114] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for those skilled in the art, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A remote sensing image classification method based on prompt learning, characterized in that: The following steps are involved: Step S1, obtaining remote sensing image samples and their true category labels, and performing preprocessing to form a remote sensing image dataset; Step S2, obtaining an image feature vector and a text feature vector of a remote sensing image sample through an image encoder and a text encoder according to the remote sensing image data set; Step S3, calculating the similarity between the image feature vector and the text feature vector of the remote sensing image sample to obtain the predicted category label of the remote sensing image sample; Step S4, calculating the classification loss and the contrast loss, repeating the steps S1 to S4, training and outputting the classification model; Step S5: classify the remote sensing image using the classification model.

2. The remote sensing image classification method based on prompt learning according to claim 1, characterized in that: In the step S1, the data preprocessing includes: Step S11, determining the category information description and the number of categories included in the remote sensing image dataset according to the real category labels; Step S12, setting a prompt template, wherein the structure of the prompt template is "This is a photo of the {pro} {cls} used for the classification task", pro is the attribute information description of the remote sensing image sample, and cls is the category information description of the remote sensing image sample; Step S13: generating a prompt for each remote sensing image sample in the remote sensing image dataset according to the true category label and the prompt template.

3. The remote sensing image classification method based on prompt learning according to claim 2, characterized in that: In the step S13, it specifically includes: Step S131, defining a category information description list and an attribute information description list, wherein the elements of the category information description list are category information descriptions corresponding to the categories contained in the remote sensing image dataset, and the elements of the attribute information description list are attribute information descriptions corresponding to the category information descriptions one by one, and the lengths of the category information description list and the attribute information description list are equal to the number of categories in the remote sensing image dataset; Step S132: according to the real category label, search the category information description list and the attribute information description list, obtain the category information description and the attribute information description corresponding to the real category label, and put them into the pro and cls positions of the prompt template respectively to form a prompt of the remote sensing image sample.

4. The remote sensing image classification method based on prompt learning according to claim 3 is characterized in that: In the step S2, it specifically includes: Step S21, obtaining an image feature vector of a remote sensing image through the image encoder, wherein the network structure of the image encoder is a Vision Transformer structure of a CLIP image branch; Step S22: obtaining a prompt text feature vector of the remote sensing image sample through the text encoder, wherein the network structure of the text encoder is the Transformer structure of the CLIP text branch.

5. The remote sensing image classification method based on prompt learning according to claim 4 is characterized in that: In step S3, the similarity between the image feature vector and the text feature vector is calculated to obtain the probability that the remote sensing image sample belongs to each category, and the category with the largest probability value is taken as the predicted category label. The probability that the remote sensing image sample belongs to category k is The calculation formula is as follows: Among them, <·> is the cosine similarity, f(·) is the image encoder, g(·) is the text encoder, f(x i ) represents the i-th remote sensing image sample x i The image feature vector, g(t j ) represents the corresponding prompt t of label j j The text feature vector of , the range of k and j is [1, K], K is the number of categories of the remote sensing image dataset, and τ is the temperature hyperparameter.

6. The remote sensing image classification method based on prompt learning according to claim 5, characterized in that: In the step S4, it specifically includes: Step S41: Calculate the classification loss L according to the category probability obtained in step S3 cls ; Step S42: Calculate the contrast loss L according to the image feature vector and text feature vector obtained in step S2. con ; Step S43: construct the total loss L total =L cls +λL con , where λ is the balance coefficient, which is used to adjust the weight between losses, L cls is the classification loss, L con is the comparison loss.

7. The remote sensing image classification method based on prompt learning according to claim 6, characterized in that: In step S41, the real category label y of the remote sensing image sample is used i and the predicted category label obtained in step S3 is the true category label y i The probability of calculating the classification loss L cls , expressed as: Among them, B is the size of the batch.

8. The remote sensing image classification method based on prompt learning according to claim 6, characterized in that: In the step S42, it specifically includes: Step S421: construct a supervision matrix gt according to the real category labels of the remote sensing image samples. The shape of gt is B×B, where B is the size of the batch and its elements are: Among them, y p and q Respectively represent the labels of images with serial numbers p and q in the batch of images, and the range of p and q is [0, B-1]; Step S4222: Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on the image feature vector and text feature vector obtained in step S2, and calculate the contrast loss L using the supervision matrix gt obtained in step S4221. con =(L i2t +L t2i ) / 2, where L i2t and L t2i They are all BCEWithLogitsLoss loss functions, expressed as: BCEWithLogitsLoss={l1,...,l N },l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))] Where N represents the number of batches, n represents the number of labels predicted for each batch, and σ(·) represents the Sigmoid function, which is expressed as:

9. The remote sensing image classification method based on prompt learning according to claim 8, characterized in that: In the step S422, it specifically includes: Step S4221, calculate the image text similarity matrix i2t_logit of the remote sensing image sample, expressed as: i2t_logit=I·T Τ Where, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing image samples, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing image samples; Step S4222, calculate the text image similarity matrix t2i_logit: t2i_logit=T·I Τ ; Step S4223: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the image-text similarity matrix i2t_logit obtained in step S4221. i2t : L i2t =BCEWithLogitsLoss(i2t_logit,gt); Step S4224: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the text image similarity matrix t2i_logit obtained in step S4222. t2i : L t2i =BCEWithLogitsLoss(t2i_logit,gt)。

Images