Remote sensing image classification method based on prompt learning

Abstract

The present invention relates to a remote sensing image classification method based on prompt learning, comprising: obtaining remote sensing image samples and their true category labels, and performing preprocessing to form a remote sensing image dataset; obtaining image feature vectors and text feature vectors of the remote sensing image samples using an image encoder and a text encoder based on the remote sensing image dataset; calculating the similarity between the image feature vectors and text feature vectors of the remote sensing image samples to obtain predicted category labels for the remote sensing image samples; calculating classification loss and contrast loss, repeatedly performing the above steps, training and outputting a classification model; and using the classification model to classify remote sensing images. The present invention alleviates the problem of insufficient semantic information in remote sensing image classification methods through the above method, thereby improving the accuracy of remote sensing image classification.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interpretation of remote sensing images, and in particular to a remote sensing image classification method based on prompt learning. Background Art

[0002] Remote sensing image classification is a key application of remote sensing technology, crucial for advancements in fields such as agricultural and environmental monitoring, urban planning, disaster relief, and military reconnaissance. Due to the rapid advancement of remote sensing imaging technology and the proliferation of remote sensing satellites, the quantity and quality of remote sensing image data have continuously increased. Researchers have developed numerous high-performing remote sensing image classification methods. Among these, convolutional neural network-based and attention-based deep learning methods have achieved remarkable performance. Convolutional neural network-based remote sensing image classification methods leverage local and global spatial features to learn feature representations of remote sensing images, improving classification performance. A convolutional neural network consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. Finally, a fully connected layer in the convolutional neural network captures abstract image features, which are then fed into a classifier for classification. These methods, such as VGG and ResNet, utilize attention mechanisms to improve classification performance by focusing on specific regions of the image. These methods, such as ViT, utilize attention mechanisms to improve classification performance. Although these methods have achieved performance improvements, they only focus on visual representation and ignore the understanding of image semantics, which hinders the performance improvement of remote sensing image classification methods. Summary of the Invention

[0003] To solve the technical problems existing in the above-mentioned prior art, the present invention provides a remote sensing image classification method based on prompt learning, comprising the following steps:

[0004] Step S1: Obtain remote sensing image samples and their true category labels, and perform preprocessing to form a remote sensing image dataset;

[0005] Step S2: Obtain image feature vectors and text feature vectors of remote sensing image samples through an image encoder and a text encoder according to the remote sensing image dataset;

[0006] Step S3: Calculate the similarity between the image feature vector and the text feature vector of the remote sensing image sample to obtain the predicted category label of the remote sensing image sample;

[0007] Step S4: Calculate the classification loss and contrast loss, repeat steps S1 to S4, train and output the classification model;

[0008] Step S5: classify the remote sensing image using the classification model.

[0009] According to a technical solution of the present invention, in step S1, the data preprocessing includes:

[0010] Step S11: determining the category information description and the number of categories included in the remote sensing image dataset based on the true category labels;

[0011] Step S12: setting a prompt template, wherein the structure of the prompt template is "This is a photo of the {pro} {cls} used for the classification task", where pro is the attribute information description of the remote sensing image sample, and cls is the category information description of the remote sensing image sample;

[0012] Step S13: generating a prompt for each remote sensing image sample in the remote sensing image dataset according to the true category label and the prompt template.

[0013] According to a technical solution of the present invention, in step S13, the following steps are specifically included:

[0014] Step S131: defining a category information description list and an attribute information description list, wherein the elements of the category information description list are category information descriptions corresponding to the categories included in the remote sensing image dataset, and the elements of the attribute information description list are attribute information descriptions corresponding one-to-one to the category information descriptions, and the length of the category information description list and the attribute information description list are equal to the number of categories in the remote sensing image dataset;

[0015] Step S122: According to the true category label, search the category information description list and the attribute information description list, obtain the category information description and attribute information description corresponding to the true category label, and put them into the pro and cls positions of the prompt template respectively to form a prompt of the remote sensing image sample.

[0016] According to a technical solution of the present invention, in step S2, it specifically includes:

[0017] Step S21: obtaining an image feature vector of a remote sensing image through the image encoder, wherein the network structure of the image encoder is a Vision Transformer structure of a CLIP image branch;

[0018] Step S22: obtaining a prompt text feature vector of the remote sensing image sample through the text encoder, wherein the network structure of the text encoder is the Transformer structure of the CLIP text branch.

[0019] According to a technical solution of the present invention, in step S3, the similarity between the image feature vector and the text feature vector is calculated to obtain the probability that the remote sensing image sample belongs to each category, and the category with the largest probability value is taken as the predicted category label. The probability that the remote sensing image sample belongs to category k is The calculation formula is as follows:

[0020]

[0021] Among them, <·> is the cosine similarity, f(·) is the image encoder, g(·) is the text encoder, f(x i ) represents the label x of the i-th remote sensing image i The image feature vector, g(t j ) represents the corresponding prompt t of label j j The text feature vector of , the range of k and j is [1, K], K is the number of categories of the remote sensing image dataset, and τ is the temperature hyperparameter.

[0022] According to a technical solution of the present invention, in step S4, it specifically includes:

[0023] Step S41: Calculate the classification loss L based on the category probability obtained in step S3 cls ;

[0024] Step S42: Calculate the contrast loss L based on the image feature vector and text feature vector obtained in step S2. con ;

[0025] Step S43: Construct the total loss L total =L cls +λL con , where λ is the balance coefficient, which is used to adjust the weight between losses, L cls is the classification loss, L con is the contrast loss.

[0026] According to a technical solution of the present invention, in step S41, the real category label y of the remote sensing image sample is used. i and the predicted category label obtained in step S3 is the true category label y i The probability of calculating the classification loss L cls , expressed as:

[0027]

[0028] Among them, B is the batch size.

[0029] According to a technical solution of the present invention, in step S422, the following steps are specifically included:

[0030] Step S4221: Construct a supervision matrix gt based on the true category labels of the remote sensing image samples. The shape of gt is B×B, where B is the size of the batch, and its elements are:

[0031]

[0032] Among them, y p and y q Respectively represent the labels of images with sequence numbers p and q in the batch of images, and the range of p and q is [0, B-1];

[0033] Step S4222: Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on the image feature vector and text feature vector obtained in step S2, and calculate the contrast loss L using the supervision matrix gt obtained in step S4221. con =(L i2t +L t2i ) / 2, where L i2t and L t2i They are all BCEWithLogitsLoss loss functions, expressed as:

[0034] BCEWithLogitsLoss={l1,...,l N},l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))]

[0035] Where N represents the number of batches, n represents the number of labels predicted for each batch, and σ(·) represents the Sigmoid function, which is expressed as:

[0036]

[0037] According to a technical solution of the present invention, in step S422, the following steps are specifically included:

[0038] Step S4221: Calculate the image-text similarity matrix i2t_logit of the remote sensing image sample, expressed as:

[0039] i2t_logit=I·T Τ

[0040] Wherein, represents matrix multiplication, I is the image feature matrix of a batch of remote sensing image samples, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing image samples;

[0041] Step S4222: Calculate the text image similarity matrix t2i_logit:

[0042] t2i_logit=T·I Τ ;

[0043] Step S4223: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the image-text similarity matrix i2t_logit obtained in step S4221. i2t :

[0044] L i2t =BCEWithLogitsLoss(i2t_logit,gt);

[0045] Step S4224: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the text image similarity matrix t2i_logit obtained in step S4222. t2i :

[0046] L t2i =BCEWithLogitsLoss(t2i_logit,gt).

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention proposes a remote sensing image classification method based on prompt learning. The remote sensing image classification method based on prompt learning converts the classification task into an image-text matching task based on the CLIP model framework, designs a prompt template that reflects the remote sensing image attributes and contextual information to convert the classification label into a text prompt, and uses contrast loss to solve the problem of too few positive sample pairs. Through prompt learning, the present invention simultaneously utilizes both image and text modal information to provide the model with richer semantic and contextual information, alleviates the problem of low classification accuracy caused by remote sensing image classification methods that only focus on visual representation while ignoring semantic information mining, and improves the accuracy of remote sensing image classification. The present invention fully utilizes the prior knowledge of the visual language model and, through contrastive learning, converts the classification task into an image and text matching task, further improving the classification accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0050] Figure 1 A model training flow chart schematically illustrating a remote sensing image classification method based on prompt learning according to an embodiment of the present invention;

[0051] Figure 2 A flowchart schematically illustrating a remote sensing image classification method based on prompt-based learning according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram schematically illustrates the structure of a remote sensing image classification model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The description of the embodiments in this specification should be combined with the corresponding drawings, which should be considered a complete part of this specification. In the drawings, the shapes and thicknesses of the embodiments may be exaggerated and indicated for simplicity or convenience. Furthermore, the various structural components in the drawings will be described separately. It is worth noting that components not shown in the drawings or not described in words are known to those of ordinary skill in the art.

[0054] The description of the embodiments herein and any references to directions and orientations are for ease of description only and are not to be construed as limiting the scope of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination. The present invention is not specifically limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0055] The proposed remote sensing image classification method based on cue learning integrates cue learning into the visual task, enabling the model to learn rich image semantic information and improving the model's classification performance. Cue learning originates from the field of natural language processing. It processes input text information according to a specific template, reconstructing the task into a form that fully leverages the processing of a pretrained language model to train the model. The cue serves as the input context. Drawing on the ideas of cue learning, the visual language model constructs a two-branch network consisting of a visual encoder and a text encoder. This uses text information as a cue for the image, allowing the model to fully learn the image's semantic information. CLIP, one of the first methods to introduce cue learning into the visual field, uses contrastive learning to reconstruct the classification task into an image-text matching task for model training. CoOp, building on CLIP, uses learnable vectors instead of manually designed cue templates. However, during training, each image in a batch is paired with only one text cue as a positive sample. This imbalance between positive and negative sample pairs can hinder the improvement of model classification performance. Furthermore, their text cue templates are designed for natural images and do not consider the characteristics of remote sensing images, making them unsuitable for direct application in remote sensing image classification. The present invention can effectively solve the above problems.

[0056] like Figures 1 to 3 As shown, a remote sensing image classification method based on prompt learning of the present invention comprises the following steps:

[0057] Step S1: Obtain remote sensing image samples and their true category labels, and perform preprocessing to form a remote sensing image dataset.

[0058] In step S1, data preprocessing includes:

[0059] Step S11: Determine the category information description and the number of categories included in the remote sensing image dataset based on the true category labels.

[0060] By accurately labeling the category of each pixel in the remote sensing image dataset, a more accurate classification model can be trained, thereby improving the accuracy of remote sensing image interpretation.

[0061] Step S12: Set a prompt template. The structure of the prompt template is "This is a photo of the {pro} {cls} used for the classification task", where pro is the attribute information description of the remote sensing image sample, and cls is the category information description of the remote sensing image sample.

[0062] Step S13: Generate a prompt for each remote sensing image sample in the remote sensing image dataset based on the true category label and the prompt template.

[0063] In step S13, it specifically includes:

[0064] Step S131, define a category information description list and an attribute information description list, the elements of the category information description list are category information descriptions corresponding to the categories contained in the remote sensing image dataset, and the elements of the attribute information description list are attribute information descriptions corresponding one-to-one to the category information descriptions. The length of the category information description list and the attribute information description list is equal to the number of categories in the remote sensing image dataset.

[0065] By defining a clear category information description for each category, we can ensure that consistent and standardized category labels are used throughout the data processing and model training process; the attribute information description list provides detailed attribute information related to each category, which helps the model better understand the characteristics and context of each category; in addition, by combining category information and attribute information, the model can more comprehensively learn the semantic content of remote sensing images and improve its in-depth understanding of image content.

[0066] Step S132: According to the true category label, search the category information description list and the attribute information description list, obtain the category information description and attribute information description corresponding to the true category label, and put them into the pro and cls positions of the prompt template respectively to form a prompt of the remote sensing image sample.

[0067] Each image sample generates a prompt containing its specific category and attribute information, allowing the model to receive more personalized and targeted information; by embedding the correct category and attribute information into the prompt, the model's ability to semantically match image content with text descriptions can be enhanced; it also helps the model better understand the context of the image, thereby making more accurate predictions in classification tasks.

[0068] In some embodiments of the present invention, after obtaining remote sensing images and labels, a prompt template is designed, and the template structure is "This is a photo of the {pro} {cls} used for the classification task", where pro is the attribute and semantic information description of the remote sensing image, and cls is the category information description of the remote sensing image.

[0069] By combining category information and attribute information into the prompt template, the model not only learns the visual features of the image, but also learns the semantic information of the image, enhancing the model's in-depth understanding of the image content; rich semantic information can help the model better distinguish different categories of remote sensing images, especially when the visual features between categories are similar, thereby improving the accuracy of classification.

[0070] Step S2: Obtain image feature vectors and text feature vectors of remote sensing image samples through an image encoder and a text encoder according to the remote sensing image dataset.

[0071] In step S2, it specifically includes:

[0072] Step S21: Obtain an image feature vector of the remote sensing image through an image encoder, wherein the network structure of the image encoder is the Vision Transformer structure of the CLIP image branch;

[0073] Step S22: Obtain a text feature vector of the prompt of the remote sensing image sample through a text encoder, where the network structure of the text encoder is the Transformer structure of the CLIP text branch.

[0074] Through the image encoder and text encoder, cross-modal features can be extracted from remote sensing images and related texts, enabling the model to understand the visual content of the image and its semantic information; the CLIP model as an encoder can utilize pre-training knowledge to improve the ability of feature extraction; the Vision Transformer structure can effectively capture the local and global features of the image, and the Transformer structure performs well in processing sequence data (such as text). The combination of the two can provide rich feature representation.

[0075] Step S3: Calculate the similarity between the image feature vector and the text feature vector of the remote sensing image sample to obtain the predicted category label of the remote sensing image sample.

[0076] In step S3, the similarity between the image feature vector and the text feature vector is calculated to obtain the probability that the remote sensing image sample belongs to each category. The category with the largest probability value is taken as the predicted category label. The probability that the remote sensing image sample belongs to category k is The calculation formula is as follows:

[0077]

[0078] Among them, <·> is the cosine similarity, f(·) is the image encoder, g(·) is the text encoder, f(x i ) represents the i-th remote sensing image sample x i The image feature vector, g(t j ) represents the corresponding prompt t of label j j The text feature vector of , the range of k and j is [1, K], K is the number of categories of the remote sensing image dataset, and τ is the temperature hyperparameter.

[0079] The model can comprehensively consider the visual features of the image and the semantic features of the text, and predict the category of the remote sensing image by calculating the similarity between them, thereby improving the accuracy and reliability of classification.

[0080] Step S4: Calculate the classification loss and contrast loss, repeat steps S1 to S4, train and output the classification model;

[0081] In step S4, it specifically includes:

[0082] Step S41: Calculate the classification loss L based on the category probability obtained in step S3 cls ;

[0083] In step S41, the real category label y of the remote sensing image sample is used i and the predicted category label obtained in step S3 is the true category label y i The probability of calculating the classification loss Lcls , expressed as:

[0084]

[0085] Among them, B is the batch size.

[0086] Step S42: Calculate the contrast loss L based on the image feature vector and text feature vector obtained in step S2. con ;

[0087] In step S42, it specifically includes:

[0088] Step S421: Construct a supervision matrix gt based on the true category labels of the remote sensing image samples. The shape of gt is B×B, where B is the size of the batch and its elements are:

[0089]

[0090] Among them, y p and y q Respectively represent the labels of images with sequence numbers p and q in the batch of images, and the range of p and q is [0, B-1];

[0091] Step S422: Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on the image feature vector and text feature vector obtained in step S2, and calculate the contrast loss L using the supervision matrix gt obtained in step S421. con =(L i2t +L t2i ) / 2, where L i2t and L t2i They are all BCEWithLogitsLoss loss functions, expressed as:

[0092] BCEWithLogitsLoss={l1,...,l N},l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))]

[0093] Where N represents the number of batches, n represents the number of labels predicted for each batch, and σ(·) represents the Sigmoid function, which is expressed as:

[0094]

[0095] In step S422, it specifically includes:

[0096] Step S4221: Calculate the image-text similarity matrix i2t_logit of the remote sensing image sample, expressed as:

[0097] i2t_logit=I·T Τ

[0098] Where, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing image samples, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing image samples;

[0099] Step S4222: Calculate the text image similarity matrix t2i_logit:

[0100] t2i_logit=T·I Τ ;

[0101] Step S4223: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the image-text similarity matrix i2t_logit obtained in step S4221. i2t :

[0102] L i2t =BCEWithLogitsLoss(i2t_logit,gt);

[0103] Step S4224: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the text image similarity matrix t2i_logit obtained in step S4222. t2i :

[0104] L t2i =BCEWithLogitsLoss(t2i_logit,gt)

[0105] Step S43: Construct the total loss L total =L cls +λL con , where λ is the balance coefficient, which is used to adjust the weight between losses, L cls is the classification loss, L con is the contrast loss.

[0106] Considering both classification loss and contrastive loss simultaneously enables the model to learn richer and more discriminative feature representations while achieving accurate classification; contrastive loss helps the model learn more robust features that not only perform well on training data but also maintain stable performance on new, unseen data; through contrastive loss, the model is encouraged to cluster similar samples together while pushing dissimilar samples apart, thereby enhancing the discrimination in the feature space; in remote sensing image classification, some categories may have fewer samples, and contrastive learning can help the model make better use of limited samples and improve model performance by learning discriminative features; by introducing the balance coefficient λ, the contribution of classification loss and contrastive loss to the total loss can be flexibly adjusted, making model training more adaptable to different data distributions and task requirements.

[0107] Step S5: classify the remote sensing image using the classification model.

[0108] Based on the advanced techniques and strategies adopted in the previous steps, step S5 can achieve high-accuracy classification.

[0109] In summary, the present invention proposes a remote sensing image classification method based on prompt learning. During training, first, a corresponding text prompt is generated for each remote sensing image based on the label of the remote sensing image and the designed prompt template. An image encoder is constructed based on the VisionTransformer network structure of the CLIP image branch, and the feature representation of the remote sensing image is obtained through the image encoder. A text encoder is constructed based on the Transformer network structure of the CLIP text branch, and the text features of the prompt are obtained using the text encoder. Then, the similarity between the image and the text is calculated to obtain the category probability and similarity matrix, and the classification task is converted into an image-text matching task. The classification loss is calculated based on the true category label and the contrast loss is calculated using the constructed supervision matrix. The two are weighted to obtain the total loss, and the model training is completed by backpropagation. The present invention uses prompt learning and a designed prompt template suitable for remote sensing images to enable the model to learn the semantics and contextual information of the remote sensing image, thereby alleviating the problem that the remote sensing image classification method only focuses on visual representation and ignores semantic information. At the same time, by constructing a supervision matrix for calculating contrast loss, the problem of too few positive sample pairs during training is avoided, so that the classification accuracy of the remote sensing image is further improved, which is of great significance to remote sensing image classification.

[0110] Furthermore, it should be noted that the present invention may be provided as a method, apparatus, or computer program product. Thus, embodiments of the present invention may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0111] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0113] It should also be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal device comprising the element.

[0114] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be noted that although the preferred embodiment of the present invention has been described, it is clear that those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles of the present invention. Such improvements and modifications should also be considered as within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A remote sensing image classification method based on prompt learning, characterized in that: The following steps are involved: Step S1: Obtain remote sensing image samples and their true category labels, and perform preprocessing to form a remote sensing image dataset; Step S2: obtaining an image feature vector of a remote sensing image sample and a text feature vector of a prompt of the remote sensing image sample through an image encoder and a text encoder according to the remote sensing image dataset; Step S3: calculating the similarity between the image feature vector and the text feature vector to obtain a predicted category label of the remote sensing image sample; Step S4: Calculate the classification loss and contrast loss, repeat steps S1 to S4, train and output the classification model; Step S5: classifying remote sensing images using the classification model; In step S1, the preprocessing includes: Step S11: accurately label each pixel of the remote sensing image sample to form the true category label, and determine the category information description and category number of the categories included in the remote sensing image dataset based on the true category label; Step S12: Setting a prompt template. The structure of the prompt template is "This is a photo of the {pro}{cls} used for the classification task", where pro is the attribute information description of the remote sensing image sample, and cls is the category information description of the remote sensing image sample. Step S13: generating a prompt for each remote sensing image sample in the remote sensing image dataset according to the true category label and the prompt template; In the step S13, it specifically includes: Step S131: defining a category information description list and an attribute information description list, wherein the elements of the category information description list are category information descriptions corresponding to the categories included in the remote sensing image dataset, and the elements of the attribute information description list are attribute information descriptions corresponding one-to-one to the category information descriptions, and the length of the category information description list and the attribute information description list are equal to the number of categories in the remote sensing image dataset; Step S132: searching the category information description list and the attribute information description list according to the true category label, obtaining the category information description and the attribute information description corresponding to the true category label, and placing them in the pro and cls positions of the prompt template, respectively, to form a prompt of the remote sensing image sample; In step S3, the similarity between the image feature vector and the text feature vector is calculated to obtain the probability that the remote sensing image sample belongs to each category. The category with the largest probability value is taken as the predicted category label. The probability that the remote sensing image sample belongs to category k is The calculation formula is as follows: Among them, <·> is the cosine similarity, f(·) is the image encoder, g(·) is the text encoder, f(x i ) represents the i-th remote sensing image sample x i The image feature vector, g(t j ) represents the corresponding prompt t of label j j The text feature vector of , the range of k and j is [1, K], K is the number of categories of the remote sensing image dataset, and τ is the temperature hyperparameter; In the step S4, it specifically includes: Step S41: Calculate the classification loss L based on the category probability obtained in step S3 cls ; Step S42: Calculate the contrast loss L based on the image feature vector and text feature vector obtained in step S2. con .

2. The remote sensing image classification method based on prompt learning according to claim 1, characterized in that In the step S2, it specifically includes: Step S21: obtaining an image feature vector of a remote sensing image through the image encoder, wherein the network structure of the image encoder is a Vision Transformer structure of a CLIP image branch; Step S22: obtaining a prompt text feature vector of the remote sensing image sample through the text encoder, wherein the network structure of the text encoder is the Transformer structure of the CLIP text branch.

3. The remote sensing image classification method based on prompt learning according to claim 2, characterized in that In the step S4, it further includes: Step S43: Construct the total loss L total =L cls +λL con , where λ is the balance coefficient, which is used to adjust the weight between losses, L cls is the classification loss, L con is the contrast loss.

4. The remote sensing image classification method based on prompt learning according to claim 3, characterized in that In step S41, the real category label y of the remote sensing image sample is used i and the predicted category label obtained in step S3 is the true category label y i The probability of calculating the classification loss L cls , expressed as: Among them, B is the batch size.

5. The remote sensing image classification method based on prompt learning according to claim 4 is characterized in that: In the step S42, it specifically includes: Step S421: Construct a supervision matrix gt based on the true category labels of the remote sensing image samples. The shape of gt is B×B, where B is the size of the batch and its elements are: Among them, y p and y q Respectively represent the labels of images with sequence numbers p and q in the batch of images, and the range of p and q is [0, B-1]; Step S422: Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on the image feature vector and text feature vector obtained in step S2, and calculate the contrast loss L using the supervision matrix gt obtained in step S421. con =(L i2t +L t2i ) / 2, where L i2t and L t2i They are all BCEWithLogitsLoss loss functions, expressed as: BCEWithLogitsLoss={l1,...,l N },l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))] Where N represents the number of batches, n represents the number of labels predicted for each batch, and σ(·) represents the Sigmoid function, which is expressed as:

6. The remote sensing image classification method based on prompt learning according to claim 5, characterized in that In the step S422, it specifically includes: Step S4221: Calculate the image-text similarity matrix i2t_logit of the remote sensing image sample, expressed as: i2t_logit=I·T Τ Where, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing image samples, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing image samples; Step S4222: Calculate the text image similarity matrix t2i_logit: t2i_logit=T·I Τ ; Step S4223: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the image-text similarity matrix i2t_logit obtained in step S4221. i2t : L i2t =BCEWithLogitsLoss(i2t_logit,gt); Step S4224: Calculate the contrast loss L based on the supervision matrix gt obtained in step S421 and the text image similarity matrix t2i_logit obtained in step S4222. t2i : L t2i =BCEWithLogitsLoss(t2i_logit,gt)。

Images