A cross-modal remote sensing image-text retrieval method based on single-mode feature modeling

Abstract

The present application is based on the requirements of multi-source multi-modal remote sensing data collaborative processing, specifically analyzes the defects of current methods which ignore the single modal feature distribution characteristics, and discloses a cross-modal remote sensing image-text retrieval method based on single modal feature modeling. Specifically, for the redundant information of remote sensing image and text data, a mask-based attention mechanism is proposed; for the internal semantic relationship between image regions and text sequences, a relationship modeling method based on Transformer is proposed; for the defects of the current mainstream loss function ternary loss which only focuses on the distance and matching relationship between cross-modal features, based on the characteristics of large amount of image information in the remote sensing field and the diversity of corresponding text description, an unsupervised classification loss function is proposed. The cross-modal remote sensing retrieval method of the present application can improve the accuracy, robustness and practicality of cross-modal remote sensing image-text retrieval.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal retrieval technology, specifically relating to a cross-modal remote sensing image and text retrieval method based on single-modal feature modeling. Background Technology

[0002] In recent years, with the rapid development of satellite observation technology, remote sensing technology has been widely applied in fields such as resource acquisition and disaster monitoring. At the same time, the rapid development of remote sensing applications has led to an explosive growth in remote sensing data. However, in stark contrast to the ability to acquire remote sensing data, the ability to process remote sensing information is extremely low, resulting in low utilization of remote sensing big data and falling into the paradox of "big data, little knowledge."

[0003] Current retrieval techniques for remote sensing big data primarily focus on single-modal designs, failing to consider the collaborative processing requirements of multi-source, multi-modal remote sensing data. To more easily retrieve data that meets user needs and interests from massive amounts of remote sensing big data, a new retrieval model that transcends different modalities is needed. This model should be able to return semantically relevant results in any modality based on the query sample—that is, cross-modal retrieval technology. For example, if a user provides image data of a specific object as a query, cross-modal retrieval results will return relevant text descriptions, audio materials, and other modal content to supplement their visual experience. Text, as a crucial aspect of human-computer interaction, has led to increasing attention being paid to cross-modal remote sensing image-text retrieval by researchers.

[0004] Existing technologies disclose a retrieval method based on multi-scale fine-grained hash networks to achieve high retrieval speed and low storage overhead. However, investigations reveal the following key issues in improving the accuracy of cross-modal image-text retrieval: 1) Compared to natural images, remote sensing images often contain more redundant information, and text itself also contains redundant information. This leads to irrelevant information interfering with image-text matching, hindering the model's understanding of the content of remote sensing images or text; 2) Currently, almost all algorithms ignore the inherent semantic relationships within a single modality of image and text. For example, in a text sequence, the relationship between "airplane" and "airport" is closer than that between "airplane" and "port"; 3) The ternary loss function often used in current algorithms has significant limitations for remote sensing image-text retrieval. It causes the model to learn only the cross-modal matching relationship between images and text, ignoring the inherent feature distribution within a single modality of images and text, which affects the model's robustness. Therefore, designing a more accurate cross-modal remote sensing image-text retrieval method that fully considers the feature distribution of a single modality is an urgent technical problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] To avoid the shortcomings of existing technologies, this invention provides a cross-modal remote sensing image and text retrieval method based on single-modal feature modeling. This method addresses the feature distribution of a single modality, reduces the impact of redundant information to a certain extent, fully explores the semantic information and intrinsic connections between images and text, and compensates for the defects of ternary loss, thereby obtaining more robust features and more accurate retrieval results.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A cross-modal remote sensing image and text retrieval method based on single-modal feature modeling, characterized by including a training process and a retrieval inference process;

[0009] The training process includes

[0010] Retrieve remote sensing images and their corresponding descriptive texts from the dataset;

[0011] A cross-modal retrieval model is constructed, including: high-dimensional feature extraction of images and text based on a backbone network; information masking of image and text feature vectors; relationship modeling of image and text features based on a Transformer encoder; and obtaining processed image and text feature vectors.

[0012] Cosine similarity calculation is performed on cross-modal features;

[0013] The calculation of model loss includes: calculating the ternary loss based on similarity results; calculating the classification loss using an unsupervised approach; obtaining the total loss by summing the ternary loss and the classification loss; and optimizing the network parameters using the loss function until convergence.

[0014] The retrieval reasoning process includes

[0015] Obtain remote sensing images or descriptive text input by the user as the data to be retrieved;

[0016] The data to be retrieved is input into a trained cross-modal retrieval model to obtain its feature vector;

[0017] The final search results are obtained by calculating cosine similarity based on feature vectors and sorting them.

[0018] A further technical solution of the present invention: During the training process, the feature extraction of the image based on the backbone network is as follows:

[0019]

[0020] In the formula, F v High-dimensional image features extracted by the backbone network, This represents the CNN network that processes images, where I represents the input remote sensing image, and θ...i These represent the learnable parameters in the CNN backbone network.

[0021] A further technical solution of the present invention: During the training process, the feature extraction of text based on the backbone network is as follows:

[0022] F t =GRU(T; θ t )

[0023] In the formula, F t The high-dimensional text features extracted by the backbone network are represented by GRU, which represents a recurrent neural network that processes text sequences. T represents the text sequence corresponding to the input image, and θ represents the high-dimensional text features extracted by the backbone network. t This represents the learnable parameters in the GRU backbone network.

[0024] A further technical solution of the present invention: the information masking strategy during the training process is as follows:

[0025] F v ′=R mask (F v ;τ v )

[0026] F t ′=R mask (F t ;τ t )

[0027] In the formula, F v ′ and F t ′ represent the image features and text features after masking, respectively, R mask The proposed masking strategy, τ v and τ t These represent the mask ratios for the image and text, respectively.

[0028] A further technical solution of the present invention: the relationship modeling of the image during the training process is as follows:

[0029]

[0030] In the formula, MSA represents the visual features of an image after relational modeling. R→R v′ represents the modeling of the relationship between image regions. i It is F v The subvector of ′ This represents a matrix of three learnable parameters.

[0031] A further technical solution of the present invention: During the training process, the text features are modeled into relationships as follows:

[0032]

[0033] In the formula, MSA represents the text sequence features obtained after relational modeling. W→W Modeling the relationships between words in a text, t′ i It is F t The subvector of ′ This represents a matrix of three learnable parameters.

[0034] A further technical solution of the present invention: the formula for calculating the cosine similarity of cross-modal features during the training process is as follows:

[0035]

[0036] In the formula, · represents the dot product of vectors, and ||·||2 represents the L2 normalization of vectors.

[0037] A further technical solution of the present invention: the formula for calculating the ternary loss based on the similarity results during the training process is as follows:

[0038]

[0039] In the formula: L triplet Represents the ternary loss function, [x] + =max(x,0), Represents paired sample pairs. Is with Unpaired image feature samples Is with Unpaired text feature samples, where α is the minimum margin defined in the ternary loss.

[0040] A further technical solution of the present invention: the formula for calculating the classification loss using an unsupervised approach during training is as follows:

[0041]

[0042]

[0043] L Entity =L visual +L textual

[0044] In the formula, L Entity The proposed unsupervised classification loss function represents the softmax activation function, and W represents the cross-entropy activation function. v W t These are the weights of the two fully connected classification layers, b1 and b2 represent the bias vectors, and l v , l tThese represent pseudo-labels generated for images and text, respectively, with lengths n. v n t Specifically, a clustering algorithm was used when generating text pseudo-tags.

[0045] A computer system is characterized by comprising: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0046] The beneficial effects of this invention are as follows:

[0047] The present invention provides an end-to-end cross-modal remote sensing image and text retrieval method based on single-modal feature modeling, which has the following advantages:

[0048] 1. This invention addresses the redundant information in remote sensing images and text data, as well as the semantic relationships within their modalities. It proposes a relationship modeling method based on masked attention, which uses the self-attention mechanism of the Transformer encoder to make the model focus on the more salient parts of the image and text. At the same time, it obtains more robust features by fully mining the intrinsic connections between modalities, thereby improving the effect of cross-modal remote sensing image and text retrieval.

[0049] 2. This invention addresses the limitation of current mainstream models that use a ternary loss function that only focuses on cross-modal feature distance. Based on the characteristics of remote sensing image and text retrieval that differ from the natural domain, namely the large amount of image information and the strong diversity of corresponding text descriptions in the remote sensing domain, a new classification loss is proposed based on the idea of ​​unsupervised learning. This ensures that the model learns the independent feature distributions of images and text in a single modality while learning the matching relationship, thereby enhancing the robustness and generality of the model and enabling high-precision cross-modal retrieval of remote sensing images and descriptive text. Attached Figure Description

[0050] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0051] Figure 1 This is a framework diagram of the method of the present invention.

[0052] Figure 2 This is a schematic diagram of the training process of the method proposed in this invention.

[0053] Figure 3 This is a flowchart illustrating the unsupervised classification pseudo-label generation process used in this invention.

[0054] Figure 4 This is a schematic diagram of the retrieval process of the method proposed in this invention.

[0055] Figure 5 This is a comparison chart showing the accuracy of the method proposed in this invention with other methods. Detailed Implementation

[0056] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, such as replacing the backbone network. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0057] The framework design of the method of this invention is as follows: Figure 1 As shown, based on the characteristics of each single modality and the three challenges identified in the background survey, corresponding solutions are proposed, the innovative content and methods are determined, and finally, experimental verification is performed on a dataset. To address the redundancy of information in remote sensing images and text data, a mask-based attention mechanism is proposed; to address the inherent semantic relationship between image regions and text sequence modalities, a Transformer-based relationship modeling method is proposed; and to address the shortcomings of the current mainstream ternary loss function, which only focuses on the distance and matching relationship between features, an unsupervised classification loss function is proposed, based on the characteristics of the large amount of image information and the diversity of corresponding text descriptions in the remote sensing field.

[0058] The method of this invention can be divided into two processes: training and retrieval reasoning.

[0059] The training process includes:

[0060] 1. Obtain the remote sensing images and corresponding descriptive text from the dataset;

[0061] 2. Build a cross-modal retrieval model, which includes:

[0062] (1) High-dimensional feature extraction is performed on images and text based on the backbone network;

[0063] (2) Perform information masking on the image and text feature vectors respectively;

[0064] (3) The relationship between image and text features is modeled based on the Transformer encoder;

[0065] (4) Obtain the processed image feature vector and text feature vector;

[0066] 3. Calculate cosine similarity for cross-modal features;

[0067] 4. Calculate the model loss, which includes:

[0068] (1) Calculate the ternary loss based on the cosine similarity calculation results;

[0069] (2) Calculate the classification loss for each modality using an unsupervised approach;

[0070] (3) The total loss is obtained by summing the ternary loss and the classification loss;

[0071] 5. Optimize the model network parameters using the loss function until convergence.

[0072] Retrieval reasoning process:

[0073] 1. Obtain remote sensing images or descriptive text input by the user as the data to be retrieved;

[0074] 2. Input the data to be retrieved into the trained cross-modal retrieval model to obtain its feature vector;

[0075] 3. Calculate the cosine similarity based on the feature vectors and sort them to obtain the final search results.

[0076] The following detailed explanation illustrates the specific implementation methods:

[0077] Training process:

[0078] See Figure 2 The training process includes the following steps: Step 1 is the data acquisition process; Steps 2, 3, and 4 are the process of building a cross-modal retrieval model; and Step 5 is the process of calculating the loss function and optimizing the model parameters.

[0079] Step 1 involves obtaining the remote sensing images and corresponding descriptive text from the dataset.

[0080] In this embodiment, the remote sensing image and text dataset is defined as D={{I1,T1},{I2,T2},...}, where I represents the remote sensing image and T represents the corresponding descriptive text.

[0081] Step 2 involves extracting high-dimensional features from both the image and text based on the backbone network.

[0082] Step 2-1: For remote sensing images, CNNs are often used as the backbone network to extract feature information. This embodiment uses ResNet18, pre-trained on the ImageNet dataset, as the image feature extractor. Given a remote sensing image... H and W represent the height and width of the image, respectively. The feature extraction process based on the backbone network is as follows:

[0083]

[0084] In the formula: F v High-dimensional image features extracted by the backbone network, The CNN network representing the image processing is ResNet18 in this embodiment, where I represents the input remote sensing image, and θ represents the input image. i These represent the learnable parameters in the CNN backbone network. h, w, and d represent the length, width, and dimension of the vector, respectively. For ease of processing, this embodiment converts the image feature vector into a two-dimensional vector. Where N = h × w.

[0085] Step 2-2: For a length of l i The descriptive text can be defined as in The text representation, using GRU as the encoder, is represented by the word vectors:

[0086]

[0087] In the formula: Emb(w) represents the embedding vector of word w, and GRU represents a recurrent neural network for processing text sequences. Ultimately, this embodiment obtains the text features. Where M represents the number of words in the text description. In this embodiment, the text feature extraction process using GRU as the encoder is defined as follows:

[0088] F t =GRU(T; θ t )

[0089] In the formula: F t The high-dimensional text features extracted by the backbone network are represented by GRU, which represents a recurrent neural network that processes text sequences. T represents the text sequence corresponding to the input image, and θ represents the high-dimensional text features extracted by the backbone network. t This represents the learnable parameters in the GRU backbone network.

[0090] Step 3 involves masking the feature vectors of both the image and the text.

[0091] Step 3-1: In this embodiment, a subset of feature vectors is randomly selected and the remaining subset is masked (i.e., deleted). Specifically, the random selection follows a uniform distribution. Random sampling with a masking rate (i.e., the ratio of removed vectors) largely eliminates unnecessary and redundant information. In this embodiment, the image masking rate is set to τ when masking the image feature vectors. v , specifically, τ v =0.5, after masking, a new image feature vector F can be obtained. v'={v1',v2',...,v n '},in

[0092] Step 3-2: When masking the text feature vector, the masking strategy is the same as in Step 3-1. In this embodiment, the text masking rate is set to τ. t , specifically, τ t =0.25, after masking, a new text feature vector F can be obtained. t '={t1',t2',...,t m '},in

[0093] Step 4 involves modeling the relationship between image and text features based on the Transformer encoder.

[0094] Step 4-1: When modeling the relationship between image regions during training, this embodiment uses the mask feature vector F v 'As input, calculate the i-th...' th Regional characteristics v i 'with the jth th Regional characteristics v j The normalized attention coefficient α between ' ij Then, each feature vector v is updated by calculating the sum of the weights of all embedding vectors and a non-linear ReLU layer. i 'arrive As shown in the formula below:

[0095]

[0096]

[0097] In the formula: To query the weight matrix (Query), This is the key weight matrix (Key). Let b1 and b2 be the value weight matrix (Value), and b1 and b2 be the bias vectors. The update procedure in this embodiment can be repeated x times, with the updated parameters... As the input to the next layer of the Transformer encoder, specifically, in this embodiment, x = 3. The final output is defined in this embodiment as... It should be noted that learnable parameters It is not shared between different layers. Specifically, in this embodiment, the relationship modeling process of the multihead self-attention (MSA) mechanism mentioned above is represented as follows:

[0098]

[0099] In the formula: MSA represents the visual features of an image after relational modeling. R→R v′ represents the modeling of the relationship between image regions. i It is F v The subvector of ′ This represents a matrix of three learnable parameters.

[0100] Step 4-2: When modeling the relationships between words in the text during training, the modeling mechanism is the same as in Step 4-1, specifically, it can be represented as follows:

[0101]

[0102] In the formula: MSA represents the text sequence features obtained after relational modeling. W→W Modeling the relationships between words in a text, t′ i It is F t The subvector of ′ This represents a matrix of three learnable parameters.

[0103] Step 5 involves optimizing the network parameters using the loss function until convergence.

[0104] This embodiment employs a loss function framework primarily composed of triplet loss and unsupervised classification loss (entity loss). Based on the traditional triplet loss, this embodiment addresses its limitation of focusing only on cross-modal feature distances by incorporating a classification layer in an unsupervised manner to classify the feature distribution within a single modality of the image / text. For example... Figure 3 As shown, specifically, since each image in remote sensing image retrieval is an independent entity, this embodiment treats each image as a category to generate pseudo-labels. Simultaneously, because the corresponding text sets contain similar or identical descriptive texts, this embodiment uses the DBSCAN clustering algorithm to measure distance and cluster to generate pseudo-labels. Specifically, the neighborhood radius of the cluster is set to 0.15. For example, after data cleaning, the RSICD dataset contains 1966 remote sensing images, each with 5 descriptive texts. In this embodiment, there are 1966 image categories, and the text categories are derived through the clustering algorithm. By calculating the cross-entropy classification loss, the model can distinguish fine-grained differences between groups, learn more realistic features, and obtain more robust model parameters.

[0105] Step 5-1: Calculate the cosine similarity of cross-modal features. The calculation formula is as follows:

[0106]

[0107] In the formula: · represents the dot product of vectors, and ||·||2 represents the L2 normalization of vectors.

[0108] Step 5-2: Calculate the ternary loss based on the similarity results. The formula for calculating the ternary loss is as follows:

[0109]

[0110] In the formula: L triplet Represents the ternary loss function, [x] + =max(x,0), F represents paired sample pairs. v Is with Unpaired image feature samples Is with Unpaired text feature samples, where α is the minimum margin defined in the ternary loss.

[0111] Step 5-3: Calculate the classification loss using an unsupervised approach.

[0112]

[0113]

[0114]

[0115]

[0116] L Entity =L visual +L textual

[0117] In the formula: L Entity The representative proposed the overall unsupervised classification loss, L visual The classification loss representing image features, L textual The classification loss represents the text features, softmax represents the cross-entropy activation function, and W... v W t These are the weights of the two fully connected classification layers, b1 and b2 represent the bias vectors, and l v , l t These represent pseudo-labels generated for images and text, respectively, with lengths n. v n t Specifically, the DBSCAN clustering algorithm was used when generating text pseudo-labels.

[0118] Step 5-4: Calculate the total loss;

[0119] L=L triplet +λL Entity

[0120] In the formula: λ represents the weight of the classification loss in the total loss.

[0121] Step 5-5: Backpropagation to update parameters;

[0122] It should be noted that during the backpropagation of the total loss L to update the parameters, the updated parameters mainly include θ. i ,θt t , W v W t .

[0123] During training, an information masking mechanism was used to address the characteristics of remote sensing images and text, enabling the model to focus more on key regions and vocabulary.

[0124] During training, an unsupervised classification loss is added to the classification layer in an unsupervised manner to classify the feature distribution within a single modality of image / text. This allows the model to distinguish fine-grained differences between groups, learn more realistic features, and obtain more robust model parameters.

[0125] In the generation of pseudo-labels for unsupervised classification loss, since each image in remote sensing image retrieval is an independent entity, this invention treats each image as a category to generate pseudo-labels. At the same time, since there are similar or identical texts in the corresponding text set, this invention uses a clustering algorithm to measure the distance of text features and cluster them to generate pseudo-labels.

[0126] The overall loss function during training consists of ternary loss and unsupervised classification loss.

[0127] Retrieval reasoning process:

[0128] See Figure 4 The retrieval reasoning process includes the following steps:

[0129] Step 1 involves obtaining the remote sensing image or descriptive text input by the user as the data to be retrieved.

[0130] In this embodiment, the remote sensing image input by the user is defined as I, and the descriptive text input by the user is defined as T.

[0131] Step 2 involves inputting the data to be retrieved into a trained cross-modal retrieval model to obtain its feature vector.

[0132] In this embodiment, the trained model is defined as MGRM-EL, and the image feature vector... and text feature vectors It can be obtained through the following formula:

[0133]

[0134] It should be noted that the masking ratio for both image regions and text words in the model is 0.

[0135] Step 3 involves calculating the cosine similarity based on the feature vectors and sorting them to obtain the final search results.

[0136] Step 3-1: Initialize the similarity matrix S;

[0137] Step 3-2: Calculate the cosine similarity matrix using the following formula:

[0138] Step 3-3: Place S cal Assign the values ​​to the similarity matrix S;

[0139] Steps 3-4: Obtain the final search results based on the sorting results of the similarity matrix.

[0140] In the experimental verification module, the sample data used in this embodiment are the publicly available multimodal remote sensing datasets RSICD and RSITMD. The RSICD dataset contains 10,921 remote sensing images, each with 5 descriptive texts, making it the largest remote sensing image-text dataset currently available. The RSITMD dataset is a reconstructed dataset, partly selected from the RSICD dataset and partly from Google Maps, containing 4,743 remote sensing images and 23,715 descriptive texts, offering higher quality than the RSICD dataset. In this embodiment, 80%, 10%, and 10% of the datasets are used as the training set, validation set, and test set, respectively, for training and inference on a single NVIDIA GeForce GTX 3090 GPU high-performance graphics card.

[0141] This embodiment uses two evaluation metrics, R@K (K = 1, 5, and 10) and mR, to evaluate the experimental results. R@K represents the proportion of the ground truth label appearing in the top K results. mR is obtained by calculating the average of all six recall rates R@K, which is more reasonable for evaluating the overall performance of the model. The experimental verification results of this embodiment can be found in [link to relevant documentation]. Figure 5The model proposed in this invention, named MGRM-EL, demonstrates significant performance on two datasets. On the RSICD dataset, the model without the improved loss function achieves an mR of 19.40. With the improved loss function, the complete model achieves an mR of 21.42, which is 2.46 higher than the best model proposed in 2022, GaLR. Specifically, in image-to-text retrieval, R@1, R@5, and R@10 increase by 0.82, 3.39, and 4.28, respectively; while in text retrieval, R@1, R@5, and R@10 increase by 1.06, 1.75, and 3.42, respectively. On the RSIMD dataset, the proposed model also exhibits highly competitive performance, with an overall mR improvement of 1.35 compared to GaLR.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A method for cross-modal remote sensing image-text retrieval based on single-mode feature modeling, characterized in that This includes the training process and the retrieval and reasoning process; The training process includes Retrieve remote sensing images and their corresponding descriptive texts from the dataset; A cross-modal retrieval model is constructed, including: high-dimensional feature extraction of images and text based on a backbone network; information masking of image and text feature vectors; relationship modeling of image and text features based on a Transformer encoder; and obtaining processed image and text feature vectors. The image feature extraction based on the backbone network is as follows: In the formula, High-dimensional image features extracted by the backbone network CNN networks, representing image processing Represents the input remote sensing image, These represent the learnable parameters in the CNN backbone network. The text feature extraction based on the backbone network is as follows: In the formula, High-dimensional text features extracted from the backbone network. Recurrent neural networks, representing the processing of text sequences, This represents the text sequence corresponding to the input image. represent Learnable parameters in the backbone network; The information masking strategy is as follows: In the formula, and These represent the image features and text features after masking, respectively. The masking strategy proposed by the representative and These represent the mask ratios for the image and text, respectively. The relationship modeling of the image is as follows: In the formula, This represents the visual features of the image after relational modeling. Modeling the relationships between regions in an image. yes subvectors, This represents the three learnable parameter matrices; The text features are modeled into relationships as follows: In the formula, This represents the text sequence features obtained after relational modeling. Modeling the relationships between words in a text. yes subvectors, This represents the three learnable parameter matrices; Cosine similarity is calculated for cross-modal features; the calculation formula is as follows: In the formula, Represents the dot product of vectors. L2 normalization of the vector; The calculation of model loss includes: calculating the ternary loss based on similarity results; calculating the classification loss using an unsupervised approach; obtaining the total loss by summing the ternary loss and the classification loss; and optimizing the network parameters using the loss function until convergence. The formula for calculating the ternary loss based on similarity results is as follows: In the formula: Represents the ternary loss function. , Represents paired sample pairs. Is with Unpaired image feature samples Is with Unpaired text feature samples, It is the minimum margin defined in the ternary loss; The formula for calculating the classification loss using the unsupervised approach is as follows: In the formula, The representative proposed the unsupervised classification loss. Represents the cross-entropy activation function. , These are the weights of the two fully connected layers for classification. , Represents the bias vector. , These represent pseudo-tags generated for images and text, respectively, with lengths of [length values ​​missing]. , Specifically, a clustering algorithm was used when generating text pseudo-tags; The retrieval reasoning process includes Obtain remote sensing images or descriptive text input by the user as the data to be retrieved; The data to be retrieved is input into a trained cross-modal retrieval model to obtain its feature vector; The final search results are obtained by calculating cosine similarity based on feature vectors and sorting them.

2. A computer system, characterized in that... include: One or more processors, a computer-readable storage medium for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of claim 1.

Images