Cross-modal retrieval method of remote sensing images based on fusion of language and visual detail features

Abstract

The present invention relates to a remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features, which includes the following steps: step 1: processing the training data of the remote sensing image-text retrieval model; step 2: constructing a multi-detail language and visual fusion model; step 3: training a multi-objective optimized detail language and visual fusion model; step 4: constructing a remote sensing image-text description feature library; step 5: completing cross-modal retrieval of remote sensing image-text description. The present invention uses a single-modal encoder to represent image and text features respectively, uses a multi-modal encoder to fuse the features of the two modalities, improves the expression of fine-grained semantic features of each encoder for the corresponding modal data through feature fusion and multi-task optimization training, and completes cross-modal retrieval through the similarity calculation of semantic features; by designing a multi-objective optimization strategy for the model, the model has the ability to express multiple details of features for remote sensing images and text descriptions at the same time.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a cross-modal retrieval method for remote sensing images based on the fusion of language and visual detail features. Background Art

[0002] In recent years, remote sensing satellite and drone technology has developed rapidly. Remote sensing technology, as its core, has achieved remarkable results in the fields of geographic positioning, disaster relief, military reconnaissance, disaster monitoring, etc. With the widespread application of remote sensing technology, remote sensing images have shown explosive growth, which has brought great difficulties to multiple tasks such as large-scale remote sensing image recognition, detection, classification and retrieval. Among them, the cross-modal retrieval task of remote sensing images refers to finding remote sensing images with the same or similar content in a large-scale remote sensing image dataset based on a given natural language description, and vice versa. Compared with traditional remote sensing image retrieval, cross-modal retrieval of images and texts presents better human-computer interaction characteristics and has stronger application value.

[0003] In the application scenario of cross-modal retrieval of remote sensing images, one of the key requirements of users is to input a description of the scene and retrieve images that are consistent or similar to the input description from the huge remote sensing image library. In this process, the query data and the data stored in the database have different modality types. There is often a huge representation gap between the feature expressions of different modalities. It is necessary to establish a connection between samples with the same semantics in different modalities. At present, the cross-modal retrieval methods for remote sensing images are mainly divided into methods based on image label retrieval and methods based on image-text feature vector retrieval. The method based on image label retrieval mainly describes each image with keywords as the feature label of the image. In the retrieval process, the description entered by the user is disassembled into keywords, matched with the keyword labels of the image, and similar target images are found. The method of image-text feature vector retrieval is to use the trained image-text encoder to encode images and texts with the same or similar semantics into feature vectors with closer distances, and vice versa. Both current retrieval methods have defects to varying degrees, which are mainly reflected in the following aspects:

[0004] 1. The image tag retrieval method relies on high-quality tag description of existing image data, which takes a lot of time and is not applicable in large-scale data retrieval.

[0005] 2. The retrieval method based on image-text feature vectors requires feature alignment of image content and text description. Due to the differences in data structures of images and texts, the process of feature extraction, alignment, and fusion of the two types of data is very difficult;

[0006] 3. Current image feature encoders often rely on a high-quality remote sensing image target recognition model to represent the details of the image. The accuracy of this model has a great impact on the overall retrieval effect, and training this target recognition model also requires more efforts.

[0007] To solve the above-mentioned problems, the present invention proposes a cross-modal retrieval method for remote sensing images based on the fusion of language and visual detail features. By designing a cross-modal retrieval framework for remote sensing images including two single-modal encoders, vision and language, and one multi-modal encoder, the single-modal encoder is used to represent the image and text features respectively, and the multi-modal encoder is used to fuse the features of the two modalities for learning. The expression ability of each encoder for the fine-grained semantic features of the corresponding modal data is improved through feature fusion and multi-task optimization training, and the cross-modal retrieval task is completed by calculating the similarity of semantic features. In order to express the detailed features of the image, the present invention designs a shallow visual transformer model to extract the local features of the image, and transforms the pipelined "target detection + retrieval" process into an end-to-end training process; this end-to-end framework bridges the gap between the target detector and the retrieval model training process, and reduces the training overhead of the entire retrieval model; the present invention designs a set of multi-objective optimization strategies for the model, and trains the overall model under this strategy, so that the model has the ability to express multiple detailed features of remote sensing images and text descriptions at the same time; the model with converged training results completes the end-to-end text-image retrieval task that does not require image labels. Summary of the invention

[0008] In order to overcome the shortcomings of the existing technology, a single-modal encoder is used to represent image and text features respectively, and a multi-modal encoder is used to fuse the features of the two modalities. Feature fusion and multi-task optimization training are used to improve the expression of fine-grained semantic features of each encoder for the corresponding modal data, and cross-modal retrieval is completed by calculating the similarity of semantic features. By designing a multi-objective optimization strategy for the model, the model has the ability to express multiple details of remote sensing images and text descriptions at the same time.

[0009] To achieve the above object, the solution adopted by the present invention is to provide a remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features, which includes the following steps:

[0010] Step 1: Process the training data of the remote sensing image and text retrieval model;

[0011] The training data of the remote sensing image and text retrieval model is based on image data and text data. The image data is the measured image, and the text data is the description information corresponding to the remote sensing image. First, the number of words in the text data is obtained. words and the number of stop words lenstops ; Delete all the items that satisfy len words =len stops and len words =len stops +1 image-text data pairs to avoid the stop words in the image-text data pairs interfering with the effect of retrieval model training; then, process the image data and the corresponding text data; finally, use the cleaned remote sensing image-text retrieval training data to train the image local encoder and global encoder;

[0012] Step 2: Build a multi-detail language and vision fusion model;

[0013] Cross-modal retrieval based on image-text features mainly extracts and expresses the features of image data and text data respectively. By optimizing the expression ability of each extractor, the feature representation of image data and text data with semantic similarity is minimized in the vector space. The overall architecture based on the multi-detail language and visual fusion model includes: remote sensing image visual encoder F enc-V , Remote Sensing Image Description Language Feature Encoder F enc-L and a multimodal encoder F based on the vision-language fusion model enc-Mul , as shown below:

[0014]

[0015] Where: I represents the input remote sensing image data; T represents the input text data; f IL Represents the local features of remote sensing images; f IGL Represents the local-global fusion features of remote sensing images; f T Represents the text features of remote sensing image description; S distance Represents the distance similarity between feature vectors; S pairwise Represents the matching probability value of the image-text pair; F enc-V represents the remote sensing image visual encoder; F enc-L F stands for remote sensing image description language feature encoder; enc-Mul represents a multimodal encoder based on the vision-language fusion model;

[0016] Step 3: Train the multi-objective optimized detail language and visual fusion model;

[0017] A multi-objective comprehensive supervision optimization method including four loss functions is constructed, and a deep supervision strategy is introduced in the middle branch; a model training process is completed, wherein the training process is mainly optimized for an image encoder, a text encoder, and a multimodal encoder; the image data and text data processed in step 1 are divided into a training set, a validation set, and a test set according to a first ratio value, and the divided training set is sent to the model constructed in step 2, and the model parameters are initialized using a normal distribution without using pre-trained parameters; the image encoder part is frozen when calculating the image-text matching loss, and the multimodal encoder is focused on being optimized;

[0018] Step 4: Construct a remote sensing image-text description feature library;

[0019] Step 41: In the retrieval task, the recall rate is used to represent the proportion of correct samples among the N candidate samples returned by the retrieval algorithm; first use R i2t represents the image-to-text retrieval recall rate and R t2i Represents the retrieval recall rate of text to image; then calculate the image-to-text recall rate R of the top1, top5, and top10 retrieval tasks on the validation set i2t @N and text-to-image recall R t2i @N; finally calculate the image to text recall rate R i2t @N and text-to-image recall R t2i @N The average value mR in the test sample i2t @N and mR i2t @N, and save the model with the highest recall rate for subsequent retrieval tasks. The specific calculation formula is as follows:

[0020]

[0021] Where: mR i2t @N and mR t2i @N represents the image-to-text recall rate R i2t @N and text-to-image recall R t2i @N is the average value in the test sample; Image k Text represents the kth image-text pair with similar semantics; k represents the kth text-image pair with similar semantics; R i2t @N(Image k ) represents the image-to-text recall rate, input image Image k ; R t2i @N(Text k ) represents the recall rate of text to image, input text Text k; k represents the image and text pair number; m represents the total number of image and text pairs; N represents the retrieval task number;

[0022] Step 42: construct an image feature database, use the image encoder trained in step 3 to extract features from all image data, and store the generated image features in the database to improve retrieval efficiency in subsequent applications;

[0023] Step 43: construct a text feature database, use the text encoder trained in step 3 to extract features from all text data, and store the generated text features in the database;

[0024] Step 5: Complete cross-modal retrieval of remote sensing images and text descriptions;

[0025] The cross-modal retrieval includes four main modules: image encoding module, text encoding module, similarity judgment recall module and multimodal reordering module; among them, the image encoding module is connected in parallel with the text encoding module, and then the similarity judgment recall module and the multimodal reordering module are cascaded; the cross-modal retrieval of remote sensing image-text description is completed through the above four modules.

[0026] Preferably, the processed image data and the corresponding text data in step 1 are specifically:

[0027] The image data processing process is as follows:

[0028] Step 111: All image data are uniformly adjusted to a size of 278x278x3;

[0029] Step 112: Perform basic data enhancement on the input image data with a probability of 50%, including random rotation and random flipping to enhance the generalization ability of the model;

[0030] Step 113: Expand the amount of input image data, adopt an image data splicing method, randomly select two images I_a and I_b in the same category according to the category label, and perform pixel-level superposition. For the text part, directly perform text description splicing, and splice T_b directly behind T_a;

[0031] Step 114: randomly cropping the image data after step 112 and step 113, and the cropping area size is 256x256x3 to adapt to the input of the subsequent neural network model;

[0032] Step 115: normalizing the image data after step 114 to transform the image grayscale range to between 0 and 1;

[0033] The processing process of the text data is as follows:

[0034] Step 121: Remove stop words from the text data, set the maximum word length to 64, truncate the text data, and discard the part exceeding the maximum word length;

[0035] Step 122: Construct the "random mask" combined with the "directional mask" strategy as follows:

[0036] Step 1221: construct a Boolean text data mask descriptor S_T, the length of which is consistent with the length of the words in the processed text data, initialized with False, and no mask operation is performed by default;

[0037] Step 1222: randomly select 15% of the positions in S_T according to Bernoulli distribution and mark them as True;

[0038] Step 1223: record the category labels imgs_cls of all images, traverse the text data, and mark the positions containing imgs_cls on S_T with True to focus on the category information of the target in subsequent mask operations;

[0039] Step 1224: construct a quantity information descriptor S_N, store English numeric words between 0 and 10, traverse the text data, and mark the position containing S_N with True on S_T to focus on the quantity information of the target in subsequent mask operations;

[0040] Step 1225: For all positions marked with True in S_T, replace the text data words with [MASK] blank with a probability of 80%; replace the original words with any other random words with a probability of 10%; the words in the remaining 10% positions are not replaced.

[0041] Preferably, the remote sensing image visual encoder in step 2 is specifically:

[0042] In the remote sensing image-text cross-modal retrieval task, the amount and complexity of the semantic information contained in the image data is much higher than that of the corresponding text data. enc-V The module M is designed to extract the global features of the image cnn-mvsa And the module M for extracting local features vit , and design a module M that integrates global features and local features midf , as shown below:

[0043]

[0044] Where: f IG Represents the global features of remote sensing images; M cnn-mvsaM represents the global feature extraction module of remote sensing images; vit M represents the local feature extraction module of remote sensing images; midf Represents a module that fuses global features and local features;

[0045] The global feature extraction module M of the remote sensing image cnn-mvsa A ResNet-50 residual convolutional neural network is used as a feature extractor, and a multi-scale self-attention model is used to optimize the effect of feature extraction. The entire image after cleaning and enhancement in step 1 is input into M cnn-mvsa In the equation f IG ;

[0046] The local feature extraction module M of the remote sensing image vit A 6-layer Vision Transformer model is used as a feature extractor to input the serialized image data in step 1 into M vit Perform feature extraction in and get f IL ;

[0047] The fusion module M of the global feature and the local feature midf is a linear function by transforming the feature f IG and f IL Perform linear addition, and the first fusion parameter a and the second fusion parameter b obtained during the training process are as follows:

[0048] M midf (f IG ,f IL ) = af IG +bf IL ;

[0049] Where: M midf Represents the fusion module of global features and local features; a represents the first fusion parameter; b represents the second fusion parameter.

[0050] Preferably, the remote sensing image description language feature encoder in step 2 is specifically:

[0051] The remote sensing image description language feature encoder F enc-L Includes a Bert-based text encoder M bert , as shown below:

[0052] f T =M bert (T);

[0053] Where: M bert Represents the standard Bert model;

[0054] During the training process, Mbert The feature expression characteristics of the model will be adjusted according to the labeled data to enhance the model's ability to express features. The text data masked in step 1 will be used as training data and input into the model M. bert In the middle; after the model training is completed, it enters the application stage M bert Express user input text data as a semantic feature vector.

[0055] Preferably, the multimodal encoder based on the vision-language fusion model in step 2 is specifically:

[0056] Multimodal encoder F based on vision-language fusion model enc-Mul It contains two modules, namely, M for calculating the cross-modal vector distance cms and M for vision-language fusion vlf , as shown below:

[0057] F enc-Mul ={M cms ,M vlf};

[0058] Where: M cms The representation module is used to calculate the distance similarity between the image feature vector and the text feature vector, using the cosine distance to measure; M vlf Indicates the use of visual-language fusion; F enc-Mul represents a multimodal encoder based on the vision-language fusion model;

[0059] M vlf The module is initialized with the last 6 layers of a BERT-based model, with an additional criss-cross attention layer to model vision-language interactions.

[0060] Preferably, in step 3, a multi-objective comprehensive supervision optimization method including four loss functions is constructed, specifically:

[0061] Step 31: The triplet loss is used to learn the representation space of image features and text features. The loss is calculated by comparing the distances between the features of the three comparison samples. The first triplet loss function L is constructed between the text feature encoder and the image feature encoder. itt1 The features encoded by the local image encoder contain more image details, which are crucial for the final image expression and the subsequent training of the multimodal encoder. A second triplet loss function L is constructed between the text feature encoder and the local image feature encoder. itt2 For deep supervision, it is as follows:

[0062]

[0063] Where: Litt1 represents the first triplet loss function; ε represents the minimum margin used to enlarge the gap between the benchmark sample and the positive / negative sample pair; Sim represents the similarity recall module; (I, T) represents the matched image-text pair features, generated by the image encoder and the text encoder respectively; T ^ Represents the text features that are not matched with image I; I ^ Indicates the image features that are not matched with the text T; L itt2 I represents the second triplet loss function; loc Indicates the generation of the local encoder of the image. The + in the lower right corner of the formula means that when the value in [] is greater than 0, the value is taken as loss, and when it is less than 0, the loss is 0;

[0064] Step 32: Construct the image-text matching loss function L between the multimodal encoder and the image feature encoder itm , by connecting a fully connected binary classification layer to predict whether a pair of input images and texts match; when selecting positive / negative samples, the similarity recall module is used to calculate the difficult samples closest to the positive samples in a single batch as negative samples to enhance the learning ability of the multimodal module. The loss calculation form is as follows:

[0065] L itm =-y itm log(p itm (I,T))+(1-y itm )log(1-p itm (I,T));

[0066] Where: L itm Represents the image-text matching loss function constructed between the multimodal encoder and the image feature encoder; y itm represents the matching label of the constructed image-text pair; p itm (I, T) represents the image-text matching probability;

[0067] Step 33: Construct a mask language model loss function L between the multimodal encoder and the image local feature encoder mlm , the masked text is represented by T ′ Indicates that the model predicts probability with p msk (I,T ′ ) is represented as follows:

[0068] L mlm =-y msk log(p msk (I,T ′ ))+(1-y msk )log(1-p msk (I,T ′ ));

[0069] Where: L mlm Represents the loss function of the mask language model constructed between the multimodal encoder and the image local feature encoder; y msk represents the predicted probability of the image-text pair; p msk represents the model prediction probability;

[0070] Step 34: The combination strategy of the above four loss functions will affect the final expression effect of the model. It is necessary to assign appropriate weight coefficients for joint optimization of multiple objectives to avoid a single task dominating the joint learning. Therefore, a dynamic update strategy of loss weight is adopted. For each training objective, the ratio of the current loss to the initial loss is considered in each round of training, and a hyperparameter λ is introduced to balance the influence of the weight, as shown below:

[0071]

[0072] Where: θ t represents the weight of task t calculated by the above formula; L i (t) represents the loss value calculated for task t during the current mini-batch iteration; L0(t) represents the loss value for task t during the initial iteration; λ represents the weight used to balance each task, which is set to 0.5; i represents the number of each round of training.

[0073] Preferably, the image-to-text recall rate R in step 41 is i2t @N and text-to-image recall R t2i @N, specifically:

[0074] The image-to-text recall is as follows:

[0075]

[0076] Where: text1, text2…text N They represent the 1st, 2nd to Nth candidate text sample sets returned by the retrieval algorithm respectively;

[0077] The text-to-image recall is as follows:

[0078]

[0079] Where: image1, image2...image N They respectively represent the 1st, 2nd to Nth candidate image sample sets returned by the retrieval algorithm.

[0080] Preferably, the cross-modal retrieval of remote sensing images and text descriptions in step 5 includes a process of searching images by text and a process of searching text by images, specifically:

[0081] The process of searching images by text is as follows: when a text description is input, the constructed cross-modal retrieval model first uses the text encoding module to calculate the features of the input text, and then uses the similarity judgment recall module to calculate the similarity between each image feature and the text feature in the image feature database constructed in step 4, and returns the top1, top5 and top10 similar features as the preliminary retrieval results; the multimodal re-ranking module is used for secondary correction of the preliminary retrieval results, and the module calculates the matching probability between each returned image and the input text, thereby realizing fine-tuning of the preliminary retrieval results;

[0082] The image-based text search process is as follows: when an image is input, the constructed cross-modal retrieval model first uses the image encoding module to calculate the features of the input image, and then uses the similarity judgment recall module in the text feature database constructed in step 4 to calculate the similarity between each text feature and the image feature, and returns the top1, top5 and top10 similar features as preliminary retrieval results; the multimodal reranking module is used for secondary correction of the preliminary retrieval results. The module calculates the matching probability between each returned text and the input image, thereby achieving fine-tuning of the preliminary retrieval results.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] (1) The present invention designs a remote sensing image cross-modal retrieval framework including two single-modal encoders for vision and language and one multi-modal encoder. The single-modal encoder is used to represent image and text features respectively, and the multi-modal encoder is used to fuse the features of the two modalities. The expression ability of each encoder for the fine-grained semantic features of the corresponding modal data is improved through feature fusion and multi-task optimization training, and the cross-modal retrieval task is completed by calculating the similarity of semantic features.

[0085] (2) In order to express the detailed features of the image and eliminate the training overhead brought by the pre-trained object recognition model, the present invention designs a shallow visual transformer model to extract the local features of the image, and transforms the pipelined "object detection + retrieval" process into an end-to-end training process; this end-to-end framework bridges the gap between the object detector and the retrieval model training process, and reduces the training overhead of the entire retrieval model;

[0086] (3) The present invention designs a set of multi-objective optimization strategies for the model and trains the overall model under this strategy, so that the model has the ability to express multiple details of the features of remote sensing images and text descriptions at the same time; the model with convergent training results completes the end-to-end text-image retrieval task without the need for image labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 A control block diagram of a remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to an embodiment of the present invention;

[0088] Figure 2 A diagram of the model training process of an embodiment of the present invention;

[0089] Figure 3 is a diagram of a cross-modal retrieval process according to an embodiment of the present invention;

[0090] Figure 4 This is a diagram of the process of searching images by text according to an embodiment of the present invention;

[0091] Figure 5 This is a diagram of the text search process using images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0093] The embodiment of the present invention uses a certain picture as a case for analysis. In the design of the visual encoder, a visual transformer module trained with a multimodal encoder is introduced to realize the ability of target detection, and the pipeline training process is transformed into an end-to-end training process, which greatly reduces the amount of training data and training time required, and reduces the convenience and accuracy of model building. In the model training process, through multimodal fusion learning and optimization of multi-objective tasks, the model has the ability to handle multi-detail tasks, and better retrieval performance is obtained compared with other related methods. The entire retrieval process is optimized. After the recall task is completed, a re-ranking model based on the language and visual fusion model is introduced. The recall results can be further optimized with less computing consumption, and the top-1 and top-5 sorting performance can be improved. Figure 1 Shown is a control block diagram of a remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to an embodiment of the present invention.

[0094] The embodiment of the present invention provides a remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features, such as Figure 2 The figure shows a model training process diagram of an embodiment of the present invention. In order to illustrate the applicability of the present invention, it is applied to an example, which specifically includes the following steps:

[0095] S1: Processing training data for remote sensing image and text retrieval model.

[0096] The training data of the remote sensing image and text retrieval model is based on image data and text data. The image data is the measured image, and the text data is the descriptive information corresponding to the remote sensing image.

[0097] The image data is processed as follows:

[0098] S111: All image data are uniformly adjusted to a size of 278x278x3.

[0099] S112: Perform basic data enhancement on the input image data with a probability of 50%, including random rotation and random flipping to enhance the generalization ability of the model.

[0100] S113: Expand the data volume of the input image data by using the image data stitching method. According to the category label, two images I_a and I_b are randomly selected within the same category and superimposed at the pixel level. For the text part, the text description is directly stitched, and T_b is directly spliced ​​behind T_a.

[0101] S114: Randomly crop the image data after S112 and S113, and the size of the cropped area is 256x256x3 to adapt to the input of the subsequent neural network model.

[0102] S115: performing normalization processing on the image data after S114 to transform the image grayscale range to between 0 and 1;

[0103] The processing of text data is as follows:

[0104] S121: Remove stop words from the text data, set the maximum word length to 64, truncate the text data, and discard the part exceeding the maximum word length.

[0105] S122: Construct a "random mask" combined with a "directional mask" strategy as follows:

[0106] S1221: Construct a Boolean type text data mask descriptor S_T, whose length is consistent with the word length in the processed text data, and is initialized to False. No mask operation is performed by default.

[0107] S1222: Randomly select 15% of the positions in S_T according to Bernoulli distribution and mark them as True.

[0108] S1223: Record the category labels imgs_cls of all images, traverse the text data, and mark the positions containing imgs_cls on S_T with True to focus on the category information of the target in subsequent mask operations.

[0109] S1224: Construct quantity information descriptor S_N, store English numeric words between 0 and 10, traverse the text data, and mark the position containing S_N with True on S_T to focus on the quantity information of the target in subsequent mask operations.

[0110] S1225: For all positions marked with True in S_T, replace the text data words with [MASK] blank with a probability of 80%; replace the original words with any other random words with a probability of 10%; the words in the remaining 10% positions are not replaced.

[0111] First, get the number of words in the text data, len words and the number of stop words len stops ; Delete all the items that satisfy len words =len stops and len words =len stops +1 image-text data pairs to avoid stop words in the image-text data pairs interfering with the effect of retrieval model training; then, process the image data and the corresponding text data; finally, use the cleaned remote sensing image-text retrieval training data to train the image local encoder and the global encoder.

[0112] S2: Construct a multi-detailed language and vision fusion model.

[0113] Cross-modal retrieval based on image-text features mainly involves extracting and expressing the features of image data and text data respectively. By optimizing the expression capabilities of each extractor, the feature representations of image data and text data with semantic similarity are minimized in the vector space, such as Figure 3 The figure shows the cross-modal retrieval process of an embodiment of the present invention; the overall architecture based on the multi-detail language and visual fusion model includes: remote sensing image visual encoder F enc-V , Remote Sensing Image Description Language Feature Encoder F enc-L and a multimodal encoder F based on the vision-language fusion model enc-Mul , as shown below:

[0114]

[0115] Where: I represents the input remote sensing image data; T represents the input text data; f IL Represents the local features of remote sensing images; fIGL Represents the local-global fusion features of remote sensing images; f T Represents the text features of remote sensing image description; S distance Represents the distance similarity between feature vectors; S pairwise Represents the matching probability value of the image-text pair; F enc-V represents the remote sensing image visual encoder; F enc-L F stands for remote sensing image description language feature encoder; enc-Mul Represents a multimodal encoder based on the vision-language fusion model.

[0116] In the remote sensing image-text cross-modal retrieval task, the amount and complexity of the semantic information contained in the image data is much higher than that of the corresponding text data. enc-V The module M is designed to extract the global features of the image cnn-mvsa And the module M for extracting local features vit , and design a module M that integrates global features and local features midf , as shown below:

[0117]

[0118] Where: f IG Represents the global features of remote sensing images; M cnn-mvsa M represents the global feature extraction module of remote sensing images; vit M represents the local feature extraction module of remote sensing images; midf Represents a module that fuses global features and local features.

[0119] Global feature extraction module M for remote sensing images cnn-mvsa A ResNet-50 residual convolutional neural network is used as a feature extractor, and a multi-scale self-attention model is used to optimize the effect of feature extraction. The cleaned and enhanced whole image in S1 is input into M cnn-mvsa In the equation f IG .

[0120] Local feature extraction module of remote sensing images vit A 6-layer Vision Transformer model is used as a feature extractor to input the serialized image data in S1 into M vit Perform feature extraction in and get f IL .

[0121] The fusion module M of global features and local features midf is a linear function by transforming the feature f IG and f ILPerform linear addition, and the first fusion parameter a and the second fusion parameter b obtained during the training process are as follows:

[0122] M midf (f IG ,f IL ) = af IG +bf IL ;

[0123] Where: M midf Represents the fusion module of global features and local features; a represents the first fusion parameter; b represents the second fusion parameter.

[0124] Remote Sensing Image Description Language Feature Encoder F enc-L Includes a Bert-based text encoder M bert , as shown below:

[0125] f T =M bert (T);

[0126] Where: M bert Represents the standard Bert model.

[0127] During the training process, M bert The feature expression characteristics of the model will be adjusted according to the labeled data to enhance the model's ability to express features. The text data masked in S1 will be used as training data and input into the model M. bert In the middle; after the model training is completed, it enters the application stage M bert Express user input text data as a semantic feature vector.

[0128] Multimodal encoder F based on vision-language fusion model enc-Mul It contains two modules, namely, M for calculating the cross-modal vector distance cms and M for vision-language fusion vlf , as shown below:

[0129] F enc-Mul ={M cms ,M vlf};

[0130] Where: M cms The representation module is used to calculate the distance similarity between the image feature vector and the text feature vector, using the cosine distance to measure; M vlf Indicates the use of visual-language fusion; F enc-Mul Represents a multimodal encoder based on the vision-language fusion model.

[0131] M vlfThe module is initialized with the last 6 layers of a BERT-based model, with an additional criss-cross attention layer to model vision-language interactions.

[0132] S3: Train a detailed language and vision fusion model for multi-objective optimization.

[0133] A multi-objective comprehensive supervised optimization method with four loss functions is constructed, and a deep supervision strategy is introduced in the middle branch.

[0134] S31: The triplet loss is used to learn the representation space of image features and text features. The loss is calculated by comparing the distance between the three comparison sample features. The first triplet loss function L is constructed between the text feature encoder and the image feature encoder. itt1 The features encoded by the local image encoder contain more image details, which are crucial for the final image expression and the subsequent training of the multimodal encoder. A second triplet loss function L is constructed between the text feature encoder and the local image feature encoder. itt2 For deep supervision, it is as follows:

[0135]

[0136] Where: L itt1 represents the first triplet loss function; ε represents the minimum margin used to enlarge the gap between the benchmark sample and the positive / negative sample pair; Sim represents the similarity recall module; (I, T) represents the matched image-text pair features, generated by the image encoder and the text encoder respectively; T ^ Represents the text features that are not matched with image I; I ^ Indicates the image features that are not matched with the text T; L itt2 I represents the second triplet loss function; loc It represents the generation of the local encoder of the image. The + in the lower right corner of the formula means that when the value in [] is greater than 0, the value is taken as the loss, and when it is less than 0, the loss is 0.

[0137] S32: Construct an image-text matching loss function L between the multimodal encoder and the image feature encoder itm , by connecting a fully connected binary classification layer to predict whether a pair of input images and texts match; when selecting positive / negative samples, the similarity recall module is used to calculate the difficult samples closest to the positive samples in a single batch as negative samples to enhance the learning ability of the multimodal module. The loss calculation form is as follows:

[0138] L itm =-y itm log(p itm (I,T))+(1-y itm )log(1-pitm (I,T));

[0139] Where: L itm Represents the image-text matching loss function constructed between the multimodal encoder and the image feature encoder; y itm represents the matching label of the constructed image-text pair; p itm (I,T) represents the image-text matching probability.

[0140] S33: Construct a mask language model loss function L between the multimodal encoder and the image local feature encoder mlm , the masked text is represented by T ′ Indicates that the model predicts probability with p msk (I,T ′ ) is represented as follows:

[0141] L mlm =-y msk log(p msk (I,T ′ ))+(1-y msk )log(1-p msk (I,T ′ ));

[0142] Where: L mlm Represents the loss function of the mask language model constructed between the multimodal encoder and the image local feature encoder; y msk represents the predicted probability of the image-text pair; p msk Represents the model prediction probability.

[0143] S34: The combination strategy of the above four loss functions will affect the final expression effect of the model. It is necessary to assign appropriate weight coefficients for joint optimization of multiple objectives to avoid a single task dominating the joint learning. Therefore, a dynamic update strategy of loss weight is adopted. For each training objective, the ratio of the current loss to the initial loss is considered in each round of training, and a hyperparameter λ is introduced to balance the influence of the weight, as shown below:

[0144]

[0145] Where: θ t represents the weight of task t calculated by the above formula; L i (t) represents the loss value calculated for task t during the current mini-batch iteration; L0(t) represents the loss value for task t during the initial iteration; λ represents the weight used to balance each task, which is set to 0.5; i represents the number of each round of training.

[0146] The model training process is completed. The training process is mainly for optimizing the image encoder, text encoder and multimodal encoder. The image data and text data processed by S1 are divided into a training set, a validation set and a test set according to a first ratio value. In a preferred embodiment, the first ratio value is set to 8:1:1, and the divided training set is sent to the model constructed by S2. The model parameters are initialized with a normal distribution, and no pre-trained parameters are used. When calculating the image-text matching loss, the image encoder part is frozen, and the focus is on optimizing the multimodal encoder.

[0147] S4: Construct a remote sensing image-text description feature library.

[0148] S41: In the retrieval task, the recall rate is used to represent the proportion of correct samples among the N candidate samples returned by the retrieval algorithm. First, use R i2t represents the image-to-text retrieval recall rate and R t2i Represents the retrieval recall rate of text to image; then calculate the image-to-text recall rate R of the top1, top5, and top10 retrieval tasks on the validation set i2t @N and text-to-image recall R t2i @N.

[0149] The image-to-text recall is shown below:

[0150]

[0151] Where: text1, text2…text N They respectively represent the 1st, 2nd to Nth candidate text sample sets returned by the retrieval algorithm.

[0152] The text-to-image recall is shown below:

[0153]

[0154] Where: image1, image2...image N They respectively represent the 1st, 2nd to Nth candidate image sample sets returned by the retrieval algorithm.

[0155] Finally, calculate the image-to-text recall rate R i2t @N and text-to-image recall R t2i @N The average value mR in the test sample i2t @N and mR i2t @N, and save the model with the highest recall rate for subsequent retrieval tasks. The specific calculation formula is as follows:

[0156]

[0157] Where: mR i2t @N and mR t2i @N represents the image-to-text recall rate R i2t @N and text-to-image recall R t2i @N is the average value in the test sample; Image k Text represents the kth image-text pair with similar semantics; k represents the kth text-image pair with similar semantics; R i2t @N(Image k ) represents the image-to-text recall rate, input image Image k ; R t2i @N(Text k ) represents the recall rate of text to image, input text Text k ; k represents the image and text pair number; m represents the total number of image and text pairs; N represents the retrieval task number.

[0158] S42: Build an image feature database, use the image encoder trained in S3 to extract features from all image data, and store the generated image features in the database to improve retrieval efficiency in subsequent applications.

[0159] S43: Build a text feature database, use the text encoder trained in S3 to extract features from all text data, and store the generated text features in the database.

[0160] S5: Complete cross-modal retrieval of remote sensing images and text descriptions.

[0161] Cross-modal retrieval includes four main modules: image encoding module, text encoding module, similarity judgment recall module and multi-modal re-ranking module. The image encoding module is connected in parallel with the text encoding module, followed by the similarity judgment recall module and the multi-modal re-ranking module. The above four modules are used to complete the cross-modal retrieval of remote sensing images and text descriptions. Cross-modal retrieval includes the process of searching images with text and searching text with images, specifically:

[0162] The process of searching images with text is as follows: when a text description is input, the constructed cross-modal retrieval model first uses the text encoding module to calculate the features of the input text, and then uses the similarity judgment recall module to calculate the similarity between each image feature and the text feature in the image feature database constructed in S4, and returns the top1, top5 and top10 similar features as the preliminary retrieval results; the multimodal re-ranking module is used for secondary correction of the preliminary retrieval results. This module calculates the matching probability between each returned image and the input text, thereby achieving fine-tuning of the preliminary retrieval results. Figure 4The figure shows the process of searching images by text in an embodiment of the present invention, taking returning the top 5 similar results as an example.

[0163] The process of searching for text with images is as follows: when an image is input, the constructed cross-modal retrieval model first uses the image encoding module to calculate the features of the input image, and then uses the similarity judgment recall module to calculate the similarity between each text feature and the image feature in the text feature database constructed in S4, and returns the top1, top5 and top10 similar features as the preliminary retrieval results; the multimodal re-ranking module is used for secondary correction of the preliminary retrieval results. This module calculates the matching probability between each returned text and the input image, thereby achieving fine-tuning of the preliminary retrieval results. Figure 5 The figure shows the text search process by image in an embodiment of the present invention, taking returning the top 5 similar results as an example.

[0164] This method has three advantages:

[0165] During the training phase, this method transforms the pipeline training process into an end-to-end training process, transforms the traditional two-stage model training into a one-stage one, and can be trained without relying on additional high-quality annotated datasets of remote sensing images based on target recognition. The model training time is half of that of the two-stage training method.

[0166] During the model training process, through multimodal fusion learning and optimization of multi-objective tasks, the model has the ability to handle multi-detail tasks and semantically align images and texts. Compared with traditional methods, the experimental data shows that the mR i2t @N and mR t2i @N, N=1, 5, 10 The evaluation indexes exceed most of the traditional methods, as shown in Table 1 below. Through the analysis of the embodiments, it can be seen that the actual application effect of this method is better than that of traditional method 1 and traditional method 2.

[0167] Table 1 Comparative analysis results between this method and traditional methods

[0168]

[0169] After the recall task is completed, a reranking model based on the language and vision fusion model is introduced to further optimize the recall results with lower computational consumption and further improve the top-1 and top-5 ranking performance. As shown in Table 2 below, the optimization effect of the reranking model is better than that of the model without reranking.

[0170] Table 2 Comparative analysis results of models with and without reordering

[0171]

[0172] In summary, the prediction results of the cross-modal retrieval method of remote sensing images based on the fusion of language and visual detail features in this case have proven to be very effective.

[0173] (1) The embodiment of the present invention designs a remote sensing image cross-modal retrieval framework including two single-modal encoders for vision and language and one multi-modal encoder. The single-modal encoder is used to represent the image and text features respectively, and the multi-modal encoder is used to fuse the features of the two modalities. The expression ability of each encoder for the fine-grained semantic features of the corresponding modal data is improved through feature fusion and multi-task optimization training, and the cross-modal retrieval task is completed by calculating the similarity of semantic features.

[0174] (2) In order to express the detailed features of the image and eliminate the training overhead brought by the pre-trained target recognition model, the embodiment of the present invention designs a shallow visual transformer model to extract the local features of the image, and transforms the pipelined "target detection + retrieval" process into an end-to-end training process; this end-to-end framework bridges the gap between the target detector and the retrieval model training process, and reduces the training overhead of the entire retrieval model.

[0175] (3) The embodiment of the present invention designs a set of multi-objective optimization strategies for the model, and trains the overall model under this strategy, so that the model has the ability to express multiple detailed features of remote sensing images and text descriptions at the same time; the model with converged training results completes the end-to-end text-image retrieval task that does not require image labels.

[0176] The embodiments described above are only descriptions of the preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features, characterized in that: It includes the following steps: Step 1: Process the training data of the remote sensing image and text retrieval model; The training data of the remote sensing image and text retrieval model includes image data and text data, wherein the image data is a measured image, and the text data is the description information corresponding to the remote sensing image. First, the number of words in the text data is obtained. words and the number of stop words len stops ; Delete all the items that satisfy len words =len stops and len words =len stops +1 image-text data pairs to avoid the stop words in the image-text data pairs interfering with the retrieval model training; then, process the image data and the corresponding text data; finally, use the cleaned remote sensing image-text retrieval training data to train the image local encoder and global encoder; Step 2: Build a multi-detail language and vision fusion model; Cross-modal retrieval based on image-text features is to extract and express the features of image data and text data respectively, and optimize the expression ability of each extractor to minimize the distance between the features of image data and text data with semantic similarity in the vector space; The overall architecture based on the multi-detail language and vision fusion model includes: remote sensing image visual encoder F enc-V , Remote Sensing Image Description Language Feature Encoder F enc-L and a multimodal encoder F based on the vision-language fusion model enc-Mul , as shown below: Where: I represents the input remote sensing image data; T represents the input text data; f IL Represents the local features of remote sensing images; f IGL Represents the local-global fusion features of remote sensing images; f T Represents the text features of remote sensing image description; S distance Represents the distance similarity between feature vectors; S pairwise Represents the matching probability value of the image-text pair; F enc-V represents the remote sensing image visual encoder; F enc-L F stands for remote sensing image description language feature encoder; enc-Mul represents a multimodal encoder based on the vision-language fusion model; Step 3: Train the multi-objective optimized detail language and visual fusion model; A multi-objective comprehensive supervision optimization method including four loss functions is constructed, and a deep supervision strategy is introduced in the middle branch; a model training process is completed, wherein the training process is mainly optimized for an image encoder, a text encoder, and a multimodal encoder; the image data and the text data processed in step 1 are divided into a training set, a validation set, and a test set according to a first ratio value, and the divided training set is sent to the model constructed in step 2, and the model parameters are initialized using a normal distribution without using pre-trained parameters; the image encoder part is frozen when calculating the image-text matching loss, and the multimodal encoder is optimized; the first ratio value is set to 8:1:1; Step 4: Construct a remote sensing image-text description feature library; Step 41: In the retrieval task, the recall rate is used to represent the proportion of correct samples among the N candidate samples returned by the retrieval algorithm; first use R i2t represents the image-to-text retrieval recall rate and R t2i Represents the retrieval recall rate of text to image; then calculate the image-to-text recall rate R of the top1, top5, and top10 retrieval tasks on the validation set i2t @N and text-to-image recall R t2i @N; finally calculate the image to text recall rate R i2t @N and text-to-image recall R t2i @N average value in the test sample mR i2t @N and mR i2t @N, and save the model with the highest recall rate for subsequent retrieval tasks. The specific calculation formula is as follows: Where: mR i2t @N and mR t2i @N represents the image-to-text recall rate R i2t @N and text-to-image recall R t2i @N is the average value in the test sample; Image k Text represents the kth image-text pair with similar semantics; k represents the kth text-image pair with similar semantics; R i2t @N(Image k ) represents the image-to-text recall rate, input image Image k ; R t2i @N(Text k ) represents the recall rate of text to image, input text Text k ; k represents the image and text pair number; m represents the total number of image and text pairs; N represents the retrieval task number; Step 42: construct an image feature database, use the image encoder trained in step 3 to extract features from all image data, and store the generated image features in the database to improve retrieval efficiency in subsequent applications; Step 43: construct a text feature database, use the text encoder trained in step 3 to extract features from all text data, and store the generated text features in the database; Step 5: Complete cross-modal retrieval of remote sensing images and text descriptions; The cross-modal retrieval includes four modules: an image encoding module, a text encoding module, a similarity judgment recall module and a multi-modal reordering module; wherein the image encoding module is connected in parallel with the text encoding module, and then the similarity judgment recall module and the multi-modal reordering module are cascaded; the cross-modal retrieval of remote sensing images and text descriptions is completed through the above four modules.

2. The remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to claim 1 is characterized in that: The processing image data and corresponding text data in step 1 are specifically: The image data processing process is as follows: Step 111: All image data are uniformly adjusted to a size of 278x278x3; Step 112: Perform basic data enhancement on the input image data with a probability of 50%, including random rotation and random flipping to enhance the generalization ability of the model; Step 113: Expand the amount of input image data, adopt an image data splicing method, randomly select two images I_a and I_b in the same category according to the category label, and perform pixel-level superposition. For the text part, directly perform text description splicing, and splice T_b directly behind T_a; Step 114: randomly cropping the image data after step 112 and step 113, and the cropping area size is 256x256x3 to adapt to the input of the subsequent neural network model; Step 115: normalizing the image data after step 114 to transform the image grayscale range to between 0 and 1; The processing process of the text data is as follows: Step 121: Remove stop words from the text data, set the maximum word length to 64, truncate the text data, and discard the part exceeding the maximum word length; Step 122: Construct the "random mask" combined with the "directional mask" strategy as follows: Step 1221: construct a Boolean text data mask descriptor S_T, the length of which is consistent with the length of the words in the processed text data, initialized with False, and no mask operation is performed by default; Step 1222: randomly select 15% of the positions in S_T according to Bernoulli distribution and mark them as True; Step 1223: record the category labels imgs_cls of all images, traverse the text data, and mark the positions containing imgs_cls on S_T with True to focus on the category information of the target in subsequent mask operations; Step 1224: construct a quantity information descriptor S_N, store English numeric words between 0 and 10, traverse the text data, and mark the position containing S_N with True on S_T to focus on the quantity information of the target in subsequent mask operations; Step 1225: For all positions marked with True in S_T, replace the text data words with [MASK] blanks with a probability of 80%; replace the original words with any other random words with a probability of 10%; The remaining 10% of the words are not replaced.

3. The remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to claim 1 is characterized in that: The remote sensing image visual encoder in step 2 is specifically: In the remote sensing image-text cross-modal retrieval task, the amount and complexity of semantic information contained in image data are higher than that of its corresponding text data. Therefore, in the visual encoder F enc-V The module M is designed to extract the global features of the image cnn-mvsa And the module M for extracting local features vit , and design a module M that integrates global features and local features midf , as shown below: Where: f IG Represents the global features of remote sensing images; M cnn-mvsa Represents the remote sensing image global feature extraction module; M vit Represents a local feature extraction module for remote sensing images; M midf Represents a module that fuses global features and local features; The global feature extraction module M of the remote sensing image cnn-mvsa A ResNet-50 residual convolutional neural network is used as a feature extractor, and a multi-scale self-attention model is used to optimize the effect of feature extraction. The entire image after cleaning and enhancement in step 1 is input into M cnn-mvsa In the equation f IG ; The local feature extraction module M of the remote sensing image vit A 6-layer Vision Transformer model is used as a feature extractor to input the serialized image data in step 1 into M vit Perform feature extraction in and get f IL ; The fusion module M of the global feature and the local feature midf is a linear function by transforming the feature f IG and f IL Perform linear addition, and the first fusion parameter a and the second fusion parameter b obtained during the training process are as follows: M midf (f IG ,f IL )=of IG +bf IL ; Where: M midf Represents the fusion module of global features and local features; a represents the first fusion parameter; b represents the second fusion parameter.

4. The remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to claim 1 is characterized in that: The remote sensing image description language feature encoder in step 2 is specifically: The remote sensing image description language feature encoder F enc-L Includes a Bert-based text encoder M bert , as shown below: f T =M bert (T); Where: M bert Represents the standard Bert model; During the training process, M bert The feature expression characteristics of the model will be adjusted according to the labeled data to enhance the model's ability to express features. The text data masked in step 1 will be used as training data and input into the model M. bert In the middle; after the model training is completed, it enters the application stage M bert Express user input text data as a semantic feature vector.

5. The remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to claim 1 is characterized in that: The multimodal encoder based on the vision-language fusion model in step 2 is specifically: Multimodal encoder F based on vision-language fusion model enc-Mul It contains two modules, namely, M for calculating the cross-modal vector distance cms and M for vision-language fusion vlf , as shown below: F enc-Mul ={M cms ,M vlf }; Where: M cms The representation module is used to calculate the distance similarity between the image feature vector and the text feature vector, using the cosine distance to measure; M vlf Indicates the use of visual-language fusion; F enc-Mul represents a multimodal encoder based on the vision-language fusion model; M vlf The module is initialized with the last 6 layers of a BERT-based model, with an additional criss-cross attention layer to model vision-language interactions.

6. The remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to claim 1 is characterized in that: In step 3, a multi-objective comprehensive supervision optimization method including four loss functions is constructed, specifically: Step 31: The triplet loss is used to learn the representation space of image features and text features. The loss is calculated by comparing the distances between the features of the three comparison samples. The first triplet loss function L is constructed between the text feature encoder and the image feature encoder. itt1 The features encoded by the local image encoder contain more image details, which are crucial for the final image expression and the subsequent training of the multimodal encoder. A second triplet loss function L is constructed between the text feature encoder and the local image feature encoder. itt2 For deep supervision, it is as follows: Where: L itt1 represents the first triplet loss function; ε represents the minimum margin used to enlarge the gap between the benchmark sample and the positive / negative sample pair; Sim represents the similarity recall module; (I, T) represents the matching image-text pair features, generated by the image encoder and text encoder respectively; T ^ Represents the text features that are not matched with image I; I ^ Indicates the image features that are not matched with the text T; L itt2 I represents the second triplet loss function; loc Indicates the generation of the local encoder of the image. The + in the lower right corner of the formula means that when the value in [] is greater than 0, the value is taken as loss, and when it is less than 0, the loss is 0; Step 32: Construct the image-text matching loss function L between the multimodal encoder and the image feature encoder itm , by connecting a fully connected binary classification layer, it is possible to predict whether a pair of input images and texts match; When selecting positive / negative samples, the similarity recall module is used to calculate the difficult samples closest to the positive samples in a single batch as negative samples to enhance the learning ability of the multimodal module. The loss calculation form is as follows: L itm =-y itm log(p itm (I,T))+(1-y itm )log(1-p itm (I,T)); Where: L itm Represents the image-text matching loss function constructed between the multimodal encoder and the image feature encoder; y itm represents the matching label of the constructed image-text pair; p itm (I, T) represents the image-text matching probability; Step 33: Construct a mask language model loss function L between the multimodal encoder and the image local feature encoder mlm , the masked text is represented by T ′ Indicates that the model predicts probability with p msk (I,T ′ ) is represented as follows: L mlm =-y msk log(p msk (I,T ′ ))+(1-y msk )log(1-p msk (I,T ′ )); Where: L mlm Represents the loss function of the mask language model constructed between the multimodal encoder and the image local feature encoder; y msk represents the predicted probability of the image-text pair; p msk represents the model prediction probability; Step 34: The combination strategy of the above four loss functions will affect the final expression effect of the model. It is necessary to assign appropriate weight coefficients for joint optimization of multiple objectives to avoid a single task dominating the joint learning. Therefore, a dynamic update strategy of loss weight is adopted. For each training objective, the ratio of the current loss to the initial loss is considered in each round of training, and a hyperparameter λ is introduced to balance the influence of the weight, as shown below: Where: θ t represents the weight of task t calculated by the above formula; L i (t) represents the loss value calculated for task t during the current mini-batch iteration; L0(t) represents the loss value for task t during the initial iteration; λ represents the weight used to balance each task, which is set to 0.5; i represents the number of each round of training.

7. The remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to claim 1 is characterized in that: The image-to-text recall rate R in step 41 i2t @N and text-to-image recall R t2i @N, specifically: The image-to-text recall is as follows: Where: text1, text2…text N They represent the 1st, 2nd to Nth candidate text sample sets returned by the retrieval algorithm respectively; The text-to-image recall is as follows: Where: image1, image2...image N They respectively represent the 1st, 2nd to Nth candidate image sample sets returned by the retrieval algorithm.

8. The remote sensing image cross-modal retrieval method based on the fusion of language and visual detail features according to claim 1 is characterized in that: The cross-modal retrieval of remote sensing images and text descriptions in step 5 includes a process of searching images by text and a process of searching text by images, specifically: The process of searching images by text is as follows: when a text description is input, the constructed cross-modal retrieval model first uses the text encoding module to calculate the features of the input text, and then uses the similarity judgment recall module to calculate the similarity between each image feature and the text feature in the image feature database constructed in step 4, and returns the top1, top5 and top10 similar features as the preliminary retrieval results; the multimodal re-ranking module is used for secondary correction of the preliminary retrieval results, and the module calculates the matching probability between each returned image and the input text, thereby realizing fine-tuning of the preliminary retrieval results; The image-based text search process is as follows: when an image is input, the constructed cross-modal retrieval model first uses the image encoding module to calculate the features of the input image, and then uses the similarity judgment recall module in the text feature database constructed in step 4 to calculate the similarity between each text feature and the image feature, and returns the top1, top5 and top10 similar features as preliminary retrieval results; the multimodal reranking module is used for secondary correction of the preliminary retrieval results. The module calculates the matching probability between each returned text and the input image, thereby achieving fine-tuning of the preliminary retrieval results.

Images