A Remote Sensing Image and Text Retrieval Method Based on Historical Experience

Abstract

This invention proposes a remote sensing image-text retrieval method based on historical experience. The steps are as follows: obtaining training and testing sample sets; constructing a remote sensing image-text retrieval network model θ based on historical experience; iteratively training the remote sensing image-text retrieval network model; and obtaining remote sensing image-text retrieval results. The historical experience network in this invention calculates pseudo-labels for true labels using current experience and related historical experience, reducing similarity measurement bias caused by modal differences, lowering the probability of generating erroneous pseudo-labels, and enhancing the understanding of the relevance between images and text. Furthermore, by encoding each remote sensing image and each piece of text to obtain global semantic information, it fully considers the complexity and scale diversity of remote sensing images. Simultaneously, the multimodal fusion encoder further explores the relationship between images and text and dynamically fuses bi-branch features to enhance modal representation and fusion capabilities, thereby improving retrieval accuracy.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology and relates to a remote sensing image and text retrieval method, specifically a remote sensing image and text retrieval method based on historical experience, which can be applied to fields such as environmental monitoring and management, urban planning and management, and disaster emergency response. Background Technology

[0002] Image-text retrieval is a technique that utilizes the relationships between images and text to retrieve information. Remote sensing image-text retrieval is the application of image-text retrieval in the field of remote sensing, specifically referring to the technique of mutual retrieval using remote sensing images and text. Remote sensing image retrieval aims to establish semantic relationships between remote sensing graphics and text, which requires exploring the global semantic information and intrinsic features within both images and text. Furthermore, due to the high intra-class diversity and inter-class similarity of remote sensing images, text describing the content of one image may contain content from other images, leading to false negative samples in the labeled training set. This makes it difficult for methods used in remote sensing image-text retrieval to accurately learn the relevance between images and text, thus reducing the accuracy of image-text retrieval.

[0003] To improve the accuracy of image-text retrieval, for example, Yuan Zhiqiang et al. proposed a remote sensing image-text retrieval method in their 2022 paper "Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval" published in the journal *IEEE Transactions on Geoscience and Remote Sensing*. This invention constructs an asymmetric multimodal feature matching network that includes visual and text encoders, multi-scale visual self-attention, and visual-guided multimodal fusion.

[0004] By incorporating modules such as the soft-margin operation in the ternary loss, the proposed model can more effectively perform image-text retrieval tasks: a CNN-based visual encoder extracts low-level and high-level features from remote sensing images; a multi-scale visual self-attention module fuses low-level and high-level features and filters redundant information in the image; an RNN-based text encoder extracts text embedding features; and a visually guided multimodal fusion module uses the obtained image features to guide the encoding of text features, thereby narrowing the semantic gap between image and text features and improving the accuracy of image-text retrieval. Furthermore, the network sets a soft margin in the ternary loss based on the similarity between the text in the image-text pair and the text matching the image in the image-text pair, reducing the impact of false negative samples on model training and further improving the accuracy of image-text retrieval.

[0005] However, due to the differences between modalities, the semantic similarity between texts and between text and images focuses on different details, which can easily generate incorrect pseudo-labels and make it difficult to mitigate the impact of false negative samples on the network's learning of image-text relevance. Furthermore, encoders based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have limitations in extracting global semantic information from remote sensing images and texts. At the same time, the cross-modal fusion architecture is relatively simple and cannot further process the complex relationship between remote sensing images and texts, thus limiting the further improvement of the network's retrieval accuracy. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a remote sensing image and text retrieval method based on historical experience to solve the technical problem of low image and text retrieval accuracy in the existing technology.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0008] (1) Obtain the training sample set and the test sample set:

[0009] Obtain M remote sensing images and C descriptive texts matching each image. Pair each remote sensing image with M×C descriptive texts to form a total of M pairs. 2 ×C image-text pairs are generated, and each remote sensing image and its matching descriptive text are labeled as the positive class of the image-text pair for that remote sensing image. The remote sensing image and its unmatched descriptive text are labeled as the negative class of the image-text pair for that remote sensing image. Finally, the image-text pairs of K remote sensing images and their corresponding labels are used to form the training sample set R, and the image-text pairs of the remaining MK remote sensing images are used to form the test sample set E, where M≥2100, C≥5, K≥0.9×M;

[0010] (2) Construct a remote sensing image and text retrieval network model θ based on historical experience:

[0011] Constructing a cascaded remote sensing image retrieval network θ ITR and historical experience network θ LPE The remote sensing image retrieval model θ, where θ ITR It includes a parallel visual encoder and a text encoder, and a multimodal fusion encoder and a classification head cascaded with the outputs of these two encoders; the visual encoder includes a cascaded visual backbone network and a multi-scale fusion module with a recurrent architecture; the multimodal fusion encoder includes two parallel branches composed of multiple multimodal fusion modules and branch fusion modules cascaded with the outputs of these two branches; and a historical experience network θ. LPE It includes a cascaded pseudo-label generation submodule and a historical experience storage module;

[0012] (3) Iteratively train the remote sensing image retrieval network model:

[0013] The training sample set R is used as input to the remote sensing image retrieval network model θ for iterative training to obtain the trained remote sensing image retrieval network model θ. * ;

[0014] (4) Obtain remote sensing image-text retrieval results:

[0015] The test sample set E is used as the trained remote sensing image retrieval network model θ. * Input, remote sensing image retrieval network θ ITR The matching probability of the image and text in each test sample is detected and sorted in descending order of probability to obtain the text that best matches each image.

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] 1. The historical experience network in this invention calculates pseudo-labels for true labels using current experience and related historical experience. Since historical experience is a joint representation of images and text, this historical experience-dependent method uses the similarity between task-oriented multimodal fusion features as the basis for generating pseudo-labels. This can reduce the similarity measurement bias caused by modal differences, reduce the probability of generating erroneous pseudo-labels, and thus enhance the network's understanding of the relevance of images and text. Compared with existing technologies, this effectively improves retrieval accuracy.

[0018] 2. This invention encodes each remote sensing image and each text in each training sample using a visual encoder and a text encoder, respectively, to obtain global semantic information. It fully considers the complexity and scale diversity of remote sensing images. At the same time, the multimodal fusion encoder adopts a more complex dual-branch fusion architecture to further explore the relationship between images and text, and dynamically fuses dual-branch features to enhance modal representation and fusion capabilities. Compared with existing technologies, this invention further improves retrieval accuracy. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0020] Figure 2 This is a schematic diagram of the remote sensing image retrieval network model of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0022] Reference Figure 1 The present invention includes the following steps:

[0023] Step 1) Obtain the training sample set and the test sample set:

[0024] Obtain M remote sensing images and C descriptive texts matching each image. Pair each remote sensing image with M×C descriptive texts to form a total of M pairs. 2 ×C image-text pairs are generated, and each remote sensing image and its matching descriptive text are labeled as the positive class of the image-text pair for that remote sensing image. The remote sensing image and its unmatched descriptive text are labeled as the negative class of the image-text pair for that remote sensing image. Finally, the image-text pairs of K remote sensing images and their corresponding labels are used to form the training sample set R, and the image-text pairs of the remaining MK remote sensing images are used to form the test sample set E, where M≥2100, C≥5, K≥0.9×M;

[0025] In this embodiment, the remote sensing image size is 256×256×3, M=2100, C=5, K=0.9×M. The training sample set R contains K image-text pairs labeled as positive, including (K 2 -1)×C image-text pairs labeled as negative.

[0026] Step 2) Construct a remote sensing image and text retrieval network model θ based on historical experience:

[0027] Constructing a cascaded remote sensing image retrieval network θ ITR and historical experience network θ LPE The remote sensing image retrieval model θ, such as Figure 2 As shown, where θ ITR It includes a parallel visual encoder and a text encoder, and a multimodal fusion encoder and a classification head cascaded with the outputs of these two encoders; the visual encoder includes a cascaded visual backbone network and a multi-scale fusion module with a recurrent architecture; the multimodal fusion encoder includes two parallel branches composed of multiple multimodal fusion modules and branch fusion modules cascaded with the outputs of these two branches; and a historical experience network θ. LPE It includes a cascaded pseudo-tag generation submodule and a historical experience storage module.

[0028] Remote sensing image retrieval network θ ITR The visual encoder contains a T-order visual backbone network. The multi-scale fusion module includes T cascaded multi-scale attention mechanism modules. Each multi-scale attention mechanism module includes three parallel sub-modules and a cascaded fully connected layer. Each sub-module consists of a layer normalization layer and a fully connected layer stacked together. The text encoder includes multiple cascaded Transformer encoders. Each multi-modal fusion module in the multi-modal fusion encoder consists of three cascaded sub-modules. The branch fusion module consists of attention sub-modules, such as...

[0029] In this embodiment, the existing Swin-Transformer network is used as the visual backbone network, which includes four cascaded stages, i.e., T=4. The existing BERT network is used as the text encoder, which consists of 12 cascaded Transformer encoders.

[0030] Step 3) Iteratively train the remote sensing image retrieval network model:

[0031] The training sample set R is used as input to the remote sensing image retrieval network model θ for iterative training to obtain the trained remote sensing image retrieval network model θ. * ;

[0032] The remote sensing image retrieval network model θ is trained iteratively, and the steps are as follows:

[0033] (3a) Initialize the number of iterations to i, the maximum number of iterations to I, I≥135, and the remote sensing image retrieval network model θ in the i-th iteration. i The weights and bias parameters in the equation are w. i b i Historical experience network The historical experience storage module stores O random data points, O≥K, and lets i=1;

[0034] (3b) Remote Sensing Image Retrieval Network The visual encoder and text encoder encode the k-th remote sensing image and the k-th text in each training sample, respectively; the multimodal fusion encoder processes the features of the encoded k-th remote sensing image. Text features Perform fusion; classify the heads to obtain multimodal representations from the fusion. A linear mapping is performed, and the features obtained by the mapping are normalized using the softmax(·) function to obtain the image-text matching probability distribution of the k-th training sample.

[0035] The visual encoder uses a visual backbone network to sequentially generate visual representations at different scales. The multi-scale fusion module utilizes a multi-scale attention mechanism to hierarchically enhance the high-level semantic features of the image. For the t-th level feature, the complete features of higher levels are fused together. As a query, the feature M at this level t As keys and values, the enhanced features at the current level It can be obtained from the following formula:

[0036]

[0037]

[0038]

[0039]

[0040] in and Let represent a linear mapping, c be the number of channels for the T-th level image feature, and softmax(·) and LN(·) represent the nonlinear activation function and layer normalization function, respectively. In this way, both current and deeper information will be thoroughly considered. When the enhanced features... Once all are obtained, they are compared with the classification token features. Integrating them together to generate a comprehensive visual representation

[0041] The multimodal fusion encoder first fuses image features through two parallel branches consisting of N multimodal fusion modules. and text features These are referred to as visual branches and text branches. Here, we'll use the nth block in the visual branch as an example to explain the forward propagation process. Specifically, given the text representation... and the output of the (n-1)th block in the visual branch The output of the nth block is calculated as follows:

[0042]

[0043]

[0044]

[0045] Here, MSA(·), MCA(·), and MLP(·) represent functions for multi-head self-attention, multi-head cross-attention, and multilayer perceptron, respectively. After passing through N fusion blocks, hybrid visual features are obtained. Similarly, text features In each fusion module within the text branch, the data sequentially undergoes multi-head self-attention, multi-head cross-attention, and a multilayer perceptron, and is combined with image features in the multi-head cross-attention process. The fusion process, after passing through N fusion blocks, yields hybrid text features.

[0046] The branch fusion module utilizes a weighted fusion strategy to fuse hybrid features. and This generates a unified multimodal representation. Specifically, given and The characteristics corresponding to the "[CLS]" token in and First, they are concatenated to obtain the joint features. Then, the learnable vector q is used. IT The fusion is accomplished using an attention mechanism for the query, with the following details:

[0047]

[0048]

[0049]

[0050]

[0051] in and Let q be a learnable matrix. In the above equation, q IT As a fundamental component for generating dynamic fusion weights, it enables the branch fusion module to flexibly provide... and Assigning importance captures the complex relationship between the two modalities while preserving the inherent differences between each feature.

[0052] (3c) Historical Experience Network θ LPE The pseudo-label generation submodule in the middle will represent multimodal characteristics. Image-text matching probability distribution And the k-th image-text pair label L k Composition of current experience E k And through E k and historical experience storage module with E k The weighted sum of relevant historical experience values, L. k pseudo-tags Simultaneously via E k Replace the historical experiences in the historical experience storage module;

[0053] Current experience E k The expression is:

[0054]

[0055] in, This represents the key used to find relevant experiences, and the value represents the pseudo-tags generated. The value of .

[0056] Pseudo-labels The calculation formula is:

[0057]

[0058]

[0059] in, These are pseudo-labels generated based on historical experience. s represents the value of each historical experience stored in the historical experience storage module. k,o Indicates the current experience E k With the o-th experience E in the historical experience storage module o The mixed similarity of the keys, defined as follows:

[0060]

[0061] in, Represents the probability distribution of image-text matching The probability corresponding to the first dimension, This represents the o-th historical experience E in the historical experience storage module. o Stored image-text matching probability distribution The first dimension corresponds to the probability, and cos(·) represents the cosine similarity function. E represents the o-th historical experience. o The stored multimodal representations, where α represents the threshold, and Filter(·,α) is the filtering function, defined as follows:

[0062]

[0063] The balancing parameter c takes the following form:

[0064] c = γ × (W) E ) r

[0065]

[0066]

[0067] Where r is a learnable parameter, and γ is the ratio of the number of historical experiences in the historical experience storage module to the total capacity O.

[0068] (3d) Using the k-th image-text pair label L k and its corresponding pseudo-tags and the probability distribution of image and text matching Calculate the loss value of the remote sensing image retrieval network model and through For weights and bias parameters w i b i The model is updated to obtain the remote sensing image retrieval network model θ for this iteration. i ;

[0069] Loss value of remote sensing image retrieval network model The calculation formula is:

[0070]

[0071]

[0072]

[0073] in, This represents the image-text matching loss function. The loss function represents the balance parameter optimization function, and MSE(·) represents the mean squared error loss function. Indicates by The classification head generates a two-dimensional image-text matching probability distribution that does not participate in gradient calculation;

[0074] Weights and bias parameters w i b i The update formulas are as follows:

[0075]

[0076]

[0077] Among them, w i+1 b i+1 They represent w respectively i b i The update result is given by α, which represents the learning rate.

[0078] (3e) Determine whether i > A is true. If yes, obtain the trained remote sensing image retrieval network model θ. * , otherwise, let i=i+1, θ i =θ and execute step (3b).

[0079] Step 4) Obtain remote sensing image-text search results:

[0080] The test sample set E is used as the trained remote sensing image retrieval network model θ. * Input, remote sensing image retrieval network θ ITR The matching probability of the image and text in each test sample is detected and sorted in descending order of probability to obtain the text that best matches each image.

Claims

1. A method for retrieving remote sensing images based on historical experience, characterized in that, Comprising the following steps: (1) Obtain a training sample set and a test sample set: Get A remote sensing image and the matching of each remote sensing image Each descriptive text, and each remote sensing image is associated with Each descriptive text is grouped into pairs, resulting in a total of [number] pairs. Each remote sensing image is assigned an image-text pair. Then, each image and its matching descriptive text are labeled as the positive class of the image-text pair for that image, and each unmatched image and its unmatched descriptive text are labeled as the negative class of the image-text pair for that image. Finally, the image-text pairs are... The training sample set consists of image-text pairs of remote sensing images and their corresponding labels. , the remaining The test sample set consists of image-text pairs of remote sensing images. ,in, , , ; (2) Constructing a remote sensing image-text retrieval network model based on historical experience : Remote sensing image-text retrieval model including cascaded remote sensing image-text retrieval network and history experience network wherein comprises a visual encoder and a text encoder arranged in parallel, and a multi-modal fusion encoder and a classification head cascaded with the output ends of the two encoders; the visual encoder comprises a visual backbone network and a multi-scale fusion module adopting a recurrent architecture cascaded; the multi-modal fusion encoder comprises two branches composed of multiple multi-modal fusion modules arranged in parallel, and a branch fusion module cascaded with the output ends of the two branches; the history experience network comprises a pseudo-label generation submodule and a history experience storage module cascaded, for calculating a pseudo-label of a true label through current experience and history experience related thereto;​ (3) Iteratively train the remote sensing image-text retrieval network model: The training sample set is obtained by iterative training on the input of the remote sensing image-text retrieval network model , to obtain a trained remote sensing image-text retrieval network model ; in the training process, the loss value of the remote sensing image-text retrieval network model is calculated through each image-text pair label and its corresponding pseudo label, and the image-text matching probability distribution. (4) Obtain a remote sensing image-text retrieval result: The test sample set is divided into a training set and a test set As the input of the trained remote sensing image-text retrieval network model The remote sensing image-text retrieval network Detects the matching probability of the image and the text in each test sample, and sorts them in descending order of probability to get the most matched text for each image.

2. The method of claim 1, wherein, The remote sensing image-text retrieval network model in step (2) wherein: Remote sensing image-text retrieval network The visual backbone network contained in the visual encoder is The multi-scale fusion module comprises cascaded Each multi-scale attention mechanism module comprises three sub-modules arranged in parallel and a fully connected layer cascaded therewith, each sub-module being stacked by a layer normalization layer and a fully connected layer; the text encoder comprises cascaded multiple Transformer encoders; each multi-modal fusion module in the multi-modal fusion encoder is composed of three sub-modules cascaded; and the branch fusion module is composed of an attention sub-module.

3. The method of claim 2, wherein, The remote sensing image retrieval network model in step (3) The iteration training is implemented, and the implementation steps are as follows: (3a) initialize the iteration number as , the maximum iteration number as , , the weight and bias parameters in the remote sensing image-text retrieval network model in the first iteration as , , , the historical experience network having stored random data in the historical experience storage module, , and letting ;​ (3b) Remote sensing image-text retrieval network The visual encoder and the text encoder respectively encode the first remote sensing image and the first text in each training sample; the multi-modal fusion encoder fuses the encoded remote sensing image features and text features; and the image-text retrieval network is trained by using the fused features and the label of the training sample as a training sample. ​​​​​ The classification head is used for classifying the multi-modal representation A linear mapping is performed, and a The features obtained by the mapping are normalized by using a The image-text matching probability distribution of the i-th training sample ​ (3c) Historical experience network The pseudo-label generation submodule in the middle will represent multimodal characteristics. Image-text matching probability distribution and the Image-text pair label Composition of current experience and through and in the historical experience storage module Relevant historical experience values Weighted sum calculation pseudo-tags At the same time, through Replace the historical experiences in the historical experience storage module; (3d) label the first image-text pair and its corresponding pseudo label , and the image-text matching probability distribution calculate the loss value of the remote sensing image-text retrieval network model , and update the weight and bias parameters , , , , to obtain the remote sensing image-text retrieval network model of this iteration ; (3e) judging whether it is true, if yes, obtaining the trained remote sensing image-text retrieval network model , otherwise, letting , and performing step (3b).

4. The method of claim 3, wherein, The visual encoder described in step (3b) encodes the first The remote sensing image is encoded, and the implementation steps are as follows: The visual backbone network extracts features of the remote sensing image at a plurality of scales for each training sample The multi-scale fusion module fuses the remote sensing image features extracted by the visual backbone network at the plurality of scales to obtain remote sensing image features at a plurality of scales ​​​​ 5. The method of claim 3, wherein, The current experience described in step (3c) whose expression is: ； wherein, represents a key for looking up relevant experience , represents a value for generating pseudo-label .

6. The method of claim 5, wherein, The pseudo label described in step (3c) The calculation formula is: ； ； where c is a learnable tuning parameter, pseudo-labels generated from historical experiences, represent values in each historical experience stored in the historical experience storage module, represents the current experience the similarity of the key of the i-th experience in the historical experience storage module, the key of the i-th experience 7. The method of claim 5, wherein, The loss value of the remote sensing image-text retrieval network model in step (3d) The calculation formula is: ； ； ； wherein, represents a graph-text matching loss function, represents a balance parameter optimization loss function, represents a mean square error loss function, represents a two-dimensional graph-text matching probability distribution generated by the classification head in which does not participate in gradient calculation.

8. The method of claim 5, wherein, The weight and bias parameters in step (3d) are updated according to the following update equations, respectively: , , ； ； wherein, , respectively represent , the update result of denotes the learning rate.

Images