A remote sensing image-text retrieval method based on guided visual semantic alignment

Abstract

A remote sensing image-text retrieval method based on guided visual semantic alignment, including preprocessing before modal input, building of image-text retrieval model and design of loss function, wherein the building of image-text retrieval model includes three parts of constructing modal input, modal interaction and similarity measurement, the target-oriented attention module of the modal interaction includes an intra-modal fusion attention module and an inter-modal guided attention module, which uses significant target features as a guide to adaptively adjust the distance of the final visual and text embedding in the latent semantic space. The invention solves the common visual-semantic imbalance problem in remote sensing image-text retrieval, and deeply excavates the correlation between vision and language, improving the accuracy of remote sensing image-text retrieval.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing, and particularly relates to a remote sensing image-text retrieval method based on guided visual semantic alignment. BACKGROUND

[0002] With the rapid development of aerospace technology and computer technology, remote sensing data and its applications are growing explosively. Reasonable use of remote sensing data can better help and improve people's life and production. Remote sensing image-text retrieval refers to obtaining more interesting and valuable data (text or image) from massive remote sensing data (image or text), which plays a significant role in resource investigation, disaster detection, agricultural production and other fields. The characteristics of semantic redundancy and inter-class similarity in remote sensing data cause serious visual-semantic imbalance in the process of remote sensing image-text retrieval. These imbalances can lead to incorrect matching of non-semantic visual and text features, reducing the accuracy of image-text retrieval.

[0003] Current remote sensing-based image-text retrieval methods are mainly divided into global visual feature-based methods and global and local visual feature-based methods according to visual representation methods. These methods pay less attention to this visual-semantic imbalance and rely too much on single visual information, resulting in decreased retrieval accuracy. SUMMARY

[0004] In order to overcome the low accuracy of the prior art, the present application provides a remote sensing image-text retrieval method based on guided visual semantic alignment, which solves the problem of visual-semantic imbalance commonly existing in remote sensing image-text retrieval and deeply excavates the correlation between vision and language.

[0005] In order to achieve the above-mentioned purpose, the technical scheme of the present application is as follows:

[0006] A remote sensing image-text retrieval method based on guided visual semantic alignment, the method comprising the following steps:

[0007] Step one, preprocessing before modal input, preprocessing the images and texts of the remote sensing image-text data set to obtain standard model input sample data;

[0008] Step two, building of the image-text retrieval model, including three parts of building modal input, modal interaction and similarity measurement, realizing visual and text modal feature extraction, modal information interaction and calculation of cosine similarity of visual and text modal;

[0009] Step three, design of the loss function, including a ternary sorting loss function and a global visual semantic loss function, and aligning the images and texts by minimizing the loss function.

[0010] Further,

[0011] The step two comprises the following sub-steps:

[0012] Step 2.1: constructing modal input; the modal input comprises two visual encoders and a text encoder; wherein the visual encoders comprise a multi-scale visual encoder and a salient target encoder;

[0013] Step 2.2: constructing modal interaction; the modal interaction comprises a target-oriented attention module; which comprises an intra-modal fusion attention module and an inter-modal guided attention module; the intra-modal fusion attention module fuses the mapped multi-scale visual features and salient target features in step 2.1.2 to obtain fused visual features; the inter-modal guided attention module uses the mapped salient target features in step 2.1.3 to guide the expression of the word-level text features to obtain visually guided text features;

[0014] Step 2.3: constructing similarity measurement; the mapped multi-scale visual features and word-level text features in step 2.1.2, and the fused visual features and visually guided text features obtained in step 2.2 are converted into corresponding embeddings using mean value processing, and the cosine similarity of the multi-scale visual embedding and the word-level text embedding, and the cosine similarity of the fused visual embedding and the visually guided text embedding are calculated respectively.

[0015] Preferably, the process of the sub-step is as follows:

[0016] Step 2.1.1: the multi-scale visual encoder uses ResNet-50 as the network skeleton, which is pre-trained on the AID dataset, for extracting multi-scale visual features in remote sensing images; a multi-layer perceptron is added after the multi-scale visual encoder and the posterior respectively to map the extracted features;

[0017] Step 2.1.2: the salient target encoder uses ResNet-50 as the network skeleton, which is pre-trained on the DOTA dataset, for extracting salient target features; a fully connected layer is added after the salient target encoder to map the extracted features;

[0018] Step 2.1.3: wherein the text encoder is composed of a Glove model and a bidirectional GRU; the Glove model is used to map the sentence vectors obtained in step 1.2 into a 300-dimensional sentence vector; the bidirectional GRU obtains the context relationship of these sentence vectors to obtain preliminary text features; a double-flow gating module is added after the text encoder to strengthen the semantic association of the deep text features to obtain word-level text features.

[0019] Further, the step three comprises the following sub-steps:

[0020] Step 3.1: The triplet ranking loss function achieves the final alignment between the image and the text by minimizing the distance between the fused visual embedding and the visually guided text embedding in the latent semantic space;

[0021] Step 3.2: The global semantic loss function minimizes the multi-scale visual embedding and word-level text embedding to ensure the invariance of the modal input to the original semantics of the image and text, serving as an external constraint for the fusion of visual embedding and visually guided text embedding.

[0022] Step 3.3: Combine the ternary ranking loss function and the global semantic loss function as the overall loss function for model training.

[0023] The process of sub-step 2.2 is as follows:

[0024] The goal-oriented attention module includes an intra-modal fusion attention module and an inter-modal guided attention module; the intra-modal fusion attention module processes the multi-scale visual features F mapped in step 2.1.2. M and salient target features F R To achieve fusion, first, F M and F R F′ is obtained after passing through a fully connected layer. M and F′ R Then, the joint score of the two features is calculated using matrix multiplication, and then the two features are activated separately. The formula is expressed as:

[0025]

[0026]

[0027]

[0028] Where S MR This represents the joint score of multi-scale visual features and salient object features. and The aggregated features are represented, and the final fused visual features F are obtained. MR The formula is expressed as:

[0029]

[0030] Among them W L and b L ∪ represents the weights and biases of the linear transformation, and ∪ represents concatenation by channel;

[0031] The intermodal guided attention module utilizes the salient target features F mapped in step 2.1.2. R To guide the expression of the mapped word-level text features FG in step 2.1.3, firstly... and The pre-processing is performed, and the mean value is calculated respectively, which is expressed by the formula:

[0032]

[0033]

[0034] Then E R and E G After a fully connected layer, F′ R and F′ G is obtained, and then the joint score of the two features is calculated, and then the text feature is activated, which is expressed by the formula:

[0035]

[0036]

[0037] wherein S RG represents the joint score of the salient target feature and the word-level text feature, represents the aggregated feature, and finally the visual-guided text feature F RG is obtained, which is expressed by the formula:

[0038]

[0039] wherein MLP represents a multi-layer perception machine.

[0040] In the sub-step 3.3, the overall loss function is expressed by the formula:

[0041]

[0042]

[0043] wherein α represents an edge parameter, [x] + ≡max(x, 0), and represent the most negative samples in the minimum batch of samples V and T in the remote sensing image and the text, and S(·, ·) represents the cosine similarity calculated in step 2.3, represents the triplet ranking loss function, represents the global semantic loss function, λ g represents the weight factor of the global visual semantic loss function.

[0044] The step one includes the following sub-steps:

[0045] Step 1.1: preprocessing of the remote sensing image; dividing the image data into a training set, a validation set and a test set; performing data enhancement processing on the training and validation data, including scaling, random cropping, random flipping and standardization processing;

[0046] Step 1.2: preprocessing of text data; five sentences of text description corresponding to each remote sensing image; text segmentation, stop word removal and one-hot encoding processing are performed on the texts to obtain corresponding word vectors; and the word vectors are recombined to obtain sentence vectors. The beneficial effects of the present application are: using significant visual features as a guide, adaptively adjusting the distance of the final visual and text embedding in the latent semantic space, reducing the dependence on single visual information, effectively solving the visual-semantic imbalance in remote sensing image-text retrieval, and improving the accuracy of remote sensing image-text retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of the image-text retrieval method of the present application.

[0048] Figure 2 is a schematic diagram of the image-text retrieval network framework of the present application.

[0049] Figure 3 is a schematic diagram of the double-flow gating module of the present application.

[0050] Figure 4 is a schematic diagram of the target-oriented attention module of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0052] Reference Figures 1-4 A remote sensing image-text retrieval method based on guided visual semantic alignment, comprising the following steps:

[0053] Step 1, preprocessing before modal input, preprocessing the images and texts of the remote sensing image-text data set to obtain standard model input sample data;

[0054] The step 1 includes the following sub-steps:

[0055] Step 1.1: preprocessing of remote sensing images; the image data is divided into training set, validation set and test set; data augmentation processing is performed on the training and validation data, including scaling, random cropping, random flipping and standardization processing;

[0056] Step 1.2: preprocessing of text data; five sentences of text description corresponding to each remote sensing image; text segmentation, stop word removal and one-hot encoding processing are performed on the texts to obtain corresponding word vectors; and the word vectors are recombined to obtain sentence vectors wherein

[0057] Step two, the construction of the image-text retrieval model, including the construction of modal input, modal interaction and similarity measurement three parts, realize the extraction of visual and text modal characteristics, modal information interaction and calculate the cosine similarity of visual and text modal;

[0058] The step S2 includes the following sub-steps:

[0059] Step 2.1: as Figure 2 The modal input includes two visual encoders and a text encoder; wherein the visual encoder includes a multi-scale visual encoder and a salient target encoder;

[0060] The process of step 2.1 is as follows:

[0061] Step 2.1.1: the multi-scale visual encoder uses ResNet-50 as the network skeleton, pre-trains on the AID dataset, and extracts multi-scale visual features in the remote sensing image Wherein N m =4, representing the semantic level features output by the multi-scale visual encoder layer1, layer2, layer3 and layer4, a multi-layer perceptron is added after the multi-scale visual encoder to map the extracted features, which can be expressed as:

[0062] F M =MLP(M v ), (1)

[0063] Wherein represents the mapped multi-scale visual features, and MLP(·) represents the multi-layer perceptron;

[0064] Step 2.1.2: the salient target encoder uses ResNet-50 as the network skeleton, pre-trains on the DOTA dataset, and extracts salient target features Wherein N r =36, a fully connected layer is added after the salient target encoder to map the extracted features, which can be expressed as:

[0065] F R =R v W r +b r , (2)

[0066] Wherein represents the mapped salient target features, W r and b r represent the weights and bias of the fully connected layer;

[0067] Step 2.1.3: The text encoder consists of a Glove model and a bidirectional GRU; the Glove model maps the sentence vectors obtained in step 1.2 into a 300-dimensional sentence vector e i = W e i (i∈[1, N c ]) where W e is the parameter matrix of Glove, and then the context relationship of these sentence vectors is obtained through the bidirectional GRU to obtain the preliminary text features, which can be expressed as:

[0068]

[0069]

[0070] where and are the i-th layer hidden layer outputs of the bidirectional GRU forward and backward, respectively. Therefore, the forward and backward text features and can be obtained as shown in the following formula: Figure 3

[0071]

[0072]

[0073]

[0074]

[0075] F G = MLP(T f⊙b +T b⊙f )+T f⊙b +T b⊙f , (9)

[0076] where and are the activated forward and backward text features, respectively, and T f⊙b and T b⊙f are the forward and backward interaction features, respectively, and ⊙ represents element-wise multiplication; GA represents the gated self-attention module, and its input is assumed to be Then and can be obtained, which represent the query, key and value, respectively, and can be expressed as follows:

[0077] ​​​

[0078] Among them W Q (W K W V ) and b Q (b K b V The weights and biases of the linear transformation are represented by ). Then, Q and V are activated through a gating mechanism, as shown below:

[0079] G=σ((Q⊙K)W A +b A (11)

[0080] Q′=G⊙Q, (12)

[0081] K′=G⊙K, (13)

[0082] Where G represents the gating activation value, W A and b A The weights and biases represent the linear transformation, σ(·) represents the sigmoid function, and Q and K are the activations of Q′ and K′. The final activations can be calculated using a click attention mechanism. The formula is expressed as follows:

[0083]

[0084] Softmax(·) represents the softmax function.

[0085] Step 2.2: As Figure 4 The target-oriented attention module shown includes an intra-modal fusion attention module and an inter-modal guided attention module. The intra-modal fusion attention module processes the multi-scale visual features F mapped in step 2.1.2. M and salient target features F R To achieve fusion, first, F M and F R Perform a linear transformation, expressed as:

[0086] F′ M =F M W M +b M (15)

[0087] F′ R =F R W R +b R (16)

[0088] Among them W M (W R ) and b M (b R) represent the weights and bias of linear transformation, respectively. Then the joint score of F and F′ M R is calculated by matrix multiplication, and then the two features are activated respectively, which can be expressed as:

[0089]

[0090]

[0091]

[0092] where S MR represents the joint score of multi-scale visual features and salient object features, and represent the aggregated features. Finally, the fused visual features F MR are obtained, which can be expressed as:

[0093]

[0094] where W L and b L represent the weights and bias of linear transformation, and represents concatenation along the channel.

[0095] The inter-modal guided attention module uses the mapped salient object features F R in step 2.1.2 to guide the expression of the mapped word-level text features F G in step 2.1.3. First, the and are preprocessed, and the mean is calculated respectively, which can be expressed as:

[0096]

[0097]

[0098] Then, linear transformation is performed on E R and E G as in equations (15) and (16), and F′ R and F′ G are obtained. Then the joint score of the two features is calculated, and then the text features are activated, which can be expressed as:

[0099]

[0100]

[0101] where S RG represents the joint score of salient object features and word-level text features, represent the aggregated features. Finally, the visual-guided text features F​RG The formula is expressed as:

[0102]

[0103] Step 2.3: Calculate the multi-scale visual features F obtained in Step 1.2. M and word-level text features F G Step 2.1 Obtained fused visual features F MR and visually guided text features F RG The average value is converted into the corresponding embedding using formulas (21) and (22), where the multi-scale visual embedding V is calculated respectively. M and word-level text embedding T G Cosine similarity, fusion visual embedding V MR and visually guided text embedded T RG Cosine similarity;

[0104] Step 3: Design of loss functions, including triplet ranking loss function and global visual semantic loss function, to achieve image and text alignment by minimizing the loss function;

[0105] Step S3 includes the following sub-steps:

[0106] Step 3.1: First, define a bidirectional triplet loss function, expressed as follows:

[0107]

[0108] Where α represents the marginal parameter, [x] + ≡max(x, 0), and S(·,·) represents the most negative sample in the smallest batch of samples V and T in the remote sensing image and text, and S(·,·) represents the cosine similarity calculated in step 2.3.

[0109] Triple sorting loss function The final alignment between the image and text is achieved by minimizing the distance between the fused visual embedding and the visually guided text embedding in the latent semantic space;

[0110] Step 3.2: Global Semantic Loss Function By minimizing multi-scale visual embedding and word-level text embedding, the invariance of the modal input to the original semantics of the image and text is guaranteed, serving as an external constraint for the fusion of visual embedding and visually guided text embedding.

[0111] Step 3.3: Combine the ternary ranking loss function and the global semantic loss function as the overall loss function for model training, expressed as:

[0112]

[0113] wherein λ g represents a weight factor of the global visual semantic loss function.

[0114] The application also provides an implementation of the remote sensing image text retrieval system based on guided visual semantic alignment, comprising the following modules:

[0115] Preprocessing before modal input, preprocessing the images and texts of the remote sensing image-text dataset to obtain standard model input sample data;

[0116] Building of the image-text retrieval model, including three parts of constructing modal input, modal interaction and similarity measurement, realizing the extraction of visual and text modal features, information interaction between modes and calculation of cosine similarity of visual and text modes;

[0117] Design of the loss function, including a triplet ranking loss function and a global visual semantic loss function, and aligning the images and texts by minimizing the loss function.

[0118] The above modules correspond to steps 1-3 of the method.

[0119] The above embodiments only express two embodiments of the application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which are all within the scope of the application. Therefore, the protection scope of the patent of the application should be subject to the appended claims.

Claims

1. A remote sensing image and text retrieval method based on guided visual semantic alignment, characterized in that, The method includes the following steps: Step 1: Preprocessing before modal input. Preprocess the images and text in the remote sensing image-text dataset to obtain standard model input sample data. Step 2: Building the image and text retrieval model, which includes three parts: constructing modal input, modal interaction, and similarity measurement, to achieve the extraction of visual and text modal features, information interaction between modalities, and calculation of the cosine similarity between visual and text modalities; Step 3: Design of loss functions, including triplet ranking loss function and global visual semantic loss function, to achieve image and text alignment by minimizing the loss function; Step two includes the following steps: Step 2.1: Construct modal input; the modal input includes two visual encoders and one text encoder; wherein the visual encoders include a multi-scale visual encoder and a salient object encoder; the process is as follows: Step 2.1.1: The multi-scale visual encoder uses ResNet-50 as the network skeleton and is pre-trained on the AID dataset to extract multi-scale visual features from remote sensing images; multilayer perceptrons are added to the multi-scale visual encoder and the network to map the extracted features. Step 2.1.2: The salient object encoder uses ResNet-50 as the network skeleton and is pre-trained on the DOTA dataset to extract salient object features; fully connected layers are added after the salient object encoder to obtain the mapping of the extracted features; Step 2.1.3: The text encoder consists of a GloVe model and a bidirectional GRU. The GloVe model is used to map sentence vectors into a 300-dimensional sentence vector. The bidirectional GRU obtains the contextual relationships of these sentence vectors to obtain preliminary text features. A dual-stream gating module is added after the text encoder to enhance the deep semantic associations of the preliminary text features and obtain word-level text features. Step 2.2: Construct modal interaction; the modal interaction includes a goal-oriented attention module; it includes an intra-modal fusion attention module and an inter-modal guided attention module; the intra-modal fusion attention module fuses the multi-scale visual features mapped in Step 2.1.1 and the salient target features mapped in Step 2.1.2 to obtain fused visual features; the inter-modal guided attention module uses the salient target features mapped in Step 2.1.2 to guide the expression of word-level text features to obtain visually guided text features; Step 2.3: Construct similarity metrics; convert the multi-scale visual features mapped in Step 2.1.1, the word-level text features mapped in Step 2.1.3, the fused visual features and visually guided text features obtained in Step 2.2 into corresponding embeddings using mean-squared processing, and calculate the cosine similarity between the multi-scale visual embedding and the word-level text embedding, and the cosine similarity between the fused visual embedding and the visually guided text embedding, respectively.

2. The remote sensing image and text retrieval method based on guided visual semantic alignment as described in claim 1, characterized in that, Step three includes the following steps: Step 3.1: The triplet ranking loss function is minimized by fusing visual embeddings. Visually Guided Text Embedding The distance in the latent semantic space achieves the final alignment between the image and the text; Step 3.2: The global semantic loss function minimizes the multi-scale visual embedding. and word-level text embedding This ensures the invariance of the modal input to the original semantics of the image and text, serving as an external constraint for text embedding that integrates visual embedding and visual guidance. Step 3.3: Combine the ternary ranking loss function and the global semantic loss function as the overall loss function for model training.

3. The remote sensing image and text retrieval method based on guided visual semantic alignment as described in claim 1, characterized in that, The process of step 2.2 is as follows: The goal-oriented attention module includes an intra-modal fusion attention module and an inter-modal guided attention module; the intra-modal fusion attention module processes the multi-scale visual features mapped in step 2.1.

1. And the salient target features mapped in step 2.1.2 To achieve integration, firstly... and After passing through a fully connected layer, it is obtained and Then, the joint score of the two features is calculated using matrix multiplication, and then the two features are activated separately. The formula is expressed as: ； ； ； in This represents the joint score of multi-scale visual features and salient object features. and This represents the aggregated features, ultimately resulting in the fused visual features. The formula is expressed as: ； in and Represents the weights and biases of the linear transformation. This indicates splicing by channel; The intermodal guided attention module utilizes the salient target features mapped in step 2.1.

2. To guide the mapping of word-level text features in step 2.1.3 The expression, first of all, is and Preprocessing is performed, and the mean is calculated separately. The formula is as follows: ； ； Then to and After passing through a fully connected layer, it is obtained and Then, the joint score of the two features is calculated, and then the text features are activated. The formula is expressed as: ； ； in This represents the joint score of salient target features and word-level text features. The aggregated features are represented, and finally, the visually guided text features are obtained. The formula is expressed as: ； in This represents a multilayer perceptron.

4. The remote sensing image and text retrieval method based on guided visual semantic alignment as described in claim 2, characterized in that, In step 3.3, the overall loss function is expressed as: ； ； in Indicates edge parameters, , and Represents the smallest batch of samples in remote sensing images and text. and The most negative sample in the sample. This indicates that step 2.3 calculates the cosine similarity. This represents the loss function for sorting triples. Represents the global semantic loss function. This represents the weighting factor of the global visual semantic loss function.

5. A remote sensing image and text retrieval method based on guided visual semantic alignment as described in any one of claims 1 to 3, characterized in that, Step one includes the following sub-steps: Step 1.1: Preprocessing of remote sensing images; dividing the image data into training, validation and test sets; performing data augmentation on the training and validation data, including scaling, random cropping, random flipping and normalization. Step 1.2: Preprocessing of text data; each remote sensing image corresponds to five text descriptions; these texts are processed by text segmentation, stop word removal and one-hot encoding to obtain the corresponding word vectors; then these word vectors are recombined to obtain sentence vectors.

Images