A remote sensing image-text retrieval method based on information processing

Abstract

A remote sensing image-text retrieval method based on information processing, including pretreatment before modal input, building of image-text retrieval model and representation alignment, wherein the building of image-text retrieval model includes designing target information enhancement module and noise information compression module, the target information enhancement module aims to fuse multiple scale features into visual features through detail multi-information by a detail guidance information attention module to enrich the detail expression of the image, and the noise information compression module compresses the image-text information to filter redundant information.The present application solves the problems of insufficient remote sensing image-text information capture and excessive attention to redundant information in existing remote sensing image-text retrieval methods, and improves the accuracy of remote sensing image-text retrieval.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing, and particularly relates to a remote sensing image-text retrieval method based on information processing. BACKGROUND

[0002] Remote sensing image-text retrieval is a key information retrieval technology, aiming to associate remote sensing images with related text information, and retrieve matching image blocks and text content therefrom. The core goal of this technology is to efficiently and accurately retrieve results matching the query text or image block from a large remote sensing image database (such as data collected by satellites or unmanned aerial vehicles). It has important application value in resource investigation, disaster monitoring, agricultural production, etc.

[0003] Compared with natural images, remote sensing images usually contain multiple targets and a large amount of background or irrelevant information for retrieval. This complexity makes it difficult to accurately capture key information or filter redundant content, resulting in reduced accuracy of image-text alignment, and thus significantly affecting retrieval efficiency.

[0004] Current remote sensing image-text retrieval methods have some limitations in information capture and processing: some methods pay too much attention to local features, ignoring the global relationship between targets, resulting in incomplete overall information representation; other methods focus on global information alignment, but fail to fully exploit the diversity of local details. In addition, some methods attempt to reduce information redundancy, but still struggle to effectively filter irrelevant content. Although the alignment mechanism and multi-modal fusion technology improve the retrieval performance to some extent, many methods still fail to completely eliminate the interference of irrelevant features, resulting in the persistence of irrelevant information, thus limiting the further improvement of overall retrieval effect. SUMMARY

[0005] In order to overcome the deficiencies of the prior art in processing remote sensing image-text information, the present application proposes a remote sensing image-text retrieval method based on information processing, which solves the problems of insufficient remote sensing image-text information capture and excessive attention to redundant information in remote sensing image-text retrieval.

[0006] In order to achieve the above-mentioned purposes, the technical solutions of the present application are as follows:

[0007] A remote sensing image-text retrieval method based on information processing, comprising the following steps:

[0008] Step 1, remote sensing image-text preprocessing, pre-processing the image and text input of the remote sensing image-text data set;

[0009] Step 2, building a remote sensing image-text retrieval model, including constructing image-text pre-encoding, constructing a target information enhancement module and a noise information compression module, and obtaining the final visual and text embedding features;

[0010] Step three, alignment representation, including similarity measure and subspace representation; calculate the cosine similarity matrix of visual and text modal features, design a contrast loss function, design an overall loss function, and realize the alignment of image and text by minimizing the overall loss function.

[0011] Further, the step two includes the following sub-steps:

[0012] Step 2.1: Constructing image-text pre-encoding; the image-text pre-encoding includes a visual encoder, a multi-information encoder and a text encoder; the process is as follows:

[0013] Step 2.1.1: using Swin Transformer network as visual encoder to extract visual global class features and local related features;

[0014] Step 2.1.2: using a pre-trained Bert model as a text encoder to extract text global class features and local related features;

[0015] Step 2.1.3: using a pre-trained ResNet50 network on AID dataset as a multi-information encoder to extract multi-scale feature vectors by taking advantage of the characteristics of different levels of receptive field of convolutional neural network, so as to capture more details of remote sensing image and enhance the visual representation ability;

[0016] Step 2.2: Constructing target information enhancement module; input the multi-scale features extracted in step 2.1.3 into the designed detail guiding information attention module to fuse the detail information into the visual features to enrich the detail expression of the image; at the same time, activate the text features through multi-layer perceptron to enhance the expression richness of the text;

[0017] Step 2.3: Constructing noise information compression module; compress the image features and text features generated in step 2.2 to obtain compressed visual features and text features; then, add the respective global class features to generate visual embedding features and text embedding features, and impose constraints on the feature compression process through alignment information bottleneck function to further optimize the feature expression;

[0018] Further, the step three includes the following sub-steps:

[0019] Step 3.1: Designing a contrast loss function; calculate the cosine similarity of the corresponding final visual embedding features and final text embedding features to obtain the contrast loss from visual to text and the contrast loss from text to visual, thereby obtaining the contrast loss;

[0020] Step 3.2: Combining the contrast loss function and the alignment information bottleneck function as the overall loss function for model training.

[0021] Further, the step one includes the following sub-steps:

[0022] Step 1.1: preprocessing of remote sensing images; the image data is preprocessed and divided into a training set, a validation set and a test set; data augmentation operations are performed on the training set and the validation set, including scaling, random cropping, random flipping and standardization processing, to improve the robustness and generalization ability of the model;

[0023] Step 1.2: preprocessing of text data; each remote sensing image corresponds to five sentences of text description. First, add a special mark [CLS] at the beginning of each text to identify the start of the sentence, and add a mark [SEP] at the end of the sentence to identify the end of the sentence; then, a word vector table is constructed, and each word is converted into a one-dimensional vector for subsequent processing.

[0024] The beneficial effects of the present application are: to solve the problems of insufficient remote sensing image information capture and excessive attention to redundant information in the existing remote sensing image retrieval method, and to improve the accuracy of remote sensing image retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of the image retrieval method of the present application.

[0026] Figure 2 is a schematic diagram of the image retrieval network framework of the present application.

[0027] Figure 3 is a schematic diagram of the detail guidance information attention module of the present application.

[0028] Figure 4 is a schematic diagram of the noise information compression module of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0030] Referring to Figures 1-4 , a remote sensing image retrieval method based on information processing includes the following steps:

[0031] Step one, remote sensing image preprocessing, pre-processing the image and text input of the remote sensing image-text data set;

[0032] The step one includes the following sub-steps:

[0033] Step 1.1: preprocessing of remote sensing images; preprocessing of image data, dividing it into training set, validation set and test set; data augmentation operation is performed on the training set and validation set, including scaling, random cropping, random flipping and standardization processing, to improve the robustness and generalization ability of the model;

[0034] Step 1.2: preprocessing of text data; each remote sensing image corresponds to five sentences of text description. First, add special mark [CLS] at the beginning of each text to identify the start of the sentence, and add mark [SEP] at the end of the sentence to identify the end of the sentence; then, build a word vector table to convert each word into a one-dimensional vector for subsequent processing.

[0035] Step two, building a remote sensing image-text retrieval model, including building image-text pre-encoding, building target information enhancement module and noise information compression module, and obtaining the final visual and text embedding features;

[0036] The step two includes the following sub-steps:

[0037] Step 2.1: build image-text pre-encoding; the image-text pre-encoding includes a visual encoder, a multi-information encoder and a text encoder;

[0038] The process of step 2.1 is as follows:

[0039] In step 2.1.1, first divide the input image into fixed-size image blocks, then use the Swin Transformer network to encode these image blocks to obtain visual global features and local related features expressed as:

[0040]

[0041] Where represents the visual encoder, is the fine-tuning weight, [·,·] represents stacking and connecting in the sequence length dimension, and m is the number of local related features;

[0042] In step 2.1.2, a pre-trained Bert is used as a text encoder to encode the text T, thereby obtaining the text global category feature and the text local feature expressed as:

[0043]

[0044] Where represents the text encoder, This refers to fine-tuning the weights, where n is the number of locally relevant features;

[0045] In step 2.1.3, convolutional layers of different depths were selected from the pre-trained ResNet50 model to obtain local detail multi-information C for different receptive fields. l :

[0046] C l =(Layer) l (I l-1 (3)

[0047] Layer l (·) represents the convolutional layers of ResNet50, a total of four convolutional layers, I l-1 This represents the output of layer l-1. Then, through a designed detail-guided information attention module, multi-detail information is incorporated into visual features to enrich the image's detail expression. Specifically, firstly, local multi-detail information C... l C is processed through pooling (max pooling for the first three layers to capture prominent low-level features, and average pooling for the last layer, as it contains more abstract and global information) and reshaping operations (including linear mapping and expansion operations). l Reconstructed into locally relevant features F v The shape, thereby obtaining pre-fused features. The formula is expressed as:

[0048]

[0049] Pool(·) represents the pooling operation, and Reshape(·) represents the reshaping operation.

[0050] Step 2.2: The extracted features at multiple scales are incorporated into the visual features through a designed detail-guided information attention module to enrich the image's detail expression; for remote sensing text, text features are activated through a multilayer perceptron to obtain richer expression; such as... Figure 3 As shown, to construct a detail-guided information attention module, firstly, image features are activated, and the input F is... v The linear mapping is represented by three vectors: query Q i Key K i Sum V i The formula is expressed as:

[0051] Q i =F v W i Q ,K i =F v W i K Vi = F v W i V (5)

[0052] where W i Q W i K W i V are trainable parameters, and i denotes the i-th attention head. Then, the activated image features in the l-th information block are obtained as The formula is expressed as:

[0053]

[0054] where where Softmax(·) is the Softmax function, h is the number of attention heads, and D is the dimension of the image feature F v . Next, the pre-fused feature is fused with the image feature , and the final fused feature v l is obtained, which is expressed by the formula

[0055] v l = [v l,1 , v l,2 ,..., v l,h ], (7)

[0056] where where W j Q W j Q W j V are learnable parameters.

[0057] Finally, the features in the four information fusion blocks are selectively summed to obtain the final image visual feature v f , which is expressed by the formula:

[0058]

[0059] where a l ∈ {0, 1}, which determines which layer of image fused features is added to the final image visual feature.

[0060] In order to enhance the features of the text, a multi-layer perceptron is used to extract more rich text information to obtain the final text feature t f , which is expressed by the formula:

[0061] t f = MLP(Ft ). (9)

[0062] Step 2.3: As Figure 4 constructing the noise information compression module; for image feature v f , using the masking method to reduce redundant information, limiting the information flow of image feature v f to compressed feature v e by a simple parameter encoder, and using the reparameterization trick to get v e , which is specifically expressed as:

[0063] v e = λ v v f + (1- λ v ) ∈ v , (10)

[0064] where is a Gaussian distribution with the same mean and variance as v f , λ v ∈(0,1), which can be obtained by a learnable parameter α v through the Sigmoid function, which is expressed as:

[0065] λ v = Sigmoid(α v ), (11)

[0066] Text compression features can be obtained by the same method. Then v cls and t cls are added to v e and t e respectively to avoid losing these important information in the compression process, obtaining the final embedding features and After that, according to the target relationship, the alignment information bottleneck function, namely the visual information bottleneck function and the text information bottleneck function, is designed, which is expressed as:

[0067]

[0068] where θ v and θ t represent the parameters required by the visual information bottleneck and the text information bottleneck respectively, β v and β t represent the compression coefficients, used to balance the compression term I(v e ,v f ; θ v ) and the fitting term I(·,·; θ) represents mutual information.

[0069] To compute the visual information bottleneck function, first compute the compression term I(v e ,v f ; θ v ):

[0070]

[0071] where D KL (·||·) denotes the KL divergence. But since the marginal distribution p(v e ; θ v ) is difficult to compute, we approximate the constraint by a Gaussian distribution where I J is the J x J identity matrix, J is the dimension of v e , and we estimate an upper bound on the compression term, given by:

[0072]

[0073] Then compute the KL divergence, given by:

[0074]

[0075] where σ and μ denote the mean and variance of p(v e |v f ; θ v ), which are obtained by sampling.

[0076] In addition, compute the visual information bottleneck fit term given by:

[0077]

[0078] Since is difficult to compute, we approximate the constraint using where g v (·) denotes a mapping used to align and , and D is the dimension of . We thus approximate a lower bound on the fit term given by:

[0079]

[0080] We then further optimize the objective by sampling from the empirical distribution Finally, normalize the embedding, and for the normalized and the log of the Gaussian probability density can be simplified to and The cosine similarity between them is proportional, so the final optimization target is obtained:

[0081]

[0082] Similarly, the text information bottleneck function can be obtained

[0083]

[0084] Finally, the alignment information bottleneck function is obtained, and the formula is:

[0085]

[0086] Step three, representation alignment, including similarity measurement and subspace representation; calculate the cosine similarity matrix of visual and text modal features, design a contrast loss function, design a total loss function, and realize the alignment of images and texts by minimizing the total loss function;

[0087] The step three includes the following sub-steps:

[0088] Step 3.1: Design a contrast loss function; given a small batch sample, randomly sample N positive pairs {(I1, T1), (I2, T2),..., (IN, TN)}, and their corresponding final visual embedding features and final text embedding features are i , i , N , N , Then calculate the cosine similarity So the contrast loss from visual to text and the contrast loss from text to visual are obtained, and the formula is:

[0089]

[0090] Where τ is the temperature parameter, and the final contrast loss is obtained, and the formula is:

[0091]

[0092] In step 3.2, the contrast loss function and the alignment information bottleneck function are combined as the total loss function for model training, and the formula is:

[0093]

[0094] Where λ is a coefficient for balancing the proportion of alignment information bottleneck loss in the total loss function.

[0095] The embodiment also provides a remote sensing image-text retrieval system based on information processing, which includes the following modules:

[0096] Preprocessing before modal input, preprocessing the image and text of the remote sensing image-text dataset to obtain standard model input sample data;

[0097] Building of the image-text retrieval model, including constructing modal input, constructing target information enhancement module and noise information compression module, realizing extraction of visual and text modal detail features and screening of redundant features, and obtaining final visual and text embedding;

[0098] Representation alignment, including similarity measurement and subspace representation; calculating the cosine similarity matrix of visual and text modal features, designing a contrast loss function, designing an overall loss function, and realizing alignment of image and text by minimizing the overall loss function.

[0099] The above modules correspond to steps 1-3 of the method.

[0100] The above-described embodiments only express two embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for retrieving remote sensing documents based on information processing, characterized in that, The method comprises the following steps: Step one, remote sensing image-text preprocessing, pre-processing the image and text input of the remote sensing image-text data set; Step two, building a remote sensing image-text retrieval model, including building an image-text pre-encoding, building a target information enhancement module and a noise information compression module, and obtaining the final visual and text embedding features; Step three, representation alignment, including similarity measurement and subspace representation; calculate the cosine similarity matrix of visual and text modal features, design a contrast loss function, design a total loss function, and realize the alignment of image and text by minimizing the total loss function; Step two includes the following sub-steps: Step 2.1: build an image-text pre-encoding; the image-text pre-encoding includes a visual encoder, a multi-information encoder and a text encoder, the process is as follows: Step 2.1.1: use Swin Transformer network as visual encoder to extract visual global class features and local related features; Step 2.1.2: use a pre-trained Bert model as a text encoder to extract text global class features and local related features; Step 2.1.3: use a pre-trained ResNet50 network on AID dataset as a multi-information encoder to extract multi-scale feature vectors by taking advantage of the characteristics of different levels of receptive field of convolutional neural network, so as to capture more details of remote sensing image and enhance the visual representation ability; Step 2.2: build a target information enhancement module; input the multi-scale features extracted in step 2.1.3 into the designed detail guidance information attention module to fuse the detail information into the visual features to enrich the detail expression of the image; at the same time, activate the text features by multilayer perceptron to improve the expression richness of the text; Step 2.3: build a noise information compression module; compress the image features and text features generated in step 2.2 to obtain compressed visual features and text features; then, add the respective global class features to them to generate visual embedding features and text embedding features, and impose constraints on the feature compression process through alignment information bottleneck function to further optimize the feature expression; In step 2.1.1, first divide the input image into fixed-size image blocks, then encode the image blocks using a Swin Transformer network to obtain visual global features and local relevant features. ​​​​ wherein denotes a visual encoder, is a fine-tuning weight, [·, ·] denotes stacking and concatenation in the sequence length dimension, and m is the number of local correlation features; In the step 2.1.2, a pre-trained Bert is used as a text encoder to encode the text T, so as to obtain the text global category feature and the text local feature which is expressed as: wherein denotes a text encoder, are fine-tuning weights, n is the number of local relevant features; The step 2.1.3, the convolutional layers of different depths are selected from the pre-trained ResNet50 model, to obtain the local detailed multi-information C of different receptive fields l : C l = (Layer l (I l-1 )), (3) Layer l (·) represents the convolutional layers of ResNet50, a total of four convolutional layers, I l-1 This represents the output of layer l-1; then, through a designed detail-guided information attention module, multi-detail information is incorporated into visual features to enrich the detail expression of the image. Specifically, firstly, local multi-detail information C... l C is processed through pooling and reshaping operations. l Reconstructed into locally relevant features F v The shape, thereby obtaining pre-fused features. The formula is expressed as: Where Pool(·) represents the pooling operation and Reshape(·) represents the reshaping operation; In step 2.2, the detail guidance information attention module is designed by the idea of the multi-head attention method to fuse feature information from different sources. First, the image features are activated, and the input F v Linear mapping is a three-vector: query Q i , key K i and value V i , which is expressed by the formula: where W i Q , are trainable parameters, i denotes the i-th attention head, and after that, we get the image features in the l-th information block that are activated The formula is represented as: wherein Softmax(·) is a Softmax function, h is the number of attention heads, D is the dimension of the image feature F v , and then the pre-fusion feature is fused with the image feature to obtain a final fusion feature v l , which is expressed by the formula: v l = [v l,1 , v l,2 ,..., v l,h ], (7) wherein W j Q , W j V are learnable parameters, and finally, the features in the four information fusion blocks are selectively summed to obtain the final image visual feature v f , which is expressed by the formula: where a l ∈ {0,1}, which determines which layer's image fusion features are added to the final image visual features; To enhance the features of the text, a multi-layer perceptron is used to extract more abundant text information to obtain the final text features t f In public representation as: t f = MLP(F t ) (9) The step 2.3, for image feature v f , uses a masking method to reduce the redundant information, limits the information flow of image feature v f to compressed feature v e by a simple parameter encoder, and uses a reparameterization trick to get v e , which is specifically expressed by the formula: v e = λ v v f + (1 - λ v ) ∈ v , (10) where i.e. with v f Gaussian distribution with the same mean and variance, λ v ∈(0, 1), can be obtained by learning a parameter a v through a sigmoid function, which is expressed as: λ v = Sigmoid(a v ), (11) Text compression features can be obtained by the same method, then v cls and t cls are added to v e and t e respectively to avoid losing these important information in the compression process, the final embedding features and 2. The remote sensing document retrieval method based on information processing according to claim 1, wherein, In step 2.3, according to the target relationship, the alignment information bottleneck function, i.e. visual information bottleneck function and text information bottleneck function, is designed, and the formula is as follows: where θ v and θ t denote parameters used in visual information bottleneck and text information bottleneck, respectively, β v and β t denote compression coefficients, used to balance the compression term I(v e ,v f ; θ v ) and the fitting term I(·,·; θ) denotes mutual information. To compute the visual information bottleneck function, first compute the compression term I(v e ,v f ; θ v ): where D KL (·||·) denotes the KL divergence, but since the marginal distribution p(v e ; θ v ) is difficult to compute, it is approximated by a Gaussian distribution with I J the J-dimensional identity matrix, J being the dimension of v e , and the upper bound of the compression term is estimated as Then calculate the KL divergence, and the formula is as follows: where σ and μ represent the mean and variance of p(v e v f ; θ v ) obtained by sampling; Further, the visual information bottleneck fit term is calculated The formula is expressed as: Due to difficult to compute, one uses to approximate the constraint, where g v (·) denotes the mapping for aligning and D is the dimension of , so that the lower bound of the fitting term is approximated, and the formula is expressed as: Then the embedding is further optimized by sampling from the empirical distribution further optimizing the objective, and finally, the embedding is normalized, and for the normalized and the log of the Gaussian probability density can be simplified as and the cosine similarity between and is proportional to, thus, the final optimization objective is obtained: Similarly, the text information bottleneck function can be obtained Finally, the alignment information bottleneck function is obtained, and the formula is as follows:

3. The remote sensing document retrieval method based on information processing according to claim 1, wherein, Step one includes the following sub-steps: Step 1.1: preprocessing of remote sensing image; pre-process the image data, divide it into training set, validation set and test set; perform data augmentation operation on the training set and validation set, including scaling, random cropping, random flipping and standardization processing, to improve the robustness and generalization ability of the model; Step 1.2: Preprocessing of text data; five sentences of text description correspond to each remote sensing image, first, add special mark [CLS] at the beginning of each text to identify the start of the sentence, and add mark [SEP] at the end of the sentence to identify the end of the sentence; Subsequently, a word vector table is constructed, and each word is converted into a one-dimensional vector for subsequent processing.

4. The remote sensing document retrieval method based on information processing according to claim 1, wherein, The step three includes the following sub-steps: Step 3.1: Designing a contrast loss function; calculating the cosine similarity of the corresponding final visual embedding features and final text embedding features to obtain the visual-to-text contrast loss and the text-to-visual contrast loss, thereby obtaining the contrast loss; Step 3.2: Combining the contrast loss function and the alignment information bottleneck function as the overall loss function for model training.

5. The remote sensing document retrieval method based on information processing according to claim 4, wherein, In the step 3.1, given a mini-batch, randomly sample N pairs {(I1, T1), (I2, T2),..., (IN, TN)} from the mini-batch, whose corresponding final visual embedding features and final text embedding features are i , i , N , N . Then calculate the cosine similarity Thus, the visual-to-text contrastive loss and the text-to-visual contrastive loss are obtained, which are expressed by the formula as follows: Where τ is the temperature parameter, and the final contrast loss is represented by the formula: In step 3.2, the contrast loss function and the alignment information bottleneck function are combined as the overall loss function for model training, which is represented by the formula: Where λ is a coefficient for balancing the proportion of the alignment information bottleneck loss in the overall loss function.

Images