Remote sensing image-text retrieval method based on information processing

Abstract

A remote sensing image-text retrieval method based on information processing comprises the steps of preprocessing before modal input, building of an image-text retrieval model and characterization alignment, and the step of building of the image-text retrieval model comprises designing a target information enhancement module and a noise information compression module. The target information enhancement module aims at features of multiple scales to integrate detail information into visual features through the detail guide information attention module so as to enrich detail expressions of images, and the noise information compression module compresses graphic and text information so as to filter redundant information. According to the remote sensing image-text retrieval method, the problems that in an existing remote sensing image-text retrieval method, remote sensing image-text information capture is insufficient, and redundant information is excessively concerned are solved, and the remote sensing image-text retrieval precision is improved.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing, and in particular relates to a remote sensing image and text retrieval method based on information processing. Background Art

[0002] Remote sensing image and text retrieval is a key information retrieval technology that aims to associate remote sensing images with relevant text information and retrieve matching image blocks and text content from them. The core goal of this technology is to efficiently and accurately retrieve results that match the query text or image blocks from large remote sensing image databases (such as data collected by satellites or drones). It has important application value in resource surveys, disaster monitoring, agricultural production and other fields.

[0003] Compared with natural images, remote sensing images usually contain multiple targets and a large amount of background or information irrelevant to retrieval. This complexity makes it impossible to accurately capture key information or filter redundant content, resulting in reduced accuracy of image-text alignment, which in turn significantly affects retrieval efficiency.

[0004] Current remote sensing image and text retrieval methods have some limitations in information capture and processing: some methods focus too much on local features and ignore the global relationship between targets, resulting in incomplete overall information representation; other methods focus on global information alignment but fail to fully tap the diversity of local details. In addition, some methods attempt to reduce information redundancy, but still find it difficult to effectively filter out irrelevant content. Although alignment mechanisms and multimodal fusion technologies have improved retrieval performance to a certain extent, many methods still fail to completely eliminate the interference of irrelevant features, resulting in the continued existence of irrelevant information, thus limiting the further improvement of the overall retrieval effect. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art in processing remote sensing image and text information, the present invention proposes a remote sensing image and text retrieval method based on information processing to solve the problems of insufficient capture of remote sensing image and text information and excessive attention to redundant information in remote sensing image and text retrieval.

[0006] In order to achieve the above-mentioned purpose, the technical solution of this application is as follows:

[0007] A remote sensing image and text retrieval method based on information processing comprises the following steps:

[0008] Step 1: Remote sensing image and text preprocessing: preprocessing the image and text input of the remote sensing image-text dataset;

[0009] Step 2: Building a remote sensing image and text retrieval model, including building image and text precoding, building a target information enhancement module and a noise information compression module, and obtaining the final visual and text embedding features;

[0010] Step 3: Representation alignment, including similarity measurement and subspace representation; calculate the cosine similarity matrix of visual and text modal features, design the contrast loss function, design the overall loss function, and achieve image and text alignment by minimizing the overall loss function.

[0011] Further, the step 2 includes the following sub-steps:

[0012] Step 2.1: construct a picture-text pre-coding; the picture-text pre-coding includes a visual encoder, a multi-information encoder and a text encoder; the process is as follows:

[0013] Step 2.1.1: Use the Swin Transformer network as a visual encoder to extract visual global category features and local related features;

[0014] Step 2.1.2: Use a pre-trained Bert model as a text encoder to extract global category features and local related features of the text;

[0015] Step 2.1.3: Use the ResNet50 network pre-trained on the AID dataset as a multi-information encoder, and use the characteristics of the receptive fields of different levels of the convolutional neural network to extract multi-scale feature vectors, thereby capturing more detailed features of remote sensing images and enhancing visual representation capabilities;

[0016] Step 2.2: Construct a target information enhancement module; input the multi-scale features extracted in step 2.1.3 into the designed detail-guided information attention module, fuse the detail information into the visual features to enrich the detail expression of the image; at the same time, activate the text features through the multi-layer perceptron to enhance the richness of the text expression;

[0017] Step 2.3: Construct a noise information compression module; compress the image features and text features generated in step 2.2 to obtain compressed visual features and text features; then, add their respective global category features to generate visual embedding features and text embedding features, and impose constraints on the feature compression process by aligning the information bottleneck function to further optimize the feature expression;

[0018] Furthermore, the step three includes the following sub-steps:

[0019] Step 3.1: Design a contrast loss function; calculate the cosine similarity of the corresponding final visual embedding features and the final text embedding features to obtain the visual-to-text contrast loss and the text-to-visual contrast loss, thereby obtaining the contrast loss;

[0020] Step 3.2: Combine the contrast loss function and the alignment information bottleneck function as the overall loss function for model training.

[0021] Furthermore, the step 1 includes the following sub-steps:

[0022] Step 1.1: Preprocessing of remote sensing images: Preprocess the image data and divide it into training set, validation set and test set; perform data augmentation operations on the training set and validation set, including scaling, random cropping, random flipping and standardization, to improve the robustness and generalization ability of the model;

[0023] Step 1.2: Preprocessing of text data; Each remote sensing image corresponds to five sentences of text description. First, add a special marker [CLS] at the beginning of each sentence to mark the beginning of the sentence, and add a marker [SEP] at the end of the sentence to mark the end of the sentence; then, construct a word vector table and convert each word into a one-dimensional vector for subsequent processing.

[0024] The beneficial effects of the present invention are: solving the problems of insufficient capture of remote sensing image and text information and excessive focus on redundant information in existing remote sensing image and text retrieval methods, and improving the accuracy of remote sensing image and text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the image-text retrieval method of the present invention.

[0026] Figure 2 It is a schematic diagram of the image-text retrieval network framework of the present invention.

[0027] Figure 3 It is a schematic diagram of the detail-guiding information attention module of the present invention.

[0028] Figure 4 It is a schematic diagram of the noise information compression module of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] Reference Figure 1 to Figure 4 , a remote sensing image and text retrieval method based on information processing, comprising the following steps:

[0031] Step 1: Remote sensing image and text preprocessing: preprocessing the image and text input of the remote sensing image-text dataset;

[0032] The step 1 includes the following sub-steps:

[0033] Step 1.1: Preprocessing of remote sensing images: Preprocess the image data and divide it into training set, validation set and test set; perform data augmentation operations on the training set and validation set, including scaling, random cropping, random flipping and standardization, to improve the robustness and generalization ability of the model;

[0034] Step 1.2: Preprocessing of text data; Each remote sensing image corresponds to five sentences of text description. First, add a special marker [CLS] at the beginning of each sentence to mark the beginning of the sentence, and add a marker [SEP] at the end of the sentence to mark the end of the sentence; then, construct a word vector table and convert each word into a one-dimensional vector for subsequent processing.

[0035] Step 2: Building a remote sensing image and text retrieval model, including building image and text precoding, building a target information enhancement module and a noise information compression module, and obtaining the final visual and text embedding features;

[0036] The step 2 includes the following sub-steps:

[0037] Step 2.1: constructing a picture-text pre-coding; the picture-text pre-coding includes a visual encoder, a multi-information encoder and a text encoder;

[0038] The process of step 2.1 is as follows:

[0039] In step 2.1.1, first input the image Divide into The image blocks are then encoded using the Swin Transformer network to obtain the global visual features. and local correlation features It is expressed as:

[0040]

[0041] in represents the visual encoder, is the fine-tuning weight, [·,·] indicates stacking and concatenation in the sequence length dimension, and m is the number of local correlation features;

[0042] In step 2.1.2, a pre-trained Bert is used as a text encoder to encode the text T, thereby obtaining the global category feature of the text and local features of text It is expressed as:

[0043]

[0044] in represents a text encoder, is the fine-tuning weight, n is the number of locally relevant features;

[0045] In step 2.1.3, convolutional layers of different depths are selected from the pre-trained ResNet50 model to obtain local detail multi-information C of different receptive fields. l :

[0046] C l =(Layer l (I l-1 )), (3)

[0047] Layer l (·) represents the convolutional layers of ResNet50, a total of four convolutional layers, I l-1 represents the output of layer l-1. Then, the detail-guided information attention module is designed to integrate the detail information into the visual features to enrich the detail expression of the image. Specifically, the local detail information C is first l C is transformed into l Reshape into local correlation feature F v shape, thereby obtaining the pre-fusion feature The formula is:

[0048]

[0049] Where Pool(·) represents the pooling operation, and Reshape(·) represents the reshaping operation.

[0050] Step 2.2: The extracted features at multiple scales are integrated into the visual features through the designed detail-guided information attention module to enrich the detailed expression of the image; for remote sensing text, the text features are activated through a multi-layer perceptron to obtain rich expression; Figure 3 As shown in the figure, we construct a detail-guided information attention module. First, we activate the image features and input F v Linear mapping to three vectors: query Q i , key K i Sum value V i , the formula is:

[0051] Q i =F v W i Q ,K i =F v W i K ,Vi =F v W i V (5)

[0052] Where W i Q ,W i K ,W i V is a trainable parameter, i represents the i-th attention head. Then, we get the image features activated in the l-th information block. The formula is:

[0053]

[0054] in Where Softmax(·) is the Softmax function, h is the number of attention heads, and D is the image feature F v Next, the pre-fused features With image features Fusion, finally get the fusion feature v l , the formula is expressed as

[0055] v l =[v l,1 ,v l,2 ,...,v l,h ], (7)

[0056] in Where W j Q , W j Q , W j V are learnable parameters.

[0057] Finally, the features in the four information fusion blocks are selectively summed to obtain the final image visual feature v f , expressed as:

[0058]

[0059] where α l ∈{0,1}, which determines which layer of image fusion features are added to the final image visual features.

[0060] In order to enhance the features of the text, a multi-layer perceptron is used to extract richer text information to obtain the final text features t f , which is expressed as:

[0061] t f =MLP(Ft ). (9)

[0062] Step 2.3: If Figure 4 Construct a noise information compression module; for image feature v f ,Use ,masking method to reduce redundant information and restrict the image features v through a simple parameter encoder f To compress the feature v e information flow, and use the reparameterization technique to obtain v e , which is specifically expressed by the formula:

[0063] v e =λ v v f +(1-λ v )∈ v , (10)

[0064] in That is, with v f Gaussian distribution with the same mean and variance, λ v ∈(0,1), can be obtained by a learnable parameter α v It is obtained through the Sigmoid function, and the formula is expressed as:

[0065] λ v =Sigmoid(α v ), (11)

[0066] The text compression feature can be obtained by the same method. Then v cls and t cls Add to v respectively e and t e In order to avoid losing these important information during the compression process, the final embedded features are obtained and Then, according to the target relationship, the alignment information bottleneck function is designed, namely the visual information bottleneck function and the text information bottleneck function, and the formula is expressed as:

[0067]

[0068] where θ v and θ t Represent the parameters required for the visual information bottleneck and the text information bottleneck, β v and β t represents the compression factor, which is used to balance the compression term I(v e ,v f θ v ) and the fitting term I(·,·;θ) denotes mutual information.

[0069] To compute the visual information bottleneck function, first compute the compression term I(v e , v f ; θ v ):

[0070]

[0071] where D KL (·||·) represents the KL divergence. However, since the marginal distribution p(v e ; θ v ) is difficult to compute, it is approximated by a Gaussian distribution , where I J is the J-dimensional identity matrix and J is the dimension of v e . Thus, an upper bound of this compression term is estimated and expressed by the formula:

[0072]

[0073] Then compute the KL divergence, expressed by the formula:

[0074]

[0075] where σ and μ represent the mean and variance of p(v e |v f ; θ v ), which are obtained by sampling.

[0076] In addition, compute the visual information bottleneck fitting term expressed by the formula:

[0077]

[0078] Since is difficult to compute, is used for approximate constraint, where g v (·) represents the mapping used to align and , and D is the dimension of . Thus, a lower bound of the fitting term is approximately obtained and expressed by the formula:

[0079]

[0080] Then further optimize the objective by sampling from the empirical distribution . Finally, normalize the embedding. For the normalized and , the logarithm of the Gaussian probability density can be simplified to and The cosine similarity between them is proportional, so the final optimization goal is obtained:

[0081]

[0082] Similarly, we can get the text information bottleneck function

[0083]

[0084] Finally, the alignment information bottleneck function is obtained, and the formula is expressed as:

[0085]

[0086] Step 3: Representation alignment, including similarity measurement and subspace representation; calculate the cosine similarity matrix of visual and text modality features, design the contrast loss function, design the overall loss function, and achieve image and text alignment by minimizing the overall loss function;

[0087] The step three includes the following sub-steps:

[0088] Step 3.1: Design contrast loss function; given a small batch of samples, randomly sample N positive pairs {(I1,T1),(I2,T2),...,(I i ,T i ),...,(I N ,T N )}, and its corresponding final visual embedding features and final text embedding features are Then calculate the cosine similarity So we get the contrast loss from vision to text and the contrast loss from text to vision, which can be expressed as:

[0089]

[0090] Where τ is the temperature parameter, and the final contrast loss is expressed as:

[0091]

[0092] In step 3.2, the contrast loss function and the alignment information bottleneck function are combined as the overall loss function for model training, which is expressed as follows:

[0093]

[0094] Where λ is a coefficient used to weigh the proportion of alignment information bottleneck loss in the overall loss function.

[0095] This embodiment also provides a remote sensing image and text retrieval system based on information processing, including the following modules:

[0096] Preprocessing before modal input: preprocess the images and texts of the remote sensing image-text dataset to obtain standard model input sample data;

[0097] The construction of the image-text retrieval model includes constructing the modal input, building the target information enhancement module and the noise information compression module, extracting the visual and text modal detail features and filtering out the redundant features, and obtaining the final visual and text embedding;

[0098] Representation alignment, including similarity measurement and subspace representation; calculate the cosine similarity matrix of visual and text modality features, design contrast loss function, design overall loss function, and achieve image and text alignment by minimizing the overall loss function.

[0099] The above modules correspond to steps 1 to 3 of the method.

[0100] The above-mentioned embodiments only express two implementation methods of the present application, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. A remote sensing image and text retrieval method based on information processing, characterized in that: The method comprises the following steps: Step 1: Remote sensing image and text preprocessing: preprocessing the image and text input of the remote sensing image-text dataset; Step 2: Building a remote sensing image and text retrieval model, including building image and text precoding, building a target information enhancement module and a noise information compression module, and obtaining the final visual and text embedding features; Step 3: Representation alignment, including similarity measurement and subspace representation; calculate the cosine similarity matrix of visual and text modal features, design the contrast loss function, design the overall loss function, and achieve image and text alignment by minimizing the overall loss function.

2. A remote sensing image and text retrieval method based on information processing as claimed in claim 1, characterized in that: The step 2 includes the following sub-steps: Step 2.1: Construct image-text precoding; the image-text precoding includes a visual encoder, a multi-information encoder and a text encoder, and the process is as follows: Step 2.1.1: Use the Swin Transformer network as a visual encoder to extract visual global category features and local related features; Step 2.1.2: Use a pre-trained Bert model as a text encoder to extract global category features and local related features of the text; Step 2.1.3: Use the ResNet50 network pre-trained on the AID dataset as a multi-information encoder, and use the characteristics of the receptive fields of different levels of the convolutional neural network to extract multi-scale feature vectors, thereby capturing more detailed features of remote sensing images and enhancing visual representation capabilities; Step 2.2: Construct a target information enhancement module; input the multi-scale features extracted in step 2.1.3 into the designed detail-guided information attention module, fuse the detail information into the visual features to enrich the detail expression of the image; at the same time, activate the text features through the multi-layer perceptron to enhance the richness of the text expression; Step 2.3: Construct a noise information compression module; compress the image features and text features generated in step 2.2 to obtain compressed visual features and text features; then, add their respective global category features to generate visual embedding features and text embedding features, and impose constraints on the feature compression process by aligning the information bottleneck function to further optimize the feature expression.

3. A remote sensing image and text retrieval method based on information processing as claimed in claim 2, characterized in that: In step 2.1.1, first input the image Divide into The image blocks are then encoded using the SwinTransformer network to obtain the global visual features. and local correlation features It is expressed as: in represents the visual encoder, are the fine-tuning weights, [·,·] denotes stacking and concatenation in the sequence length dimension, and m is the number of locally relevant features.

4. A remote sensing image and text retrieval method based on information processing as claimed in claim 2 or 3, characterized in that: In step 2.1.2, a pre-trained Bert is used as a text encoder to encode the text T, thereby obtaining the global category feature of the text and local features of text It is expressed as: in represents a text encoder, is the fine-tuning weight and n is the number of locally relevant features.

5. A remote sensing image and text retrieval method based on information processing as claimed in claim 2 or 3, characterized in that: In step 2.1.3, convolutional layers of different depths are selected from the pre-trained ResNet50 model to obtain local detail multi-information C of different receptive fields. l : C l =(Layer l (I l-1 )), (3) Layer l (·) represents the convolutional layers of ResNet50, a total of four convolutional layers, I l-1 represents the output of layer l-1; then, the detail-guided information attention module is designed to integrate the detail information into the visual features to enrich the detail expression of the image. Specifically, the local detail information C l Through pooling and reshaping operations, C l Reshape into local correlation feature F v shape, thereby obtaining the pre-fusion feature The formula is: Where Pool(·) represents the pooling operation, and Reshape(·) represents the reshaping operation.

6. A remote sensing image and text retrieval method based on information processing as claimed in claim 2 or 3, characterized in that: In step 2.2, the idea of ​​multi-head attention method is used to design details to guide the information attention module to fuse feature information from different sources. First, the image features are activated and the input F v Linear mapping to three vectors: query Q i , key K i Sum value V i , the formula is: Where W i Q , is a trainable parameter, i represents the i-th attention head, and then the image features activated in the l-th information block are obtained. The formula is: in Softmax(·) is the Softmax function, h is the number of attention heads, and D is the image feature F v Next, the pre-fusion feature With image features Fusion, finally get the fusion feature v l , the formula is: v l =[v l,1 ,v l,2 ,...,v l,h ], (7) in W j Q , W j Q , W j V is a learnable parameter. Finally, the features in the four information fusion blocks are selectively summed to obtain the final image visual feature v f , expressed as: where α l ∈{0,1}, which determines which layer of image fusion features are added to the final image visual features; In order to enhance the features of the text, a multi-layer perceptron is used to extract richer text information to obtain the final text features t f , which is expressed as: t f =MLP(F t ) (9)。 7. A remote sensing image and text retrieval method based on information processing as claimed in claim 2 or 3, characterized in that: In step 2.3, for the image feature v f , using masking methods to reduce redundant information and restricting image features v through a simple parameter encoder f To compress the feature v e information flow, and use the reparameterization technique to obtain v e , which is specifically expressed by the formula: v e =λ v v f +(1-l v )∈ v , (10) in That is, with v f Gaussian distribution with the same mean and variance, λ v ∈(0,1), can be obtained by a learnable parameter α v It is obtained through the Sigmoid function, and the formula is expressed as: l v =Sigmoid(α v ), (11) The text compression feature can be obtained by the same method, and then v cls and t cls Add to v respectively e and t e In order to avoid losing these important information during the compression process, the final embedded features are obtained and Then, according to the target relationship, the alignment information bottleneck function is designed, namely the visual information bottleneck function and the text information bottleneck function, and the formula is expressed as: where θ v and θ t Represent the parameters required for the visual information bottleneck and the text information bottleneck, β v and β t represents the compression factor, which is used to balance the compression term I(v e ,v f θ v ) and the fitting term I(·,·;θ) represents mutual information; To calculate the visual information bottleneck function, we first calculate the compression term I(v e ,v f θ v ): Where D KL (·||·) represents the KL divergence, but due to the marginal distribution p(v e θ v ) is difficult to calculate, so we use Gaussian distribution To approximate the constraint, where I J is the J-dimensional identity matrix, J is v e The dimension of , so the upper bound of the compression term is estimated, and the formula is expressed as: Then calculate the KL divergence, the formula is expressed as: where σ and μ represent the mean and variance of p(v e |v f ; θ v ), which are obtained by sampling; In addition, the computational visual information bottleneck fitting term The formula is: because It is difficult to calculate, so we use To make approximate constraints, where g v (·) indicates alignment and The mapping of The dimension of The lower bound of is expressed as: Then by using the empirical distribution Sampling further optimizes the target. Finally, the embedding is normalized. and The Gaussian probability density The logarithm of can be simplified to and The cosine similarity between them is proportional, so the final optimization goal is obtained: Similarly, we can get the text information bottleneck function Finally, the alignment information bottleneck function is obtained, and the formula is expressed as:

8. A remote sensing image and text retrieval method based on information processing as claimed in any one of claims 1 to 3, characterized in that: The step 1 includes the following sub-steps: Step 1.1: Preprocessing of remote sensing images: Preprocess the image data and divide it into training set, validation set and test set; perform data augmentation operations on the training set and validation set, including scaling, random cropping, random flipping and standardization, to improve the robustness and generalization ability of the model; Step 1.2: Preprocessing of text data; Each remote sensing image corresponds to five sentences of text description. First, a special marker [CLS] is added at the beginning of each sentence to mark the beginning of the sentence, and a marker [SEP] is added at the end of the sentence to mark the end of the sentence; Then, a word vector table is constructed to convert each word into a one-dimensional vector for subsequent processing.

9. A remote sensing image and text retrieval method based on information processing as claimed in any one of claims 1 to 3, characterized in that: The step three includes the following sub-steps: Step 3.1: Design a contrast loss function; calculate the cosine similarity of the corresponding final visual embedding features and the final text embedding features to obtain the visual-to-text contrast loss and the text-to-visual contrast loss, thereby obtaining the contrast loss; Step 3.2: Combine the contrast loss function and the alignment information bottleneck function as the overall loss function for model training.

10. A remote sensing image and text retrieval method based on information processing as claimed in claim 9, characterized in that: In step 3.1, given a small batch of samples, randomly sample N positive pairs {(I1,T1),(I2,T2),...,(I i ,T i ),...,(I N ,I N )}, and its corresponding final visual embedding features and final text embedding features are Then calculate the cosine similarity So we get the contrast loss from vision to text and the contrast loss from text to vision, which can be expressed as: Where τ is the temperature parameter, and the final contrast loss is expressed as: In step 3.2, the contrast loss function and the alignment information bottleneck function are combined as the overall loss function for model training, which is expressed as follows: Where λ is a coefficient used to weigh the proportion of alignment information bottleneck loss in the overall loss function.

Images