Remote sensing cross-modal image-text retrieval method and device

Abstract

The invention relates to the technical field of remote sensing, in particular to a remote sensing cross-modal image-text retrieval method and device, and the method comprises the steps: constructing a non-foreground restoration loss function and a prediction segmentation mask loss function through a remote sensing image and a real segmentation mask corresponding to the remote sensing image, and constructing a triple loss function through the remote sensing image and a text corresponding to the remote sensing image; and constructing a total loss function to train the remote sensing image-text retrieval model, so that the trained remote sensing image-text retrieval model retrieves the remote sensing image-text data. According to the method, the precision and efficiency of cross-modal remote sensing image-text retrieval are improved, and the prediction segmentation image of the remote sensing image is additionally output.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a remote sensing cross-modal image-text retrieval method and device. Background Art

[0002] With the rapid development of remote sensing satellite technology, remote sensing data has achieved a qualitative leap in both quantity and quality. How to quickly retrieve remote sensing data has become increasingly important. Among all retrieval forms, the two retrieval methods of "retrieving images by text" and "retrieving text by images" have become a classic and common information retrieval method in the remote sensing scenario due to their cross-modal data retrieval capabilities and efficient and convenient data retrieval characteristics, and are collectively referred to as the remote sensing cross-modal image-text retrieval task.

[0003] Currently, the remote sensing image-text cross-modal retrieval task mainly relies on deep neural networks to extract features from remote sensing images and text descriptions respectively, and then realizes the similarity matching between images and texts by constructing a unified feature embedding space. Common methods include using convolutional neural networks or Transformer models to extract image features, combining pre-trained language models to obtain text semantics, and then realizing cross-modal association through contrast learning, cross-attention or multi-modal alignment mechanisms. However, such methods often have poor effects when processing remote sensing images. For example, Patent CN117932101A discloses a cross-modal remote sensing image-text retrieval method based on single-modal feature modeling to achieve robust and effective cross-modal image-text retrieval performance. However, in terms of improving the accuracy and interpretability of cross-modal image-text retrieval, the following key problems still exist in this solution: (1) The existing data set lacks the corresponding relationship between entities in the text and targets in the image, and there will be problems of feature misalignment in remote sensing scenarios containing a large amount of noise, which further leads to feature misalignment problems in the similarity calculation process, affecting the further improvement of the model retrieval accuracy and lacking interpretability; (2) In addition to foreground targets with extremely extreme scale distributions in remote sensing scenarios, there are also more complex and extensive background information, which together participate in the encoding process of foreground targets, introducing interference from background noise and causing problems such as mottled features and poor robustness, affecting the retrieval performance of the model.

[0004] Therefore, the existing remote sensing image-text cross-modal retrieval task has problems of feature misalignment and mottled features, resulting in reduced retrieval accuracy. Summary of the Invention

[0005] For this reason, the technical problem to be solved by the present invention is to overcome the problem in the prior art that the accuracy of remote sensing image-text cross-modal retrieval is reduced due to feature misalignment and mottled features.

[0006] To solve the above technical problems, the present invention provides a remote sensing cross-modal image-text retrieval method, including: The remote sensing image and its corresponding text description are respectively passed through an image encoder and a text encoder to obtain image features and text features; A background mask image is obtained according to the remote sensing image and its corresponding true segmentation mask, and the background mask image is passed through an image encoder and a pixel restoration decoder to obtain a restored background region; a non-foreground restoration loss function is constructed based on the restored background region; The image features are passed through an image decoder to obtain a predicted segmentation mask; the image features are segmented region enhanced according to the predicted segmentation mask to obtain weighted features after segmented region enhancement; the weighted features after segmented region enhancement are passed through an image decoder to obtain a target predicted segmentation mask; a predicted segmentation mask loss function is constructed according to the target predicted segmentation mask and the true segmentation mask; The weighted features after segmented region enhancement are input into a hierarchical feature learning module, and each hierarchical feature of the weighted features after segmented region enhancement is sampled, mapped and weighted to obtain target image features; the similarity between the target image features and the text features is calculated to construct a triplet loss function; A total loss function is constructed with the non-foreground restoration loss function, the predicted segmentation mask loss function and the triplet loss function; Based on the total loss function, the image encoder, text encoder, pixel restoration decoder, hierarchical feature learning module and image decoder are trained; a remote sensing image-text retrieval model is constructed using the trained image encoder, text encoder, hierarchical feature learning module and image decoder to retrieve remote sensing image-text data.

[0007] Preferably, using the remote sensing image-text retrieval model to retrieve remote sensing image-text data includes: Inputting the query text into the remote sensing image-text retrieval model to output a predicted segmentation image of the remote sensing image matching the query text, including: The query text passes through the text encoder to obtain query text features; Several remote sensing images in the remote sensing image database pass through the image encoder of the remote sensing image-text retrieval model to obtain several image features; Each image feature passes through the hierarchical feature learning module to obtain its corresponding target image features; Calculate the similarity between the query text features and each target image feature, and obtain the remote sensing image matching the query text according to the similarity; Input the image features of the remote sensing image matching the query text into the image decoder to obtain a predicted segmentation mask of the remote sensing image matching the query text; Output a predicted segmentation image of the remote sensing image matching the query text according to the remote sensing image matching the query text and its predicted segmentation mask; Input the query remote sensing image into the remote sensing image-text retrieval model to output a predicted text matching the query remote sensing image, including: The query remote sensing image passes through an image encoder to obtain query image features; A number of texts in the text database pass through the text encoder of the remote sensing text and image retrieval model to obtain a number of text features; Calculate the similarity between the query image features and each text feature; Output the predicted text that matches the query remote sensing image according to the similarity.

[0008] Preferably, a background mask image is obtained according to the remote sensing image and its corresponding true segmentation mask, and the formula is: ; Wherein, represents the remote sensing image, represents the background mask image; represents the mask strategy adopted after reading the true segmentation mask corresponding to the remote sensing image, and is used to randomly retain a number of pixels in the image blocks masked by the true segmentation mask; represents the mask rate; represents the mask operation.

[0009] Preferably, a non-foreground restoration loss function is constructed based on the restored background region, and the formula is: ; Wherein, represents the non-foreground restoration loss function, represents the restored background region, represents the remote sensing image, represents the background mask image.

[0010] Preferably, the image features are strengthened in the segmentation region according to the predicted segmentation mask to obtain the weighted features after segmentation region strengthening, and the formula is: ; Wherein, represents the weighted features after segmentation region strengthening, represents the weighting operation, represents the normalization operator, represents the mapping operator, represents the downsampling operator; represents the image features, represents the predicted segmentation mask, represents the weighting factor.

[0011] Preferably, a predicted segmentation mask loss function is constructed according to the target predicted segmentation mask and the true segmentation mask, and the formula is: ; Wherein, represents the predicted segmentation mask loss function, represents the target prediction segmentation mask; represents the nth ground-truth segmentation mask block, represents the ground-truth segmentation mask; and are the height and width of the ground-truth segmentation mask, respectively.

[0012] Preferably, the weighted features after segmentation region enhancement are input into the hierarchical feature learning module, and each hierarchical feature of the weighted features after segmentation region enhancement is sampled, mapped, and weighted to obtain the target image features. The formula includes: ; ; ; where represents the lth hierarchical feature of the weighted features after segmentation region enhancement, represents the weighted features after segmentation region enhancement, is the total number of hierarchies; represents the sampling operator, represents the features after sampling compression, represents the lth hierarchical feature of the features after sampling compression; represents the mapping operator, represents the features after mapping, represents the lth hierarchical feature of the features after mapping; represents the weighting operator, represents the target image features, represents the lth hierarchical feature of the target image features.

[0013] Preferably, the similarity between the target image features and the text features is calculated to construct a triplet loss function, including: The similarity between the target image features and the text features is calculated using the classical Pearson correlation coefficient. The formula is: ; where represents the target image features, represents the text features, represents the similarity between the target image features and the text features; and represent the number of channels of the target image features and the text features, respectively; represents the kth component of the target image features, represents the jth component of the text features; The triplet loss function is constructed based on the similarity between the target image features and the text features. The formula is: ; Among them, represents the triplet loss function, represents the boundary value, and respectively represent the negative sample features of the target image feature and the text feature; is used to ensure that the result is non - negative.

[0014] Preferably, a total loss function is constructed with a non - foreground restoration loss function, a predicted segmentation mask loss function, and a triplet loss function. The formula is: ; Among them, represents the total loss function, represents the triplet loss function, represents the predicted segmentation mask loss function, represents the non - foreground restoration loss function, and respectively represent the weighting coefficients.

[0015] The present invention also provides a remote - sensing cross - modal image - text retrieval device, including: An encoding unit, configured to obtain an image feature and a text feature by respectively passing a remote - sensing image and its corresponding text description through an image encoder and a text encoder; A non - foreground restoration loss function construction unit, configured to obtain a background mask image according to a remote - sensing image and its corresponding true segmentation mask, and pass the background mask image through an image encoder and a pixel restoration decoder to obtain a restored background region; construct a non - foreground restoration loss function based on the restored background region; A predicted segmentation mask loss function construction unit, configured to obtain a predicted segmentation mask by passing an image feature through an image decoder; perform segmentation region enhancement on the image feature according to the predicted segmentation mask to obtain a weighted feature after segmentation region enhancement; obtain a target predicted segmentation mask by passing the weighted feature after segmentation region enhancement through an image decoder; construct a predicted segmentation mask loss function according to the target predicted segmentation mask and the true segmentation mask; A triplet loss function construction unit, configured to input the weighted feature after segmentation region enhancement into a hierarchical feature learning module, sample, map, and weight each hierarchical feature of the weighted feature after segmentation region enhancement to obtain a target image feature; calculate the similarity between the target image feature and the text feature, and construct a triplet loss function; A total loss function construction unit, configured to construct a total loss function with a non - foreground restoration loss function, a predicted segmentation mask loss function, and a triplet loss function; A retrieval unit is used to train an image encoder, a text encoder, a pixel restoration decoder, a hierarchical feature learning module, and an image decoder based on a total loss function; a remote sensing image-text retrieval model is constructed using the trained image encoder, text encoder, hierarchical feature learning module, and image decoder to retrieve remote sensing image-text data.

[0016] The above technical solution of the present invention has the following beneficial effects compared with the prior art: In the remote sensing cross-modal image-text retrieval method of the present invention, a true segmentation mask of a remote sensing image is introduced during the training process of the remote sensing image-text retrieval model. First, a background mask image is obtained using the true segmentation mask, and then the background mask image is encoded and restored by the image encoder and the pixel restoration decoder in sequence to obtain a restored background region, and a non-foreground restoration loss function is constructed based on this, guiding the image encoder of the remote sensing image-text retrieval model to learn the background distribution information of the remote sensing image, filtering out the noisy background in the image features during the retrieval process, improving the alignment degree between the named entities in the text description and the corresponding foreground targets in the remote sensing image, and enhancing the understanding ability of the remote sensing image-text retrieval model for cross-modal data; Secondly, the present invention introduces a semantic segmentation task of pixel-level prediction in the remote sensing image-text retrieval model. After obtaining the predicted segmentation mask using the image decoder, the image features are segmented regionally enhanced according to the predicted segmentation mask to obtain weighted features after segmented region enhancement. The weighted features after segmented region enhancement are passed through the image decoder again to obtain a target predicted segmentation mask, and a predicted segmentation mask loss function is constructed based on this, enabling the remote sensing image-text retrieval model to further output a predicted segmentation image of the remote sensing image when retrieving the remote sensing image with a query text, and the output predicted segmentation image has finer-grained foreground features, improving the accuracy of remote sensing image segmentation; Finally, the present invention uses the hierarchical feature learning module to learn the feature information of different sizes of the weighted features after segmented region enhancement to obtain target image features. During the training process, the similarity between the target image features and the text features is calculated to construct a triplet loss function, which can introduce more underlying texture information, enrich the scale diversity of the features, further improve the fault tolerance rate of feature alignment and the stability of image-text similarity comparison, effectively improve the learning efficiency of the remote sensing image-text retrieval model for foreground targets of different scales, and improve the accuracy of remote sensing image-text retrieval. Description of the Drawings

[0017] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to the specific embodiments of the present invention in combination with the drawings, where: Figure 1 is a flowchart of a remote sensing cross-modal image-text retrieval method of the present invention; Figure 2It is a schematic diagram of the result of remote sensing image segmentation by the remote sensing image-text retrieval model, where Figure 2 Columns (a) and (d) in Figure 2 are the input remote sensing images, Figure 2 Column (b) in Figure 2 is the true segmentation mask corresponding to the remote sensing image in column (a), Figure 2 Column (c) in Figure 3 is the predicted segmentation image corresponding to the remote sensing image in column (a); Figure 3 Column (a) in Figure 3 is the query text, Figure 4 is a schematic diagram of obtaining a predicted segmentation image by inputting the query text into the remote sensing image-text retrieval model; where Figure 4 Column (a) in Figure 4 is the query remote sensing image, Figure 5 is a curve graph of the total loss function changing with the value of the weighting coefficient, where Figure 5 In (a) of is the curve graph of the total loss function changing with the Figure 5 In (b) of is the curve graph of the total loss function changing with the Specific implementation mode

[0018] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments cited are not intended to limit the present invention.

[0019] Embodiment 1:

[0020] Referring to Figure 1 shown, the present invention provides a remote sensing cross-modal image-text retrieval method, including: S1: The remote sensing image and its corresponding text description are respectively passed through an image encoder and a text encoder to obtain an image feature and a text feature .

[0021] The formula for obtaining the image feature by passing the remote sensing image through the image encoder is ; The formula for obtaining text features from the text description through the text encoder is expressed as . Among them, represents the image encoder, represents the text encoder, represents text preprocessing, including cleaning and embedding the text description.

[0022] Preferably, the image encoder in this embodiment selects the Resnet series network model, and the text encoder selects the BiGRU or Transformer model.

[0023] S2: In the non-foreground restoration branch of the segmentation supervision region prediction module, according to the remote sensing image and its corresponding ground truth segmentation mask obtain the background mask image, and pass the background mask image through the image encoder and the background decoder to extract and restore the features of the background mask image respectively, obtaining the restored background region; construct the non-foreground restoration loss function based on the restored background region.

[0024] The non-foreground restoration branch of the segmentation supervision region prediction module is only used during the model training process.

[0025] Specifically, the formula for obtaining the background mask image according to the remote sensing image and its corresponding ground truth segmentation mask is: ; Among them, represents the remote sensing image, represents the background mask image; represents the mask strategy adopted after reading the ground truth segmentation mask corresponding to the remote sensing image, which is used to randomly retain several pixels in the image blocks masked by the ground truth segmentation mask; represents the mask rate, and its value range is from 0 to 1. A smaller value can improve the performance, while an excessive value will reduce the performance; represents the mask operation.

[0026] By obtaining the background mask image, the model can be guided to learn the feature representation of the background region during the training process, and the distinguishability between the background and the foreground can be improved.

[0027] Pass the background mask image through the image encoder to obtain the background mask image features; pass the background mask image features through the background decoder to obtain the restored background region.

[0028] Construct the non-foreground restoration loss function based on the restored background region, and the formula is: ; Among them, Denotes the non - foreground restoration loss function, Denotes the restored background region, Is the L1 norm.

[0029] During the model training process, the true segmentation mask of the remote sensing image is introduced. First, the background mask image is obtained by using the true segmentation mask, and then the background mask image is encoded and restored successively by the image encoder and the pixel restoration decoder to obtain the restored background region. Based on this, a non - foreground restoration loss function is constructed to guide the image encoder to learn the background distribution information of the remote sensing image, optimize the foreground restoration task, filter out the noisy background in the image features during the retrieval process, improve the alignment degree between the named entities in the text description and the corresponding foreground targets in the remote sensing image, and enhance the model's understanding ability of cross - modal data.

[0030] S3: In the segmentation prediction branch of the segmentation supervision region prediction module, the image features are passed through the image decoder to obtain the predicted segmentation mask; the image features are segmented region - enhanced according to the predicted segmentation mask to obtain the weighted features after segmentation region enhancement; the weighted features after segmentation region enhancement are passed through the image decoder to obtain the target predicted segmentation mask; a predicted segmentation mask loss function is constructed according to the target predicted segmentation mask and the true segmentation mask.

[0031] The segmentation prediction branch of the segmentation supervision region prediction module is only used during the model training process.

[0032] Specifically, passing the image features through the image decoder to obtain the predicted segmentation mask, which is expressed by the formula: , where Denotes the image decoder, Denotes the predicted segmentation mask.

[0033] Segmenting the image features region - enhanced according to the predicted segmentation mask to obtain the weighted features after segmentation region enhancement, which is expressed by the formula: ; Among them, Denotes the weighted features after segmentation region enhancement, Denotes the weighting operation, Denotes the normalization operator, Denotes the mapping operator, Denotes the down - sampling operator; Denotes the image features, Denotes the predicted segmentation mask; Denotes the weighting factor, and its value range is from 0 to 1.

[0034] In this embodiment, The model performance is the best when the value is 0.6.

[0035] Construct a prediction segmentation mask loss function based on the target prediction segmentation mask and the ground truth segmentation mask, which is expressed by the formula: ; where represents the prediction segmentation mask loss function, represents the target prediction segmentation mask; represents the nth ground truth segmentation mask block, represents the ground truth segmentation mask; and are the height and width of the ground truth segmentation mask respectively.

[0036] In the present invention, a semantic segmentation task of pixel-level prediction is introduced into the model. After obtaining the prediction segmentation mask by using the image decoder, the image features are enhanced in the segmentation region according to the prediction segmentation mask, and the weighted features after the segmentation region enhancement are obtained. The weighted features after the segmentation region enhancement are passed through the image decoder again to obtain the target prediction segmentation mask. Based on this, a prediction segmentation mask loss function is constructed, enabling the remote sensing image-text retrieval model to further output the predicted segmentation image of the remote sensing image when retrieving the remote sensing image by the query text, and the output predicted segmentation image has finer-grained foreground features, improving the accuracy of remote sensing image segmentation.

[0037] S4: Input the weighted features after the segmentation region enhancement obtained by the segmentation prediction branch of the segmentation supervision region prediction module into the hierarchical feature learning module, sample, map, and weight each hierarchical feature of the weighted features after the segmentation region enhancement to obtain the target image features; calculate the similarity between the target image features and the text features, and construct a triplet loss function.

[0038] Specifically, input the weighted features after the segmentation region enhancement into the hierarchical feature learning module, sample, map, and weight each hierarchical feature of the weighted features after the segmentation region enhancement to obtain the target image features. The formula includes: ; ; ; where represents the feature of the lth layer of the weighted features after the segmentation region enhancement, represents the weighted features after the segmentation region enhancement, is the total number of layers; represents the sampling operator, represents the feature after sampling compression, represents the feature of the lth layer of the feature after sampling compression; represents the mapping operator, represents the feature after mapping, Represents the feature of the l-th level of the mapped feature; Represents a weighting operator to make the dimensions of each target image feature the same as those of the text feature for subsequent similarity calculation and averaging; Represents the target image feature, Represents the feature of the l-th level of the target image feature.

[0039] In this embodiment, the similarity between the target image feature and the text feature is calculated using the classical Pearson correlation coefficient, and the formula is: ; Wherein, Represents the target image feature, Represents the text feature, Represents the similarity between the target image feature and the text feature; And Represent the number of channels of the target image feature and the text feature respectively; Represents the k-th component of the target image feature, Represents the j-th component of the text feature.

[0040] A triplet loss function is constructed based on the similarity between the target image feature and the text feature, and the formula is: ; Wherein, Represents the triplet loss function, Represents the boundary value, And Represent the negative sample features of the target image feature and the text feature respectively; Is used to ensure that the result is non-negative.

[0041] The triplet loss function can learn the mapping relationship between the target image feature and the text feature, maximize the similarity of the positive sample pair, and minimize the similarity of the negative sample pair.

[0042] The present invention uses a hierarchical feature learning module to learn the feature information of different size levels of the weighted features after segmentation region enhancement to obtain the target image feature. During the training process, the similarity between the target image feature and the text feature is calculated to construct a triplet loss function, which can introduce more underlying texture information, enrich the scale diversity of the features, further improve the fault tolerance rate of feature alignment and the stability of image-text similarity comparison, so as to effectively improve the learning efficiency of the remote sensing image-text retrieval model for foreground targets of different scales and improve the accuracy of retrieval.

[0043] S5: Construct a total loss function with a non-foreground restoration loss function, a predicted segmentation mask loss function, and a triplet loss function, and the formula is expressed as: ; Among them, represents the total loss function, represents the triplet loss function, represents the predicted segmentation mask loss function for the remote sensing image, represents the non-foreground restoration loss function, and respectively represent the weighting coefficients.

[0044] In this embodiment, , .

[0045] S6: Train the image encoder, text encoder, pixel restoration decoder, hierarchical feature learning module, and image decoder based on the total loss function; use the trained image encoder, text encoder, hierarchical feature learning module, and image decoder to construct a remote sensing image-text retrieval model to retrieve remote sensing image-text data.

[0046] Specifically, using the remote sensing image-text retrieval model to retrieve remote sensing image-text data includes: Input the query text into the remote sensing image-text retrieval model, and output the predicted segmentation image of the remote sensing image that matches the query text; Input the query remote sensing image into the remote sensing image-text retrieval model, and output the predicted text that matches the query remote sensing image.

[0047] Specifically, inputting the query text into the remote sensing image-text retrieval model and outputting the predicted segmentation image of the remote sensing image that matches the query text includes: The query text passes through the text encoder to obtain the query text feature; A number of remote sensing images in the remote sensing image database pass through the image encoder of the remote sensing image-text retrieval model to obtain a number of image features; Each image feature passes through the hierarchical feature learning module to obtain its corresponding target image feature; Calculate the similarity between the query text feature and each target image feature, and obtain the remote sensing image that matches the query text according to the similarity; Input the image feature of the remote sensing image that matches the query text into the image decoder to obtain the predicted segmentation mask of the remote sensing image that matches the query text; Output the predicted segmentation image of the remote sensing image that matches the query text according to the remote sensing image that matches the query text and its predicted segmentation mask.

[0048] Specifically, inputting the query remote sensing image into the remote sensing image-text retrieval model and outputting the predicted text that matches the query remote sensing image includes: The query remote sensing image passes through the image encoder to obtain the query image feature; A number of texts in the text database pass through the text encoder of the remote sensing image-text retrieval model to obtain a number of text features; Calculate the similarity between the query image feature and each text feature; Output the predicted text that matches the query remote sensing image according to the similarity.

[0049] Figure 2 It is a schematic diagram of the result of segmenting a remote sensing image by the remote sensing image-text retrieval model, where Figure 2 Columns (a) and (d) in are the input remote sensing images, Figure 2 Column (b) in is the true segmentation mask corresponding to the remote sensing image in column (a), Figure 2 Column (c) in is the predicted segmentation image corresponding to the remote sensing image in column (a), Figure 2 Column (e) in is the true segmentation mask corresponding to the remote sensing image in column (d), Figure 2 Column (f) in is the predicted segmentation image corresponding to the remote sensing image in column (d).

[0050] Figure 3 It is a schematic diagram of obtaining a predicted segmentation image by inputting a query text into the remote sensing image-text retrieval model; where Figure 3 Column (a) in is the query text, Figure 3 Columns (b), (c), (d), (e) and (f) in are the predicted segmentation images of the remote sensing images that match the query text.

[0051] Figure 4 It is a schematic diagram of obtaining a predicted text by inputting a query remote sensing image into the remote sensing image-text retrieval model, where Figure 4 Column (a) in is the query remote sensing image, Figure 4 Column (b) in is the predicted text that matches the query remote sensing image. The green text represents the correctly retrieved text, and the black text represents the incorrectly retrieved text.

[0052] The data or images collected in this application are all from public data sets. Through experiments, it is found that on the RSITMD data set, the mR value of the present invention is increased by 6.12 / 7.83 compared with the existing baseline method. On the RSICD and UCM data sets, the mR values of the present invention are increased by 4.16 / 6.06 and 2.57 / 5.84 respectively compared with the existing baseline methods. After introducing a segmentation prediction branch into the remote sensing image-text retrieval model, the performance of the model is further improved, and the foreground category can be accurately located.

[0053] This embodiment also experiments on the values of the weighting coefficients and in the total loss function. The curve graph of the total loss function changing with the values of the weighting coefficients is referred to Figure 5 as shown, where Figure 5 in (a) is the total loss function with respect to Curve graph of value changes Figure 5 In (b) of is the curve graph of the total loss function changing with the value changes. Experiments show that in a wide range greater than 1, has little impact on mIoU and shows a convex change on mR (optimal when = 4). As continues to decrease below 1, both mR and mIoU will decrease rapidly. When changes from 0 to 1, mR shows a convex change (optimal when

[0054] = 0.5). The results show that as the model continuously learns the feature representation of non-foreground information, the method of the present invention can better distinguish foreground objects, has stable and excellent segmentation effects on objects with large scales and extreme aspect ratios in the segmentation task, and can establish entity-foreground level associations in the retrieval task, providing strong support for the retrieval results. Secondly, the present invention introduces a semantic segmentation task of pixel-level prediction in the remote sensing cross-modal image-text retrieval model. After obtaining the predicted segmentation mask using the image decoder, the image features are strengthened in the segmentation region according to the predicted segmentation mask to obtain the weighted features after segmentation region strengthening. The weighted features after segmentation region strengthening are passed through the image decoder again to obtain the target predicted segmentation mask, and a predicted segmentation mask loss function is constructed accordingly, so that the remote sensing cross-modal image-text retrieval model can further output the predicted segmentation image of the remote sensing image when retrieving the remote sensing image with the query text, and the output predicted segmentation image has finer-grained foreground features, improving the accuracy of remote sensing image segmentation. Finally, the present invention uses the hierarchical feature learning module to learn the feature information of different size levels of the weighted features after segmentation region strengthening to obtain the target image features. During the training process, the similarity is calculated between the target image features and the text features to construct a triplet loss function, which can introduce more underlying texture information, enrich the scale diversity of the features, further improve the fault tolerance rate of feature alignment and the stability of image-text similarity comparison, so as to effectively improve the learning efficiency of the remote sensing cross-modal image-text retrieval model for foreground objects of different scales and improve the accuracy of remote sensing cross-modal image-text retrieval.

[0055] Embodiment 2:

[0056] Based on the remote sensing cross-modal image-text retrieval method described in Embodiment 1, this embodiment provides a remote sensing cross-modal image-text retrieval device, including: An encoding unit, configured to obtain image features and text features by respectively passing a remote sensing image and its corresponding text description through an image encoder and a text encoder; A non-foreground restoration loss function construction unit, configured to obtain a background mask image according to the remote sensing image and its corresponding true segmentation mask, and pass the background mask image through an image encoder and a pixel restoration decoder to obtain a restored background region; construct a non-foreground restoration loss function based on the restored background region; A predicted segmentation mask loss function construction unit, configured to obtain a predicted segmentation mask by passing the image features through an image decoder; perform segmentation region enhancement on the image features according to the predicted segmentation mask to obtain weighted features after segmentation region enhancement; pass the weighted features after segmentation region enhancement through the image decoder to obtain a target predicted segmentation mask; construct a predicted segmentation mask loss function according to the target predicted segmentation mask and the true segmentation mask; A triplet loss function construction unit, configured to input the weighted features after segmentation region enhancement into a hierarchical feature learning module, sample, map, and weight each hierarchical feature of the weighted features after segmentation region enhancement to obtain target image features; calculate the similarity between the target image features and the text features, and construct a triplet loss function; A total loss function construction unit, configured to construct a total loss function with the non-foreground restoration loss function, the predicted segmentation mask loss function, and the triplet loss function; A retrieval unit, configured to train an image encoder, a text encoder, a pixel restoration decoder, a hierarchical feature learning module, and an image decoder based on the total loss function; construct a remote sensing image-text retrieval model by using the trained image encoder, text encoder, hierarchical feature learning module, and image decoder, and perform retrieval on remote sensing image-text data.

[0057] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 means for implementing the functions specified in one block or multiple blocks.

[0059] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 means for implementing the functions specified in one block or multiple blocks.

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 means for implementing the functions specified in one block or multiple blocks.

[0061] Obviously, the above embodiments are merely examples for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A remote sensing cross-modal image-text retrieval method, characterized in that Including: The remote sensing image and its corresponding text description are respectively passed through an image encoder and a text encoder to obtain image features and text features; A background mask image is obtained according to the remote sensing image and its corresponding true segmentation mask, and the background mask image is passed through an image encoder and a pixel restoration decoder to obtain a restored background region; A non-foreground restoration loss function is constructed based on the restored background region; The image features are passed through an image decoder to obtain a predicted segmentation mask; The image features are segmented region enhanced according to the predicted segmentation mask to obtain weighted features after segmented region enhancement; The weighted features after segmented region enhancement are passed through an image decoder to obtain a target predicted segmentation mask; A predicted segmentation mask loss function is constructed according to the target predicted segmentation mask and the true segmentation mask; The weighted features after segmented region enhancement are input into a hierarchical feature learning module, and each hierarchical feature of the weighted features after segmented region enhancement is sampled, mapped, and weighted to obtain target image features; The similarity between the target image features and the text features is calculated to construct a triplet loss function; A total loss function is constructed with the non-foreground restoration loss function, the predicted segmentation mask loss function, and the triplet loss function; Based on the total loss function, an image encoder, a text encoder, a pixel restoration decoder, a hierarchical feature learning module, and an image decoder are trained; A remote sensing image-text retrieval model is constructed using the trained image encoder, text encoder, hierarchical feature learning module, and image decoder to retrieve remote sensing image-text data.

2. The remote sensing cross-modal image-text retrieval method according to claim 1, wherein Using the remote sensing image-text retrieval model to retrieve remote sensing image-text data, including: Inputting the query text into the remote sensing image-text retrieval model, and outputting a predicted segmentation image of the remote sensing image matching the query text, including: The query text passes through the text encoder to obtain query text features; Several remote sensing images in the remote sensing image database pass through the image encoder of the remote sensing image-text retrieval model to obtain several image features; Each image feature passes through the hierarchical feature learning module to obtain its corresponding target image features; The similarity between the query text features and each target image feature is calculated, and the remote sensing image matching the query text is obtained according to the similarity; The image features of the remote sensing image matching the query text are input into the image decoder to obtain a predicted segmentation mask of the remote sensing image matching the query text; According to the remote sensing image matching the query text and its predicted segmentation mask, a predicted segmentation image of the remote sensing image matching the query text is output; Inputting the query remote sensing image into the remote sensing image-text retrieval model, and outputting a predicted text matching the query remote sensing image, including: The query remote sensing image passes through the image encoder to obtain query image features; Several texts in the text database pass through the text encoder of the remote sensing image-text retrieval model to obtain several text features; The similarity between the query image features and each text feature is calculated; The predicted text matching the query remote sensing image is output according to the similarity.

3. A remote sensing cross-modal image-text retrieval method according to claim 1, characterized in that, A background mask image is obtained according to the remote sensing image and its corresponding true segmentation mask, and the formula is: ； Among them, represents a remote sensing image, represents a background mask image; represents a masking strategy adopted after reading the true segmentation mask corresponding to the remote sensing image, and is used to randomly retain several pixels in the image blocks masked by the true segmentation mask; represents the masking rate; represents the masking operation.

4. A remote sensing cross-modal image-text retrieval method according to claim 1, characterized in that, A non-foreground restoration loss function is constructed based on the restored background region, and the formula is: ； Among them, represents the non-foreground restoration loss function, represents the restored background area, represents the remote sensing image, represents the background mask image.

5. A remote sensing cross-modal image-text retrieval method according to claim 1, characterized in that, The image features are segmented region enhanced according to the predicted segmentation mask to obtain weighted features after segmented region enhancement, and the formula is: ； Among them, represents the weighted feature after the segmentation region is strengthened, represents the weighting operation, represents the normalization operator, represents the mapping operator, represents the downsampling operator; represents the image feature, represents the predicted segmentation mask, represents the weighting factor.

6. A remote sensing cross-modal image-text retrieval method according to claim 1, characterized in that Construct a prediction segmentation mask loss function based on the target prediction segmentation mask and the ground truth segmentation mask. The formula is as follows: ； Among them, represents the predicted segmentation mask loss function, represents the target predicted segmentation mask; represents the nth true segmentation mask block, represents the true segmentation mask; and are the height and width of the true segmentation mask respectively.

7. A remote sensing cross-modal image-text retrieval method according to claim 1, characterized in that Input the weighted features after segmentation region enhancement into the hierarchical feature learning module, sample, map, and weight each hierarchical feature of the weighted features after segmentation region enhancement to obtain the target image features. The formula includes: ； ； ； Among them, represents the feature of the l-th level of the weighted feature after the segmentation region enhancement, represents the weighted feature after the segmentation region enhancement, is the total number of levels; represents the sampling operator, represents the feature after sampling compression, represents the feature of the l-th level of the feature after sampling compression; represents the mapping operator, represents the feature after mapping, represents the feature of the l-th level of the feature after mapping; represents the weighting operator, represents the target image feature, represents the feature of the l-th level of the target image feature.

8. A remote sensing cross-modal image-text retrieval method according to claim 1, characterized in that Calculate the similarity between the target image features and the text features, and construct a triplet loss function, including: Calculate the similarity between the target image features and the text features using the classical Pearson correlation coefficient. The formula is as follows: ； Among them, represents the target image feature, represents the text feature, represents the similarity between the target image feature and the text feature; and respectively represent the number of channels of the target image feature and the text feature; represents the k-th component of the target image feature, represents the j-th component of the text feature; Construct a triplet loss function based on the similarity between the target image features and the text features. The formula is as follows: ； Among them, represents the triplet loss function, represents the boundary value, and respectively represent the negative sample features of the target image feature and the text feature; is used to ensure that the result is non - negative.

9. A remote sensing cross-modal image-text retrieval method according to claim 1, characterized in that Construct a total loss function using the non-foreground restoration loss function, the prediction segmentation mask loss function, and the triplet loss function. The formula is as follows: ； Among them, represents the total loss function, represents the triplet loss function, represents the predicted segmentation mask loss function, represents the non-foreground restoration loss function, and represent the weighting coefficients respectively.

10. A remote sensing cross-modal image-text retrieval device, characterized in that, Including: An encoding unit for obtaining image features and text features by passing the remote sensing image and its corresponding text description through an image encoder and a text encoder respectively; A non-foreground restoration loss function construction unit for obtaining a background mask image based on the remote sensing image and its corresponding ground truth segmentation mask, passing the background mask image through an image encoder and a pixel restoration decoder to obtain a restored background region, and constructing a non-foreground restoration loss function based on the restored background region; A prediction segmentation mask loss function construction unit for obtaining a predicted segmentation mask by passing the image features through an image decoder, enhancing the segmentation region of the image features based on the predicted segmentation mask to obtain weighted features after segmentation region enhancement, obtaining a target predicted segmentation mask by passing the weighted features after segmentation region enhancement through the image decoder, and constructing a prediction segmentation mask loss function based on the target predicted segmentation mask and the ground truth segmentation mask; A triplet loss function construction unit for inputting the weighted features after segmentation region enhancement into the hierarchical feature learning module, sampling, mapping, and weighting each hierarchical feature of the weighted features after segmentation region enhancement to obtain the target image features, calculating the similarity between the target image features and the text features, and constructing a triplet loss function; A total loss function construction unit for constructing a total loss function using the non-foreground restoration loss function, the prediction segmentation mask loss function, and the triplet loss function; A retrieval unit for training the image encoder, the text encoder, the pixel restoration decoder, the hierarchical feature learning module, and the image decoder based on the total loss function, constructing a remote sensing image-text retrieval model using the trained image encoder, text encoder, hierarchical feature learning module, and image decoder, and retrieving remote sensing image-text data.

Images