Remote sensing image cross-modal retrieval method and device and electronic equipment

Abstract

The invention provides a remote sensing image cross-modal retrieval method and device and electronic equipment. The method comprises the following steps: acquiring a user description text, a plurality of remote sensing images and a pre-constructed retrieval model; a remote sensing image feature extraction network based on the model extracts global image features and multi-scale features of each remote sensing image, performs fusion enhancement on the multi-scale features, and fuses the fused and enhanced multi-scale features and the global image features to obtain remote sensing image features; extracting text features of the user description text based on a text feature extraction network; respectively calculating the similarity between the image feature of each remote sensing image and the text feature of the text input by the user; and screening the remote sensing images of which the similarity meets a preset condition from the plurality of remote sensing images to obtain a target remote sensing image. The target remote sensing image highly related to the user description text can be quickly and accurately retrieved from a large number of remote sensing images, so that the retrieval accuracy of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a remote sensing image cross-modal retrieval method, device and electronic equipment. Background Art

[0002] With the rapid development of earth observation capabilities, the amount of remote sensing data currently acquired is growing exponentially, and remote sensing data is showing a trend of diversification and mass production. Remote sensing earth observation has entered the era of big data (NASA's earth observation satellites transmit TB-level data to the earth every day, and the Sentinel series of satellites receive more than 10 TB of data every day while in orbit). The problem of "massive data and overwhelming information" in remote sensing big data is becoming increasingly prominent. It is necessary to design an effective remote sensing image retrieval method to quickly obtain the required data from massive remote sensing data and improve the management and use efficiency of remote sensing data.

[0003] According to the difference in the modality of the query data, remote sensing image retrieval technology can be divided into single-modal retrieval and cross-modal retrieval. The main difference between the two is whether the modality of the query data and the modality of the data to be retrieved are the same. If they are the same, it is single-modal retrieval, otherwise it is cross-modal retrieval. For remote sensing image retrieval, single-modal retrieval means that the query and retrieved remote sensing data are the same type of remote sensing images. For example, the deep adversarial hashing method is used to retrieve optical remote sensing images, and the convolutional neural network (CNN) is used to learn the domain-invariant features between synthetic aperture radar images and optical remote sensing images, so as to retrieve synthetic aperture radar images. In practical applications, due to the diversity of data modalities, it is often necessary to use data of other modalities (such as text, voice, etc.) rather than just the same type of remote sensing images to retrieve remote sensing images. For example, the prior art also proposes a deep visual-audio network that uses audio to directly retrieve remote sensing images. The network is based on pre-trained CNN and deep audio network, and uses neural networks to fuse and classify speech features and remote sensing image features. Compared with unimodal retrieval of remote sensing images, cross-modal retrieval of remote sensing images, especially cross-modal retrieval between remote sensing images and text, still faces many challenges due to the "heterogeneous gap" problem caused by the different distribution spaces of query and retrieval data. Summary of the invention

[0004] The present invention provides a remote sensing image cross-modal retrieval method, device and electronic device, which are used to solve the problem that the remote sensing image cross-modal retrieval method in the prior art does not make sufficient use of the multi-scale information in the remote sensing image, and still cannot fully extract this information, which affects the accuracy of the remote sensing image-text retrieval task to a certain extent, and improves the retrieval accuracy. The technical solution proposed by the present invention is as follows: In a first aspect, the present invention provides a remote sensing image cross-modal retrieval method, comprising: Obtaining user description text, multiple remote sensing images and a pre-built retrieval model; wherein the retrieval model includes a remote sensing image cross-modal retrieval network and a text feature extraction network; Extracting global image features and multi-scale features of each remote sensing image based on a remote sensing image feature extraction network, fusing and enhancing the multi-scale features, and fusing the fused and enhanced multi-scale features with the global image features to obtain remote sensing image features; Extracting text features of the user description text based on a text feature extraction network; Calculating the similarity between the remote sensing image features and the text features of each remote sensing image respectively; The remote sensing images whose similarity meets the preset conditions are screened out from the multiple remote sensing images to obtain the target remote sensing image.

[0005] Optionally, the remote sensing image cross-modal retrieval network includes a remote sensing image feature extraction network, a multi-scale feature fusion module and a significant feature enhancement module; The remote sensing image feature extraction network is used to extract global image features and multi-scale features of each remote sensing image, the multi-scale features are fused and enhanced, and the fused and enhanced multi-scale features are fused with the global image features to obtain remote sensing image features, including: For each remote sensing image, perform the following steps: Based on the remote sensing image feature extraction network, the remote sensing image is subjected to feature extraction to obtain global image features and multi-scale features; Based on a multi-scale feature fusion module, the multi-scale features are fused to obtain a fused visual feature with multi-scale information; Based on a salient feature enhancement module, the fused visual feature is fused with a first scale feature in the multi-scale feature to obtain a salient information feature; The remote sensing image feature is obtained by fusing the significant information feature and the global image feature.

[0006] Optionally, the multi-scale feature fusion module includes a channel attention module, and the multi-scale feature further includes a second scale feature and a third scale feature; The step of fusing the multi-scale features based on the multi-scale feature fusion module to obtain a fused visual feature with multi-scale information includes: Performing dimensionality reduction on the first scale feature and the second scale feature, and fusing the third scale feature and the first scale feature and the second scale feature after dimensionality reduction to obtain a preliminary fused feature; The preliminary fusion features are processed using the channel attention module, and the processed preliminary fusion features are converted into fusion visual features with the same number of channels as the third-scale features.

[0007] Optionally, the fusing the fused visual feature and the first scale feature in the multi-scale feature to obtain a salient information feature based on the salient feature enhancement module includes: Using the fused visual feature as a reference feature and using the first scale feature in the multi-scale feature as an enhanced feature; Processing the enhanced features, and performing element-wise multiplication of the baseline features using the processed enhanced features to generate initial information features; The initial information features are subjected to convolution, maximum pooling and spatial dimension reduction processing to obtain significant information features.

[0008] Optionally, the text feature extraction network includes a text encoder, a Transformer encoder and a fully connected layer; The extracting of text features of the user description text based on a text feature extraction network includes: Using the text encoder to perform word segmentation and word vector encoding on the user description text to obtain a word encoding vector of a corresponding word block, and generate a mask vector; Inputting the word encoding vector and the mask vector into the Transformer encoder to obtain semantic features; The semantic features are converted into text features with the same dimension as the remote sensing image features based on the fully connected layer.

[0009] Optionally, the retrieval model is trained in the following manner: Each remote sensing image is set as a category, and the user description text corresponding to the remote sensing image is classified into the same category; For all samples in a batch, the similarity between the remote sensing image features and the text features of the samples is calculated, and a similarity matrix is ​​constructed; wherein the samples are pairs of remote sensing images and user description texts, each row of the similarity matrix represents the similarity between a remote sensing image and all user description texts, and each column represents the similarity between a user description text and all remote sensing images; Calculate the similarity between each sample and the true category of the sample; Taking the similarity matrix and the similarities between all samples and the true categories of the samples as inputs of a pre-constructed loss function, calculating the cross entropy loss of the image retrieval text task; Taking the transpose of the similarity matrix and the similarities between all samples and the true category of the samples as inputs of a pre-constructed loss function, calculating the cross entropy loss of the text retrieval task image; Taking the average of the cross entropy loss of the image retrieval text task and the cross entropy loss of the text retrieval task image as the model loss; Model parameters of the retrieval model are trained based on the model loss.

[0010] In a second aspect, the present invention provides a remote sensing image cross-modal retrieval device, comprising: An acquisition module, used to acquire user description text, multiple remote sensing images and a pre-built retrieval model; wherein the retrieval model includes a remote sensing image cross-modal retrieval network and a text feature extraction network; An image feature extraction module is used to extract global image features and multi-scale features of each remote sensing image based on a remote sensing image feature extraction network, fuse and enhance the multi-scale features, and fuse the fused and enhanced multi-scale features with the global image features to obtain remote sensing image features; A text feature extraction module, used for extracting text features of the user description text based on a text feature extraction network; A calculation module, used for respectively calculating the similarity between the remote sensing image feature of each remote sensing image and the text feature; The screening module is used to screen out remote sensing images whose similarity meets preset conditions from the multiple remote sensing images to obtain a target remote sensing image.

[0011] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the remote sensing image cross-modal retrieval method as described in the first aspect is implemented.

[0012] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the remote sensing image cross-modal retrieval method as described in the first aspect.

[0013] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the remote sensing image cross-modal retrieval method as described in the first aspect.

[0014] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: The cross-modal retrieval method for remote sensing images provided by the present invention can fully utilize the multi-scale information of remote sensing images and improve the retrieval accuracy by fusing the multi-scale features of remote sensing images. The fused features with multi-scale information are enhanced, and the feature representation of the image is further enriched by the extraction and fusion enhancement of multi-scale features, and the expression ability of the target information of the remote sensing image is enhanced. At the same time, the model's recognition ability for smaller targets is improved as much as possible through multi-scale information. By combining the cross-modal retrieval network for remote sensing images and the text feature extraction network, the present invention can quickly and accurately retrieve target remote sensing images that are highly related to the user's description text from a large number of remote sensing images, and obtain more accurate and comprehensive remote sensing image features, thereby improving the retrieval accuracy of the model.

[0015] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the cross-modal retrieval method for remote sensing images provided by the present invention.

[0019] Figure 2 It is a structural schematic diagram of the remote sensing image cross-modal retrieval network provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the remote sensing image feature extraction network provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the MFF module provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the SFE module provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the text feature extraction network provided by the present invention.

[0024] Figure 7 It is a schematic diagram of the structure of the remote sensing image cross-modal retrieval device provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The inventor found in his work that early remote sensing image text retrieval was carried out by manually annotating text or keywords, that is, each remote sensing image was manually annotated in advance, and then the retrieval was performed by comparing the similarity between the predefined text annotation and the input text. This method requires a lot of manpower for manual annotation, the retrieval efficiency is limited, and there are certain requirements for the professionalism of the annotation personnel, which cannot cope with the challenges brought by the rapid growth of the number of remote sensing images. Therefore, more and more researchers have begun to study the generation of remote sensing image descriptions. For example, in the prior art, a remote sensing image description framework is constructed using a fully convolutional network to generate corresponding text descriptions for remote sensing images. This method based on generating remote sensing image descriptions successfully solves the cost problem of manual annotation. However, this retrieval method that processes remote sensing images and texts separately is still affected by the "heterogeneous gap", resulting in low retrieval accuracy. How to construct a similarity measurement model between different modal information to solve the problem that the similarity between two modal information is difficult to directly measure is the key to realizing cross-modal retrieval of remote sensing image text.

[0028] In recent years, with the rapid growth of multimodal data and the continuous development of deep learning technology, the research on cross-modal retrieval in the field of natural images has achieved fruitful results. At the same time, in the field of remote sensing, more and more researchers have begun to explore the cross-modal retrieval of remote sensing images based on deep learning, which can be mainly divided into two categories: methods based on semantic alignment and methods based on multi-scale information enhancement.

[0029] The semantic alignment-based method focuses on mining the potential correspondence between remote sensing images and texts, and uses semantic alignment to correspond image information with text information, strengthen the semantic relationship between remote sensing images and texts, and thus improve the accuracy of cross-modal retrieval. In the prior art, a deep semantic alignment network is designed, which uses an attention mechanism to enhance the correspondence between images and texts, and uses a gate function to filter unnecessary information to obtain discriminative visual features. There is also a cross-attention mechanism that combines sentence-level text information and region-level image information to achieve cross-modal information interaction. There is also an interactive enhancement feature Transformer proposed, which uses a feature embedding module to simultaneously process visual features and text features to reduce the semantic inconsistency of the two modalities, and uses an information interaction enhancement module to perform cross-modal information interaction. Although the semantic alignment-based method can fully mine the deep relationship between images and texts and improve the similarity between modalities, due to the wide coverage of remote sensing images and the lack of target information, it is often affected by redundant information when performing semantic alignment, which in turn affects the accuracy of retrieval.

[0030] Although there are many studies on cross-modal retrieval methods of remote sensing images based on multi-scale information enhancement, the following problems still exist: (1) Existing cross-modal retrieval methods for remote sensing images do not make sufficient use of the multi-scale information in remote sensing images and are still unable to fully extract this information, which to some extent affects the accuracy of remote sensing image-text retrieval tasks.

[0031] (2) Existing methods have poor results in identifying target information in remote sensing images. Most current methods focus on how to interact information between remote sensing images and text, but ignore the filtering of redundant features in remote sensing images, making it difficult to extract significant features, which hinders the understanding of the content of remote sensing images. The inventors made the present invention after further research and development.

[0032] Combine the following Figure 1-Figure 8 The invention describes a remote sensing image cross-modal retrieval method, device and electronic device.

[0033] Figure 1 is a flow chart of the cross-modal retrieval method for remote sensing images provided by the present invention, referring to Figure 1 As shown, the method includes the following: Step 101: Obtain user description text, multiple remote sensing images and a pre-built retrieval model.

[0034] The user description text is usually obtained through a user interface such as a web form, a mobile application interface, etc. The user input may be a keyword, a phrase, a sentence, or a more complex natural language description, which is used to describe the remote sensing image content to be retrieved.

[0035] The above remote sensing images can come from a variety of data sources, including satellites, drones and other remote sensing platforms. These data are usually stored in remote sensing data centers or professional institutions. After obtaining remote sensing images, they can be preprocessed, including denoising, geometric correction, image enhancement, etc., to improve image quality and retrieval accuracy.

[0036] The above retrieval model includes a remote sensing image cross-modal retrieval network and a text feature extraction network. The remote sensing image cross-modal retrieval network is responsible for mapping remote sensing images and user description texts into the same feature space for similarity calculation and retrieval. The text feature extraction network is used to extract semantic features from user description texts, which can accurately reflect the user's query intentions and needs.

[0037] Step 102: extract global image features and multi-scale features of each remote sensing image based on a remote sensing image feature extraction network, fuse and enhance the multi-scale features, and fuse the fused and enhanced multi-scale features with the global image features to obtain remote sensing image features.

[0038] First, the initial part of the remote sensing image feature extraction network, such as the shallow network layer, is used to extract features from the input remote sensing image. This process usually involves convolutional layers, activation functions (such as ReLU), and pooling layers to extract basic visual features of the image, such as edges, corners, etc. These features constitute the global image features of the remote sensing image and provide a basis for subsequent processing.

[0039] Next, the deep network layers in the remote sensing image feature extraction network are used to extract multi-scale features of the remote sensing image. Multi-scale features refer to image features extracted at different scales or resolutions, which can capture different details and contextual information in the image. The deep network layers can extract high-level semantic features in the image, while the shallow network layers focus more on the local details of the image. By combining these features at different scales, the content of the remote sensing image can be more fully understood.

[0040] Then, the extracted multi-scale features are fused and enhanced. The purpose of this step is to integrate feature information of different scales, highlight important features, and suppress irrelevant information. In the fusion process, attention mechanisms such as channel attention and spatial attention can also be introduced to dynamically adjust the importance of different features.

[0041] Finally, the fused and enhanced multi-scale features are fused with the global image features. The purpose of this step is to combine the rich information in the multi-scale features with the basic visual features in the global image features to form a more comprehensive and accurate remote sensing image feature representation. The fused features contain both the local detail information of the image and the global contextual information and high-level semantic information, which is helpful for subsequent classification, recognition and other tasks.

[0042] Step 103: extracting text features of the user description text based on a text feature extraction network.

[0043] Select appropriate text feature extraction networks according to specific task requirements, such as convolutional neural networks (CNN), recurrent neural networks (RNN), Transformer, etc. These networks can capture semantic information in text and convert it into high-dimensional feature vectors. Specifically, after obtaining the user description text, first, preprocess the user description text, including word segmentation, stop word removal, stem extraction, part-of-speech tagging, etc., to improve the accuracy and efficiency of feature extraction. The preprocessed text is input into the text feature extraction network, and the numerical features of the text are obtained through network calculation. These features usually include word embedding vectors, sentence embedding vectors, etc., which can reflect the key information and semantic relationships in the text. According to the specific task requirements, the extracted features are optimized, such as feature dimensionality reduction, feature selection, etc., to reduce the amount of calculation and improve the retrieval performance.

[0044] By extracting the text features of user description texts, the semantic information, key words, sentiment tendencies, etc. in the user description texts are converted into numerical features, and the natural language descriptions input by users can be converted into numerical features that can be understood by computers. These features can represent the intention and core content of the user's query. By extracting the text features of user description texts, a bridge between text and images is built, so that data of different modalities can be compared and matched in the same feature space. Moreover, by accurately extracting text features, the needs of user queries can be more accurately reflected, thereby improving the accuracy and efficiency of remote sensing image retrieval.

[0045] Step 104: Calculate the similarity between the remote sensing image features and the text features of each remote sensing image.

[0046] Traverse all remote sensing images and calculate the similarity between their respective remote sensing image features and the text features of the user description text. This step is the core of cross-modal retrieval, which allows comparison and matching between different data modalities, namely images and text. Similarity measurement methods such as cosine similarity, Euclidean distance, Manhattan distance, Pearson correlation coefficient, etc. can be selected to calculate the similarity between features.

[0047] Step 105: Filter out remote sensing images whose similarity meets a preset condition from the multiple remote sensing images to obtain a target remote sensing image.

[0048] According to the actual application scenario and requirements, the conditions for similarity judgment are set, such as a similarity threshold T. Only when the similarity between the remote sensing image feature and the text feature is greater than or equal to T, the remote sensing image is considered to have a high similarity with the user description text. Remote sensing images with similarity greater than or equal to T are selected as target remote sensing images.

[0049] The whole process realizes the comprehensive extraction and enhancement of remote sensing image features through multi-scale feature extraction, fusion enhancement and fusion with global image features. This method can make full use of the multi-scale information in remote sensing images, improve the accuracy and robustness of feature expression, and provide strong support for subsequent image analysis and application.

[0050] The cross-modal retrieval method for remote sensing images provided by the present invention can quickly retrieve the information that users want from massive multimodal media data by combining a cross-modal retrieval network for remote sensing images and a text feature extraction network, thereby solving the problem of insufficient or inaccurate retrieval results that may be caused by traditional tag or keyword-based retrieval methods. Cross-modal retrieval can learn the semantic associations between different modal data, namely remote sensing images and user description texts, to achieve more accurate semantic matching, thereby improving the accuracy and relevance of retrieval. Moreover, the cross-modal retrieval method reduces the need for manual annotation of image data, reducing labor costs and time costs. This advantage is particularly evident in large-scale data sets. Through automated feature extraction and similarity calculation, cross-modal retrieval in the present invention can achieve efficient management and utilization of multimedia data.

[0051] The cross-modal retrieval method for remote sensing images provided by the present invention allows users to use different modalities, namely text and images, as query samples, to meet the diverse retrieval needs of users, and enhance the flexibility and convenience of user experience. Applications such as cross-modal retrieval of images and texts present better human-computer interaction characteristics. Users can quickly find relevant content by inputting natural language descriptions, which improves the intuitiveness and effectiveness of the interaction. This cross-modal retrieval technology is suitable for retrieval between images and texts, and can provide support for applications in multiple fields, such as text-based image semantic segmentation, multimodal medical image processing, and multimodal named entity recognition tasks.

[0052] In an optional embodiment, in order to make full use of the multi-scale information in remote sensing images and solve the problem that the target information of remote sensing images is not prominent, the present method designs a multi-scale feature fusion (Multi-scale Feature Fusion, MFF) module and a salient feature enhancement (SFE) module respectively. The above-mentioned remote sensing image cross-modal retrieval network includes a remote sensing image feature extraction network, an MFF module and an SFE module. The method uses InceptionResnet V2 and Bidirectional Encoder Representations from Transformers (BERT) to extract remote sensing image features and text features respectively, and enhances the remote sensing image features through the MFF module and the SFE module. Finally, the similarity function is used to calculate the similarity between the remote sensing image features and the text features. The network mainly includes four parts: remote sensing image feature extraction, text feature extraction, MFF module, SFE module and similarity measurement. The overall structure refers to Figure 2 shown.

[0053] The above-mentioned S102 extracts global image features and multi-scale features of each remote sensing image based on the remote sensing image feature extraction network, fuses and enhances the multi-scale features, and fuses the fused and enhanced multi-scale features with the global image features to obtain remote sensing image features, specifically including the following steps: For each remote sensing image, perform the following steps: S1021. Extract features of the remote sensing image based on a remote sensing image feature extraction network to obtain global image features and multi-scale features.

[0054] The input remote sensing image is defined as , extracting multi-scale features and global image features of remote sensing images through remote sensing image feature extraction network , where the multi-scale features include the first visual feature, the second visual feature and the third visual feature, which are hereinafter referred to as low-level visual features , mid-level visual features and high-level visual features This method uses Inception Resnet V2 as the remote sensing image feature extraction network of the model, and selects the outputs of the Stem block, Reduction-A block, and Reduction-B block as the low-level visual features in the multi-scale features. , mid-level visual features and high-level visual features , whose structure is as follows Figure 3 shown.

[0055] S1022, fusing the multi-scale features based on the MFF module to obtain a fused visual feature with multi-scale information .

[0056] Compared with natural images, remote sensing images often contain rich target information. For larger targets, global features can be used to express them. However, for smaller targets, as the number of convolutional network layers increases, their information in remote sensing image features may gradually decrease or even disappear. In order to better extract multi-scale information in remote sensing images and retain the feature information of smaller targets, this method constructs the following Figure 4 The MFF module shown in Figure 2. This module fuses multi-scale features and uses a channel attention mechanism to enhance the multi-scale information processing capability.

[0057] S1023, based on the SFE module, the fused visual features Compared with the low-level visual features in the multi-scale features Fusion to obtain significant information features ; S1024: the significant information features and the global image features Fusion is performed to obtain the remote sensing image features .

[0058] Reference Figure 2 As shown in the figure, the output features of the remote sensing image feature extraction network are the above global image features. With the enhanced salient information features Multiply by variable coefficients , After passing through the average pooling layer, the Dropout layer with an inactivation rate of 0.8, and the Softmax layer, the final remote sensing image features are obtained. .

[0059] In an optional embodiment, the MFF module includes a channel attention module; The step of fusing the multi-scale features based on the MFF module to obtain a fused visual feature with multi-scale information in S1022 includes: S10221, the low-level visual features and the mid-level visual features Perform dimensionality reduction, fuse the high-level visual features and the low-level visual features and mid-level visual features after dimensionality reduction to obtain a preliminary fusion feature .

[0060] Due to the multi-scale features obtained by the remote sensing image feature extraction network , and The dimensions are not uniform. The MFF module first and Perform a 7×7 and 3×3 convolution respectively (with step sizes of 4 and 2 respectively). In order to reduce the amount of calculation and the number of parameters, refer to Figure 4 As shown in the figure, the 7×7 and 3×3 convolutions are split into the series of 1×7 and 7×1 convolutions and the series of 1×3 and 3×1 convolutions, respectively, and their step sizes are changed to (1, 4), (4, 1) and (1, 2), (2, 1) respectively. Then, in order to enhance the nonlinear expression ability of the network, the MFF module And after dimensionality reduction , Both perform 1×1 convolution and parameterized rectified linear units ( Figure 4 Finally, the three are spliced ​​together to obtain the initial fusion feature .

[0061] S10222, using the channel attention module to Processing is performed and the processed preliminary fusion features are converted into the high-level visual features Fusion visual features with the same number of channels .

[0062] Due to the initial fusion features Different channels have different abilities to express multi-scale information. This method can perform preliminary fusion feature Channel attention is used to adaptively assign weights to each channel, thereby improving the MFF module's ability to utilize multi-scale information. At the same time, in order to reduce the feature dimension and facilitate subsequent processing, this method uses 1×1 convolution to reduce the number of channels to the same level as the high-level visual features. Same, finally get the fusion visual features .

[0063] In an optional embodiment, due to the complexity of the target in the remote sensing image, the fusion of visual features There is still redundancy in . This makes the target information unable to stand out, so it is necessary to further optimize the fusion visual features. , filtering out irrelevant and redundant information, so as to better highlight the target information in the remote sensing image. To this end, this method designs the following Figure 5 The SFE module shown in the above S1023 is used to fuse the fused visual features and the low-level visual features in the multi-scale features to obtain the significant information features. ,include: First, the fusion visual features As the reference feature, the low-level visual features in the multi-scale features are The enhanced features are processed, and the processed enhanced features are used to perform element-wise multiplication of the baseline features to generate a significant information feature gate vector containing target information, hereinafter referred to as the initial information feature Then, the initial information features Convolution and maximum pooling are performed to enhance the salient features and reduce the spatial dimension to obtain the final salient information feature gate vector, which is referred to as the salient information feature below. .

[0064] Specifically, refer to Figure 5 As shown in the figure, in order to improve the enhancement effect of the enhanced features on the baseline features, the SFE module first processes the enhanced features through four different convolution branches, then concatenates the outputs of the four branches and uses 1×1 convolution to reduce the dimension, thereby obtaining enhanced features with more information, and then uses the Sigmoid activation function to suppress useless information in the enhanced features. On the other hand, in order to unify the feature dimensions, the module processes the baseline features Upsampling and 1×1 convolution are performed, and then multiplied with the processed enhanced features to obtain the initial information features Finally, the module uses 3×3 convolution and adaptive maximum pooling to ensure the initial information features With datum feature The dimensions of the features are consistent, and the significant information in the features is further enhanced and redundant information is filtered out, thereby obtaining the final significant information features. .

[0065] In an optional embodiment, the method uses the BERT model as a text feature extraction network to obtain text features of a specified dimension. , where each user description text corresponds to a text feature. Figure 6 As shown, the text feature extraction network includes a text encoder, a Transformer encoder, and a fully connected layer. The BERT pre-trained text encoder is used for text processing, and the fully connected layer is combined to adjust the feature dimension for subsequent processing or comparison with image features. Transformer encoder; The above-mentioned step 103 of extracting text features of the user description text based on a text feature extraction network includes: S1031. Use the text encoder to perform word segmentation and word vector encoding on the user description text to obtain word encoding vectors of corresponding word blocks, and generate mask vectors.

[0066] User description text for input , first preprocessing is performed, including removing useless characters and normalizing punctuation marks. This method uses the BERT pre-trained text encoder for word segmentation, which can decompose words into sub-word units and convert the word segmentation results into corresponding word chunk IDs. These word chunk IDs are converted into word vectors with fixed dimensions such as 768 dimensions through the BERT embedding layer to obtain the word encoding vectors of the corresponding word chunks. , these word vectors contain rich semantic information. The BERT pre-training process can refer to the description in the prior art and will not be repeated here.

[0067] The above mask vector is used in the BERT model to indicate which word blocks in the user description text are valid and which are invalid word blocks added due to padding. This allows the model to ignore the padding blocks in the self-attention mechanism and only focus on the valid blocks, thereby avoiding the impact of the padding blocks on the model output features. The mask vector is generated in the following way: The length of the mask vector is made the same as the length of the chunk sequence of the input user description text. For valid chunks, the value of the corresponding position in the mask vector is usually 1; for invalid chunks such as filler chunks, the value is 0.

[0068] S1032. Input the word encoding vector and the mask vector into the Transformer encoder to obtain semantic features.

[0069] The word encoding vector and mask vector are input into the Transformer encoder part of BERT. BERT's Transformer encoder deeply processes the word encoding vector through multi-layer self-attention mechanism and position encoding to capture the contextual information and semantic relationship in the text. Finally, the BERT model outputs the text feature vector of each word block, and the dimension of each vector is 768 dimensions.

[0070] S1033. Considering that the text features generated by the BERT model are all 768-dimensional vectors, the text features need to be fused or compared with the remote sensing image features in the future, and the dimensions of the image features may be different from the text feature dimensions output by BERT. This method converts the semantic features into vectors of the same dimension as the remote sensing image features through a fully connected layer, thereby obtaining the final text features. This conversion process usually includes the multiplication of the weight matrix and the application of the activation function (such as ReLU). Its dimension is consistent with the image features and can be used for subsequent multimodal processing tasks such as cross-modal retrieval, multimodal classification, etc.

[0071] The above BERT model has learned a wealth of language knowledge and semantic relationships through pre-training, and can generate high-quality text feature vectors, which helps improve the performance of downstream tasks. By adjusting the dimension of text features through the fully connected layer, text features can be compared or fused with features of other modalities such as image features in the same dimensional space, which facilitates cross-modal processing and analysis. Combining the adjusted text features with remote sensing image features can make full use of information from both text and image modalities, improving the accuracy and efficiency of multimodal retrieval.

[0072] The cross-modal retrieval process of remote sensing images of the present invention is as follows: First, Inception Resnet V2 is used as the remote sensing image feature extraction network to obtain global image features , while extracting multi-scale features of remote sensing images , , , and combine multi-scale features with low-level visual features Input the MFF module and SFE module respectively; then the enhanced significant information features With global image features Add and reduce the dimension to get the final remote sensing image features .

[0073] Secondly, use the BERT model as a text feature extraction network to obtain text features of specified dimensions , where each user description text corresponds to a text feature.

[0074] At the same time, in order to make full use of the multi-scale information in remote sensing images and solve the problem that the target information of remote sensing images is not prominent, this method designs MFF modules and SFE modules respectively. The MFF module uses the multi-scale features extracted by the remote sensing image feature extraction network, performs convolution operations on them respectively and then connects them, and then uses the channel attention mechanism to adaptively optimize the weights of each channel to obtain preliminary fusion features with multi-scale information. The SFE module combines the initial fusion features obtained by the MFF module As the baseline feature, the low-level visual features As an enhanced feature, the initial information feature is obtained through convolution and other processing , thereby further optimizing the fusion of visual features , obtain significant information features that can highlight the target information of remote sensing images .

[0075] Finally, the salient information features The output of the remote sensing image feature extraction network Add together to get the final remote sensing image features , and the remote sensing image features and text features Perform similarity measurement and ultimately achieve more accurate cross-modal retrieval of remote sensing images.

[0076] In summary, in response to the problem of insufficient utilization of multi-scale information in existing methods, this method designs a more effective MFF module to integrate the low-, medium- and high-level features of remote sensing images, fully utilize the multi-scale information of remote sensing images, and improve retrieval accuracy; at the same time, in order to solve the problem that existing methods have poor recognition effect on target information in remote sensing images, this method constructs an SFE module, which uses the low-level visual features in the multi-scale features to enhance the fused visual features with multi-scale information extracted by the MFF module, strengthen the expression ability of remote sensing image target information, and at the same time, improve the model's recognition ability of smaller targets as much as possible through multi-scale information; finally, a cross-modal retrieval method for remote sensing images based on MFF-SFE is proposed, which obtains more accurate and comprehensive remote sensing image features by combining the MFF module and the SFE module, thereby improving the retrieval accuracy of the model.

[0077] In an optional embodiment, in order to measure information of different modalities in a unified feature space, this method uses cosine similarity to calculate the similarity between remote sensing image features and text features, and uses a cross entropy loss function to optimize the training retrieval model. The specific process is as follows: S201, each remote sensing image is set as a category, and the user description text corresponding to the remote sensing image is classified into the same category, that is, the remote sensing image and its corresponding user description text should have the same feature representation in the common feature space. The calculation formulas of the cosine similarity and cross entropy loss function are as follows: , Formula 1

[0078] , Formula 2

[0079] In the above formula: represents the 2-norm of the calculated vector, represents the feature index, Indicates the batch size, Represents the category index, represents the total number of categories, Indicates The sample and The similarity of the categories, Representation sample The real category.

[0080] Since the present invention regards each remote sensing image as an independent category and assumes that the corresponding user description text also belongs to the same category, the sample in the present invention actually refers to the remote sensing image-user description text pair, and the true category of the sample refers to the category corresponding to this remote sensing image-user description text pair. In the image retrieval text task, the true category of the sample is the category of the remote sensing image itself. In the text retrieval image task, the true category of the sample is the category of the user description text itself.

[0081] The cosine similarity formula is used to calculate the similarity between image features and text features. Cosine similarity measures the degree of proximity between two vectors in direction. and text features When they are exactly the same, When, according to formula 1, and The cosine similarity is equal to 1; on the contrary, if and The greater the difference, the closer their cosine similarity will be to -1.

[0082] S202. For all samples in a batch, the cosine similarity between the remote sensing image features and the text features of the samples is calculated by formula 1, and a similarity matrix is ​​constructed; wherein each row of the similarity matrix represents the similarity between a remote sensing image and all user description texts, and each column represents the similarity between a user description text and all remote sensing images.

[0083] S203: Calculate the similarity between each sample and the true category of the sample.

[0084] In the image retrieval text task, for each sample, a vector representation of the remote sensing image features and text features of the sample is first generated, which is referred to as the first vector. Then a vector representation of the remote sensing image features of the remote sensing avatar in the sample and the text features of the real description text is generated, which is referred to as the second vector. The real description text refers to the user description text consistent with the remote sensing image category. Finally, the similarity between the first vector and the second vector is calculated. When generating the vector representation, the remote sensing image features and the text features can be concatenated. Similarly, in the text retrieval image task, for each sample, a vector representation of the text features and remote sensing image features of the sample is first generated, which is referred to as the first vector. Then a vector representation of the text features of the user description text in the sample and the remote sensing image features of the real remote sensing image is generated, which is referred to as the second vector. The real remote sensing image refers to the remote sensing image consistent with the user description text category. Finally, the similarity between the first vector and the second vector is calculated.

[0085] S204, the cosine similarity between all remote sensing image features and text features in the same batch, that is, the similarity matrix mentioned above, is used as , and the similarity between all samples and the true category of the sample is used as the input of the above formula 2 to calculate the cross entropy loss of the image retrieval text task. It can be seen from formula 2 that in the When there are samples, the more accurately the sample matches the category, that is, the similarity with the true category. The higher the similarity with other error categories The lower it is, the smaller the negative log likelihood of this sample is; if the number of correctly matched samples in the batch is more, the smaller the average of the negative log likelihood of all samples is, that is, the cross entropy loss The smaller (closer to 0), the smaller The bigger.

[0086] S205, taking the transpose of the similarity matrix and the similarity between all samples and the true category of the samples as inputs of the above formula 2, and calculating the cross entropy loss of the text retrieval task image. This method takes the similarity matrix between the remote sensing image features and the text features as the image-text similarity, and calculates the cross entropy loss of the image retrieval text task; at the same time, taking the transpose of the above similarity matrix as the text-image similarity, and calculating the cross entropy loss of the text retrieval image task.

[0087] S206. Taking the average of the cross entropy loss of the image retrieval text task and the cross entropy loss of the text retrieval task image as the model loss.

[0088] S207: Training model parameters of the retrieval model based on the model loss.

[0089] Through the back-propagation algorithm, the model parameters are updated according to the gradient of the model loss to minimize the loss function. This process is repeated until the performance of the model on the validation set is no longer significantly improved or the preset training rounds are reached. The performance of the model is then evaluated on an independent test set, using indicators such as accuracy, recall, and F1 score to measure the effects of image retrieval text and text retrieval image tasks.

[0090] The present invention maps remote sensing images and user description texts to a unified feature space and calculates the similarity between them. This method realizes cross-modal retrieval capabilities, that is, related texts can be queried based on images, and related images can be queried based on texts. Using cosine similarity as a similarity metric and optimizing it in combination with a cross-entropy loss function can make the model more focused on improving the similarity of true matching pairs and reducing the similarity of unmatched pairs during training, thereby improving the accuracy of retrieval. Since this method does not rely on specific image or text content, but instead learns general feature representations and similarity metrics, it has strong generalization capabilities and can be applied to cross-modal retrieval tasks in different fields and scenarios. This method can process large-scale data sets in a batch processing manner, and utilize the parallel computing capabilities of modern deep learning frameworks to accelerate the training and reasoning processes of the model.

[0091] The remote sensing image cross-modal retrieval device provided by the present invention is described below. The remote sensing image cross-modal retrieval device described below and the remote sensing image cross-modal retrieval method described above can be referred to each other. Figure 7 As shown, the device comprises: The acquisition module 301 is used to acquire user description text, multiple remote sensing images and a pre-built retrieval model; wherein the retrieval model includes a remote sensing image cross-modal retrieval network and a text feature extraction network; An image feature extraction module 302 is used to extract global image features and multi-scale features of each remote sensing image based on a remote sensing image feature extraction network, fuse and enhance the multi-scale features, and fuse the fused and enhanced multi-scale features with the global image features to obtain remote sensing image features; A text feature extraction module 303, used to extract text features of the user description text based on a text feature extraction network; A calculation module 304, used to calculate the similarity between the remote sensing image feature of each remote sensing image and the text feature; The screening module 305 is used to screen out remote sensing images whose similarity meets a preset condition from the multiple remote sensing images to obtain a target remote sensing image.

[0092] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the remote sensing image cross-modal retrieval method.

[0093] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-modal retrieval method for remote sensing images provided by the above-mentioned methods.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the remote sensing image cross-modal retrieval method provided by the above methods.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image cross-modal retrieval method, characterized in that: include: Obtaining user description text, multiple remote sensing images and a pre-built retrieval model; wherein the retrieval model includes a remote sensing image cross-modal retrieval network and a text feature extraction network; Extracting global image features and multi-scale features of each remote sensing image based on a remote sensing image feature extraction network, fusing and enhancing the multi-scale features, and fusing the fused and enhanced multi-scale features with the global image features to obtain remote sensing image features; Extracting text features of the user description text based on a text feature extraction network; Calculating the similarity between the remote sensing image features and the text features of each remote sensing image respectively; The remote sensing images whose similarity meets the preset conditions are screened out from the multiple remote sensing images to obtain the target remote sensing image.

2. The remote sensing image cross-modal retrieval method according to claim 1, characterized in that: The remote sensing image cross-modal retrieval network includes a remote sensing image feature extraction network, a multi-scale feature fusion module and a significant feature enhancement module; The remote sensing image feature extraction network is used to extract global image features and multi-scale features of each remote sensing image, the multi-scale features are fused and enhanced, and the fused and enhanced multi-scale features are fused with the global image features to obtain remote sensing image features, including: For each remote sensing image, perform the following steps: Based on the remote sensing image feature extraction network, the remote sensing image is subjected to feature extraction to obtain global image features and multi-scale features; Based on a multi-scale feature fusion module, the multi-scale features are fused to obtain a fused visual feature with multi-scale information; Based on a salient feature enhancement module, the fused visual feature is fused with a first scale feature in the multi-scale feature to obtain a salient information feature; The remote sensing image feature is obtained by fusing the significant information feature and the global image feature.

3. The remote sensing image cross-modal retrieval method according to claim 2, characterized in that: The multi-scale feature fusion module includes a channel attention module, and the multi-scale feature also includes a second scale feature and a third scale feature; The step of fusing the multi-scale features based on the multi-scale feature fusion module to obtain a fused visual feature with multi-scale information includes: Performing dimensionality reduction on the first scale feature and the second scale feature, and fusing the third scale feature and the first scale feature and the second scale feature after dimensionality reduction to obtain a preliminary fused feature; The preliminary fusion features are processed using the channel attention module, and the processed preliminary fusion features are converted into fusion visual features with the same number of channels as the third-scale features.

4. The remote sensing image cross-modal retrieval method according to claim 2, characterized in that: The method of fusing the fused visual feature and the first scale feature in the multi-scale feature to obtain a salient information feature based on a salient feature enhancement module includes: Using the fused visual feature as a reference feature and using the first scale feature in the multi-scale feature as an enhanced feature; Processing the enhanced features, and performing element-wise multiplication of the baseline features using the processed enhanced features to generate initial information features; The initial information features are subjected to convolution, maximum pooling and spatial dimension reduction processing to obtain significant information features.

5. The remote sensing image cross-modal retrieval method according to claim 1, characterized in that: The text feature extraction network includes a text encoder, a Transformer encoder and a fully connected layer; The extracting of text features of the user description text based on a text feature extraction network includes: Using the text encoder to perform word segmentation and word vector encoding on the user description text to obtain a word encoding vector of a corresponding word block, and generate a mask vector; Inputting the word encoding vector and the mask vector into the Transformer encoder to obtain semantic features; The semantic features are converted into text features with the same dimension as the remote sensing image features based on the fully connected layer.

6. The remote sensing image cross-modal retrieval method according to claim 1, characterized in that: The retrieval model is trained in the following manner: Each remote sensing image is set as a category, and the user description text corresponding to the remote sensing image is classified into the same category; For all samples in a batch, the similarity between the remote sensing image features and the text features of the samples is calculated, and a similarity matrix is ​​constructed; wherein the samples are pairs of remote sensing images and user description texts, each row of the similarity matrix represents the similarity between a remote sensing image and all user description texts, and each column represents the similarity between a user description text and all remote sensing images; Calculate the similarity between each sample and the true category of the sample; Taking the similarity matrix and the similarities between all samples and the true categories of the samples as inputs of a pre-constructed loss function, calculating the cross entropy loss of the image retrieval text task; Taking the transpose of the similarity matrix and the similarities between all samples and the true category of the samples as inputs of a pre-constructed loss function, calculating the cross entropy loss of the text retrieval task image; Taking the average of the cross entropy loss of the image retrieval text task and the cross entropy loss of the text retrieval task image as the model loss; Model parameters of the retrieval model are trained based on the model loss.

7. A remote sensing image cross-modal retrieval device, characterized in that: include: An acquisition module, used to acquire user description text, multiple remote sensing images and a pre-built retrieval model; wherein the retrieval model includes a remote sensing image cross-modal retrieval network and a text feature extraction network; An image feature extraction module is used to extract global image features and multi-scale features of each remote sensing image based on a remote sensing image feature extraction network, fuse and enhance the multi-scale features, and fuse the fused and enhanced multi-scale features with the global image features to obtain remote sensing image features; A text feature extraction module, used for extracting text features of the user description text based on a text feature extraction network; A calculation module, used for respectively calculating the similarity between the remote sensing image feature of each remote sensing image and the text feature; The screening module is used to screen out remote sensing images whose similarity meets preset conditions from the multiple remote sensing images to obtain a target remote sensing image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the remote sensing image cross-modal retrieval method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the remote sensing image cross-modal retrieval method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the remote sensing image cross-modal retrieval method according to any one of claims 1 to 6 is implemented.

Images