A Remote Sensing Image and Text Retrieval Method and System Guided by Frequency Domain Features

Abstract

This invention relates to the fields of artificial intelligence and computer vision, and discloses a remote sensing image-text retrieval method and system based on frequency domain feature guidance. The method includes: S1, performing super-resolution enhancement on low-quality remote sensing images; performing discrete wavelet transform to decompose the images into multiple frequency domain sub-images; and obtaining multi-frequency features; S2, feeding the enhanced image and input text into a pre-trained cross-modal encoder to extract high-level features; S3, obtaining enhanced remote sensing image features through a frequency domain saliency-driven weighted fusion mechanism; S4, constructing an intermediate state feature space, and using a cross-attention mechanism to achieve fine-grained semantic interaction between text features and various image features, generating corresponding text interaction features; S5, calculating similarity in the multi-feature space, and summing the results to obtain the final image-text matching score. This invention effectively improves image detail representation and feature discrimination, enhances image-text semantic alignment accuracy, and significantly improves retrieval performance in complex remote sensing scenarios.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision, and specifically relates to a remote sensing image and text retrieval method and system based on frequency domain features. Background Technology

[0002] Marine remote sensing data, characterized by its all-day, all-weather, and all-directional nature, is widely used in important fields such as pollution monitoring and target identification. Cross-modal retrieval of marine remote sensing data aims to represent different marine objects by retrieving information from different modalities, effectively capturing the matching relationships between data from different modalities. Currently, traditional remote sensing cross-modal text-image retrieval (RSCTIR) has received unprecedented attention in recent years due to its advantages of flexible input and efficient querying of massive amounts of remote sensing images.

[0003] Existing cross-modal image retrieval methods for marine remote sensing images have four main shortcomings: First, spatial domain feature modeling mixes structural and textural information, resulting in insufficient detail representation capabilities. Traditional methods directly encode the original image in the spatial domain, failing to explicitly distinguish between low-frequency global structure and high-frequency detailed texture. For example, in marine remote sensing image scenes, small targets (such as fishing boats, buoys, and offshore platforms) often occupy only a few pixels, with only slight spectral differences from the background. Under uniform convolution or Transformer processing, these fine-grained signals are treated as noise and weakened. Therefore, the model cannot maintain the energy distribution of details in the feature space, causing high-frequency texture information to be lost during encoding, reducing the discriminative power and target sensitivity of retrieval.

[0004] Second, there is a lack of effective modeling for multi-scale features. Existing methods use VIT to extract features at a uniform scale, making it difficult to capture the collaborative information between large-scale backgrounds and small-scale targets that coexist in remote sensing images. Images contain both large areas of sea and land structures, as well as sparsely distributed ship targets. Although VIT has global modeling capabilities, it cannot effectively balance the information fusion between large-scale scene structures and micro-targets at a single scale, resulting in a lack of modeling sensitivity to multi-scale information.

[0005] Third, the semantics of the image-text modality lacks deep interaction, resulting in a large semantic gap. Images and text are fed into the shared space through separate encoders, lacking fine-grained alignment mechanisms between modalities. This is especially prone to mismatches when image backgrounds are complex and semantics are unclear. For example, when a user inputs the text "Several container ships are docked at the port," there may be multiple images with the structures of "ships" and "port" in the background. However, without fine-grained interaction at the image-text fragment level in the shared semantic space, the model cannot accurately distinguish between "container ships" and "fishing boats," and may mistakenly match images with similar semantics but different semantic focuses as positive samples.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] To address or at least alleviate one or more of the above problems, a remote sensing image and text retrieval method and system based on frequency domain features is provided.

[0008] To achieve the above objectives, according to a first aspect of the present invention, a remote sensing image and text retrieval method based on frequency domain features is provided, comprising: S1: Perform super-resolution enhancement on the input low-quality remote sensing image to obtain an enhanced image; perform discrete wavelet transform on the enhanced image to decompose it into multiple frequency domain sub-images; extract features from each frequency domain sub-image to obtain corresponding multi-frequency features; S2: Input the enhanced image and the input text data into the pre-trained cross-modal encoder respectively to extract image features and text features; S3: Perform adaptive weighting and spatial attention fusion on the multi-frequency features based on frequency domain saliency to obtain the fused enhanced remote sensing image features; Specifically, in the channel dimension, channel attention is applied to each of the multi-frequency features and weight coefficients are assigned, and adaptive weighted summation is performed to obtain the weighted frequency domain fusion feature; the frequency domain fusion feature and the image feature are concatenated in the spatial dimension, and a saliency response map is generated through a fully connected layer and a Sigmoid function; based on the saliency response map, the frequency domain fusion feature and the image feature are fused, and then averaged with the image feature to obtain the final enhanced remote sensing image feature; S4: Construct an intermediate state feature space, and through a cross-attention mechanism, enable the text features to perform fine-grained semantic interaction with each image feature to generate a set of text interaction features aligned with the image features; S5: In multiple feature spaces, calculate the similarity between the enhanced remote sensing image features and each text interaction feature, as well as between the image features and the text features, and summarize to obtain the final image-text matching score.

[0009] To achieve the above objectives, according to a second aspect of the present invention, a remote sensing image and text retrieval system guided by frequency domain features is provided, the remote sensing image and text retrieval system comprising: A frequency domain information-guided feature extraction module is used to perform super-resolution enhancement on the input low-quality remote sensing image to obtain an enhanced image; the enhanced image is subjected to discrete wavelet transform to decompose it into multiple frequency domain sub-images containing low-frequency structures and high-frequency details in different directions; features are extracted from each frequency domain sub-image to obtain corresponding multi-frequency features; The image and text feature extraction module is used to input the enhanced image and the input text data into a pre-trained cross-modal encoder to extract image features and text features, respectively. A frequency domain saliency-driven multi-frequency feature weighting and spatial attention fusion module is used to perform adaptive weighting and spatial attention fusion on the multi-frequency features based on frequency domain saliency to obtain fused enhanced image features; In the channel dimension, channel attention is applied to each of the multi-frequency features and weight coefficients are assigned. Adaptive weighted summation is then performed to obtain the weighted frequency domain fusion feature. The frequency domain fusion feature is then concatenated with the image feature in the spatial dimension, and a saliency response map is generated through a fully connected layer and a Sigmoid function. Based on the saliency response map, the frequency domain fusion feature is fused with the image feature, and then averaged with the image feature to obtain the final enhanced remote sensing image feature. The intermediate state feature space module is used to construct the intermediate state feature space. Through the cross attention mechanism, the text features and each image feature are used to perform fine-grained semantic interaction to generate a set of text interaction features aligned with the image features. The similarity matching module is used to calculate the similarity between the enhanced image features and each text interaction feature, as well as between the image features and the text features, in multiple feature spaces, and summarize them to obtain the final image-text matching score.

[0010] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: This invention combines super-resolution enhancement and frequency domain feature extraction to significantly improve the detail representation and feature recognition capabilities of remote sensing images. For example, traditional remote sensing image retrieval methods typically encode the original image directly in the image domain without enhancing its structural details. This results in small targets (such as ships, dock structures, buoys, etc.) or texture structures in the image not being fully captured, affecting modal alignment accuracy. This invention introduces super-resolution image reconstruction, enhancing the details of the low-resolution image before frequency domain modeling to improve edge sharpness and texture representation. Subsequently, the image is decomposed into four low-frequency and high-frequency sub-images in the frequency domain, capturing the structural contours and directional details in the frequency domain respectively. Compared to the original method's lack of image structure perception capability, this invention effectively enhances the modeling capability of multi-scale heterogeneous features in remote sensing images through image enhancement and wavelet decomposition mechanisms, ensuring fine-grained target representation even in complex remote sensing scenarios, thereby significantly improving cross-modal retrieval performance.

[0011] This invention proposes a multi-frequency feature adaptive weighting mechanism and intermediate-state feature space modeling to improve the accuracy and robustness of modal semantic alignment. For example, traditional methods directly map images and text to a unified embedding space after encoding, lacking deep semantic interaction between modalities, which can easily lead to mismatches, especially in complex backgrounds or semantically similar categories (such as "fishing boat" and "cargo ship"). This invention proposes a multi-frequency feature adaptive weighting mechanism in the feature fusion stage, which significantly weights the multi-frequency image features extracted from wavelet subgraphs, adjusts the attention intensity of information in different directions in the channel dimension, and focuses the spatial dimension on the semantic core region, effectively suppressing the interference of redundant background on image-text alignment. Furthermore, this invention constructs an intermediate-state feature space and uses a cross-attention mechanism to perform fine-grained semantic interaction between images and text, guiding text semantics closer to image semantics. Compared to traditional methods that lack modeling of image-text semantic correspondence, this invention significantly improves the modeling accuracy and semantic alignment robustness of complex remote sensing semantic relationships through an inter-modal semantic remapping mechanism, thereby improving overall retrieval accuracy.

[0012] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0013] The accompanying drawings, as part of this invention, are provided to further illustrate the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation thereof. Clearly, the drawings described below are merely some embodiments; those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0014] In the attached diagram: Figure 1This is a schematic diagram of the architecture of the remote sensing image and text retrieval system based on frequency domain features in this specific embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0016] Example 1: Please refer to Figure 1 This invention provides a remote sensing image and text retrieval method based on frequency domain feature guidance, comprising: S1. Perform super-resolution enhancement on the input low-quality remote sensing image to obtain an enhanced image; perform discrete wavelet transform on the enhanced image to decompose it into multiple frequency domain sub-images; extract features from each frequency domain sub-image. S2: Input the enhanced image and the input text data into the pre-trained cross-modal encoder respectively to extract image features and text features; S3: Perform adaptive weighting and spatial attention fusion on the multi-frequency features based on frequency domain saliency to obtain the fused enhanced remote sensing image features; Specifically, in the channel dimension, channel attention is applied to each of the multi-frequency features and weight coefficients are assigned, followed by adaptive weighted summation to obtain weighted frequency domain fusion features. The frequency domain fusion features are then concatenated with the image features in the spatial dimension, and a saliency response map is generated through a fully connected layer and a Sigmoid function. Based on the saliency response map, the frequency domain fusion features are fused with the image features, and then averaged with the image features to obtain the final enhanced remote sensing image features. S4: Construct an intermediate state feature space, and through a cross-attention mechanism, enable the text features to perform fine-grained semantic interaction with each image feature to generate a set of text interaction features aligned with the image features; S5: In multiple feature spaces, calculate the similarity between the enhanced remote sensing image features and each text interaction feature, as well as between the image features and the text features, and summarize to obtain the final image-text matching score.

[0017] In some embodiments, the super-resolution enhancement of the input low-quality remote sensing image to obtain the enhanced image includes: Input low-quality remote sensing images SR technology can be used to enhance low-quality remote sensing images to obtain enhanced images with richer details. , is represented as: ; in, It is a low-quality remote sensing image. It uses the Real-ESRGAN model to process low-quality remote sensing images. Using super-resolution technology, output remote sensing images containing richer details. .

[0018] In some embodiments, performing discrete wavelet transform on the enhanced image to decompose it into multiple frequency domain sub-images containing low-frequency structures and high-frequency details in different directions includes: Image enhancement using Haar wavelet basis functions Perform row and column transformations, first on the enhanced image. The specific formula for performing row transformations is as follows: Low-pass filtering is represented as: ; High-pass filtering is represented as: ; Downsampling is represented as: ; ; in, Represents the input 2D augmented image , This represents the result after row low-pass filtering. This represents the result after row high-pass filtering. and These represent the coefficients of the low-pass and high-pass filters. Indicates the index of the downsampled signal. Indicates the vertical position of the image. Indicates the horizontal position of the image; Subsequently, The specific formula for performing column transformation is as follows: Low-pass filtering is represented as: ; High-pass filtering is represented as: ; Downsampling is represented as: ; ; in, This indicates the result after low-pass filtering. This represents the result after high-pass filtering. Indicates the index of the downsampled signal; Subsequently, the obtained row-direction low-pass filtering results were processed. , Row direction high-pass filtering results Column direction low-pass filtering results High-pass filtering results in the column direction Combining these elements yields four frequency domain subgraphs: the low-frequency subgraph. Horizontal high-frequency subgraph, vertical high-frequency subgraph With diagonal high-frequency subgraph , is represented as: ; ; ; ; in, , , and This represents the grayscale sub-image obtained from the wavelet transform.

[0019] In some embodiments, inputting the frequency domain sub-graphs into the feature extraction network to obtain the corresponding multi-frequency features includes: Will , , and First, channel expansion is performed using convolutional layers to create three-channel pseudo-color images. Then, these images are input into a Transformer for feature extraction, resulting in four multi-frequency features guided by frequency domain information. The specific formulas are as follows: ; ; ; ; ; ; ; ; in, , , , This represents a pseudo-color image generated by assigning the corresponding pixel values ​​of the original grayscale image to all three channels (it is still displayed in grayscale, but meets the RGB format requirements to adapt to the RGB input format of the pre-trained Transformer model). This refers to the Vision Transformer encoder. , , , This represents the multi-frequency features extracted from the grayscale image. , This represents a three-channel pseudo-color image obtained by copying the corresponding grayscale sub-image along the three channel dimensions. This represents the index form of a three-dimensional tensor, where the first two dimensions... The corresponding spatial coordinates of the image are rows and columns, and the third dimension. Corresponding color channels Indicates the color channel index. express The values ​​are 1, 2, and 3.

[0020] To address the problems of existing methods that directly encode raw images in the spatial domain, lacking structural enhancement and multi-scale feature modeling, resulting in difficulty in recognizing small targets, edge textures, and other details, and insufficient semantic expression, this invention proposes a frequency-domain-aware structural enhancement and multi-scale modeling mechanism in the feature extraction stage. First, a super-resolution reconstruction model is introduced to pre-enhance the input remote sensing image, significantly improving the image's detail clarity and local texture contrast while maintaining global semantic consistency, allowing small targets and edge structures to be preserved in subsequent feature encoding. This step achieves pre-reconstruction from low-quality images to high-precision structural features, effectively solving the problem of missing image detail information and its impact on semantic expression. Second, frequency decomposition is performed on the enhanced image based on Discrete Wavelet Transform (DWT) to obtain... , , and Four frequency domain sub-images correspond to low-frequency structural information and three types of high-frequency detail information, respectively. By extracting features independently from different sub-images in the frequency domain, hierarchical modeling and directionality enhancement of multi-scale structures are achieved. Unlike traditional methods that use uniform scale modeling in the spatial domain, this method can adaptively allocate attention weights in the frequency domain, thereby establishing a collaborative perception relationship between the "global background and local target," effectively improving target recognition accuracy and feature robustness.

[0021] In some embodiments, the step of inputting the enhanced image and the input text data into a pre-trained cross-modal encoder (including a visual Transformer encoder for processing images and a Transformer encoder for processing text) to extract image features and text features includes: A visual Transformer encoder is used to extract enhanced images. Image features The formula is as follows: ; in, This represents a visual Transformer encoder; Then, text feature extraction is performed on the input text data. Similarly, a Transformer encoder is used to extract text features. The specific formula is as follows: ; in, This represents the input text data. This indicates a Transformer encoder.

[0022] In some embodiments, the step of performing adaptive weighting and spatial attention fusion on the multi-frequency features based on frequency domain saliency to obtain the fused enhanced remote sensing image features includes: get , , , ,in, The corresponding low-frequency components contain the main structural information of the image; while , , These correspond to high-frequency components in the horizontal, vertical, and diagonal directions, respectively, and include texture, edge, and fine-grained structural features. Since high-frequency components often contain noise or irrelevant textures, to prevent model attention shift, a frequency-domain adaptive weighting mechanism is first introduced in the channel dimension, as shown in the following formula: ; in, For frequency domain fusion features, These are weighting coefficients. It is a channel attention function. The four multi-frequency features obtained after processing by the feature extraction module guided by frequency domain information are: , , , , The values ​​are 1, 2, 3, and 4. Subsequently, and Spatial attention is used to further enhance semantically related regions through saliency response mapping, and the calculation formula is as follows: ; in, This represents the output features after spatial attention. This is the weight matrix of the fully connected layer, representing the feature concatenation operation. It is the sigmoid function. It is a bias vector. This represents element-wise multiplication; Finally and Averaging is performed to fuse information from two semantic or modalities while maintaining consistent feature scales, as shown below: ; in, This represents the enhanced remote sensing image features after fusion.

[0023] For the first time, explicit redistribution of attention energy is achieved in the frequency domain feature space. Unlike traditional spatial attention, this module performs multi-frequency feature weighting and spatial guidance based on frequency domain energy saliency, enabling the model to distinguish between "high-frequency noise" and "high-frequency semantics," and adaptively strengthen sub-band features related to text semantics. By performing attention fusion operations in the frequency domain, this invention achieves a leap from "spatial domain saliency" to "frequency saliency," allowing the model to maintain global structural consistency while adaptively focusing on multi-frequency information at the semantic level. This effectively solves the problems of strong background interference, redundant features, and semantic misalignment in remote sensing images, thereby significantly improving the semantic discrimination capability and retrieval accuracy of cross-modal retrieval.

[0024] In some embodiments, the image features, each of the multi-frequency features, and the text features are subjected to nonlinear transformations to obtain transformed features; the transformed text features are used as a query matrix, and the transformed image features are used as a key matrix and a value matrix, respectively, and cross-attention calculations are performed sequentially; through the cross-attention calculations, a set of text interaction features aligned with the fine-grained values ​​of each image feature is generated, including: First of all and The input to the multilayer perceptron is adjusted to change the feature dimension and introduce non-linear expressive power, as shown in the following formula: ; ; ; ; ; ; in, and These are the weights of two fully connected layers. and It is a bias term. It is the sigmoid activation function. , , , , and These are the image and text features transformed by the multilayer perceptron; Subsequently, a cross-modal attention mechanism is employed to establish fine-grained connections between image features and text features, integrating text features... Considered Transformed image features , , , , Considered and Calculate through cross-modal attention mechanism and The similarity, i.e. the association between text words and image regions, is used to weight the similarity. That is, image region features, calculated using the following formula: ; ; ; ; ; in, It is the dimension of image features. This represents the query matrix, used to find the most relevant feature responses in the image feature space. , , , , They represent the corresponding image features respectively. , , , , The key matrix is ​​used to calculate attention weights; , , , , The Value matrix represents the values ​​of the corresponding features, indicating the actual semantic content of the features; Softmax represents the normalization exponential function, used to normalize the attention score so that the sum of the weights at all positions is 1. 、 、 、 、 These represent the text features after weighted aggregation using the cross-attention mechanism.

[0025] To address the issue that images and text are often fed into a shared space via separate encoders, lacking fine-grained alignment mechanisms between modalities, and prone to mismatches, especially when image backgrounds are complex or semantics are unclear, we construct an intermediate-state feature space module for interactive guidance. This module introduces a cross-attention mechanism within the shared space, dynamically guiding image features to align text semantics with image semantics, effectively narrowing the modal gap and improving matching accuracy. This solves the problem of the original method lacking deep interaction between image and text modal semantics and resulting in a large semantic gap.

[0026] In some embodiments, the step of calculating the similarity between the enhanced image features and each text interaction feature, and between the image features and the text features, in multiple feature spaces, and summing them to obtain the final image-text matching score includes: All generated text features , , , Enhanced remote sensing image features between, and Similarity matching is performed between them, and the specific formula is as follows: ; ; ; ; ; ; in, Represents similarity calculation, express and The similarity measure between them, correspondingly, , , , , They represent and , , 、 、 A measure of similarity between them; Final similarity score The calculation formula is as follows: .

[0027] In some embodiments, the loss function is as follows: ; in, Indicates the similarity between an image and its corresponding text. Representing the same image With negative sample text The similarity between them, that is, the semantic similarity between mismatched image-text pairs. Represents negative sample images With correct text The similarity between them, that is, another form of mismatched image-text pairs, Representation and Image Mismatched negative sample text, Representation and text Mismatched negative sample images This represents a preset boundary used to control the minimum acceptable distance difference between positive and negative samples. This represents the ReLU function.

[0028] Example 2: Based on the same inventive concept, the present invention also provides a remote sensing image and text retrieval system guided by frequency domain features, combined with... Figure 1 As shown, the remote sensing image and text retrieval system includes: A frequency domain information-guided feature extraction module is used to perform super-resolution enhancement on the input low-quality remote sensing image to obtain an enhanced image; the enhanced image is subjected to discrete wavelet transform to decompose it into multiple frequency domain sub-images containing low-frequency structures and high-frequency details in different directions; features are extracted from each frequency domain sub-image to obtain corresponding multi-frequency features; The image and text feature extraction module is used to input the enhanced image and the input text data into a pre-trained cross-modal encoder to extract image features and text features, respectively. A frequency domain saliency-driven multi-frequency feature weighting and spatial attention fusion module is used to perform adaptive weighting and spatial attention fusion on the multi-frequency features based on frequency domain saliency to obtain fused enhanced remote sensing image features; In the channel dimension, channel attention is applied to each of the multi-frequency features and weight coefficients are assigned. Adaptive weighted summation is then performed to obtain the weighted frequency domain fusion feature. The frequency domain fusion feature is then concatenated with the image feature in the spatial dimension, and a saliency response map is generated through a fully connected layer and a Sigmoid function. Based on the saliency response map, the frequency domain fusion feature is fused with the image feature, and then averaged with the image feature to obtain the final enhanced remote sensing image feature. The intermediate state feature space module is used to construct the intermediate state feature space. Through the cross attention mechanism, the text features and each image feature are used to perform fine-grained semantic interaction to generate a set of text interaction features aligned with the image features. The similarity matching module is used to calculate the similarity between the enhanced image features and each text interaction feature, as well as between the image features and the text features, in multiple feature spaces, and summarize them to obtain the final image-text matching score.

[0029] The functional implementation and specific data processing methods of each module can be found in the method steps described in Example 1, and will not be repeated here.

[0030] In summary, the innovation of this invention lies in: In the frequency domain-guided feature extraction module, addressing the problem that traditional methods directly encode the original image at a single scale in the spatial domain, leading to difficulty in identifying small targets, edge textures, and other details, and a decrease in retrieval accuracy, this invention introduces a super-resolution reconstruction model before feature extraction. This model enhances the details and reconstructs the structure of low-quality remote sensing images, making fine-grained textures more prominent. Subsequently, discrete wavelet transform is used to decompose the image into four sub-images: LL, LH, HL, and HH, corresponding to the low-frequency global structure and three types of high-frequency directional details, respectively, achieving multi-scale structural modeling in the frequency domain. This solves the problem that the original method, which only uses VIT for uniform-scale feature extraction, struggles to capture the collaborative information between the large-scale background and small-scale targets simultaneously present in remote sensing images.

[0031] In the frequency-domain saliency-driven multi-frequency feature weighting and spatial attention fusion module, traditional methods primarily rely on spatial feature correlation for attention calculations in remote sensing images, which often involve background areas such as sea surfaces, clouds, and farmland. This lack of explicit modeling of the relationship between frequency-domain feature energy and semantic relevance leads to over-attention of high-proportion but low-semantic-value background areas, resulting in attention bias. This invention introduces a saliency-weighted attention fusion mechanism in the frequency domain. By adaptively modeling the energy distribution of frequency-domain features, it dynamically suppresses high-frequency noise channels and strengthens semantically salient directional features (such as vertical and horizontal edges) in the channel dimension. Simultaneously, it introduces saliency mapping in the spatial dimension, guiding the model to focus on regions strongly correlated with the text based on frequency-domain feature energy. This mechanism achieves a leap from spatial saliency to frequency saliency, enabling the model to adaptively redistribute energy and focus semantically in the multi-frequency semantic space, effectively improving semantic discrimination ability and retrieval robustness.

[0032] In the intermediate state feature space module, we construct an intermediate state feature space to guide modal interaction. We introduce a cross-attention mechanism in the shared space, enabling image features to dynamically guide text semantics closer to image semantics, effectively narrowing the modal gap and improving matching accuracy. This addresses the problem in the original method where images and text are mostly fed into the shared space through separate encoders, lacking a fine-grained alignment mechanism between modalities, which easily leads to mismatches, especially when the image background is complex and the semantics are unclear.

[0033] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-described technical content to create equivalent embodiments without departing from the scope of the present invention. The implementation schemes in the above embodiments can also be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A remote sensing image and text retrieval method based on frequency domain features, characterized in that, include: S1: Perform super-resolution enhancement on the input low-quality remote sensing image to obtain the enhanced image; The enhanced image is subjected to discrete wavelet transform and decomposed into multiple frequency domain sub-images; Feature extraction is performed on each of the frequency domain sub-graphs to obtain the corresponding multi-frequency features; S2: Input the enhanced image and the input text data into the pre-trained cross-modal encoder respectively to extract image features and text features; S3: Perform adaptive weighting and spatial attention fusion on the multi-frequency features based on frequency domain saliency to obtain the fused enhanced remote sensing image features; Specifically, in the channel dimension, channel attention is applied to each of the multi-frequency features and weight coefficients are assigned, and adaptive weighted summation is performed to obtain the weighted frequency domain fusion feature; the frequency domain fusion feature and the image feature are concatenated in the spatial dimension, and a saliency response map is generated through a fully connected layer and a Sigmoid function; based on the saliency response map, the frequency domain fusion feature and the image feature are fused, and then averaged with the image feature to obtain the final enhanced remote sensing image feature; S4: Construct an intermediate state feature space, and through a cross-attention mechanism, enable the text features to perform fine-grained semantic interaction with each image feature to generate a set of text interaction features aligned with the image features; S5: In multiple feature spaces, calculate the similarity between the enhanced remote sensing image features and each text interaction feature, as well as between the image features and the text features, and summarize to obtain the final image-text matching score.

2. The method according to claim 1, characterized in that, The enhanced image is subjected to discrete wavelet transform and decomposed into multiple frequency domain sub-images, including: Using Haar wavelet basis functions, the enhanced image I of size H×W is subjected to row and column transformations sequentially, and each transformation is downsampled by a factor of 2, resulting in four frequency domain sub-images of size H / 2×W / 2: Low-frequency sub-image LL: Row low-pass filtering followed by column low-pass filtering, containing low-frequency structural information of the image; Horizontal high-frequency sub-image (LH): Row low-pass filtering followed by column high-pass filtering, containing high-frequency details in the horizontal direction of the image; Vertical high-frequency sub-image HL: Row high-pass filtering followed by column low-pass filtering, containing high-frequency details in the vertical direction of the image; Diagonal high-frequency sub-image HH: Row high-pass filtering followed by column high-pass filtering, containing high-frequency details in the diagonal direction of the image.

3. The method according to claim 2, characterized in that, The frequency domain subgraphs are input into the feature extraction network to obtain the corresponding multi-frequency features, including: , , and First, channel expansion is performed through convolutional layers to create three-channel pseudo-color images. Then, these images are input into a Transformer encoder for feature extraction, resulting in four multi-frequency features obtained after feature extraction guided by frequency domain information.

4. The method according to claim 3, characterized in that, The enhanced image and the input text data are respectively input into a pre-trained cross-modal encoder to extract image features and text features, including: using a visual Transformer encoder to extract the enhanced image. Image features For the input text data A Transformer encoder is used to extract text features. .

5. The method according to claim 4, characterized in that, The multi-frequency features are subjected to adaptive weighting based on frequency domain saliency and spatial attention fusion to obtain the fused enhanced image features, including: First, a frequency-domain adaptive weighting mechanism is introduced at the channel level, expressed as: ； in, For frequency domain fusion features, These are weighting coefficients. It is a channel attention function. The four multi-frequency features obtained after feature extraction guided by frequency domain information are: , , , , The values ​​are 1, 2, 3, and 4. right and Using spatial attention, it can be represented as: ； in, This represents the output features after spatial attention. It is the weight matrix of the fully connected layer. Representative feature splicing operation, It is the sigmoid function. It is a bias vector. This represents element-wise multiplication; Finally and The average is expressed as: ； in, This represents the enhanced remote sensing image features after fusion.

6. The method according to claim 5, characterized in that, The intermediate state feature space is constructed, and through a cross-attention mechanism, the text features and image features undergo fine-grained semantic interaction to generate a set of text interaction features aligned with the image features, including: The image features, the multi-frequency features, and the text features are subjected to nonlinear transformations to obtain the transformed features. The transformed text features are used as the query matrix, and the transformed image features are used as the key matrix and value matrix, respectively. Cross attention is calculated sequentially. Through the cross-attention calculation, a set of text interaction features is generated that are finely aligned with the features of each image.

7. The method according to claim 6, characterized in that, The intermediate state feature space is constructed, and through a cross-attention mechanism, the text features and image features undergo fine-grained semantic interaction to generate a set of text interaction features aligned with the image features, including: First, the image features are... The multi-frequency features , , , and the text features The transformed image features are obtained by adjusting the feature dimensions and introducing nonlinear expressive power into a multilayer perceptron. , , , , and text features ; Subsequently, a cross-attention mechanism is employed to establish fine-grained relationships between image features and text features, integrating text features... Considered Transformed image features , , , , Considered and Calculate through cross-attention mechanism and The similarity, i.e. the association between text words and image regions, is used to weight the similarity. That is, image region features, represented as: ； ； ； ； ； in, It is the dimension of image features. This represents the query matrix, used to find the most relevant feature responses in the image feature space. , , , , They represent the corresponding image features respectively. , The key matrix is ​​used to calculate attention weights; , , , , The Value matrix represents the values ​​of the corresponding features, indicating the actual semantic content of the features; Softmax represents the normalization exponential function, used to normalize the attention score so that the sum of the weights at all positions is 1. 、 、 、 、 These represent the text interaction features after weighted aggregation using the cross-attention mechanism.

8. The method according to claim 7, characterized in that, The process involves calculating the similarity between the enhanced remote sensing image features and each text interaction feature, as well as between image features and text features, within multiple feature spaces, and summarizing these similarities to obtain the final image-text matching score. This includes: All generated text interaction features 、 、 、 、 With the enhanced remote sensing image features between, and Performing similarity matching between them is represented as: ； ； ； ； ； ； in, Represents similarity calculation, express and The similarity measure between them, correspondingly, 、 、 、 、 They represent and 、 、 、 、 A measure of similarity between them; The final similarity score is obtained by summing the results. , represented as: 。 9. The method according to claim 1, characterized in that, The loss function is expressed as: ； in, Indicates the similarity between an image and its corresponding text. Representing the same image With negative sample text The similarity between them, that is, the semantic similarity between mismatched image-text pairs. Represents negative sample images With correct text The similarity between them, that is, another form of mismatched image-text pairs, Representation and Image Mismatched negative sample text, Representation and text Mismatched negative sample images This represents the preset boundary hyperparameters used to control the minimum expected distance difference between positive and negative samples. This represents the ReLU function.

10. A remote sensing image and text retrieval system guided by frequency domain features, characterized in that, The remote sensing image retrieval system includes: A frequency domain information-guided feature extraction module is used to perform super-resolution enhancement on the input low-quality remote sensing image to obtain an enhanced image; the enhanced image is subjected to discrete wavelet transform to decompose it into multiple frequency domain sub-images containing low-frequency structures and high-frequency details in different directions; features are extracted from each frequency domain sub-image to obtain corresponding multi-frequency features; The image and text feature extraction module is used to input the enhanced image and the input text data into a pre-trained cross-modal encoder to extract image features and text features, respectively. A frequency domain saliency-driven multi-frequency feature weighting and spatial attention fusion module is used to perform adaptive weighting and spatial attention fusion on the multi-frequency features based on frequency domain saliency to obtain fused enhanced remote sensing image features; In the channel dimension, channel attention is applied to each of the multi-frequency features and weight coefficients are assigned. Adaptive weighted summation is then performed to obtain the weighted frequency domain fusion feature. The frequency domain fusion feature is then concatenated with the image feature in the spatial dimension, and a saliency response map is generated through a fully connected layer and a Sigmoid function. Based on the saliency response map, the frequency domain fusion feature is fused with the image feature, and then weighted and averaged with the image feature to obtain the final enhanced remote sensing image feature. The intermediate state feature space module is used to construct the intermediate state feature space. Through the cross attention mechanism, the text features and each image feature are used to perform fine-grained semantic interaction to generate a set of text interaction features aligned with the image features. The similarity matching module is used to calculate the similarity between the enhanced image features and each text interaction feature, as well as between the image features and the text features, in multiple feature spaces, and summarize them to obtain the final image-text matching score.

Images