Remote sensing image-text retrieval method based on channel dimension multi-scale attention
Patent Information
- Application Number
- CN202611077152.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-20
- Publication Date
- 2026-10-20
AI Technical Summary
[0005]本发明的目的在于解决随着遥感用户与遥感影像数据量的爆炸式增长，如何根据用户个性化的自然语言描述为其精准检索感兴趣的遥感影像以及如何根据用户给予的遥感影像生成自然语言描述的问题，同时缓解提升跨模态检索精度的同时伴随的模块参数量过大、计算效率低的问题
[0050]1、针对现有遥感图像-文本检索方法中多尺度特征融合与跨模态对齐耦合性差、模块参数量大的问题，本发明提出了一种适用于跨模态检索的通道维度多尺度注意力融合方法，通过通道分组、双分支并行变换及跨组矩阵点积融合，在保持轻量化的同时实现通道维度的多尺度语义建模，为遥感数据智能检索提供了新的技术路径。本发明在保持检索精度的前提下避免了传统方法全连接层带来的参数冗余，显著降低了模型参数量与计算开销，实现了检索精度与模型效率的优化平衡。
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention belongs to the field of cross-modal retrieval technology, specifically relating to a remote sensing image-text retrieval method based on channel-dimensional multi-scale attention. Background Technology
[0002] With the rapid development of aerospace remote sensing technology and the dense deployment of high-resolution Earth observation systems and commercial remote sensing satellite constellations, the amount of remote sensing data generated globally every day has reached the petabyte level, covering multiple modalities such as images, text, and radar. This multimodal remote sensing data has been widely used in key areas such as urban planning, environmental monitoring, disaster emergency response, and national defense security, becoming a core foundational data source supporting the construction of digital earth and smart cities.
[0003] However, facing massive and heterogeneous remote sensing data, efficiently and accurately retrieving the information users need has become a key bottleneck restricting its in-depth application. Cross-modal retrieval technology allows users to retrieve one type of data (such as natural language) from another type of data (such as remote sensing images), realizing a leap from "data retrieval" to "knowledge retrieval," significantly improving information acquisition efficiency and lowering the barrier to entry. This technology has shown broad application prospects in scenarios such as ground feature interpretation, geographic information updates, and intelligent question answering.
[0004] Nevertheless, in remote sensing image-text cross-modal retrieval tasks, due to factors such as large differences in the scale of ground objects in remote sensing images, complex scene semantics, and difficulties in cross-modal feature alignment, existing methods still face challenges such as low retrieval accuracy and excessive module parameters, making it difficult to balance performance and efficiency in practical deployments, which seriously restricts the application of this technology. Summary of the Invention
[0005] The purpose of this invention is to address the challenges of accurately retrieving remote sensing images of interest based on users' personalized natural language descriptions, and generating natural language descriptions from user-provided images, amidst the explosive growth in remote sensing users and the amount of remote sensing image data. Simultaneously, it aims to alleviate the problems of excessive module parameters and low computational efficiency that accompany improving cross-modal retrieval accuracy. To this end, this invention proposes a remote sensing image-text retrieval method based on channel-dimensional multi-scale attention.
[0006] The technical solution adopted in this invention is as follows:
[0007] A remote sensing image-text retrieval method based on channel-dimensional multi-scale attention includes the following steps:
[0008] Step 1: Obtain the standard dataset required for the remote sensing image-text retrieval task. The standard dataset includes remote sensing image samples, natural language text description samples, and matching labels between remote sensing images and natural language text.
[0009] Step 2: Construct a remote sensing image-text retrieval network model; the remote sensing image-text retrieval network model includes an image encoder, a text encoder, a cross-modal alignment module, and a channel attention module;
[0010] Step 3: Train the remote sensing image-text retrieval network model using a standard dataset;
[0011] Step 4: Input all remote sensing images and texts involved in the user's retrieval into the image encoder and text encoder of the remote sensing image-text retrieval network model, respectively. The image encoder outputs multi-scale features and fuses them. The image features are obtained through the channel attention module, and the text encoder outputs global text features. Then, calculate the cosine similarity between each image feature and each global text feature.
[0012] Step 5: For the task of retrieving text from an image, sort the cosine similarity between the image features of the image to be retrieved and the global text features of each candidate text in descending order, and return the top K texts; for the task of retrieving images from text, sort the cosine similarity between the global text features of the text to be retrieved and the image features of each candidate image in descending order, and return the top K images.
[0013] Furthermore, the image encoder uses a pre-trained ResNet50 as the backbone network, and the image encoder operates as follows:
[0014] (1) Adjust the size of the input image to 224×224×3, normalize the pixel values, and obtain the preprocessed image tensor. Set batch size B;
[0015] (2) Convert the preprocessed image tensor The input is a ResNet50, which undergoes initial downsampling to reduce the spatial resolution to 1 / 4 of the input. Then, four processing layers sequentially extract four output feature maps. , , , The spatial resolution is reduced to 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the input, respectively;
[0016] (3) To , , Using max pooling, Average pooling is used; after pooling, each output feature map is flattened into a one-dimensional vector with shapes (B,256), (B,512), (B,1024), and (B,2048), respectively.
[0017] (4) Four independent linear fully connected layers are used to process the four flattened one-dimensional vectors respectively to obtain four image feature vectors. , , , All of them have a shape of (B, 768).
[0018] Furthermore, the text encoder uses a pre-trained BERT-base-uncased model as the backbone network, and the text encoder works as follows:
[0019] (1) Preprocess the input text, add [CLS] and [SEP] markers, uniformly pad or truncate to a fixed length L, and generate input_ids and attention_mask. Input_ids is the numerical index sequence of each word in the text in the BERT vocabulary, and attention_mask is a 0 / 1 mask sequence of the same length as input_ids, which is used to distinguish effective words from padding positions in the multi-head self-attention calculation of BERT.
[0020] (2) Input input_ids and attention_mask into the BERT-base-uncased model and output sequence features sequence_output;
[0021] (3) Extract the vector corresponding to the [CLS] marker at position 0 from sequence_output and use it as the global feature vector of the text. ;
[0022] (4) Extract the vectors corresponding to positions 1 to L-1 as local feature vectors of the text. .
[0023] Furthermore, the cross-modal alignment module performs cross-modal alignment on each image feature vector output by the image encoder. For k=1,2,3,4, perform the following operations:
[0024] Will Expanded into query vector ;
[0025] Will as a key vector Sum value vector ;
[0026] By analyzing local features of the text Perform multi-head cross-attention to obtain aggregated text features; among which, the query vector... Guided by image features, starting from local text features It retrieves and aggregates semantic information related to image content to achieve fine-grained alignment between images and text; through key vectors... With query vector Calculate attention weights to measure the relevance of each word in the text to image features; value vector Based on attention weights, the semantic information of related words is weighted and aggregated to obtain aggregated text features;
[0027] Aggregate text features and Add them together to obtain the merged text representation;
[0028] Integrating text representation with Element-wise multiplication is performed, followed by a fully connected layer and sigmoid activation, to output a matching score matrix. .
[0029] Furthermore, the channel attention module operates as follows:
[0030] (1) Combine the four image feature vectors , , , Adding them together yields the fused feature tensor. The shape is (B, 768); The semantic space is uniformly divided into G=32 semantic subspaces along the channel dimension, with each group containing 768 / G=24 channels. The feature representation after grouping is as follows: ;
[0031] (2) Perform parallel processing of the grouped features using channel scaling branch and channel transformation branch;
[0032] In the channel scaling branch, horizontal and vertical global average pooling are performed on each group of features. The horizontal and vertical pooling results are then concatenated and fused using channel blending projection. The fused result is then split into two components, which are activated by Sigmoid and multiplied to obtain a group-level scaling factor. This scaling factor is then multiplied element-wise with the original grouped features. The multiplied features are then grouped and normalized to output linear projection features. ;
[0033] In the channel transformation branch, a learnable fully connected projection matrix is used to perform linear mixing of the intra-channel features of each group, outputting linear projection features. ; and Same shape;
[0034] (3) To and Global average pooling is performed separately to obtain the global description of each group, followed by Softmax normalization, and then... and Reshape the matrix into matrix form, calculate the interaction between the two matrices through matrix dot product, and then add the results to generate the comprehensive attention weights. ;
[0035] (4) After Sigmoid activation, and grouped features Element-wise multiplication yields the weighted features, which are then restored to their shape (B, 768). Residual connections are then used to fuse the original features. Adding the weighted features together yields the enhanced fusion features. , The shape is (B, 768).
[0036] Furthermore, in step 3, the AdamW optimizer is used to perform end-to-end training on the remote sensing image-text retrieval network model; batch size B=32, initial learning rate... The weight decay coefficient is 0.01; cosine annealing learning rate scheduling or ReduceLROnPlateau strategy is used; the total number of training rounds is 80; in each iteration, the total loss is calculated. All trainable parameters are updated through backpropagation until the network converges.
[0037] Of which, total loss The calculation method is as follows:
[0038]
[0039] In the formula, The value range is 5 to 25. The value range is 0 to 30;
[0040] The multi-scale contrast loss is calculated as follows:
[0041]
[0042] Where B is the batch size. For the first The matching score of a positive sample pair. For the first Image and the first The matching score of the sentence text;
[0043] The cross-scale consistency loss is calculated as follows:
[0044]
[0045] Where KL represents relative entropy, which is used to measure the difference between two probability distributions, and softmax represents softmax normalization;
[0046] The triplet loss is calculated as follows:
[0047]
[0048] in, This represents the margin, with a value ranging from 0.4 to 0.6. It is the first Enhanced fusion features of each image Indicates the first Positive sample text features corresponding to each image Indicates the first Negative sample text features corresponding to each image It is the cosine similarity of positive sample pairs, with an expected value close to 1. It is the cosine similarity of negative sample pairs, with an expected value approaching 0.
[0049] Compared with the prior art, the present invention has the following advantages:
[0050] 1. To address the problems of poor coupling between multi-scale feature fusion and cross-modal alignment, and large module parameter count in existing remote sensing image-text retrieval methods, this invention proposes a channel-dimensional multi-scale attention fusion method suitable for cross-modal retrieval. Through channel grouping, bi-branch parallel transformation, and cross-group matrix dot product fusion, multi-scale semantic modeling of the channel dimension is achieved while maintaining lightweight design, providing a new technical path for intelligent retrieval of remote sensing data. This invention avoids parameter redundancy caused by fully connected layers in traditional methods while maintaining retrieval accuracy, significantly reducing the number of model parameters and computational overhead, and achieving an optimized balance between retrieval accuracy and model efficiency.
[0051] 2. This invention fully considers the drawback of general models in simultaneously capturing coarse-grained semantic strength and fine-grained structural dependencies. It utilizes the semantic association between multi-scale image features and text features along the channel dimension to achieve multi-level semantic modeling from the global to the local. Users only need to input existing image / text data, and the model will calculate the similarity between the image and text, returning the top K closest text / images matching the user's needs as search results. This method employs channel grouping, pointwise linear projection, and matrix dot product fusion, avoiding the parameter redundancy caused by large-scale fully connected layers in traditional methods. Therefore, the retrieval process is extremely lightweight and highly suitable for the remote sensing field. Attached Figure Description
[0052] Figure 1This is a schematic diagram of remote sensing image-text retrieval in an example of the present invention.
[0053] Figure 2 This is a schematic diagram illustrating the principle of remote sensing image-text retrieval in an example of the present invention.
[0054] Figure 3 This is a schematic diagram of the multi-scale image and text feature extraction model in an example of the present invention.
[0055] Figure 4 This is a schematic diagram of the multi-scale cross-modal alignment model in an example of the present invention.
[0056] Figure 5 This is a schematic diagram of the channel-based multi-scale attention network model in an example of the present invention. Detailed Implementation
[0057] The specific embodiments of the present invention will now be described with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention.
[0058] A remote sensing image-text retrieval method based on channel-dimensional multi-scale attention, such as... Figure 1 and 2 As shown, it includes the following steps:
[0059] (1) Obtain the RSITMD remote sensing public dataset as the standard dataset required for the remote sensing image-text retrieval task. This dataset contains remote sensing image samples, natural language text description samples, and matching labels (i.e., matching positive sample pairs) between remote sensing images and natural language text.
[0060] (2) Construct a remote sensing image-text retrieval network model, including an image encoder, a text encoder, a cross-modal alignment module, and a channel attention module; such as Figure 3 As shown, the image encoder is used for remote sensing image feature extraction, and the text encoder is used for natural language text feature extraction.
[0061] The image encoder uses a pre-trained ResNet50 as the backbone network (ResNet50 is pre-trained on the ImageNet dataset) to extract feature maps at four scales: 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the input spatial resolution. , , , After global pooling and fully connected layers, the data is mapped to a unified feature dimension, outputting multi-scale image feature vectors. , , , The specific method is as follows:
[0062] (201) Adjust the input image size to 224×224×3, normalize the pixel values, and set the batch size B according to the hardware conditions.
[0063] (202) Convert the preprocessed image tensor Inputting a ResNet50 network, after initial downsampling (7×7 convolution with a stride of 2, batch normalization, ReLU, and 3×3 max pooling with a stride of 2), we obtain a feature map with a spatial resolution reduced to 1 / 4 of the input. Then, we extract the output feature maps from four key stages sequentially:
[0064] Layer 1: Output feature map The spatial resolution is reduced to 1 / 8 of the input. This layer's feature map retains rich spatial details and is suitable for capturing small-scale targets (such as vehicles and trees).
[0065] Layer 2: Output feature map The spatial resolution is reduced to 1 / 16 of the input. This layer's feature map incorporates semantic information at a medium scale.
[0066] Layer 3: Output Feature Map The spatial resolution is reduced to 1 / 32 of the input. This layer's feature map has strong semantic abstraction capabilities and is suitable for capturing large-scale targets (such as buildings and airports).
[0067] Layer 4: Output Feature Map The spatial resolution is reduced to 1 / 64 of the input. This layer's feature map has the highest semantic level but the coarsest spatial details, making it suitable for representing global scene categories.
[0068] The above four feature maps , , , This refers to the extracted multi-scale visual features, which correspond to different granularities from fine to coarse.
[0069] (203) Perform global pooling on each feature map: , , Using max pooling, Average pooling is used. After pooling, each feature map is flattened into a one-dimensional vector, resulting in shapes (B,256), (B,512), (B,1024), and (B,2048), respectively.
[0070] (204) Use four independent fully connected layers (linear layers) to process the vectors at each scale respectively:
[0071]
[0072]
[0073]
[0074]
[0075] Finally, the image encoder outputs four multi-scale image feature vectors. , , , They encode multi-level information ranging from local details to global semantics.
[0076] The text encoder uses a pre-trained BERT-base-uncased model as the backbone network (the BERT-base-uncased model is pre-trained on large-scale unlabeled text) to construct the text encoder, which is used to extract global and local text feature vectors from the input natural language description. and The specific method is as follows:
[0077] (211) Preprocess the input text: Add [CLS] and [SEP] markers, uniformly pad or truncate to a fixed length L=31 (which can be adjusted according to the dataset), and generate input_ids and attention_mask. Among them, input_ids is the sequence of numerical indices corresponding to each word in the text in the BERT vocabulary ([CLS] and [SEP] also correspond to specific indices respectively), and attention_mask is a 0 / 1 mask sequence of the same length as input_ids, which is used to distinguish effective words (marked as 1) and padding bits (marked as 0) in the multi-head self-attention calculation of BERT, to ensure that the model only focuses on the actual text content.
[0078] (212) Input input_ids and attention_mask into the BERT-base-uncased model and output sequence features sequence_output.
[0079] (213) Extract the vector corresponding to position 0 ([CLS] marker) from sequence_output as the global feature vector of the text. .
[0080] (214) Extract the vectors corresponding to positions 1 to L-1 as local feature vectors of the text. These vectors correspond to the semantic representation of each word (excluding [CLS]) in the input text.
[0081] like Figure 4As shown, the cross-modal alignment module is used to perform cross-modal alignment between remote sensing images and natural language text, aligning the feature vectors of the remote sensing images at each scale. Perform the following operations:
[0082] Will Expanded into query vector , where the query vector Used in cross-attention, guided by image features, to extract local text features It retrieves and aggregates semantic information related to image content to achieve fine-grained alignment between images and text;
[0083] Will as a key vector Sum value vector ,in Used with query vectors Calculate attention weights to measure the relevance of each word in the text to image features. Based on these weights, the semantic information of related words is weighted and aggregated into the output;
[0084] By analyzing local features of the text Perform multi-head cross-attention to obtain aggregated text features;
[0085] Aggregate text features and Add them together to obtain the merged text representation;
[0086] Integrating text representation with Element-wise multiplication is performed, followed by a fully connected layer and sigmoid activation, to output a matching score matrix. .
[0087] The four matching matrices will serve as inputs for subsequent loss function calculations.
[0088] like Figure 5 As shown, the channel attention module first adds the four multi-scale image feature vectors element by element to form a fused feature tensor. Subsequently, the fused features are uniformly divided into G semantic subspaces along the channel dimension, and the grouped features are represented as follows: This achieves independent decoupling of different semantic concepts. Based on this, a dual-branch parallel transformation architecture is adopted: the first branch is the channel scaling branch, which performs horizontal and vertical global context aggregation, dual-component decoupling activation, and grouping normalization on each set of features to obtain linear projection features. The first branch is used to evaluate the overall importance of each semantic subspace; the second branch is the intra-group linear projection branch, which uses a learnable fully connected matrix to linearly mix the channels within each group to obtain linear projection features. This yields fine-grained semantic structure dependencies. Next, through a cross-group semantic fusion mechanism, the outputs of the two branches are subjected to global pooling, softmax normalization, and matrix dot product interaction to establish global dependencies between different semantic subspaces, generating comprehensive attention weights. Finally, the weights are activated by the Sigmoid function and multiplied element-wise with the original grouped features. This is then followed by spatial dimension recovery and residual connection (adding the weighted features to the initial fused features) to output the enhanced fused features. The entire process, while maintaining a lightweight parameter count, achieves progressive semantic enhancement from coarse-grained intensity evaluation to fine-grained structural modeling and then to cross-group global interaction, which can significantly improve feature discrimination capabilities. The specific steps are as follows:
[0089] (221) Add the feature vectors of the four multi-scale images to obtain the fused feature tensor. The shape is (B, 768):
[0090]
[0091] Then, The semantic space is divided into G=32 subspaces along the channel dimension, with each group containing 768 / G=24 channels. The features after grouping are represented as follows: The organization consists of B×G independent groups, each containing feature vectors with 24 channels. The specific value of G needs to be adapted to the change in channel dimension. At the same time, considering the finer granularity of the semantic subspace, G can be increased, which will help decouple more refined semantic concepts, but will increase the computational cost because the number of groups increases, and the number of convolution and normalization operations will increase.
[0092] (222) The grouped features are subjected to a two-branch parallel transformation, including channel scaling and channel transformation, which correspond to coarse-grained semantic strength evaluation and fine-grained semantic modeling, respectively.
[0093] In the channel scaling part, horizontal global average pooling is performed on each group of features:
[0094]
[0095] And vertical global average pooling:
[0096]
[0097] Then, the horizontal and vertical pooling results are concatenated, and intermediate features are obtained by feature fusion through channel blending projection. :
[0098]
[0099] Then Split into two components , After being activated by Sigmoid, the groups are multiplied to obtain the group-level scaling factor. :
[0100]
[0101] Then, the scaling factor is multiplied element-wise by the original grouped features, and finally, the weighted features are grouped and normalized to output the result. :
[0102]
[0103] In the channel transformation part, a learnable fully connected projection matrix is applied to each group of features to achieve linear mixing of the channels within the group, and then the linear projection features are output. :
[0104]
[0105] Two branches process the same input in parallel Output and They have the same shape.
[0106] (223) Cross-group semantic fusion: for and Global average pooling is performed separately to obtain the global description of each group, which is then normalized using Softmax. and Remodeled into matrix form:
[0107]
[0108]
[0109] The interaction between the two branch outputs is then calculated using matrix dot product, and the results from both paths are summed to generate the overall attention weight. :
[0110]
[0111] This weight captures the global dependencies between different semantic subspaces.
[0112] (224) Feature enhancement: After Sigmoid activation, and grouped features Element-wise multiplication yields the weighted features, which are then restored to their shape (B, 768). Residual connections are then used to fuse the original features. Weighted features (optionally passed through a fully connected layer) The two components are added together to obtain the final enhanced fusion feature. :
[0113]
[0114] The shape is (B, 768).
[0115] The lightweight analysis is as follows:
[0116] To verify the lightweight advantage of this method, the parameter count of the aforementioned channel-dimensional attention module is compared with that of widely used channel attention modules in existing technologies (such as the SENet structure). With the same input dimension (C=768), the SENet module (containing two fully connected layers) has approximately 590k trainable parameters, while the module of this invention (with G=32 groups) has approximately 200k trainable parameters, representing a reduction of approximately 66%. This comparison shows that this invention fundamentally avoids large-scale parameter redundancy by replacing traditional fully connected layers with channel grouping and point-by-point linear projection. From a computational complexity perspective, the core operations of this invention's module are concentrated in linear projection within groups and cross-group matrix dot products. Its floating-point operations (FLOPs) are linearly related to the number of input channels, while those of traditional fully connected layers are quadratic. Therefore, the computational efficiency advantage of this invention is more significant in scenarios with a large number of channels. In summary, this invention significantly reduces the model's storage and computational overhead while maintaining feature representation capabilities, providing a theoretical basis for deployment on resource-constrained platforms.
[0117] (3) The AdamW optimizer was used to train the entire network (including the image encoder, text encoder, cross-modal alignment module, and channel attention module) end-to-end. The training hyperparameters were set as follows: batch size B = 32. Initial learning rate... The weight decay factor is 0.01. Cosine annealing learning rate scheduling or the ReduceLROnPlateau strategy is used (patience=5, factor=0.5). The total number of training epochs is 80. In each iteration, the total loss is calculated. Backpropagation updates all trainable parameters until the network converges (the average retrieval accuracy mR on the validation set no longer improves).
[0118] The total loss is obtained by weighted summation of the three losses:
[0119]
[0120] and These two parameters can be adjusted reasonably, but usually... Between 5 and 25, Between 2 and 15. If the task places more emphasis on cross-scale consistency (such as the coexistence of multiple scale objectives), it can be... Raise to Quite comparable or even larger; if only single-scale performance is considered, it can be Set it to 0 (i.e., remove the loss item).
[0121] The three loss functions in the total loss are multi-scale contrast loss, cross-scale consistency loss, and triplet loss:
[0122] Multi-scale contrast loss: the matching matrix for each scale A contrastive learning constraint is applied so that the scores of positive sample pairs (diagonal elements) approach 1, and the scores of negative sample pairs (off-diagonal elements) approach 0. The calculation formula is:
[0123]
[0124] Cross-scale consistency loss: matching matrix at the largest scale To ensure cross-scale semantic consistency, the distribution of the smaller-scale matching matrix is constrained to align with the teacher signal. KL divergence is used for calculation, with the following formula:
[0125]
[0126] Triplet loss: Based on the final fused image features and text global features Construct a triplet loss to shorten the distance between matched sample pairs and widen the distance between unmatched sample pairs. The calculation formula is as follows:
[0127]
[0128] in This is the margin, which can be set to 0.5.
[0129] (4) For all remote sensing images and texts involved in the user's retrieval, input them into the image encoder and text encoder respectively. The image encoder outputs multi-scale features and fuses them. The final image features are obtained through the channel multi-scale attention module. The text encoder outputs global text features. ,calculate and The cosine similarity indicates the degree of similarity between an image and text.
[0130] In this stage, the cross-modal alignment module is no longer needed. Instead, the pre-trained image encoder, channel attention module, and text encoder are used to extract features F and T from the query image or text, respectively, and their cosine similarity is calculated.
[0131] (5) When a user searches, for an image-to-text search task, the similarity between the image features of the image to be searched and the text features of all candidate texts is calculated, and the texts are sorted in descending order of similarity. The top K texts are returned. Similarly, for a text-to-image search task, the top K images are returned. The top K texts / images will be returned to the user as search results. K is a preset positive integer, which can be set to 1, 2, 3, 4, 5, etc., according to user needs.
[0132] As is well known, the core of remote sensing image-text retrieval is modeling cross-modal semantic associations, that is, extracting visual features from remote sensing images and linguistic features from text descriptions, and achieving accurate retrieval matching through feature alignment and fusion. While existing cross-modal retrieval methods have achieved some success, their direct application to remote sensing image-text retrieval tasks still faces the following limitations:
[0133] (1) Large number of module parameters and high deployment cost. Existing methods use channel attention mechanism in multi-scale image feature fusion, which generates channel weights through fully connected layers, resulting in a large number of fusion module parameters and serious computational redundancy. This makes it difficult to deploy the model on resource-constrained platforms such as spaceborne, UAV, and edge devices, thus limiting the practical application scope of remote sensing image-text retrieval technology.
[0134] (2) Channel attention interaction methods are singular and cannot take into account both coarse-grained and fine-grained semantics. Conventional channel attention only models the inter-channel dependencies through a globally fully connected layer, lacking the ability to decouple different semantic subspaces. It cannot assess the overall importance of each group of semantics (coarse-grained) nor learn the linear correlation structure of channels within a group (fine-grained). In remote sensing images, the target scale varies greatly, and its semantic features present a complex hierarchical structure in the channel dimension, which is difficult to fully capture with a single interaction method.
[0135] (3) Lack of synergistic optimization between feature fusion and cross-modal alignment. Existing methods treat the feature fusion module and the cross-modal interaction module as independent parts. The fused features are only used for the final retrieval and fail to utilize the fused information in the multi-scale interaction process, resulting in limited expressive power and retrieval performance of the overall model. In addition, the alignment distribution of features at different scales lacks consistency constraints, which can easily lead to semantic conflicts between scales.
[0136] To address the aforementioned issues, this invention innovatively proposes an efficient channel-dimensional multi-scale attention module based on a holistic multi-scale retrieval architecture. Specifically: First, multi-scale image features are added and fused; then, the image is divided into multiple semantic subspaces along the channel dimension to achieve semantic decoupling. Next, a dual-branch parallel transformation architecture is employed—a channel scaling branch is used to evaluate the overall importance of each semantic group, while an intra-group linear projection branch is used to learn the linear dependencies between channels within the group. Then, a cross-group semantic fusion mechanism is used to establish global dependencies between different semantic subspaces using matrix dot products, generating comprehensive attention weights. Finally, enhanced fused features are output through residual connections. The entire process achieves progressive semantic enhancement from coarse-grained intensity evaluation to fine-grained structural modeling and then to cross-group global interaction.
[0137] In summary, this invention implements a remote sensing image-text retrieval method based on channel-dimensional multi-scale attention. First, a pre-trained ResNet50 is used to extract image feature maps at four scales, which are then mapped to a unified dimension via global pooling and fully connected layers to obtain multi-scale image feature vectors. A pre-trained BERT is used to extract global and local text features, representing sentence-level semantics and word-level details respectively, effectively addressing the heterogeneity problem between image and text at the feature representation level. Second, a cross-modal alignment module is designed, using image features as queries and local text features as keys and values. Multi-head cross-attention generates a multi-scale matching matrix, achieving fine-grained semantic interaction between image and text. Further, an efficient channel-dimensional multi-scale attention module is constructed. Multi-scale image features are added and fused, then grouped along channels. After parallel transformation via channel scaling and linear projection, cross-group matrix dot product fusion generates comprehensive attention weights. Finally, residual connections are used to output enhanced fused features, achieving progressive semantic enhancement from coarse-grained intensity evaluation to fine-grained structure modeling and then to cross-group global interaction. Meanwhile, a joint optimization framework of multi-scale contrastive loss, cross-scale consistency loss, and triplet loss is constructed to improve retrieval accuracy while maintaining lightweight design. Comparative experiments on public datasets such as RSITD, RSICD, and UCM Caption show that this method achieves a better balance in terms of average retrieval accuracy, parameter count, and inference speed.
[0138] This invention achieves efficient multi-scale feature fusion, fine-grained semantic interaction between images and text, and progressive semantic enhancement along the channel dimension by constructing four modules: multi-scale image and text feature extraction, cross-modal alignment, channel-dimensional multi-scale attention, and joint loss optimization. This ensures both retrieval accuracy and model lightweighting, making it particularly suitable for resource-constrained remote sensing image-text retrieval scenarios. This invention can support accurate semantic matching between massive amounts of remote sensing data and natural language descriptions.
Claims
1. A remote sensing image-text retrieval method based on channel-dimensional multi-scale attention, characterized in that, Includes the following steps: Step 1: Obtain the standard dataset required for the remote sensing image-text retrieval task. The standard dataset includes remote sensing image samples, natural language text description samples, and matching labels between remote sensing images and natural language text. Step 2: Construct a remote sensing image-text retrieval network model; the remote sensing image-text retrieval network model includes an image encoder, a text encoder, a cross-modal alignment module, and a channel attention module; Step 3: Train the remote sensing image-text retrieval network model using a standard dataset; Step 4: Input all remote sensing images and texts involved in the user's retrieval into the image encoder and text encoder of the remote sensing image-text retrieval network model, respectively. The image encoder outputs multi-scale features and fuses them. The image features are obtained through the channel attention module, and the text encoder outputs global text features. Then, calculate the cosine similarity between each image feature and each global text feature. Step 5: For the task of retrieving text from an image, sort the cosine similarity between the image features of the image to be retrieved and the global text features of each candidate text in descending order, and return the top K texts; for the task of retrieving images from text, sort the cosine similarity between the global text features of the text to be retrieved and the image features of each candidate image in descending order, and return the top K images.
2. The remote sensing image-text retrieval method based on channel-dimensional multi-scale attention according to claim 1, characterized in that, The image encoder uses a pre-trained ResNet50 as the backbone network, and the image encoder works as follows: (1) Adjust the size of the input image to 224×224×3, normalize the pixel values, and obtain the preprocessed image tensor. Set batch size B; (2) Convert the preprocessed image tensor Inputting ResNet50, the spatial resolution is reduced to 1 / 4 of the input through initial downsampling; Then, four output feature maps are extracted sequentially through four processing layers. , , , The spatial resolution is reduced to 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the input, respectively; (3) To , , Using max pooling, Average pooling is used; after pooling, each output feature map is flattened into a one-dimensional vector with shapes (B,256), (B,512), (B,1024), and (B,2048), respectively. (4) Four independent linear fully connected layers are used to process the four flattened one-dimensional vectors respectively to obtain four image feature vectors. , , , All of them have a shape of (B, 768).
3. The remote sensing image-text retrieval method based on channel-dimensional multi-scale attention according to claim 1, characterized in that, The text encoder uses a pre-trained BERT-base-uncased model as its backbone network, and the text encoder works as follows: (1) Preprocess the input text, add [CLS] and [SEP] markers, uniformly pad or truncate to a fixed length L, and generate input_ids and attention_mask. Input_ids is the numerical index sequence of each word in the text in the BERT vocabulary, and attention_mask is a 0 / 1 mask sequence of the same length as input_ids, which is used to distinguish effective words from padding positions in the multi-head self-attention calculation of BERT. (2) Input input_ids and attention_mask into the BERT-base-uncased model and output sequence features sequence_output; (3) Extract the vector corresponding to the [CLS] marker at position 0 from sequence_output and use it as the global feature vector of the text. ; (4) Extract the vectors corresponding to positions 1 to L-1 as local feature vectors of the text. .
4. The remote sensing image-text retrieval method based on channel-dimensional multi-scale attention according to claim 1, characterized in that, The cross-modal alignment module performs cross-modal alignment on each image feature vector output by the image encoder. For k=1,2,3,4, perform the following operations: Will Expanded into query vector ; Will as a key vector Sum value vector ; By analyzing local features of the text Perform multi-head cross-attention to obtain aggregated text features; among which, the query vector... Guided by image features, starting from local text features It retrieves and aggregates semantic information related to image content to achieve fine-grained alignment between images and text; through key vectors... With query vector Calculate attention weights to measure the relevance of each word in the text to image features; value vector Based on attention weights, the semantic information of related words is weighted and aggregated to obtain aggregated text features; Aggregate text features and Add them together to obtain the merged text representation; Integrating text representation with Element-wise multiplication is performed, followed by a fully connected layer and sigmoid activation, to output a matching score matrix. .
5. The remote sensing image-text retrieval method based on channel-dimensional multi-scale attention according to claim 1, characterized in that, The channel attention module operates as follows: (1) Combine the four image feature vectors , , , Adding them together yields the fused feature tensor. The shape is (B, 768); The semantic space is uniformly divided into G=32 semantic subspaces along the channel dimension, with each group containing 768 / G=24 channels. The feature representation after grouping is as follows: ; (2) Perform parallel processing of the grouped features using channel scaling branch and channel transformation branch; In the channel scaling branch, horizontal and vertical global average pooling are performed on each group of features. The horizontal and vertical pooling results are then concatenated and fused using channel blending projection. The fused result is then split into two components, which are activated by Sigmoid and multiplied to obtain a group-level scaling factor. This scaling factor is then multiplied element-wise with the original grouped features. The multiplied features are then grouped and normalized to output linear projection features. ; In the channel transformation branch, a learnable fully connected projection matrix is used to perform linear mixing of the intra-channel features of each group, outputting linear projection features. ; and Same shape; (3) To and Global average pooling is performed separately to obtain the global description of each group, followed by Softmax normalization, and then... and Reshape the matrix into matrix form, calculate the interaction between the two matrices through matrix dot product, and then add the results to generate the comprehensive attention weights. ; (4) After Sigmoid activation, and grouped features Element-wise multiplication yields the weighted features, which are then restored to their shape (B, 768). Residual connections are then used to fuse the original features. Adding the weighted features together yields the enhanced fusion features. , The shape is (B, 768).
6. The remote sensing image-text retrieval method based on channel-dimensional multi-scale attention according to claim 1, characterized in that, In step 3, the AdamW optimizer is used to train the remote sensing image-text retrieval network model end-to-end; Batch size B=32, initial learning rate The weight decay coefficient is 0.01; cosine annealing learning rate scheduling or ReduceLROnPlateau strategy is adopted. The total number of training rounds is 80; in each iteration, the total loss is calculated. All trainable parameters are updated through backpropagation until the network converges. Of which, total loss The calculation method is as follows: In the formula, The value range is 5 to 25. The value range is 0 to 30; The multi-scale contrast loss is calculated as follows: Where B is the batch size. For the first The matching score of a positive sample pair. For the first Image and the first The matching score of the sentence text; The cross-scale consistency loss is calculated as follows: Where KL represents relative entropy, which is used to measure the difference between two probability distributions, and softmax represents softmax normalization; The triplet loss is calculated as follows: in, This represents the margin, with a value ranging from 0.4 to 0.
6. It is the first Enhanced fusion features of each image Indicates the first Positive sample text features corresponding to each image Indicates the first Negative sample text features corresponding to each image It is the cosine similarity of positive sample pairs, with an expected value close to 1. It is the cosine similarity of negative sample pairs, with an expected value approaching 0.