A remote sensing image language description generation method based on location awareness and reinforcement learning

Abstract

The application discloses a remote sensing image language description generation method based on position perception and reinforcement learning. First, a multi-scale feature extraction module is used to obtain global image embedding, CLIP image embedding containing language features and regional image embedding of the remote sensing image respectively. Second, position perception attention processing is performed, the global and regional features are compressed and matrix interaction is performed to generate regional enhanced features containing spatial topological relationship. Then, multi-scale feature fusion is performed, the attention mechanism with random drop strategy and the gating mechanism are used to deeply fuse the regional enhanced features and the CLIP image embedding. Finally, the multi-scale fused features are input into an improved Transformer remote sensing image language description model to generate accurate and spatially logical remote sensing image natural language description. In addition, the application adopts a supervised pre-training and reinforcement learning fine-tuning combined training mode to improve the generation accuracy of the model.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and computer vision, specifically to a method for generating language descriptions of remote sensing images based on location awareness and reinforcement learning. This method can be widely applied to scenarios such as intelligent interpretation of remote sensing images, automated annotation of geographic information systems, and analysis of Earth observation data. Background Technology

[0002] Remote sensing image description generation is a highly challenging cross-modal task. Its goal is to establish a semantic mapping between the visual content of remote sensing images and their natural language descriptions, enabling computers to "describe images" like humans, automatically generating accurate, fluent sentences rich in information about ground features. With the rapid development of Earth observation technology, massive amounts of remote sensing data urgently require intelligent processing. Automated image description generation technology has enormous application value in fields such as disaster monitoring, urban planning, and military reconnaissance.

[0003] With the evolution of deep learning technology, the field has undergone a significant shift from early template matching and rule-driven methods to deep generative models based on encoder-decoder architectures. Early research often used convolutional neural networks to extract image features and combined them with recurrent neural networks to generate text. Subsequently, the introduction of the Transformer architecture further improved the ability to model long sequences. However, facing the complex scenes unique to remote sensing images, such as high field of view, wide coverage, and huge differences in the scale of ground objects, existing mainstream technologies still have obvious limitations. On the one hand, traditional methods often struggle to take into account both the macroscopic scene background and the details of microscopic small objects during feature extraction. The single-scale feature representation results in a lack of hierarchy in the generated descriptions, failing to accurately reflect the remote sensing characteristic of "small objects within a large scene." On the other hand, although attention mechanisms have been widely used, most models employ a general self-attention calculation method, lacking explicit perception and modeling of the spatial topology and relative positional relationships of ground objects in remote sensing images. This leads to deviations in the generated text regarding the location description and spatial logic of ground objects. Furthermore, due to the scarcity of high-quality remote sensing images and texts, existing models often suffer from insufficient semantic alignment when transferring large-scale general pre-trained knowledge to specific remote sensing domains. This makes it difficult to effectively overcome the overfitting risk caused by small sample sizes, and the fine-grained interaction between visual features and language decoding remains coarse, leading to object omissions or attribute misjudgments when processing densely packed scenes. To address these issues, this invention proposes a method for generating language descriptions of remote sensing images based on location awareness and reinforcement learning. Summary of the Invention

[0004] This invention discloses a method for generating language descriptions of remote sensing images based on location awareness and reinforcement learning, characterized by the following steps: S1. Perform multi-scale feature extraction on the remote sensing image to be described, including: extracting global feature embeddings of the remote sensing image through a ResNet-152 residual network. The system extracts regional feature embeddings T from remote sensing images using a region feature extraction module (LIE), and extracts CLIP image embeddings containing linguistic features using a pre-trained CLIP image encoder. ; S2. Perform location-aware attention processing, including: processing the extracted region features Two compression operations were performed to obtain two attention blocks. and They are then multiplied to obtain the region feature attention matrix. ; global features Compressed into and combine it with the region feature attention matrix Multiply, then multiply the result and add it to the product. Perform residual connections to obtain region enhancement features. ; S3. Perform multi-scale feature fusion, including: integrating region enhancement features. Flattening the dimensions yields the flattened result. Then, the CLIP image embedding is broadened through a copy operation. The dimensions are used to obtain the CLIP image embedding matrix. and enhance regional features and CLIP image embedding matrix By projecting the region onto a space of dimension d through a linear transformation, the corresponding region-enhanced projection features are obtained. CLIP image embedding projection matrix Finally, a multi-head self-attention mechanism with a random drop-out strategy is used to process the data. The region fusion feature Z is obtained and then combined with... Further fusion yields multi-scale fusion features. ; S4. Fusing features across multiple scales Inputting into the improved Transformer remote sensing image language description model generates a natural language description of the remote sensing image; S5. A joint training model combining supervised pre-training and baseline-based reinforcement learning fine-tuning is adopted; The pre-trained CLIP image encoder refers to the Contrastive Language–Image Pre-training (CLIP) neural network model proposed by OpenAI, specifically the paper "Radford, A., Kim, JW, Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), Vol. 139, pp. 8748–8763". The ResNet-152 residual network refers to a general framework for deep residual learning of images proposed in the paper "He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770–778." The LIE is a region feature extraction module based on ResNet-152 residual network proposed in this invention; Furthermore, the multi-scale feature extraction in step S1 specifically includes: S1.1 Adjust the remote sensing image to be described to a 224×224 resolution on 3 channels to obtain a standard remote sensing image. ; S1.2 will use standard remote sensing images The input is fed into a ResNet-152 residual network for processing to obtain the global feature embedding of the remote sensing image. , , in, This represents the ResNet-152 residual network used for global feature extraction, where N is the number of feature channels generated by ResNet-152, and H is the spatial size of each feature. for Features on the Nth channel, Represents the set of real numbers; S1.3 performs feature extraction through the Region Feature Extraction (LIE) module. The specific processing procedure is as follows: Based on the feature space size H of the ResNet-152 residual network, the remote sensing image to be described is... Divide the image into segments of size H×H, and for each segment... Features were extracted using the ResNet-152 network to obtain... Regional characteristics Then to Perform adaptive average pooling to obtain the pooling result. Finally, the pooling results of all segmented images are aggregated to obtain the region feature embedding T of the remote sensing image. The calculation process is as follows: , , , , in, This represents the ResNet-152 residual network in the region feature extraction module (LIE). Representing remote sensing images , express indivual Element collection This indicates that the dimension is The set of elements Combined into a dimension Matrix; This represents adaptive average pooling, specifically, it involves pooling elements of dimension 1 to 1. Matrix average pooling is performed on a matrix of dimension 1. The row vector; S1.4 The remote sensing image to be described The image is fed into a pre-trained CLIP image encoder for extraction, resulting in CLIP image embeddings containing language features. ,in This indicates the dimension of the CLIP image embedding vector.

[0005] Furthermore, the position-aware attention processing in step S2 specifically includes: S2.1 Embedding regional features The signal is compressed into two attention blocks along the channel using a 1×1 convolution. and Then The two-dimensional spatial dimension H×H of each channel is flattened into a one-dimensional sequence length. ,get and Finally, the region feature attention matrix is ​​obtained through the following matrix multiplication. : , S2.2 Global Features The attention block is compressed along the channel using a 1×1 convolution. and flatten it into Then Attention matrix with regional features Multiplying yields the region-enhanced flattening features. : , S2.3 will The last dimension Reshaping it into a two-dimensional space of dimension H×H, we get the reshaped ,Will Repeat the process 8 times along the channel dimension and connect them to restore N channels, thus obtaining the attention-enhanced region features. : , in, Indicates will splicing along the passage; S2.4 Enhance attention to regional features With global feature embedding Perform residual connections to obtain region enhancement features. : , in," "" indicates adding point by point.

[0006] Furthermore, the multi-scale feature fusion in step S3 specifically includes: S3.1 Enhance regional features The two-dimensional spatial dimension H×H on each channel is flattened into a one-dimensional sequence length. to be flattened Then embed the CLIP image. Copying the image N times along the channel dimension yields the CLIP image embedding matrix. ; S3.2 will The dimension of each channel is projected onto a space of dimension d through a linear transformation to obtain the corresponding region-enhanced projection feature. CLIP image embedding projection matrix : , , in, and The linear transformation parameter matrix, and There are two biases; S3.3 uses a multi-head self-attention mechanism with a random drop-out strategy. Enhanced projection features for the region By integrating, regional integration characteristics can be obtained. The calculation process is as follows: , in, For the linear transformation parameter matrix, concat(•) denotes the join operation. This is the result of the nth multi-head self-attention, where n is... The number of heads for self-attention, and for the i-th multi-head self-attention... The calculation process is as follows: , in, , For normalization function, Indicates to Perform matrix transpose. for The linear projection parameters of the query, key, and value for the i-th self-attention head. Let i be the feature dimension of the i-th self-attention head. Let be the random drop function for the i-th head. Given its parameters, we have: , in, express The function selects the number of elements from each row of P in descending order of their values, where s1 and s2 are the row and column numbers of parameter P, respectively. express In line s1 A set of elements in descending order. express for One of the elements; S3.4 Embed the region fusion feature Z with the CLIP image into the projection matrix. The data is concatenated and then filtered out for useless information using a gating mechanism based on the sigmoid activation function to obtain multi-scale fused features. The calculation process is as follows: , in, It is a linear transformation parameter matrix. There are two biases. This indicates element-wise multiplication. This represents the sigmoid activation function.

[0007] Furthermore, the improved Transformer remote sensing image language description model in step S4 specifically includes: S4.1 The improved Transformer remote sensing image language description model Mode(•) specifically consists of an improved Transformer encoder (•) with B layers, an improved Transformer decoder (•) with B layers, and a classification layer Classification (•): , The B represents the number of overlapping layers in the encoder (•) and decoder (•); The specific improvements to the Transformer encoder (•) described in S4.2 are as follows: (1) Add a dilated convolutional DC with a kernel size of 3×3 before the standard Transformer coding layer to transform the input of the multi-head attention mechanism, while keeping the rest unchanged. For the i-th coding layer, the multi-head attention is calculated as follows: , in, This represents the result of the multi-head attention calculation for the i-th coding layer. For standard scaled dot product multi-head attention, Let be the input of the i-th coding layer, where , , For multi-head attention in the i-th encoding layer Learnable projection weight parameters, This represents a dilated convolution with a kernel size of 3×3. The multi-scale fusion features generated in step S3.4; (2) Put the output results of all coding layers of Encoder(•) into a set to obtain the multi-scale encoder output. The calculation process is as follows: , in, This is the output of the B-th coding layer; The specific improvements to the Transformer decoder (•) described in S4.3 are as follows: (1) Improve the multi-head attention in each layer of the standard Transformer decoder into a mesh cross-attention MCA, while keeping the rest unchanged, to form the structure of the improved Transformer decoder Decoder(•): , in, For the standard masked multi-head self-attention sublayer in the i-th layer of Decoder(•), For the improved mesh cross-attention sublayer in the i-th layer of Decoder(•), For the standard feedforward neural network sublayer in the i-th layer of Decoder(•); (2) Decoder(•) in the first layer The calculation process is as follows: , , , in, For the first layer of masked multi-head self-attention sublayer The processed text sequence representation generated by the decoder (•) The length of the text sequence. This is the output of the first encoding layer in Encoder(•). yes The parameter matrix of the linear transformation. For the attention of the bulls The calculation results yes The parameter matrix of the linear transformation. yes The bias in for The correlation measurement between them for The calculation results; (3) Through the first layer of Decoder(•) After processing, the output of the first layer in Decoder(•) is obtained. ; (4) For layers i>1, Decoder(•) in the i-th layer and its results The calculation and processing steps are similar to steps (2) and (3), only the formulas (20) and (21) need to be changed. Replace with the output of the previous decoding layer This yields the output of the i-th layer in Decoder(•). ; (5) The output of the B-th layer of Decoder(•) As the output feature of the decoder (•), i.e. = : (6) For the t-th time step, the model Use the output features of the decoder (•) at this time step. Classify and calculate the model The output description word at time step t .

[0008] Furthermore, the joint training mode in step S5 specifically includes: S5.1 uses supervised pre-training to train the model. In this stage, a word-level cross-entropy loss function is used. : , in, This represents the sequence of actual descriptive words from time step 1 to time step t-1. Let TM be the actual descriptive word at time step t, and TM be the maximum time step size. Representation Model All learnable parameters; Representation Model by As input to the decoder (•), the true description word at time step t. The predicted probability, Represents the natural logarithm; the model The overall model proposed in this invention includes all processing modules in steps S1 to S4 and the improved Transformer remote sensing image language description model. ; S5.2 Fine-tuning the model using reinforcement learning based on supervised pre-training. In the reinforcement learning fine-tuning phase, a beam search strategy is used to select the top sk candidate description sentences with the highest probabilities. Then, the CIDEr scores of these description sentences are calculated as reward signals. Finally, the following reinforcement learning loss function with baseline is used. : , , Where sk is the number of samples. Let represent the i-th descriptive sentence sampled in the beam, and r(·) be the reward function based on the CIDEr score. To reward the baseline, The function for calculating the CIDEr score is defined as follows: CIDEr is an abbreviation for "cosine similarity between TF-IDF vectors of generated and referencesentences," and the CIDEr score is an image / text evaluation index based on n-gram co-occurrence. The beam search strategy is a greedy + breadth-first sequence generation strategy. The beam sampling is performed by randomly sampling within the candidate beams of the beam search, rather than directly selecting the one with the highest probability.

[0009] The present invention has the following advantages: (1) By using a local image embedding strategy, the image is divided into semantically perceptual segments and features are extracted independently, which effectively makes up for the shortcomings of traditional global feature extraction in the identification of small-sized, low-recognition ground objects, and preserves spatial layout and object-level details. (2) The designed image location-aware attention module and multi-scale feature fusion module realize the deep fusion of local features, global features and CLIP pre-trained semantic features. The Top-k sparse attention mechanism reduces computational overhead while improving feature discrimination. (3) The introduction of a large-scale image-text pair pre-trained CLIP model as a global context prior effectively alleviates the semantic modeling bias caused by the scarcity of remote sensing annotation data; (4) The model generation accuracy was improved by adopting a joint training mode of supervised pre-training and reinforcement learning fine-tuning. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the LIE (Local Feature Extraction) module structure; Figure 3 This is a schematic diagram of the location-aware attention module (IPAA). Figure 4This is a schematic diagram of the CMA (Multi-Scale Feature Fusion) module structure; Figure 5 A schematic diagram of the improved Transformer remote sensing image language description model. Detailed Implementation

[0011] The present invention will be further described below with reference to specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments.

[0012] according to Figures 1 to 5 The flowchart and module structure diagram of the present invention shown illustrate the following steps for generating a language description of a remote sensing image based on location awareness and reinforcement learning: S1. Perform multi-scale feature extraction on the remote sensing image to be described, including: extracting global feature embeddings of the remote sensing image through a ResNet-152 residual network. The system extracts regional feature embeddings T from remote sensing images using a region feature extraction module (LIE), and extracts CLIP image embeddings containing linguistic features using a pre-trained CLIP image encoder. ; S2. Perform location-aware attention processing, including: processing the extracted region features Two compression operations were performed to obtain two attention blocks. and They are then multiplied to obtain the region feature attention matrix. ; global features Compressed into and combine it with the region feature attention matrix Multiply, then multiply the result and add it to the product. Perform residual connections to obtain region enhancement features. ; S3. Perform multi-scale feature fusion, including: integrating region enhancement features. Flattening the dimensions yields the flattened result. Then, the CLIP image embedding is broadened through a copy operation. The dimensions are used to obtain the CLIP image embedding matrix. and enhance regional features and CLIP image embedding matrix By projecting the region onto a space of dimension d through a linear transformation, the corresponding region-enhanced projection features are obtained. CLIP image embedding projection matrix Finally, a multi-head self-attention mechanism with a random drop-out strategy is used to process the data. The region fusion feature Z is obtained and then combined with... Further fusion yields multi-scale fusion features. ; S4. Fusing features across multiple scales Inputting into the improved Transformer remote sensing image language description model generates a natural language description of the remote sensing image; S5. A joint training model combining supervised pre-training and baseline-based reinforcement learning fine-tuning is adopted; The pre-trained CLIP image encoder refers to the Contrastive Language–Image Pre-training (CLIP) neural network model proposed by OpenAI, specifically the paper "Radford, A., Kim, JW, Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), Vol. 139, pp. 8748–8763". The ResNet-152 residual network refers to a general framework for deep residual learning of images proposed in the paper "He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770–778." The LIE is a region feature extraction module based on ResNet-152 residual network proposed in this invention; Furthermore, the multi-scale feature extraction in step S1 specifically includes: S1.1 Adjust the remote sensing image to be described to a 224×224 resolution on 3 channels to obtain a standard remote sensing image. ; S1.2 will use standard remote sensing images The input is fed into a ResNet-152 residual network for processing to obtain the global feature embedding of the remote sensing image. , , in, This represents the ResNet-152 residual network used for global feature extraction, where N is the number of feature channels generated by ResNet-152, and H is the spatial size of each feature. for Features on the Nth channel, Represents the set of real numbers; S1.3 performs feature extraction through the Region Feature Extraction (LIE) module. The specific processing procedure is as follows: Based on the feature space size H of the ResNet-152 residual network, the remote sensing image to be described is... Divide the image into segments of size H×H, and for each segment... Features were extracted using the ResNet-152 network to obtain... Regional characteristics Then to Perform adaptive average pooling to obtain the pooling result. Finally, the pooling results of all segmented images are aggregated to obtain the region feature embedding T of the remote sensing image. The calculation process is as follows: , , , , in, This represents the ResNet-152 residual network in the region feature extraction module (LIE). Representing remote sensing images , express indivual Element collection This indicates that the dimension is The set of elements Combined into a dimension Matrix; This represents adaptive average pooling, specifically, it involves pooling elements of dimension 1 to 1. Matrix average pooling is performed on a matrix of dimension 1. The row vector; S1.4 The remote sensing image to be described The image is fed into a pre-trained CLIP image encoder for extraction, resulting in CLIP image embeddings containing language features. ,in This indicates the dimension of the CLIP image embedding vector.

[0013] Furthermore, the position-aware attention processing in step S2 specifically includes: S2.1 Embedding regional features The signal is compressed into two attention blocks along the channel using a 1×1 convolution. and Then The two-dimensional spatial dimension H×H of each channel is flattened into a one-dimensional sequence length. ,get and Finally, the region feature attention matrix is ​​obtained through the following matrix multiplication. : , S2.2 Global Features The attention block is compressed along the channel using a 1×1 convolution. and flatten it into Then Attention matrix with regional features Multiplying yields the region-enhanced flattening features. : , S2.3 will The last dimension Reshaping it into a two-dimensional space of dimension H×H, we get the reshaped ,Will Repeat the process 8 times along the channel dimension and connect them to restore N channels, thus obtaining the attention-enhanced region features. : , in, Indicates will splicing along the passage; S2.4 Enhance attention to regional features With global feature embedding Perform residual connections to obtain region enhancement features. : , in," "" indicates adding point by point.

[0014] Furthermore, the multi-scale feature fusion in step S3 specifically includes: S3.1 Enhance regional features The two-dimensional spatial dimension H×H on each channel is flattened into a one-dimensional sequence length. to be flattened Then embed the CLIP image. Copying the image N times along the channel dimension yields the CLIP image embedding matrix. ; S3.2 will The dimension of each channel is projected onto a space of dimension d through a linear transformation to obtain the corresponding region-enhanced projection feature. CLIP image embedding projection matrix : , , in, and The linear transformation parameter matrix, and There are two biases; S3.3 uses a multi-head self-attention mechanism with a random drop-out strategy. Enhanced projection features for the region By integrating, regional integration characteristics can be obtained. The calculation process is as follows: (12), in, For the linear transformation parameter matrix, concat(•) denotes the join operation. This is the result of the nth multi-head self-attention, where n is... The number of heads for self-attention, and for the i-th multi-head self-attention... The calculation process is as follows: , in, , For normalization function, Indicates to Perform matrix transpose. for The linear projection parameters of the query, key, and value for the i-th self-attention head. Let i be the feature dimension of the i-th self-attention head. Let be the random drop function for the i-th head. Given its parameters, we have: , Where k represents The function selects the number of elements from each row of P in descending order of their values, where s1 and s2 are the row and column numbers of parameter P, respectively. express The set of k descending elements in line s1. express for One of the elements; S3.4 Embed the region fusion feature Z with the CLIP image into the projection matrix. The data is concatenated and then filtered out for useless information using a gating mechanism based on the sigmoid activation function to obtain multi-scale fused features. The calculation process is as follows: (15), in, It is a linear transformation parameter matrix. There are two biases. This indicates element-wise multiplication. This represents the sigmoid activation function.

[0015] Furthermore, the improved Transformer remote sensing image language description model in step S4 specifically includes: S4.1 The improved Transformer remote sensing image language description model Mode(•) specifically consists of an improved Transformer encoder (•) with B layers, an improved Transformer decoder (•) with B layers, and a classification layer Classification (•): , The B represents the number of overlapping layers in the encoder (•) and decoder (•); The specific improvements to the Transformer encoder (•) described in S4.2 are as follows: (1) Add a dilated convolutional DC with a kernel size of 3×3 before the standard Transformer coding layer to transform the input of the multi-head attention mechanism, while keeping the rest unchanged. For the i-th coding layer, the multi-head attention is calculated as follows: , in, This represents the result of the multi-head attention calculation for the i-th coding layer. For standard scaled dot product multi-head attention, Let be the input of the i-th coding layer, where , , For multi-head attention in the i-th encoding layer Learnable projection weight parameters, This represents a dilated convolution with a kernel size of 3×3. The multi-scale fusion features generated in step S3.4; (2) Put the output results of all coding layers of Encoder(•) into a set to obtain the multi-scale encoder output. The calculation process is as follows: , in, This is the output of the B-th coding layer; The specific improvements to the Transformer decoder (•) described in S4.3 are as follows: (1) Improve the multi-head attention in each layer of the standard Transformer decoder into a mesh cross-attention MCA, while keeping the rest unchanged, to form the structure of the improved Transformer decoder Decoder(•): , in, For the standard masked multi-head self-attention sublayer in the i-th layer of Decoder(•), For the improved mesh cross-attention sublayer in the i-th layer of Decoder(•), For the standard feedforward neural network sublayer in the i-th layer of Decoder(•); (2) Decoder(•) in the first layer The calculation process is as follows: , , , in, For the first layer of masked multi-head self-attention sublayer The processed text sequence representation generated by the decoder (•) The length of the text sequence. This is the output of the first encoding layer in Encoder(•). yes The parameter matrix of the linear transformation. For the attention of the bulls The calculation results yes The parameter matrix of the linear transformation. yes The bias in for The correlation measurement between them for The calculation results; (3) Through the first layer of Decoder(•) After processing, the output of the first layer in Decoder(•) is obtained. ; (4) For layers i>1, Decoder(•) in the i-th layer and its results The calculation and processing steps are similar to steps (2) and (3), only the formulas (20) and (21) need to be changed. Replace with the output of the previous decoding layer This yields the output of the i-th layer in Decoder(•). ; (5) The output of the B-th layer of Decoder(•) As the output feature of the decoder (•), i.e. = : (6) For the t-th time step, the model Use the output features of the decoder (•) at this time step. Classify and calculate the model The output description word at time step t .

[0016] Furthermore, the joint training mode in step S5 specifically includes: S5.1 uses supervised pre-training to train the model. In this stage, a word-level cross-entropy loss function is used. : , in, This represents the sequence of actual descriptive words from time step 1 to time step t-1. Let TM be the actual descriptive word at time step t, and TM be the maximum time step size. Representation Model All learnable parameters; Representation Model by As input to the decoder (•), the true description word at time step t. The predicted probability, Represents the natural logarithm; the model The overall model proposed in this invention includes all processing modules in steps S1 to S4 and the improved Transformer remote sensing image language description model. ; S5.2 Fine-tuning the model using reinforcement learning based on supervised pre-training. In the reinforcement learning fine-tuning phase, a beam search strategy is used to select the top sk candidate description sentences with the highest probabilities. Then, the CIDEr scores of these description sentences are calculated as reward signals. Finally, the following reinforcement learning loss function with baseline is used. : , , Where sk is the number of samples. Let represent the i-th descriptive sentence sampled in the beam, and r(·) be the reward function based on the CIDEr score. To reward the baseline, The function for calculating the CIDEr score is defined as follows: CIDEr is an abbreviation for "cosine similarity between TF-IDF vectors of generated and referencesentences," and the CIDEr score is an image / text evaluation index based on n-gram co-occurrence. The beam search strategy is a greedy + breadth-first sequence generation strategy. The beam sampling is performed by randomly sampling within the candidate beams of the beam search, rather than directly selecting the one with the highest probability.

[0017] Application Examples 1. Instance Environment Example uses Figures 1 to 5 The processing flow and model structure implementation are shown in Table 1. Table 1 Hyperparameters of the Instance .

[0018] 2. Dataset This example evaluates and compares three RSIC datasets (https: / / github.com / 201528014227051 / RSICD_optimal): Sydney-Captioning, UCM Captioning, and RSICD. Detailed information about the datasets is shown in Table 2. Table 2 Evaluation Dataset .

[0019] 3. Comparison Methods This example compares the model of this invention with six aspect-level sentiment classification methods, as shown below: • GVFGA + LSGA [1] utilizes the CNN-RNN framework and combines two attention mechanisms to better utilize image and text information. First, through GVFGA, the model filters out redundant information and emphasizes the most important regions in the image. Second, through LSGA, the model further adjusts the fusion of image and text information according to the current language state to generate more accurate and consistent explanatory text. RASG [2] integrates multi-scale visual features extracted by the ESP module. It designs a recurrent attention mechanism to capture high-level attention mappings and a semantic gate to merge the semantic information of two LSTMs. RSGPT [3] is a large-scale visual language model specifically designed for the field of remote sensing, trained on the high-quality remote sensing dataset RSICap. MLAT [4] integrates multi-scale visual features from different convolutional layers and feeds them into a Transformer architecture. This method uses LSTM to aggregate information from all encoder layers and then feeds it into the Transformer decoder to generate descriptive text. ·M 2 The Transformer [5] embeds prior knowledge of the relationships between objects into the self-attention mechanism of the Transformer encoder, which is responsible for encoding image features into semantic representations and modeling the relationships between objects. Based on the output of the encoder, the decoder enhances its understanding of the input image through multi-layer self-attention and positional encoding mechanisms, and finally generates the corresponding descriptive text. ·PKG-Transformer [6] explicitly models the correlation and differences between different objects and scene regions by integrating the attention mechanism and the feature enhancement module of the graph neural network; then, through the attention module enhanced by prior knowledge, the scene-object relationship is embedded as prior knowledge into the encoder to dynamically establish the object relationship network and filter the objects most relevant to a specific scene region.

[0020] References: [1] Zhang Z, Zhang W, Yan M, et al. Global visual feature and linguistic state guided attention for remote sensing image captioning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-16 [2] Mehta S, Rastegari M, Caspi A, et al. Espnet: Efficient spatialpyramid of dilated convolutions for semantic segmentation[C] / / Proceedings ofthe european conference on computer vision (ECCV). 2018: 552-568 [3] Hu Y, Yuan J, Wen C, et al. Rsgpt: A remote sensing visionlanguage model and benchmark[J]. ISPRS Journal of Photogrammetry and RemoteSensing, 2025, 224: 272-286 [4] Liu C, Zhao R, Shi Z. Remote-sensing image captioning based onmultilayer aggregated transformer[J]. IEEE Geoscience and Remote SensingLetters, 2022, 19: 1-5 [5]Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, RitaCucchiara; Meshed-Memory Transformer for Image Captioning[C] / / Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR),2020, pp. 10578-10587 [6]Meng L, Wang J, Yang Y, et al. Prior knowledge-guided transformerfor remote sensing image captioning[J]. IEEE Transactions on Geoscience andRemote Sensing, 2023, 61: 1-13 4. Case Comparison Results The examples and comparison models were tested on the Sydney-subtitle, UCM subtitle, and RSICD test sets, respectively, and the comparison results are shown in Tables 3, 4, and 5. Table 3. Quantitative comparison results of the Sydney-Captions dataset. , Table 4. Quantitative comparison results of the University of California, Merced dataset (UCM-Captions) , Table 5. Quantitative comparison results of Remote Sensing Image Description Dataset (RSICD) .

[0021] The method proposed in this invention achieves state-of-the-art performance on all evaluation metrics (BLEU-4, METEOR, ROUGE-L, CIDEr-D, SPICE, etc.) across all test datasets, significantly outperforming other comparative models. This demonstrates the effectiveness and sophistication of the proposed method in combining location-aware attention with multi-scale feature fusion, and employing a joint training mode of supervised pre-training and reinforcement learning fine-tuning. It can generate more accurate and semantically richer remote sensing image descriptions, particularly demonstrating significant advantages in capturing image details, spatial relationships, and generating high-quality descriptive text.

Claims

1. A method for generating language descriptions of remote sensing images based on location awareness and reinforcement learning, characterized in that... Includes the following steps: S1. Perform multi-scale feature extraction on the remote sensing image to be described, including: extracting global feature embeddings of the remote sensing image through a ResNet-152 residual network. The system extracts regional feature embeddings T from remote sensing images using a region feature extraction module (LIE), and extracts CLIP image embeddings containing linguistic features using a pre-trained CLIP image encoder. ; S2. Perform location-aware attention processing, including: processing the extracted region features Two compression operations were performed to obtain two attention blocks. and They are then multiplied to obtain the region feature attention matrix. ; global features Compressed into and combine it with the region feature attention matrix Multiply, then multiply the result and add it to the product. Perform residual connections to obtain region enhancement features. ; S3. Perform multi-scale feature fusion, including: integrating region enhancement features. Flattening the dimensions yields the flattened result. Then, the CLIP image embedding is broadened through a copy operation. The dimensions are used to obtain the CLIP image embedding matrix. and enhance regional features and CLIP image embedding matrix By projecting the region onto a space of dimension d through a linear transformation, the corresponding region-enhanced projection features are obtained. CLIP image embedding projection matrix Finally, a multi-head self-attention mechanism with a random drop-out strategy is used to process the data. The region fusion feature Z is obtained and then combined with... Further fusion yields multi-scale fusion features. ; S4. Fusing features across multiple scales Inputting into the improved Transformer remote sensing image language description model generates a natural language description of the remote sensing image; S5. A joint training model combining supervised pre-training and baseline-based reinforcement learning fine-tuning is adopted; The pre-trained CLIP image encoder refers to the Contrastive Language–Image Pre-training (CLIP) neural network model proposed by OpenAI. The ResNet-152 residual network is a general framework for image deep residual learning; The LIE is a region feature extraction module based on ResNet-152 residual network proposed in this invention; The multi-scale feature extraction in step S1 specifically includes: S1.1 The remote sensing image to be described Adjusted to 224×224 resolution on 3 channels, a standard remote sensing image was obtained. ; S1.2 will use standard remote sensing images The input is fed into a ResNet-152 residual network for processing to obtain the global feature embedding of the remote sensing image. : (1)， in, This represents the ResNet-152 residual network used for global feature extraction, where N is the number of feature channels generated by ResNet-152, and H is the spatial size of each feature. for Features on the Nth channel, Represents the set of real numbers; S1.3 performs feature extraction through the Region Feature Extraction (LIE) module. The specific processing procedure is as follows: Based on the feature space size H of the ResNet-152 residual network, the remote sensing image to be described is... Divide the image into segments of size H×H, and for each segment... Features were extracted using the ResNet-152 network, resulting in... Regional characteristics Then to Perform adaptive average pooling to obtain the pooling result. Finally, the pooling results of all segmented images are aggregated to obtain the region feature embedding T of the remote sensing image. The calculation process is as follows: (2)， (3)， (4)， ， in, This represents the ResNet-152 residual network in the region feature extraction module (LIE). Representing remote sensing images , express indivual Element collection This indicates that the dimension is The set of elements Combined into a dimension Matrix; This represents adaptive average pooling, specifically, it involves pooling elements of dimension 1 to 1. Matrix average pooling is performed on a matrix of dimension 1. The row vector; S1.4 The remote sensing image to be described The image is fed into a pre-trained CLIP image encoder for extraction, resulting in CLIP image embeddings containing language features. ,in This represents the dimension of the CLIP image embedding vector; The position-aware attention processing in step S2 specifically includes: S2.1 Embedding regional features The signal is compressed into two attention blocks along the channel using a 1×1 convolution. and Then The two-dimensional spatial dimension H×H of each channel is flattened into a one-dimensional sequence length. ,get and Finally, the region feature attention matrix is ​​obtained through the following matrix multiplication. : (6)， S2.2 Global Features The attention block is compressed along the channel using a 1×1 convolution. and flatten it into Then Attention matrix with regional features Multiplying yields the region-enhanced flattening features. : (7)， S2.3 will The last dimension Reshaping it into a two-dimensional space of dimension H×H, we get the reshaped ,Will Repeat the process 8 times along the channel dimension and connect them to restore N channels, thus obtaining the attention-enhanced region features. : (8)， in, Indicates will splicing along the passage; S2.4 Enhance attention to regional features With global feature embedding Perform residual connections to obtain region enhancement features. : (9)， in," " indicates addition point by point; The multi-scale feature fusion in step S3 specifically includes: S3.1 Enhance regional features The two-dimensional spatial dimension H×H of each channel, when flattened into a one-dimensional sequence length, to be flattened Then embed the CLIP image. The CLIP image embedding matrix is ​​obtained by copying it N times along the channel dimension. ; S3.2 will The dimension of each channel is projected onto a space of dimension d through a linear transformation to obtain the corresponding region-enhanced projection feature. CLIP image embedding projection matrix : (10)， (11)， in, and The linear transformation parameter matrix, and There are two biases; S3.3 uses a multi-head self-attention mechanism with a random drop-out strategy. Enhanced projection features for the region By integrating, regional integration characteristics can be obtained. The calculation process is as follows: (12)， in, For the linear transformation parameter matrix, concat(•) denotes the join operation. This is the result of the nth multi-head self-attention, where n is... The number of heads for self-attention, and for the i-th multi-head self-attention... The calculation process is as follows: ， in, , For normalization function, Indicates to Perform matrix transpose. for The linear projection parameters of the query, key, and value for the i-th self-attention head. Let i be the feature dimension of the i-th self-attention head. Let be the random drop function for the i-th head. Given its parameters, we have: ， Where k represents The function selects the number of elements from each row of P in descending order of their values, where s1 and s2 are the row and column numbers of parameter P, respectively. express The set of k descending elements in line s1. express for One of the elements; S3.4 Embed the region fusion feature Z with the CLIP image into the projection matrix. The data is concatenated and then filtered out for useless information using a gating mechanism based on the sigmoid activation function to obtain multi-scale fused features. The calculation process is as follows: (15)， in, It is a linear transformation parameter matrix. There are two biases. This indicates element-wise multiplication. This represents the sigmoid activation function; The improved Transformer remote sensing image language description model in step S4 specifically includes: The improved Transformer remote sensing image language description model described in S4.1 Specifically, it consists of an improved Transformer encoder (•) with B layers, an improved Transformer decoder (•) with B layers, and a classification layer (•): (16)， The B represents the number of overlapping layers in the encoder (•) and decoder (•); The specific improvements to the Transformer encoder Encoder(•) described in S4.2 are as follows: (1) Add a dilated convolutional DC with a kernel size of 3×3 before the standard Transformer coding layer to transform the input of the multi-head attention mechanism, while keeping the rest unchanged. For the i-th coding layer, the multi-head attention is calculated as follows: (17)， in, This represents the result of the multi-head attention calculation for the i-th coding layer. For standard scaled dot product multi-head attention, Let be the input of the i-th coding layer, where , , For multi-head attention in the i-th encoding layer Learnable projection weight parameters, This represents a dilated convolution with a kernel size of 3×3. The multi-scale fusion features generated in step S3.4; (2) Put the output results of all coding layers of Encoder(•) into a set to obtain the multi-scale encoder output. The calculation process is as follows: (18)， in, This is the output of the B-th coding layer; The specific improvements to the Transformer decoder (•) described in S4.3 are as follows: (1) Improve the multi-head attention in each layer of the standard Transformer decoder into a mesh cross-attention MCA, while keeping the rest unchanged, to form the structure of the improved Transformer decoder Decoder(•): (19)， in, For the standard masked multi-head self-attention sublayer in the i-th layer of Decoder(•), For the improved mesh cross-attention sublayer in the i-th layer of Decoder(•), For the standard feedforward neural network sublayer in the i-th layer of Decoder(•); (2) Decoder(•) in the first layer The calculation process is as follows: (20)， (21)， (22)， in, For the first layer of masked multi-head self-attention sublayer The processed text sequence representation generated by the decoder (•) The length of the text sequence. This is the output of the first encoding layer in Encoder(•). yes The parameter matrix of the linear transformation. For the attention of the bulls The calculation results yes The parameter matrix of the linear transformation. yes The bias in for The correlation measurement between them for The calculation results; (3) Through the first layer of Decoder(•) After processing, the output of the first layer in Decoder(•) is obtained. ; (4) For layers i>1, Decoder(•) in the i-th layer and its results The calculation and processing steps are similar to steps (2) and (3), only the formulas (20) and (21) need to be changed. Replace with the output of the previous decoding layer This yields the output of the i-th layer in Decoder(•). ; (5) The output of the B-th layer of Decoder(•) As the output feature of the decoder (•), i.e. = : (6) For the t-th time step, the model Use the output features of the decoder (•) at this time step. Classify and calculate the model The output description word at time step t ; The joint training mode in step S5 specifically includes: S5.1 uses supervised pre-training to train the model. In this stage, a word-level cross-entropy loss function is used. : ， in, This represents the sequence of actual descriptive words from time step 1 to time step t-1. Let TM be the actual descriptive word at time step t, and TM be the maximum time step size. Representation Model All learnable parameters; Representation Model by As input to the decoder (•), the true description word is given at time step t. The predicted probability, Represents the natural logarithm; the model The overall model proposed in this invention includes all processing modules in steps S1 to S4 and the improved Transformer remote sensing image language description model. ; S5.2 Fine-tuning the model using reinforcement learning based on supervised pre-training. In the reinforcement learning fine-tuning phase, a beam search strategy is used to select the top sk candidate description sentences with the highest probabilities. Then, the CIDEr scores of these description sentences are calculated as reward signals. Finally, the following reinforcement learning loss function with baseline is used. : ， (25)， Where sk is the number of samples. Let represent the i-th descriptive sentence sampled in the beam, and r(·) be the reward function based on the CIDEr score. To reward the baseline, The function for calculating the CIDEr score is defined as follows: the CIDEr score is an image / text evaluation metric based on n-gram co-occurrence; the beam search strategy is a greedy + breadth-first sequence generation strategy; and the sampling within the beam is performed by randomly sampling within the candidate beams of the beam search, rather than directly selecting the beam with the highest probability.

Images