Remote sensing image subtitle generation method and device based on channel-space attention and Mama network

Abstract

The invention provides a remote sensing image subtitle generation method and device based on channel-space attention and a Mama network, and relates to the field of remote sensing image processing. The method comprises the following steps: inputting a remote sensing image into a pre-trained convolutional neural network, and outputting a first image feature according to the pre-trained convolutional neural network; inputting the first image feature to a first module, and outputting a second image feature based on the first module; inputting the second image feature to a second module, and outputting a third image feature based on the second module; extracting global image features based on the second image features, and extracting global text features based on the text features; and based on the global image features and the global text features, combining position coding and a mask mechanism to generate an image subtitle text. According to the remote sensing image subtitle generation method, the problem that the generated remote sensing image subtitles are easy to express inaccurately, lack of semantics or redundantly describe when the image content has significant structural diversity, non-uniform semantic distribution or local region noise interference in the traditional method is solved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing image processing, and in particular to a method and device for generating remote sensing image captions based on channel-spatial attention and Mamba network. Background Art

[0002] Remote sensing image captioning (RSIC) is a cross-modal task that combines computer vision and natural language processing to convert remote sensing images into natural language descriptions so that non-experts can intuitively understand the image content. In recent years, with the development of remote sensing technology, RSIC, as a subtask of image captioning, has attracted widespread attention in fields such as geographical research and environmental monitoring.

[0003] Traditional methods typically generate remote sensing image captions by modeling image features using only a single-directional information flow. This ignores the bidirectional dependencies of the image's spatial structure and lacks explicit modeling of local details and channel semantics, resulting in insufficient modeling capabilities in complex scenarios. This is especially true when image content exhibits significant structural diversity, uneven semantic distribution, or localized noise interference. Generated remote sensing image captions are prone to inaccurate expression, missing semantics, or redundant descriptions.

[0004] Therefore, there is an urgent need for a remote sensing image caption generation method and device based on channel-spatial attention and Mamba network. Summary of the Invention

[0005] The present application provides a method and device for generating remote sensing image captions based on channel-spatial attention and Mamba network, which solves the problem of generating remote sensing image captions by modeling image features only through information flow in a single direction. When the image content has significant structural diversity, uneven semantic distribution, or local area noise interference, the generated remote sensing image captions are prone to inaccurate expression, missing semantics, or redundant descriptions.

[0006] In a first aspect of the present application, a remote sensing image caption generation method based on channel-spatial attention and Mamba network is provided, the method comprising: inputting a remote sensing image into a pre-trained convolutional neural network, and outputting a first image feature according to the pre-trained convolutional neural network; inputting the first image feature into a first module, and outputting a second image feature based on the first module; inputting the second image feature into a second module, and outputting a third image feature based on the second module; inputting the third image feature into a CSA Mamba block, and extracting global image features based on the CSA Mamba block; inputting the annotation sentence corresponding to the remote sensing image into a Transformer module, and extracting global text features based on the Transformer module; generating image caption text based on the global image features and global text features, combined with position encoding and masking mechanisms.

[0007] Optionally, the remote sensing image is input into a pre-trained convolutional neural network, and the first image feature is output according to the pre-trained convolutional neural network, specifically including: inputting the remote sensing image into the pre-trained convolutional neural network, encoding the remote sensing image through the backbone network of the convolutional neural network, and obtaining the first image feature.

[0008] Optionally, the first image feature is input into the first module, and the second image feature is output based on the first module, specifically including: flattening the first image feature in the spatial dimension and the channel dimension for normalization processing; mapping the normalized first image feature into a first embedded feature and a second embedded feature through two different linear transformations respectively; performing forward state space modeling on the first embedded feature as input to obtain a first direction image coding feature, and performing backward state space modeling on the second embedded feature as input to obtain a second direction image coding feature; and performing feature fusion on the first direction image coding feature and the second direction image coding feature to obtain a second image feature.

[0009] Optionally, forward state space modeling is performed with the first embedded feature as input to obtain a first direction image coding feature, and backward state space modeling is performed with the second embedded feature as input to obtain a second direction image coding feature, specifically including: using the state space model, taking the first embedded feature as input, performing image encoding on the normalized first image feature starting from the upper left corner to obtain the first direction image coding feature; using the state space model, taking the second embedded feature as input, performing image encoding on the normalized first image feature starting from the lower right corner to obtain the second direction image coding feature.

[0010] Optionally, the second image feature is input into the second module, and the third image feature is output based on the second module, specifically including: inputting the second image feature into the second module, and obtaining the global statistical features and hidden features in the second image feature; performing average pooling processing, full connection mapping processing and activation function processing on the global statistical features through the channel attention mechanism to obtain channel weighted features; performing convolution processing, average pooling processing and activation function processing on the hidden features through the spatial attention mechanism to obtain spatial weighted features; outputting the channel weighted features and the spatial weighted features and splicing them, and inputting them into the self-attention module for attention weighted calculation; after the weighted calculation, performing linear transformation operations and normalization operations to obtain the text feature and the third image feature.

[0011] Optionally, based on global image features and global text features, combined with position encoding and masking mechanisms, image caption text is generated, specifically including: semantically aligning global image features and global text features through image-text cross-modal alignment loss; generating image caption text based on the semantically aligned global image features and global text features through position encoding and masking mechanisms.

[0012] Optionally, semantic alignment is performed on global image features and global text features through image-text cross-modal alignment loss, specifically including: constructing an alignment loss from image to text through the following formula: ; in, is the alignment loss from image to text, represents the number of image-text pairs in a training batch, represents the global image features, represents the global text features, Indicates the first The text features corresponding to the text, Indicates the first The text features corresponding to the text, is the hyperparameter of normalized softmax, represents the total time step, It is used to construct the exponential part in softmax and construct the alignment loss from text to image through the following formula: ; in, is the alignment loss from text to image, Indicates the first The image features corresponding to the images, Indicates the first The image features corresponding to the images are used to construct the image-text cross-modal alignment loss through the alignment loss from image to text and the alignment loss from text to image: ; in, The image-text cross-modal alignment loss is used to semantically align global image features and global text features.

[0013] In a second aspect of the present application, a remote sensing image caption generation device based on channel-spatial attention and Mamba network is provided, the device including an acquisition module and a processing module, wherein: An acquisition module is used to input the remote sensing image into a pre-trained convolutional neural network and output a first image feature based on the pre-trained convolutional neural network; input the first image feature into a first module and output a second image feature based on the first module; input the second image feature into a second module and output a third image feature based on the second module.

[0014] The processing module is used to input the third image features into the CSA Mamba block and extract global image features based on the CSA Mamba block; input the annotation sentences corresponding to the remote sensing image into the Transformer module and extract global text features based on the Transformer module; and generate image caption text based on the global image features and global text features in combination with position encoding and masking mechanisms.

[0015] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs any of the methods described above.

[0016] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to perform any of the above methods.

[0017] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By inputting the remote sensing image into a pre-trained convolutional neural network and outputting the first image feature according to the pre-trained convolutional neural network; inputting the first image feature into the first module, and outputting the second image feature based on the first module; inputting the second image feature into the second module, and outputting the third image feature based on the second module; inputting the third image feature into the CSA Mamba block, and extracting the global image feature based on the CSA Mamba block; inputting the annotation sentence corresponding to the remote sensing image into the Transformer module, and extracting the global text feature based on the Transformer module; based on the global image feature and the global text feature, combining the position encoding and masking mechanism, the image caption text is generated. By combining the channel-spatial attention mechanism to enhance the fine-grained image features, the bidirectional Mamba block to deeply model the image spatial structure, and the Transformer module to comprehensively extract the contextual semantics, high-quality semantic alignment between image and text features is achieved, which greatly improves the accuracy of the generated remote sensing image captions and reduces the problems of semantic missing or redundant descriptions.

[0018] 2. Input the second image features into the second module, and obtain the global statistical features and hidden features in the second image features; perform average pooling processing, fully connected mapping processing and activation function processing on the global statistical features through the channel attention mechanism to obtain channel-weighted features; perform convolution processing, average pooling processing and activation function processing on the hidden features through the spatial attention mechanism to obtain spatial weighted features; output the channel-weighted features and spatial weighted features, and input them into the self-attention module for attention weighted calculation; after weighted calculation, obtain the third image feature of the text feature through linear transformation operation and normalization operation, so as to strengthen the joint perception ability of the salient target area and key channel information on the basis of maintaining the semantic integrity of the image, make the generated text features more targeted and semantically cohesive, and significantly improve the contextual accuracy and syntactic consistency of subtitle generation.

[0019] 3. Based on global image features and global text features, combined with position encoding and masking mechanisms, image subtitles are generated. Specifically, global image features and global text features are semantically aligned through image-text cross-modal alignment loss; image subtitles are generated based on semantically aligned global image features and global text features through position encoding and masking mechanisms, thereby enhancing the semantic consistency between images and text during the generation process, guiding the decoder to more accurately associate image content with language expression, and further improving the contextual coherence and semantic accuracy of subtitle generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a remote sensing image caption generation method based on channel-spatial attention and Mamba network provided in an embodiment of the present application; Figure 2 This is a schematic diagram of a bidirectional Mamba module provided in an embodiment of the present application; Figure 3 This is a schematic diagram of the structure of a CSA Mamba block and a self-attention fusion module provided in an embodiment of the present application; Figure 4 This is a schematic diagram of an image-text cross-modal alignment loss structure provided by an embodiment of the present application; Figure 5 This is a schematic diagram of the overall structure of the CSA Mamba provided in the embodiment of the present application; Figure 6 This is a schematic diagram of the visualization results provided by the embodiment of the present application; Figure 7 This is a module diagram of a remote sensing image caption generation device based on channel-spatial attention and Mamba network provided in an embodiment of the present application; Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0021] Explanation of the reference numerals: 71, acquisition module; 72, processing module; 801, processor; 802, communication bus; 803, user interface; 804, network interface; 805, memory. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0023] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application, the singular expressions "a", "an", "said", "above", "the", and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more of the listed items.

[0024] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0025] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0026] Please refer to Figure 1 , which shows a flow chart of a remote sensing image caption generation method based on channel-spatial attention and Mamba network provided in an embodiment of the present application. The flow chart mainly includes the following steps: S101 to S106.

[0027] Step S101: input a remote sensing image into a pre-trained convolutional neural network, and output a first image feature according to the pre-trained convolutional neural network.

[0028] Specifically, the pre-trained convolutional neural network is a ResNet18 structure, which was pre-trained on a large-scale image dataset and has strong general image feature extraction capabilities. The remote sensing image is first scaled to a fixed size and normalized to adapt to the network input requirements. It is then input into the ResNet18 network, where it undergoes multiple residual convolution units and pooling operations to extract the multi-level spatial structure and semantic features of the image. The feature map output by the network's final convolutional module is considered the first image feature, which retains key information from the original image while achieving spatial compression, providing basic representation support for subsequent encoder processing.

[0029] In a possible implementation, step S101 further includes: inputting the remote sensing image into a pre-trained convolutional neural network, encoding the remote sensing image through a backbone network of the convolutional neural network, and obtaining a first image feature.

[0030] Specifically, the convolutional neural network uses the ResNet18 structure as the backbone network. The backbone network includes a convolution extraction channel composed of multiple residual block layers. Each residual block consists of two consecutive convolutional layers and their residual connection paths to enhance the stability of gradient propagation and the ability to preserve image structure. During execution, the remote sensing image is first resampled to a specified resolution to match the input layer requirements of the backbone network, and the pixel values ​​are normalized to stabilize their distribution within the pre-trained parameter adaptation range. Starting from the input end, the backbone network sequentially encodes the original remote sensing image into a high-dimensional feature representation through initial convolution, batch normalization, maximum pooling, and multiple residual stacking units. At the output end of the backbone network, the multi-channel feature map output by its last set of residual modules is extracted as the first image feature. This feature map is compressible in the spatial dimension and retains the ability to respond to image edge contours, structural semantics, and regional distribution in the channel dimension. The first image feature serves as the basic semantic input for subsequent encoder modeling. It has good regional discriminability and texture adaptability, and can support the deep modeling requirements of structural elements and position distribution in image semantic generation tasks.

[0031] Step S102: input the first image feature into the first module, and output the second image feature based on the first module.

[0032] Specifically, the first module is the bidirectional Mamba module. The original Mamba block was designed primarily for one-dimensional sequence processing, with its working mechanism focusing on sequential analysis and processing of data in a single dimension. However, in the field of visual tasks, the information contained in images has significant spatial characteristics. An image is not a simple one-dimensional arrangement, but rather consists of pixels distributed at specific spatial positions on a two-dimensional plane, with close spatial correlations between each pixel. This spatial information encompasses important aspects such as an object's shape, position, orientation, and relative spatial relationships. The first image feature is first flattened and normalized in the spatial dimension and channel dimension, and then mapped into forward and backward embedded features through two linear transformations, which are input into the forward and backward state space modeling paths respectively; by simulating the bidirectional scanning process of the image from the upper left corner to the lower right corner and from the lower right corner to the upper left corner, a bidirectional image coding feature containing the global context is obtained, and finally the forward and backward outputs are fused to obtain the second image feature. In a possible implementation, step S102 also includes: flattening the first image feature in the spatial dimension and channel dimension for normalization; mapping the normalized first image feature into a first embedded feature and a second embedded feature through two different linear transformations; performing forward state space modeling with the first embedded feature as input to obtain a first direction image coding feature, and performing backward state space modeling with the second embedded feature as input to obtain a second direction image coding feature; and performing feature fusion on the first direction image coding feature and the second direction image coding feature to obtain a second image feature.

[0033] Specifically, the Vim method is used to enable the bidirectional Mamba module to better understand the spatial correlation of image information through the bidirectional scanning mode. First, the first image feature in the remote sensing image is encoded into , the first image feature Flatten to And normalize it, then map the normalized first image feature into the first embedding feature and the second embedding feature through two different linear transformations, namely and After that, after one-dimensional convolution, the first embedded feature is used as input to perform forward state space modeling to obtain the first direction image coding feature, and the second embedded feature is used as input to perform backward state space modeling to obtain the second direction image coding feature, that is, image coding is performed from two directions according to SSM (State Space Models). Please refer to Figure 2, which shows a schematic diagram of a bidirectional Mamba module provided by an embodiment of the present application. In the figure, the state-space model is used to take the first embedded feature as input and the normalized first image feature is image-encoded starting from the upper left corner to obtain the first directional image coding feature; the state-space model is used to take the second embedded feature as input and the normalized first image feature is image-encoded starting from the lower right corner to obtain the second directional image coding feature. SSM represents a class of sequence models with linear complexity, and its discrete representation can be expressed as follows:

[0034]

[0035] in, represents the hidden state, represents input, and , , Represents a discretized linear transformation, and discretization is achieved through the time scale parameter. The output of the master gate after feature fusion is used as the second image feature, that is, .

[0036] Step S103: input the second image feature into the second module, and output the third image feature based on the second module.

[0037] Specifically, in the field of computer vision, channel attention and spatial attention constitute a complementary attention mechanism system. Channel attention weights the importance of each channel through global statistical features to achieve adaptive screening of feature channels; spatial attention generates a spatial weight map based on channel dimension feature aggregation to accurately locate the salient areas in the image. As a new architecture based on linear self-attention, the Mamba module has advantages in long-distance dependency modeling, but has the problem of insufficient local feature capture capability. In an embodiment of the present application, the second module is a channel-spatial attention Mamba module, which performs multi-dimensional feature enhancement on the second image features by introducing the channel attention mechanism and the spatial attention mechanism in parallel; wherein, the channel attention is used to dynamically adjust the importance of different channel features, and the spatial attention is used to highlight the salient area information in the image. After the two are fused, their cross-correlation is modeled through the self-attention mechanism, and the text generation semantic features containing global context and local structure expression are output.

[0038] In a possible embodiment, step S103 also includes: inputting the second image feature into the second module, and obtaining the global statistical features and hidden features in the second image feature; performing average pooling processing, fully connected mapping processing, and activation function processing on the global statistical features through the channel attention mechanism to obtain channel weighted features; performing convolution processing, average pooling processing, and activation function processing on the hidden features through the spatial attention mechanism to obtain spatial weighted features; outputting the channel weighted features and the spatial weighted features and splicing them, and inputting them into the self-attention module for attention weighted calculation; after the weighted calculation, obtaining the text feature and the third image feature through linear transformation operation and normalization operation.

[0039] Specifically, by introducing the channel attention mechanism, the model can dynamically adjust the weights of each channel feature to enhance the expression of key semantic information; while the spatial attention mechanism improves the model's sensitivity to local details through refined spatial feature selection. The two mechanisms effectively make up for the limitations of the Mamba module in local feature extraction through cross-enhancement of feature dimensions and spatial dimensions. This multi-dimensional attention fusion strategy significantly improves the performance accuracy and robustness of the model in image caption generation tasks. Please refer to Figure 3 , which presents a schematic diagram of the structure of a CSA Mamba block and a self-attention fusion module structure provided by an embodiment of the present application. In the figure, the CSA Mamba block uses parallel channel attention and spatial attention to refine the image encoding, and uses the self-attention mechanism to fuse the output features of the two attentions to achieve the importance selection of channel attention and spatial attention.

[0040] The channel attention mechanism uses average pooling, fully connected mapping, and activation function processing on global statistical features to obtain channel-weighted features. Channel attention uses global statistical features to weight the importance of each channel, achieving adaptive screening of feature channels. It then performs average pooling on spatially normalized image features. After passing through two fully connected layers and a Sigmoid activation operation, the channel dimension weights are obtained. Finally, a multiplication operation is applied to obtain refined features, namely channel-weighted features.

[0041]

[0042]

[0043]

[0044]

[0045] in, Represents a fully connected operation, indicating Rectified Linear Unit (ReLU), Represents the Sigmoid operation, represents the average pooling operation, represents the global statistical characteristics of the input, 、 、 Represents the intermediate features in the channel attention mechanism processing process, represents the channel weighted feature, Input tensor representing weighted image features.

[0046] The hidden features are convolved, average pooled, and activated using a spatial attention mechanism to obtain spatially weighted features: To obtain the position weights of the hidden feature map, the input features are first convolved. Subsequently, average pooling is performed along the channel dimension, followed by a sigmoid operation, allowing the model to make key distinctions in the spatial information of image features.

[0047]

[0048]

[0049]

[0050] in, represents the convolution operation, is the hidden feature of the input, and Represents the intermediate features in the spatial attention mechanism processing process. Afterwards, the channel weighted features and spatial weighted features are output and spliced, and input into the self-attention module for attention weighted calculation; after the weighted calculation, the text features and the third image features are obtained through linear transformation and normalization operations:

[0051]

[0052]

[0053]

[0054]

[0055] in, Indicates the output concatenation of channel weighted features and spatial weighted features. Indicates dimension splicing, 、 、 Represent the linear transformation matrices corresponding to Q, K, and V, respectively. Q, K, and V represent the query, key, and value of the self-attention mechanism, respectively. represents the softmax operation, Representation layer normalization operation, represents a linear layer, Represents the third image feature of the final output.

[0056] Step S104: input the third image feature into the CSA Mamba block, and extract the global image feature based on the CSA Mamba block.

[0057] Specifically, the CSA Mamba block introduces a channel attention mechanism and a spatial attention mechanism to act in parallel on the third image feature, obtains the channel importance weight using global average pooling and nonlinear mapping in the channel dimension, and generates a salient region response map through convolution operation and channel aggregation in the spatial dimension. The channel-weighted features and spatial-weighted features obtained by the two are concatenated and input into the self-attention mechanism to capture the correlation between their dimensions; then, the CSA Mamba block restores the feature dimension through linear transformation and layer normalization, and uses global pooling operation or specific feature identification vector (such as CLStoken) to semantically compress the above fused features, thereby extracting global image features with overall semantic representativeness to represent the main visual intent of the entire image.

[0058] Step S105 : inputting the annotation sentence corresponding to the remote sensing image into the Transformer module, and extracting the global text features based on the Transformer module.

[0059] Specifically, the annotation sentences corresponding to remote sensing images can be pre-labeled by manual annotation or machine learning annotation. First, the word embedding module maps it to a fixed-dimensional embedding vector sequence, and adds position encoding to preserve syntactic order information. Subsequently, the sequence is input into the multi-layer self-attention encoder of the Transformer module for context modeling. In this process, the masking mechanism is not used to ensure that the model can perceive the complete sentence structure. The Transformer module calculates the dependencies between words through multi-head self-attention and selects a global semantic feature vector from the output sequence as a sentence-level representation, usually the first embedding (CLS token) or the average pooling result. This is used as a global text feature for the overall semantic expression of the text, so as to perform representation comparison and similarity learning with image features in cross-modal alignment.

[0060] Step S106 : generating image caption text based on global image features and global text features in combination with position encoding and masking mechanisms.

[0061] Specifically, the global image features and global text features are semantically aligned through the image-text cross-modal alignment loss; through position encoding and masking mechanisms, image caption text is generated based on the semantically aligned global image features and global text features.

[0062] In a possible implementation, step S106 further includes: Figure 4 , which shows a schematic diagram of the image-text cross-modal alignment loss structure provided by an embodiment of the present application, and constructs the alignment loss from image to text using the following formula: ; in, is the alignment loss from image to text, represents the number of image-text pairs in a training batch, represents the global image features, represents the global text features, Indicates the first The text features corresponding to the text, Indicates the first The text features corresponding to the text, is the hyperparameter of normalized softmax, represents the total time step, It is used to construct the exponential part in softmax, which is used to amplify the difference of the input value and ensure non-negativity, thereby constructing the probability distribution. The alignment loss from text to image is constructed by the following formula: ; in, is the alignment loss from text to image, Indicates the first The image features corresponding to the images, Indicates the first The image features corresponding to the images are used to construct the image-text cross-modal alignment loss through the alignment loss from image to text and the alignment loss from text to image: ; in, The image-text cross-modal alignment loss is used to semantically align global image features and global text features. The cross-entropy loss is the main loss function for the image caption generation task and can be expressed as:

[0063] in, Indicates the The label word at the time step, From the first time step to the The total sequence of label words at time steps, Indicates Make conditional probability predictions for the model parameters. Finally, the total loss can be expressed as:

[0064] in, is a hyperparameter used to control the importance of image-text cross-modal alignment loss.

[0065] Please refer to Figure 5 , which shows the overall structure diagram of CSA Mamba provided by the embodiment of this application. To verify the performance of the proposed method, this application also conducted an image subtitle generation experiment and compared the method with current mainstream advanced methods to evaluate its superiority in remote sensing image subtitle generation tasks. Specific examples are as follows: The performance evaluation of image caption generation tasks focuses on the semantic consistency between the generated captions and the image content and the fluency of the language expression. Common evaluation indicators include BLEU, METEOR, ROUGE and CIDEr. BLEU measures the degree of N-gram overlap between the predicted captions and the reference captions. METEOR further considers word form changes and stem similarity, making the similarity evaluation more flexible. ROUGE evaluates the coverage of generated captions by calculating the longest common subsequence, while combining precision and recall. CIDEr is designed specifically for image caption generation tasks and evaluates the content fit of generated captions based on a TF-IDF weighted N-gram matching mechanism. Common datasets for remote sensing image caption generation include: The UCM-Captions dataset is based on the UCM-Merced land use dataset and contains 21 land feature categories, 2,100 images, and five captions per image, covering 368 different vocabulary terms, for a total of 10,500 captions. Images are uniformly sized at 256×256 pixels. The Sydney-Captions dataset, cropped from Google Earth, contains 613 images categorized into seven categories. Each image is sized at 500×500 pixels and is accompanied by five text descriptions covering 237 vocabulary terms.

[0066] The RSICD dataset is designed for remote sensing captioning tasks and contains 10,921 images from multiple mapping platforms, divided into 30 categories. Each image is accompanied by 5 descriptions and the image size is resized to 224×224 pixels.

[0067] To verify the performance of the model, the CSA Mamba method was compared with the Soft attention, MLAT and HCNet methods on the UCM-Captions, Sydney-Captions and RSICD datasets.

[0068] For visualization results, please refer to Figure 6 , which presents a schematic diagram of the visualization results provided by the embodiments of this application. (a), (b), and (c) are from UCM-Captions, (d), (e), and (f) are from Sydney-Captions, and (g), (h), and (i) are from RSICD. GT represents caption annotation, and CSA Mamba represents the model generation result. Table 1 shows the metric comparison of all methods on the UCM-CAPTIONS dataset, Table 2 shows the metric comparison of all methods on the SYDNEY-CAPTIONS dataset, and Table 3 shows the metric comparison of all methods on the RSICD dataset.

[0069] [Table 1]

[0070] [Table 2]

[0071] [Table 3]

[0072] On the UCM-Captions and Sydney-Captions datasets, due to the small number of images and simple content structure, the generated captions are highly consistent with the annotated captions, demonstrating the model's good fitting ability. On the RSICD dataset, CSA Mamba demonstrates stronger generalization. The generated captions in (g) accurately match the reference description, the generated description of "resort" in (h) aligns with the image semantics, and the building in (i) is correctly identified as "terminal," demonstrating the model's strong semantic understanding and representation capabilities in complex scenes. Comparison results with soft attention, MLAT, and HCNet methods on the UCM-Captions and Sydney-Captions datasets are shown in Tables I and II. The CSA Mamba method significantly outperforms other methods in the BLEU, METEOR, and ROUGE metrics, demonstrating its superior generated captions in both semantic consistency and language quality. Although CIDEr's performance on the Sydney-Captions dataset is slightly lower than MLAT's, it still performs well overall. Table 4 shows the comparison of indicators of ablation experiments on the UCM-CAPTIONS dataset, and Table 5 shows the comparison of indicators of ablation experiments on the SYDNEY-CAPTIONS dataset.

[0073] [Table 4]

[0074] [Table 5]

[0075] On the RSICD dataset, Table III shows that the CSA Mamba method achieves the best performance in most metrics, particularly in the BLEU, METEOR, and ROUGE metrics, demonstrating its robustness in large-scale, multi-category remote sensing imagery. CIDEr, while slightly inferior to MLAT, still significantly outperforms other methods.

[0076] Ablation experiments were conducted on the UCM-Captions and Sydney-Captions datasets to verify the effectiveness of the bidirectional Mamba block and the channel-spatial attention module. The results presented in Tables IV and V show that removing either module resulted in a significant decrease in all evaluation metrics. This demonstrates that these two modules play a key role in global semantic modeling and local feature enhancement, making a substantial contribution to overall model performance.

[0077] The present application adopts the above method, inputs the remote sensing image into a pre-trained convolutional neural network, and outputs a first image feature according to the pre-trained convolutional neural network; inputs the first image feature into the first module, and outputs a second image feature based on the first module; inputs the second image feature into the second module, and outputs a third image feature based on the second module; inputs the third image feature into the CSA Mamba block, and extracts global image features based on the CSA Mamba block; inputs the annotation sentence corresponding to the remote sensing image into the Transformer module, and extracts global text features based on the Transformer module; based on the global image feature and the global text feature, combined with the position encoding and mask mechanism, generates image caption text, thereby achieving high-quality semantic alignment between image and text features by combining the channel-spatial attention mechanism for fine-grained enhancement of image features, the bidirectional Mamba block for deep modeling of image spatial structure, and the Transformer module for comprehensive extraction of contextual semantics, thereby greatly improving the accuracy of the generated remote sensing image captions and reducing the problems of semantic missing or redundant descriptions.

[0078] Please refer to Figure 7 , which shows a module schematic diagram of a remote sensing image caption generation device based on channel-spatial attention and Mamba network provided by an embodiment of the present application, the device includes an acquisition module 21 and a processing module 72, wherein, The acquisition module 71 is used to input the remote sensing image into the pre-trained convolutional neural network and output the first image feature according to the pre-trained convolutional neural network; input the first image feature into the first module and output the second image feature based on the first module; input the second image feature into the second module and output the third image feature based on the second module.

[0079] Processing module 72 is configured to input the third image feature into the CSA Mamba block and extract global image features based on the CSA Mamba block; input the annotation sentence corresponding to the remote sensing image into the Transformer module and extract global text features based on the Transformer module; and generate image caption text based on the global image features and global text features in combination with position encoding and masking mechanisms.

[0080] In one possible embodiment, the acquisition module 71 is used to input the remote sensing image into a pre-trained convolutional neural network and output the first image feature based on the pre-trained convolutional neural network, specifically including: inputting the remote sensing image into the pre-trained convolutional neural network, encoding the remote sensing image through the backbone network of the convolutional neural network, and obtaining the first image feature.

[0081] In a possible embodiment, the acquisition module 71 is used to input the first image feature into the first module and output the second image feature based on the first module, specifically including: flattening the first image feature in the spatial dimension and the channel dimension for normalization processing; mapping the normalized first image feature into a first embedded feature and a second embedded feature through two different linear transformations respectively; performing forward state space modeling with the first embedded feature as input to obtain a first direction image coding feature, and performing backward state space modeling with the second embedded feature as input to obtain a second direction image coding feature; and performing feature fusion on the first direction image coding feature and the second direction image coding feature to obtain a second image feature.

[0082] In one possible implementation, the acquisition module 71 is used to perform forward state space modeling with the first embedded feature as input to obtain a first-direction image coding feature, and to perform backward state space modeling with the second embedded feature as input to obtain a second-direction image coding feature, specifically including: using the state space model, taking the first embedded feature as input, performing image encoding on the normalized first image feature starting from the upper left corner to obtain the first-direction image coding feature; using the state space model, taking the second embedded feature as input, performing image encoding on the normalized first image feature starting from the lower right corner to obtain the second-direction image coding feature.

[0083] In one possible embodiment, the acquisition module 71 is used to input the second image feature into the second module and output the third image feature based on the second module, specifically including: inputting the second image feature into the second module and obtaining the global statistical feature and hidden feature in the second image feature; performing average pooling processing, fully connected mapping processing and activation function processing on the global statistical feature through the channel attention mechanism to obtain channel weighted features; performing convolution processing, average pooling processing and activation function processing on the hidden feature through the spatial attention mechanism to obtain spatial weighted features; outputting the channel weighted features and the spatial weighted features and splicing them, and inputting them into the self-attention module for attention weighted calculation; after the weighted calculation, obtaining the text feature and the third image feature through linear transformation operation and normalization operation.

[0084] In one possible implementation, the processing module 72 is used to generate image subtitle text based on global image features and global text features in combination with position encoding and masking mechanisms, specifically including: semantically aligning global image features and global text features through image-text cross-modal alignment loss; generating image subtitle text based on the semantically aligned global image features and global text features through position encoding and masking mechanisms.

[0085] In one possible implementation, the processing module 72 is configured to semantically align global image features and global text features using an image-text cross-modal alignment loss, specifically including: constructing an alignment loss from image to text using the following formula: ; in, is the alignment loss from image to text, represents the number of image-text pairs in a training batch, represents the global image features, represents the global text features, Indicates the first The text features corresponding to the text, Indicates the first The text features corresponding to the text, is the hyperparameter of normalized softmax, represents the total time step, It is used to construct the exponential part in the softmax function and construct the alignment loss from text to image through the following formula: ; in, is the alignment loss from text to image, Indicates the first The image features corresponding to the images, Indicates the first The image features corresponding to the images are used to construct the image-text cross-modal alignment loss through the alignment loss from image to text and the alignment loss from text to image: ; in, The image-text cross-modal alignment loss is used to semantically align global image features and global text features.

[0086] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0087] This application also provides an electronic device. Figure 8 , Figure 88 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. The electronic device may include: at least one processor 801, at least one communication bus 802, a user interface 803, at least one network interface 804, and a memory 805.

[0088] The communication bus 802 is used to implement the connection and communication between these components.

[0089] The user interface 803 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 803 may also include a standard wired interface and a wireless interface.

[0090] The network interface 804 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0091] The processor 801 may include one or more processing cores. Using various interfaces and circuits, the processor 801 connects to various components within the server. It executes instructions, programs, code sets, or instruction sets stored in the memory 805, as well as accesses data stored in the memory 805, to perform various server functions and process data. Optionally, the processor 801 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 801 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display screen; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 801 but implemented as a separate chip.

[0092] Among them, the memory 805 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 805 includes a non-transitory computer-readable storage medium. The memory 805 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 805 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 805 may also optionally be at least one storage device located away from the aforementioned processor 801. Refer to Figure 8 , as a computer storage medium, the memory 805 may include an operating system, a network communication module, a user interface module, and a remote sensing image caption generation application based on channel-spatial attention and Mamba network.

[0093] exist Figure 8 In the electronic device shown, the user interface 803 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 801 can be used to call the remote sensing image subtitle generation application based on channel-spatial attention and Mamba network stored in the memory 805. When executed by one or more processors 801, the electronic device executes one or more methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited to the described order of actions, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0094] The present application also provides a computer-readable storage medium storing instructions, which, when executed by one or more processors, enable an electronic device to execute one or more of the methods described in the above embodiments.

[0095] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0097] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0098] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this application. The aforementioned memory includes various media that can store program code, such as USB flash drives, mobile hard drives, magnetic disks, or optical disks.

[0100] The above descriptions are merely exemplary embodiments disclosed in this application and are not intended to limit the scope of this application. That is, any equivalent changes and modifications made based on the teachings disclosed in this application are still within the scope of this application.

[0101] This application is intended to cover any modifications, uses or adaptations disclosed in this application, which follow the general principles disclosed in this application and include common knowledge or customary technical means in the technical field not disclosed in this application.

Claims

1. A remote sensing image caption generation method based on channel-spatial attention and Mamba network, characterized in that: The method comprises: Inputting the remote sensing image into a pre-trained convolutional neural network, and outputting a first image feature according to the pre-trained convolutional neural network; inputting the first image feature into a first module, and outputting a second image feature based on the first module; inputting the second image feature into a second module, and outputting a third image feature based on the second module; Inputting the third image feature into a CSA Mamba block, and extracting a global image feature based on the CSA Mamba block; Inputting the annotation sentence corresponding to the remote sensing image into a Transformer module, and extracting global text features based on the Transformer module; Based on the global image features and the global text features, image caption text is generated in combination with position encoding and masking mechanisms.

2. The method according to claim 1, characterized in that Inputting the remote sensing image into a pre-trained convolutional neural network, and outputting a first image feature according to the pre-trained convolutional neural network, specifically includes: The remote sensing image is input into a pre-trained convolutional neural network, and the remote sensing image is encoded through the backbone network of the convolutional neural network, and the first image feature is obtained.

3. The method according to claim 1, characterized in that Inputting the first image feature into the first module and outputting the second image feature based on the first module specifically includes: Flattening the first image feature in the spatial dimension and the channel dimension for normalization; Mapping the normalized first image feature into a first embedded feature and a second embedded feature through two different linear transformations respectively; Taking the first embedded feature as input, performing forward state space modeling to obtain a first directional image coding feature, and taking the second embedded feature as input, performing backward state space modeling to obtain a second directional image coding feature; Feature fusion is performed on the first directional image coding feature and the second directional image coding feature to obtain the second image feature.

4. The method according to claim 3, characterized in that The performing forward state space modeling using the first embedded feature as input to obtain a first directional image coding feature, and performing backward state space modeling using the second embedded feature as input to obtain a second directional image coding feature, specifically includes: Using a state-space model, taking the first embedded feature as input, and performing image encoding on the normalized first image feature starting from the upper left corner to obtain a first directional image encoding feature; Using the state space model and taking the second embedded feature as input, image encoding is performed on the normalized first image feature starting from the lower right corner to obtain the second directional image encoding feature.

5. The method according to claim 1, wherein Inputting the second image feature into the second module, and outputting the third image feature based on the second module, specifically includes: Inputting the second image features into the second module, and obtaining global statistical features and hidden features in the second image features; The global statistical features are subjected to average pooling processing, full connection mapping processing, and activation function processing through a channel attention mechanism to obtain channel-weighted features; The hidden features are subjected to convolution processing, average pooling processing, and activation function processing through a spatial attention mechanism to obtain spatially weighted features; The channel weighted features and spatial weighted features are output and spliced, and input into the self-attention module for attention weighted calculation; After performing the weighted calculation, the third image feature is obtained through linear transformation and normalization operations.

6. The method according to claim 1, characterized in that The generating of image caption text based on the global image features and the global text features in combination with position encoding and masking mechanism specifically includes: Semantically aligning the global image features and the global text features through image-text cross-modal alignment loss; The image caption text is generated through the position encoding and the mask mechanism and based on the semantically aligned global image features and the global text features.

7. The method according to claim 1, characterized in that The semantic alignment of the global image features and the global text features using the image-text cross-modal alignment loss specifically includes: The alignment loss from image to text is constructed as follows: ; in, is the image-to-text alignment loss, represents the number of image-text pairs in a training batch, represents the global image feature, represents the global text feature, Indicates the number of The text features corresponding to the text, Indicates the number of The text features corresponding to the text, is the hyperparameter of normalized softmax, represents the total time step, It is used to construct the exponential part in softmax and construct the alignment loss from text to image through the following formula: ; in, is the alignment loss from text to image, Indicates the number of The image features corresponding to the images, Indicates the number of The image features corresponding to the images are used to construct the image-text cross-modal alignment loss through the alignment loss from image to text and the alignment loss from text to image: ； in, The image-text cross-modal alignment loss is used to semantically align the global image features and the global text features.

8. A remote sensing image caption generation device based on channel-spatial attention and Mamba network, characterized in that: The device includes an acquisition module and a processing module, wherein: The acquisition module is configured to input the remote sensing image into a pre-trained convolutional neural network and output a first image feature based on the pre-trained convolutional neural network; input the first image feature into a first module and output a second image feature based on the first module; input the second image feature into a second module and output a third image feature based on the second module; The processing module is configured to input the third image feature into a CSA Mamba block and extract global image features based on the CSA Mamba block; input the remote sensing image into a Transformer module and extract global text features based on the Transformer module; and generate image caption text based on the global image feature and the global text feature in combination with a position encoding and masking mechanism.

9. An electronic device, characterized in that: The electronic device comprises a processor, a communication bus, a user interface, a network interface and a memory, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images