A remote sensing image description generation method, system, device and medium based on remote sensing image-text contrast pre-training features, modal alignment and multi-scale extraction

Abstract

A modal alignment and multi-scale extraction remote sensing image description generation method, system, device and medium based on remote sensing text and image contrast pre-training features, the method comprises the following steps: obtaining a remote sensing caption generation dataset NWPU-Captions; preprocessing the text and image of the training set, validation set and test set of the remote sensing caption generation dataset NWPU-Captions to obtain a word table, text features and image features; building a remote sensing caption generation network based on a Transformer model; training the remote sensing caption generation network based on the Transformer model using the training set, then inputting the validation set into each round of the model for validation to select the best model; inputting the image features in the test set into the best model to obtain the corresponding captions of the images; the system, device and medium are used to implement the method; the image encoder, multi-scale fusion extraction module and text encoding module pre-trained by contrast are used in the application, which can extract more semantic and multi-scale features, and the generated description can contain more advanced and detailed semantic words.

Description

Technical Field

[0001] This invention belongs to the field of image-language interaction technology, and specifically relates to a method, system, device, and medium for modal alignment and multi-scale extraction of remote sensing image descriptions based on pre-trained features of remote sensing image-text comparison. Background Technology

[0002] With the development of sensor technology, the amount of high-resolution remote sensing image data has increased significantly. Accurate and rapid understanding of remote sensing images is crucial for practical applications such as ground feature image retrieval and land resource monitoring. The focus of remote sensing image captioning is generating a reliable sentence to describe the semantic content of an image containing a scene, including scene category and semantic relationships between target elements. Because remote sensing images contain rich ground feature information, directly using pre-trained image encoders based on object detection and classification tasks in the natural image captioning field to extract image features is not suitable for remote sensing image captioning tasks. These methods can only simply describe ground feature information and cannot identify high-level semantic content, such as schools or ports; furthermore, they cannot represent the relationships between ground features.

[0003] The mainstream approach in remote sensing image captioning is based on an encoder-decoder architecture. The encoder extracts image features and maps them to a high-dimensional space, while the text is mapped to this high-dimensional space via a decoder. Sentences are then generated based on the encoded image features. The most common encoder-decoder framework uses a Convolutional Neural Network (CNN) or ViT as the encoder to extract features from the remote sensing image, and a Long Short-Term Memory (LSTM) or Transformer as the decoder to generate captions. However, the encoder and decoder cannot be trained simultaneously. Typically, pre-trained image encoders on other tasks and datasets are used. These encoders are limited by predefined labels, and directly using them to extract remote sensing image features can lead to the neglect of complex and diverse details in the image. Furthermore, during training, we found significant differences between image and text features, meaning they are not well aligned. This also results in poor caption quality, such as incomplete extraction of ground features and incorrect counting of key targets.

[0004] Patent application CN115035508A discloses a method for training a ViT image encoder using a remote sensing image scene classification pre-training task. This method utilizes the scene labels inherent in the dataset to classify images, thereby alleviating the aforementioned data mismatch problem. However, this method still uses a classification task for pre-training, and although it can learn some high-level semantic features, these are limited to high-level semantic features specific to scenes.

[0005] Although Meng et al. used a pre-trained image encoder in their paper "A Multiscale Grouping Transformer With CLIP Latents for Remote Sensing Image Captioning", the encoder was only pre-trained on natural images and still had the data mismatch problem mentioned above.

[0006] In their paper "Aware-Transformer: A Novel Pure Transformer-Based Model for Remote Sensing Image Captioning", Cao et al. used a multi-scale information fusion model. Before the image features were input into the decoder, the multi-scale features had already been fused and extracted. However, this model could not perform targeted image feature extraction based on text features. That is, it extracted low-level image features at the low-level text feature layer and high-level image features at the high-level text feature layer.

[0007] In their paper "HCNet: Hierarchical Feature Aggregation and Cross-Modal Feature Alignment for Remote Sensing Image Captioning," Yang et al. proposed a cross-modal feature interaction module and a cross-modal feature alignment loss to facilitate feature alignment. However, the authors used an LSTM (Long Short-Term Memory) decoder, which only facilitated the alignment of a single word generated in the previous step with image features. The cross-modal feature alignment loss solution had limited effectiveness.

[0008] Based on the above analysis, the shortcomings of existing technologies are as follows:

[0009] First, existing methods using image encoders often fail to effectively extract high-level semantic features, or the natural images used for pre-training differ significantly from remotely sensed images. In short, the image encoders used in the target method are not well-suited for the task. Second, existing methods' multi-scale feature fusion techniques only perform fusion without effectively extracting features from different layers of the decoder. Third, existing methods' multi-modal feature alignment methods do not adequately align the entire image features with the entire text features. Summary of the Invention

[0010] In order to overcome the shortcomings of the existing technology, the present invention aims to provide a method, system, device and medium for generating remote sensing image descriptions based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text comparison. By using a comparison pre-trained image encoder, a multi-scale fusion extraction module and a text encoding module, more semantic and multi-scale features can be extracted, and the generated descriptions can contain higher-level and more detailed semantic vocabulary.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A method for generating remote sensing image descriptions based on modal alignment and multi-scale extraction using pre-trained features for remote sensing image-text comparison includes the following steps:

[0013] Step 1: Obtain the remote sensing caption generation dataset NWPU-Captions, including the training set, validation set, and test set.

[0014] Step 2: Preprocess the text and images of the training, validation, and test sets of the remote sensing caption generation dataset NWPU-Captions obtained in Step 1. Text preprocessing includes serialization, padding, and vocabulary generation to obtain vocabulary and text features. Image preprocessing includes scaling to a uniform size and normalization to obtain image features.

[0015] Step 3: Build a remote sensing caption generation network based on the Transformer model, including: an image encoder module, a text encoder module, and a Transformer decoder module; and set the network's hyperparameters, including: the number of feature extraction layers in the image encoder module, the number of layers in the Transformer decoder module; the number of layers in the text encoder module, the number of heads in the multi-head attention, the dimension of the hidden layers, and settings related to the learning process, including optimizer, learning rate, and warm-up steps.

[0016] Step 4: Input the image features and text features of the training set preprocessed in Step 2 into the remote sensing caption generation network based on the Transformer model built in Step 3. Train the generated descriptions using cross-entropy loss to obtain the trained remote sensing caption generation network based on the Transformer model. Then, input the validation set into the model obtained in each round for validation. Based on the performance on the validation set, select the model with the best performance.

[0017] Step 5: Input the image features from the test set after preprocessing in Step 2 into the model with the best performance in Step 4 to obtain the captions corresponding to the images.

[0018] The text preprocessing in step 2 includes:

[0019] Step 2.1: Sort all words in the text of the training set, validation set and test set in descending order of word frequency, replace all words with a frequency lower than 1 to 10 with unknown words [UNK], and then encode the words starting from zero according to word frequency to generate a word list.

[0020] Step 2.2: After the vocabulary list is generated, the words in the original sentence are sequenced according to the vocabulary list, and then the lengths are standardized. Long words are truncated and short words are filled in to obtain text features.

[0021] Image preprocessing includes: first scaling the images of the training set, validation set, and test set to 224×224, and then normalizing the three RGB channels according to the mean [0.406, 0.423, 0.390] and standard deviation [0.188, 0.175, 0.185] to obtain image features.

[0022] The remote sensing caption generation network based on the Transformer model in step 3 includes an image encoder module, a text encoder module, and a Transformer decoder module.

[0023] Image encoder module: Used to extract image features. This encoder fine-tunes the ViT model that has been pre-trained on the natural image dataset using the publicly available RS5M remote sensing image dataset. The image encoder module consists of 32 Transformer sub-layers to extract multi-layer image features, utilizing features at different levels. The feature dimension of each layer is 256×1280.

[0024] The text encoder module is used to encode text features to obtain context-dependent text features. The text encoder module includes a self-attention layer and a two-layer feedforward layer. The input is the text features preprocessed in step 2, which are then passed through a word embedding layer and subjected to positional encoding. In the self-attention layer, the text features serve as the Q, K, and V of attention, and are learned according to the context to learn the relationship between different words. Two feedforward layers are introduced to first change the dimension to 2048, and then back to 512, so that the input and output dimensions of the entire module remain unchanged.

[0025] Each layer of the text encoder module is represented by the formula:

[0026]

[0027] Y = LayerNorm(X + T) (1.2)

[0028] Z=FFN(Y)=(YW1+b1)W2+b2 (1.3)

[0029] S = LayerNorm(Z+Y) (1.4)

[0030] Where T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, SelfAttn is the self-attention, FFN is the feedforward layer, and LayerNorm is the layer normalization.

[0031] The Transformer decoder module is used for adaptive extraction of multi-scale features and fusion of text features. It consists of a self-attention layer, a multi-scale feature fusion extraction layer, a cross-attention layer, and a feedforward layer. The input consists of features output from the text encoding module and multi-layered image features extracted by the image encoder module. The self-attention layer, feedforward layer, and text encoder module are identical. The multi-scale feature fusion extraction layer includes two steps: fusion and extraction. The fusion step directly concatenates the extracted multi-layered image features along the channel dimension, and then adjusts the dimension back to 512 dimensions through a linear layer. The extraction step applies channel attention and spatial attention to the fused features, resulting in channel-weighted and spatially weighted features. In the cross-attention layer, image features are used as K and V, and text features as Q. Cross-attention is applied to each feature, and the resulting features are processed through a linear layer and a sigmoid function to calculate their corresponding weights. Each feature is then weighted using its own weight to obtain the fused text and image features. The fused features have both spatial and channel dimensions; these two features are summed and then passed through a feedforward layer to obtain the output.

[0032] Each layer of the Transformer decoder module is represented by the following formula:

[0033] X = SelfAttn(TW) q TW k TW v (2.1)

[0034] Y = LayerNorm(X + T) (2.2)

[0035] I=Mullayer_Fusion(V1,V2,V3)=FFN(Concat(V1,V2,V3) (2.3)

[0036] F S =SpatialAttention(I),F c =ChannelAttention(I) (2.4)

[0037] SCA = CrossAttention(Y,F) S ,F S ), CCA = CrossAttention(Y,F c ,F c(2.5)

[0038] σ s =Sigmoid(FFN(Concat(Y,SCA))),σ c =Sigmoid(FFN(Concat(Y,CCA))) (2.6)

[0039] Fsion(SCA,CCA)=SCA⊙σ s +CCA⊙σ c (2.7)

[0040] H=LayerNorm(Fsion(SCA,CCA)+Y) (2.8)

[0041] Z = FFN(H + Y) (2.9)

[0042] S = LayerNorm(Z+H) (2.10)

[0043] Where T represents the input text features, W is the weight matrix, b is the bias, d is the scaling factor, V1, V2, V3 are the multi-layer image features extracted from the image encoding, LayerNorm is the layer normalization, SpatialAttention is the spatial attention, ChannelAttention is the channel attention, CrossAttention is the cross attention, FFN is the linear layer, Concat is the concatenation operation, "+" represents element-wise addition, and ⊙ represents element-wise multiplication.

[0044] The loss during the training process in step 4 is the cross-entropy loss L. XE The details are as follows:

[0045] Cross-entropy loss L XE :

[0046]

[0047] In the formula, T represents the sentence length. Indicates prediction words The true labels of the first t-1 words, θ represents the model parameters, and p θ Indicates prediction words The probability distribution.

[0048] This invention also provides a remote sensing image description generation system based on modality alignment and multi-scale extraction of remote sensing image descriptions using pre-trained features for remote sensing image-text comparison, comprising:

[0049] The dataset acquisition module is used to acquire the remote sensing caption generation dataset NWPU-Captions, including the training set, validation set, and test set.

[0050] The dataset processing module is used to preprocess the text and images of the training, validation, and test sets of the acquired remote sensing caption generation dataset NWPU-Captions. Text preprocessing includes serialization, padding, and vocabulary generation to obtain the vocabulary and text features; image preprocessing includes scaling to a uniform size and normalization to obtain image features.

[0051] This module is for building a remote sensing caption generation network based on the Transformer model. It includes an image encoder module, a text encoder module, and a Transformer decoder module. It also sets the network's hyperparameters, including the number of feature extraction layers in the image encoder module, the number of layers in the Transformer decoder module, the number of layers in the text encoder module, the number of heads in the multi-head attention mechanism, the dimension of the hidden layers, and settings related to the learning process, including the optimizer, learning rate, and warm-up steps.

[0052] The model training module is used to input the image and text features of the preprocessed training set into the remote sensing caption generation network based on the Transformer model. The generated descriptions are trained using cross-entropy loss to obtain the trained remote sensing caption generation network based on the Transformer model. The validation set is then input into the model for each round for validation. Based on the performance on the validation set, the model with the best performance is selected.

[0053] The output module is used to input the image features from the preprocessed test set into the model with the best performance to obtain the captions corresponding to the images.

[0054] This invention also provides a remote sensing image description generation device based on modality alignment and multi-scale extraction of pre-trained features for remote sensing image-text comparison, comprising:

[0055] Memory: A computer program that stores the above-mentioned method for modal alignment and multi-scale extraction of remote sensing image description generation based on pre-trained features of remote sensing image-text comparison, and is a computer-readable device.

[0056] Processor: Used to implement the method for generating remote sensing image descriptions based on modality alignment and multi-scale extraction of pre-trained features for remote sensing image-text comparison when executing the computer program.

[0057] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for modal alignment and multi-scale extraction of remote sensing image descriptions based on pre-trained features of remote sensing image-text comparison.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] 1. Compared with existing methods that use classification-pre-trained image encoders, this invention uses remote sensing image pre-trained image encoders to extract image features, which can extract high-level semantic features and exceed the limitations of predefined labels, making it more suitable for remote sensing description generation tasks.

[0060] 2. In existing methods, the fusion of multi-scale features does not take into account the different textual information that different layers of the decoder focus on, and the fusion of multi-scale features is performed directly before the decoder. In contrast, this invention uses a multi-scale feature fusion and extraction module to fuse and extract multi-scale features in different layers of the Transformer decoder module, which can adaptively learn to focus on specific regions and channels of the image.

[0061] 3. This invention, through its proposed text encoder module, can alleviate the problem of poor description generation caused by misalignment of image and text features. In the lower layers of the decoder, high-level image features and text features are directly fused, and excessive differences between images and text lead to low utilization of image information. The text encoder proposed in this invention directly solves this problem.

[0062] In summary, the remote sensing caption generation model based on CLIP image-text comparison pre-trained features of the Transformer proposed in this invention, compared with existing technologies, fully considers the differences between the image encoder used in existing methods and the data and tasks being performed, making it more suitable for remote sensing image captioning tasks. Simultaneously, the proposed text encoder aligns image and text features to a certain extent, making better use of image and text features. Finally, the proposed multi-scale fusion extraction model is the first in the field to extract semantic features at different levels in different layers of the Transformer. Compared with the description generation performance of the ResNet+Transformer baseline model, on the NWPU-Captions dataset, the number of scene description errors generated by the proposed model is reduced from 214 to 65, and the key object counting error is also significantly improved. Attached Figure Description

[0063] Figure 1 This is a flowchart of the implementation method of the present invention.

[0064] Figure 2 This is a network structure diagram proposed in an embodiment of the present invention. Detailed Implementation

[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0066] This invention discloses a remote sensing caption generation method based on CLIP image-text comparison pre-trained features using a Transformer, primarily addressing three problems existing in current technologies for remote sensing image caption generation. First, there is the mismatch between the image encoder used and the task, as well as data mismatch. Regarding the task, most currently used image encoders are pre-trained for classification, which is incompatible with the caption generation task. Regarding data, most current image encoders are pre-trained on natural images, but this invention works on remote sensing images, which differ in target scale, viewpoint, and spatial resolution. This mismatch leads to extracted features that are unsuitable for remote sensing caption generation. Second, there is the problem of unaligned image and text features, resulting in low utilization of image features and poor generation quality. Third, current methods using the Transformer model for remote sensing image caption generation typically only fuse multi-scale features without fully extracting them in the decoder, leading to missed detections of both small and large targets. The proposed solution involves using a GeoRSCLIP encoder, pre-trained with CLIP on remote sensing image and text data, to extract multi-layered image features. These multi-layered image features are then input into a Transformer-based model improved in this invention for text generation. In this model, an additional text encoding layer is added to the decoder, allowing text features to learn global context and deep semantic information before being fused with high-level visual features, mitigating the misalignment issue between image and text features. Furthermore, a multi-scale fusion extraction module is proposed, incorporating channel attention and spatial attention to extract semantic information from different positions and channels at different decoder layers, making the decoder more suitable for remote sensing captioning tasks. Compared to existing technologies, the proposed Transformer-based remote sensing captioning model, based on image-text comparison pre-trained features, fully considers the differences between the image encoder used in existing methods and the data and task being performed, making it more suitable for remote sensing image captioning tasks. Simultaneously, the proposed text encoder aligns image and text features to a certain extent, better utilizing both image and text features. Finally, the proposed multi-scale fusion extraction module is the first in this field to extract different levels of semantic features at different layers of the Transformer.

[0067] like Figure 1 As shown, a method for generating remote sensing image descriptions based on modal alignment and multi-scale extraction using pre-trained features for remote sensing image-text comparison includes the following steps:

[0068] Step 1: Obtain the remote sensing caption generation dataset NWPU-Captions from a public website. This dataset contains 31,500 256×256 images, each with 5 corresponding texts, and has been divided into training, validation, and test sets in an 8:1:1 ratio.

[0069] Step 2 involves preprocessing the text and images from the training, validation, and test sets of the remote sensing caption generation dataset NWPU-Captions obtained in Step 1. Text preprocessing includes serialization, padding, and vocabulary generation to obtain the vocabulary and text features for direct use later. Image preprocessing includes scaling to a uniform size and normalization to obtain image features; this is necessary because the image encoder used requires images of a fixed size, and normalization ensures a more uniform distribution of the input data.

[0070] The text preprocessing in step 2 includes:

[0071] Step 2.1: Sort all words in the text of the training set, validation set and test set in descending order of word frequency, replace all words with a word frequency lower than 5 with unknown words [UNK], and then encode the words from zero according to word frequency to generate a word list;

[0072] Step 2.2: After the vocabulary list is generated, the words in the original sentence are sequenced according to the vocabulary list, and then the length is uniformly set to 16. Long words are directly truncated, and short words are padded to 16 to obtain text features.

[0073] Image preprocessing includes: first scaling the images of the training set, validation set, and test set to 224×224, and then normalizing the three RGB channels according to the mean [0.406, 0.423, 0.390] and standard deviation [0.188, 0.175, 0.185] to obtain image features.

[0074] like Figure 2 As shown, step 3 involves building a remote sensing caption generation network based on the Transformer model, including an image encoder module, a text encoder module, and a Transformer decoder module; and setting the network's hyperparameters, including the number of feature extraction layers in the image encoder module, the number of layers in the Transformer decoder module, the number of layers in the text encoder module, the number of heads in the multi-head attention, the dimension of the hidden layers, and settings related to the learning process, including the optimizer, learning rate, and warm-up steps.

[0075] The remote sensing caption generation network based on the Transformer model in step 3 includes an image encoder module, a text encoder module, and a Transformer decoder module.

[0076] Image encoder module: Used to extract image features. This encoder uses a CLIP image encoder fine-tuned on the RS5M remote sensing image dataset, specifically ViT-H, i.e., fine-tuned by comparing remote sensing image data with a pre-trained CLIP image encoder based on natural image data. The feature dimension of an image output by this image encoder is (256, 1024), where 256 is the number of image tokens and 1024 is the channel dimension of the features. This encoder has 32 layers. This invention extracts features from multiple layers to obtain ground features at different scales. Experiments show that fusing features extracted from 20, 25, and 32 layers yields better results. The extracted multi-layer image features are concatenated along the channel dimension, resulting in an image with a feature dimension of (256, 3072).

[0077] The image encoder module consists of 32 Transformer sub-layers to extract multi-layer image features. This invention utilizes features from different layers, namely the 20th, 25th, and 32nd layers, with each layer having a feature dimension of 256×1280. This invention selects to use an image encoder pre-trained on remote sensing image data to extract image features. Experiments show that the image encoder used in this invention achieves good results.

[0078] Text encoder module: used to encode text features to obtain context-related text features; the text encoder module includes a self-attention layer and a two-layer feedforward layer. Inspired by CoCa, the original Transformer decoder fuses text and images from the beginning, but the text has not yet learned to obtain context features well. Therefore, this invention draws on the structure of the CoCa model and designs a text encoder before fusing text and images.

[0079] The text encoder is constructed similarly to the Transformer encoder. The serialized text features are first processed through a word embedding layer to obtain word vectors for each word. This word embedding layer has a dimension of 512, meaning each word vector is 512-dimensional. These vectors are then input into the text encoder. Each layer of the text encoder includes a multi-head masked self-attention layer and two feedforward layers. The number of heads for self-attention is set to 8, and the dimensions of the feedforward layers are set to 2048 and 512 respectively, ensuring that the input and output dimensions of this layer are consistent. It is worth noting that the self-attention calculation here should be masked, meaning that the model should not receive information about ungenerated words during attention calculation. The text encoding layers are stacked four times to ensure that a good contextual vector is learned when fusing with image features. The maximum length of all sentences in this step is set to 17; sentences shorter than 17 are padded with 0s, and sentences longer than 17 are truncated. The resulting feature dimension is (17, 512).

[0080] The text encoder module takes as input the preprocessed text features from step 2, then passes through a word embedding layer and adds positional encoding. Self-attention requires masking because it's a generation task; generating a word means it can't see ungenerated words. In the self-attention layer, text features serve as the Q, K, and V values ​​for attention, learning relationships between different words based on context. To increase the non-linearity of the features, two feedforward layers are introduced: first, the dimension is changed to 2048, then back to 512, keeping the input and output dimensions constant. Each layer of the text encoder module is represented by the following formula:

[0081]

[0082] Y = LayerNorm(X + T) (1.2)

[0083] Z=FFN(Y)=(YW1+b1)W2+b2 (1.3)

[0084] S = LayerNorm(Z+Y) (1.4)

[0085] Where T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, SelfAttn is the self-attention, FFN is the feedforward layer, and LayerNorm is the layer normalization.

[0086] The Transformer decoder module is used for adaptive extraction of multi-scale features and fusion of text features. It consists of a self-attention layer, a multi-scale feature fusion and extraction layer, a cross-attention layer, and a feedforward layer. In the original Transformer decoder, only the features of the top layer of the Transformer are used. However, research shows that image features from different sub-layers of the Transformer contain different scale information; and the text features in different sub-layers focus on different words. Therefore, this invention utilizes multi-layer features from the image encoder and performs different fusions in the decoders of different layers, dynamically extracting key information from the image based on the focus of the text features.

[0087] The Transformer decoder module takes as input the features output from the text encoder module and the multi-layer image features extracted by the image encoder module. The self-attention layer, feedforward layer, and text encoder module are identical and will not be described further. The multi-scale feature fusion extraction layer includes two steps: fusion and extraction. The fusion step directly concatenates the extracted multi-layer image features along the channel dimension, passes them through a linear layer to adjust the dimension back to 512, and then adds them to the last layer of image features to ensure gradient stability. The resulting image features are then subjected to channel attention and spatial attention separately, yielding weighted image and spatial features. The linear layer for channel attention in the spatial attention layer is set to one layer, with the input and output dimensions being the same, resulting in an image dimension of (256, 512). The text features input to this layer pass through a multi-head self-attention layer, which also includes a mask to prevent the model from seeing ungenerated words, maintaining the same input and output dimensions. Then, the text features are used as Q for multi-head cross-attention, and the image features as K and V for feature fusion. The two fused features are each passed through a linear layer to calculate their corresponding weights, and then weighted and fused separately. Finally, there are two feedforward layers with dimensions of 2048 and 512 respectively, and a total of four decoder layers are stacked.

[0088] Each layer of the Transformer decoder module is represented by the following formula:

[0089] X = SelfAttn(TW) q TW k TW v (2.1)

[0090] Y = LayerNorm(X + T) (2.2)

[0091] I=Mullayer_Fusion(V1,V2,V3)=FFN(Concat(V1,V2,V3) (2.3)

[0092] F S =SpatialAttention(I),F c =ChannelAttention(I) (2.4)

[0093] SCA = CrossAttention(Y,F) S ,F S ), CCA = CrossAttention(Y,F c ,F c (2.5)

[0094] σ s=Sigmoid(FFN(Concat(Y,SCA))),σ c =Sigmoid(FFN(Concat(Y,CCA))) (2.6)

[0095] Fsion(SCA,CCA)=SCA⊙σ s +CCA⊙σ c (2.7)

[0096] H=LayerNorm(Fsion(SCA,CCA)+Y) (2.8)

[0097] Z = FFN(H + Y) (2.9)

[0098] S = LayerNorm(Z+H) (2.10)

[0099] Where T represents the input text features, W is the weight matrix, b is the bias, d is the scaling factor, V1, V2, V3 are the multi-layer image features extracted from the image encoding, LayerNorm is the layer normalization, SpatialAttention is the spatial attention, ChannelAttention is the channel attention, CrossAttention is the cross attention, FFN is the linear layer, Concat is the concatenation operation, "+" represents element-wise addition, and ⊙ represents element-wise multiplication.

[0100] The fused features are generated, and the feature dimension output by the Transformer decoder module is (256, 512). After passing through a linear layer, it is mapped to a vocabulary dimension of 2678, and then passed through a Softmax function to obtain the probability distribution of the generated words.

[0101] Network hyperparameter settings: Transformer decoder module has 4 layers; text encoder module has 4 layers; multi-head attention has 8 heads; hidden layer dimension is 512.

[0102] Training parameter settings: The cross-entropy loss function is used for the entire model, the Noamopt optimizer is selected, the initial learning rate is set to 10E-5, the number of warm-up steps is set to 800, and the batch size is set to 150.

[0103] Step 4: Input the image features and text features of the training set preprocessed in Step 2 into the remote sensing caption generation network based on the Transformer model built in Step 3. Train the generated descriptions using cross-entropy loss. Use 25,200 images from the NWPU-Captions training set for training the above model. According to simulation experiments, the model converges when the cross-entropy loss is reduced to about 0.4. At this point, the best index can be obtained on the validation set, i.e., select the model with the best performance.

[0104] The network loss during the training process is the cross-entropy loss L. XE The details are as follows:

[0105] Cross-entropy loss L XE :

[0106]

[0107] In the formula, T represents the sentence length. Indicates prediction words The true labels of the first t-1 words, θ represents the model parameters, and p θ Indicates prediction words The probability distribution.

[0108] Step 5: Input the image features from the test set after preprocessing in Step 2 into the model with the best performance in Step 4 to obtain the captions corresponding to the images.

[0109] The overall technical architecture problem that this invention aims to solve is the mismatch between the image encoders used in existing technologies for remote sensing image caption generation and the task being performed. This mismatch manifests in two main aspects: First, in terms of the task, most currently used image encoders are pre-trained for classification, which is incompatible with the task of caption generation, resulting in extracted features that are unsuitable for the caption generation task. Second, in terms of data, most currently used image encoders are pre-trained on natural images, but the task of this invention is performed on remote sensing images, and there are differences between natural images and remote sensing images in terms of target scale, viewpoint, spatial resolution, etc.

[0110] The first specific technical problem this invention aims to solve is the lack of alignment between image features and text features when using the Transformer model to generate captions for remote sensing images. This leads to low utilization of image features and poor generation results. To address this problem, this invention proposes a text encoder for better text encoding. Compared to the original Transformer decoder, which directly starts fusing image features after a single layer of self-attention, this invention adds an additional text encoding layer to the decoder. This allows text features to learn global context and deep semantic information before fusing with high-level visual features, thus mitigating the problem of misalignment between image and text features.

[0111] The second specific technical problem to be solved by this invention is that when using the Transformer model to generate captions for remote sensing images, only multi-scale features are usually fused, but the multi-scale features are not fully extracted in the decoder. To address this second specific technical problem, this invention proposes a multi-scale feature fusion and extraction module that fuses multi-scale features from different layers of the image encoder and extracts semantic features at different levels from different layers of the decoder.

[0112] To address the overall technical architecture issues, this invention employs an image encoder that is pre-trained on a ViT (Vision Transformer) model through comparison with a text-image dataset of natural images, and then fine-tuned on a remote sensing text-image dataset without prior pre-training. This solves the data mismatch problem and alleviates the task mismatch problem, as research has shown that models pre-trained through text-image comparison can extract features containing high-level semantics.

[0113] By combining the above methods, the remote sensing caption generation network based on the Transformer model of this invention can recognize high-level semantic targets, alleviate the problem of misclassification of targets and scenes, and generate more accurate captions.

[0114] The key points and protection points of this invention are as follows:

[0115] 1. This invention proposes using a pre-trained image encoder that is compared with a remote sensing image dataset, which achieves data and task adaptation;

[0116] 2. This invention designs a text encoder to better encode text, aligning image features and text features;

[0117] 3. This invention designs a multi-scale feature fusion extraction module, which utilizes features from different layers of the image encoder to improve the ability to extract multi-scale features.

[0118] The effects of the present invention will be further explained below with reference to simulation experiments.

[0119] 1. Simulation experimental conditions:

[0120] The hardware testing platform for the simulation experiment of this invention is:

[0121] CPU: Xeon(R)CPU E5-2630v3@2.40GHz×16

[0122] Graphics card: NVIDIA Corporation GM200×2 24G

[0123] Operating System: Ubuntu 22.04.3LTS

[0124] Development environment: Python 3.9.13, cuDann 8.9.1.23, PyTorch 2.1.0

[0125] 2. Simulation content and result analysis:

[0126] Table 1 provides detailed information about the NWPU-Captions dataset;

[0127] Table 2 shows the ablation experiments for different modules on the NWPU-Captions dataset;

[0128] Table 3 compares the metrics describing remote sensing images generated using the method of this invention and other methods;

[0129] The simulation experiments of this invention were conducted on the NWPU-Captions dataset proposed by the Northwestern Polytechnical University team. Compared with other publicly available datasets, this dataset has a large number of images and rich descriptions. Detailed parameters of the data used in this invention are shown in Table 1.

[0130] Table 1. Detailed information about the NWPU-Captions dataset.

[0131] Number of images 31500 Image size 256x256 Image spatial resolution 0.2～30m Number of descriptions per image 5 Different vocabulary 3149 Number of scenes 45

[0132] The simulation experiments of this invention were conducted on the NWPU-Captions dataset. The ablation experiments for different modules are shown in Table 2. To illustrate the effectiveness of the image encoder used in this invention, it was compared with the commonly used image encoder ResNet-101, which underwent 1000-class pre-training on ImageNet. The first and second rows of Table 2 demonstrate the effectiveness of the proposed image encoder. Compared to the commonly used ResNet-101, the proposed image encoder shows significant improvements in several metrics. Similarly, the comparisons between the second and third rows, and between the third and fourth rows of Table 2, respectively show the performance improvements of the proposed text encoder and the multi-scale feature fusion extraction module.

[0133] Table 2 shows the ablation experimental results of different modules on the NWPU-Captions dataset of this invention.

[0134] method BLEU1 BLEU4 METEOR ROUGE CIDER ResNet-101+TransDec 89.0 70.5 46.2 78.4 203.3 GeoRSCLIP+TransDec 91.7 75.9 47.9 82.2 220.2 GeoRSCLIP+TextDec+TransDec 91.9 76.2 48.1 82.3 221.4 GeoRSCLIP+TextDec+MulSC+TransDec 92.7 77.2 48.2 83 225.4

[0135] TextDec: Text decoder; MulSC: Multi-scale feature fusion and extraction layer.

[0136] TransDec:Transformer Decoder

[0137] The simulation experiments were conducted on the NWPU-Captions dataset. To illustrate the comparative effect of the proposed method with other advanced methods in the field, the comparative experiments are shown in Table 3.

[0138] Table 3 Comparison of metrics for remote sensing image description generated by this invention and other advanced methods in NWPU-Captions

[0139] method BLEU1 BLEU4 METEOR ROUGE CIDER MLCA-Net 74.5 47.8 33.7 60.1 126.4 BITA 88.54 67.6 45.2 78.5 197 HCNet 89.5 71.6 47.4 80.5 209.2 Ours 92.7 77.2 48.2 83 225.4

[0140] This invention was compared with other advanced methods in the field. MLCA-Net: CNN image encoder + LSTM decoder; BITA: CLIP image encoder + LLM; HCNet: CNN image encoder + image-text alignment module + LSTM decoder. As shown in Table 3, the method of this invention achieved the best experimental results on all metrics of the NWPU-Captions dataset.

[0141] This invention also provides a remote sensing image description generation system based on modality alignment and multi-scale extraction of remote sensing image descriptions using pre-trained features for remote sensing image-text comparison, comprising:

[0142] The dataset acquisition module is used to acquire the remote sensing caption generation dataset NWPU-Captions in step 1, including the training set, validation set and test set;

[0143] The dataset processing module is used to preprocess the text and images of the training set, validation set, and test set of the remote sensing caption generation dataset NWPU-Captions obtained in step 1 in step 2. The text preprocessing includes serialization, padding, and vocabulary generation to obtain the vocabulary and text features; the image preprocessing includes scaling to a uniform size and normalization to obtain image features.

[0144] The module for building a remote sensing caption generation network based on the Transformer model is used to implement the construction of the remote sensing caption generation network based on the Transformer model in step 3. It includes: an image encoder module, a text encoder module, and a Transformer decoder module; and sets the hyperparameters of the network, including: the number of feature extraction layers in the image encoder module, the number of layers in the Transformer decoder module; the number of layers in the text encoder module, the number of heads in the multi-head attention, the dimension of the hidden layers, and settings related to the learning process, including optimizer, learning rate, and warm-up steps.

[0145] The model training module is used to input the image features and text features of the training set preprocessed in step 2 into the remote sensing caption generation network based on the Transformer model built in step 3 in step 4, train the generated descriptions using cross-entropy loss, obtain the trained remote sensing caption generation network based on the Transformer model, and then input the validation set into each round to obtain the model for validation. Based on the performance on the validation set, the model with the best performance is selected.

[0146] The result output module is used to input the image features from the test set after preprocessing in step 2 into the model with the best performance in step 4 in step 5, so as to obtain the captions corresponding to the images.

[0147] This invention also provides a remote sensing image description generation device based on modality alignment and multi-scale extraction of pre-trained features for remote sensing image-text comparison, comprising:

[0148] Memory: A computer program that stores the above-mentioned method for modal alignment and multi-scale extraction of remote sensing image description generation based on pre-trained features of remote sensing image-text comparison, and is a computer-readable device;

[0149] Processor: Used to implement the method for generating remote sensing image descriptions based on modality alignment and multi-scale extraction of pre-trained features for remote sensing image-text comparison when executing the computer program.

[0150] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for modal alignment and multi-scale extraction of remote sensing image descriptions based on pre-trained features of remote sensing image-text comparison.

Claims

1. A method for generating remote sensing image descriptions based on modal alignment and multi-scale extraction of remote sensing image-text contrast pre-trained features, characterized in that, The method comprises the following steps: Step 1, obtaining a remote sensing caption generation dataset NWPU-Captions, including a training set, a validation set and a test set; Step 2, preprocessing the text and image of the training set, the validation set and the test set of the remote sensing caption generation dataset NWPU-Captions obtained in step 1, the text preprocessing including serialization, padding and generating a word table, obtaining a word table and text features; the image preprocessing including scaling to a uniform size, normalization, obtaining image features; Step 3, building a remote sensing caption generation network based on a Transformer model, including an image encoder module, a text encoder module and a Transformer decoder module; and setting the hyperparameters of the network, including the number of feature extraction layers of the image encoder module, the number of layers of the Transformer decoder module; the number of layers of the text encoder module, the number of attention heads, the dimension of the hidden layer, and the settings related to the learning process, including the optimizer, the learning rate, the number of warm-up steps; The remote sensing caption generation network based on the Transformer model in step 3 comprises an image encoder module, a text encoder module and a Transformer decoder module; The image encoder module is used for extracting image features, and the encoder uses a public RS5M remote sensing image-text dataset to fine-tune a ViT model pre-trained on a natural image-text dataset; the image encoder module is composed of 32 Transformer sub-layers to extract multi-layer image features, and the feature dimension of each layer is 256x1280; The text encoder module is used for encoding text features to obtain context-related text features; the text encoder module comprises a self-attention layer and two feedforward layers, and the input is the text features preprocessed in step 2, and then the text features are input into a word embedding (WordEmbed) layer and added with position encoding; in the self-attention layer, the text features are used as the Q, K and V of attention, and the relationship between different words is learned according to the context, and two feedforward layers are introduced to change the dimension from 2048 to 512, so that the input and output dimensions of the entire module remain unchanged; Each layer of the text encoder module is expressed by a formula as follows: Wherein, T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, SelfAttn is the self-attention, FFN is the feedforward layer, and LayerNorm is the layer normalization. The Transformer decoder module is used for adaptive extraction of multi-scale features and fusion of text features; the Transformer decoder module is composed of a self-attention layer, a multi-scale feature fusion extraction layer, a cross-attention layer and a feedforward layer, and the input is the features output by the text encoding module and the multi-layer image features extracted by the image encoder module; the self-attention layer, the feedforward layer and the text encoder module are consistent; the multi-scale feature fusion extraction layer includes two steps of fusion and extraction: the fusion step directly splices the extracted multi-layer image features in the channel dimension, and adjusts the dimension to 512 through a linear layer; the extraction step is to respectively perform channel attention and spatial attention on the fused features to obtain channel-weighted and spatial-weighted features; in the cross-attention layer, the image features are used as K and V, and the text features are used as Q to perform cross-attention respectively, and the obtained features are respectively subjected to a linear layer and a Sigmoid function to calculate the corresponding weights, and the obtained features are respectively weighted by their own weights to obtain a fusion feature of the text and the image; the fusion feature has a spatial dimension and a channel dimension, and the two features are added to obtain an output through the feedforward layer; Each layer of the Transformer decoder module is expressed by a formula as follows: wherein T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, V1, V2, V3 are the multi-layer image features extracted by image encoding, LayerNorm is the layer normalization, for spatial attention for channel attention, CrossAttention represents cross-attention, FFN represents a linear layer, Concat represents a concatenation operation, "+" represents element-wise addition, represents element-wise multiplication; Step 4: input the image features and the text features of the training set after the preprocessing in step 2 into the remote sensing caption generation network based on the Transformer model built in step 3, perform cross-entropy loss training on the generated description, obtain a trained remote sensing caption generation network based on the Transformer model, and then input the verification set into the model obtained in each round for verification, and select the best model according to the performance on the verification set; Step 5: input the image features in the test set after the preprocessing in step 2 into the best model in step 4 to obtain the caption corresponding to the image.

2. The method of claim 1, wherein the method is based on remote sensing image description generation based on pre-training features of remote sensing text-image contrast, modal alignment and multi-scale extraction. The text preprocessing in step 2 includes: Step 2.1: sort all the words in the texts of the training set, the verification set and the test set according to the word frequency from large to small, replace all the words with a word frequency lower than 5 with an unknown word [UNK], and then encode the words from zero according to the word frequency to generate a word table; Step 2.2: after the word table is generated, serialize the words in the original sentence according to the word table, and unify the length, directly truncate the long ones and fill the short ones to obtain text features; The image preprocessing includes: first, scale the images in the training set, the verification set and the test set to 224x224, and then normalize the three channels RGB according to the mean [0.406, 0.423, 0.390] and the standard deviation [0.188, 0.175, 0.185] to obtain image features.

3. The method of claim 1, wherein the method is based on remote sensing image description generation based on pre-training features of remote sensing image-text contrast, modal alignment and multi-scale extraction. The loss in the step 4 training process is cross-entropy loss Specifically as follows: cross-entropy loss : where T denotes the sentence length, denotes the predicted word denotes the true label of the previous t-1 words, denotes the model parameters, denotes the probability distribution of the predicted word .

4. A system for modal alignment and multi-scale extraction of remote sensing image description generation based on remote sensing image-text contrast pre-training features based on any one of claims 1-3, characterized in that, The data set acquisition module is used for acquiring a remote sensing caption generation data set NWPU-Captions, including a training set, a verification set and a test set; ​ A data set processing module is configured to preprocess the text and image of the training set, the validation set and the test set of the obtained remote sensing caption generation data set NWPU-Captions, the text preprocessing includes serialization, padding and generating a word table, and the word table and the text features are obtained; the image preprocessing includes scaling to a uniform size and normalization, and the image features are obtained; A remote sensing caption generation network building module based on a Transformer model is configured to build a remote sensing caption generation network based on a Transformer model, including an image encoder module, a text encoder module and a Transformer decoder module; and setting the hyperparameters of the network, including the number of feature extraction layers of the image encoder module, the number of layers of the Transformer decoder module, the number of layers of the text encoder module, the number of heads of multi-head attention, the dimension of the hidden layer, and the settings related to the learning process, including the optimizer, the learning rate, the number of warm-up steps; A model training module is configured to input the image features and the text features of the preprocessed training set into the built remote sensing caption generation network based on a Transformer model, perform cross-entropy loss training on the generated description, obtain the trained remote sensing caption generation network based on a Transformer model, input the validation set into the model obtained in each round for validation, and select the best model according to the performance on the validation set; A result output module is configured to input the image features in the preprocessed test set into the best model to obtain the captions corresponding to the images.

5. A device for generating remote sensing image description based on modal alignment and multi-scale extraction of remote sensing image-text contrast pre-training features, characterized in that, It comprises: a memory configured to store a computer program of a remote sensing image description generation method based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text contrast, which is readable by a computer; a processor configured to execute the computer program to implement the remote sensing image description generation method based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text contrast.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the remote sensing image description generation method based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text contrast.

Images