Modal alignment and multi-scale extraction remote sensing image description generation method, system and device based on remote sensing image-text comparison pre-training features and medium

Abstract

The invention discloses a modal alignment and multi-scale extraction remote sensing image description generation method, system and device based on remote sensing image-text comparison pre-training features and a medium. The method comprises the steps of obtaining a remote sensing subtitle generation data set NWPU-Captions; preprocessing texts and images of a training set, a verification set and a test set of the remote sensing subtitle generation data set NWPU-Captions to obtain word lists, text features and image features; building a remote sensing subtitle generation network based on a Transform model; training the remote sensing subtitle generation network based on the Transform model by using the training set, inputting the verification set into the models obtained in each round for verification, and selecting the model with the best effect; inputting the image features in the test set into the model with the best effect to obtain subtitles corresponding to the image; the system, the equipment and the medium are used for implementing the method. Compared with a pre-trained image encoder, a multi-scale fusion extraction module and a text encoding module, features with more semantics and more scales can be extracted, and generated descriptions can contain semantic vocabularies with higher levels and more details.

Description

Technical Field

[0001] The present invention belongs to the technical field of image language interaction, and in particular relates to a method, system, device and medium for generating remote sensing image description based on modal alignment and multi-scale extraction of remote sensing image text comparison pre-training features. Background Art

[0002] With the development of sensor technology, the amount of high-resolution remote sensing image data has increased significantly. Accurate and rapid understanding of remote sensing images is crucial for practical applications, such as object image retrieval and land resource monitoring. The focus of remote sensing image captioning is to generate a reliable sentence to describe the semantic content of the image containing the scene, including scene categories and semantic relationships of target elements. Since remote sensing images have rich object information, directly using pre-trained image encoders based on target detection and classification tasks in the field of natural image captioning to extract image features is not suitable for remote sensing image captioning tasks. These methods can only simply describe object information and cannot identify high-level semantic content, such as schools, ports, etc.; and cannot represent the correlation between objects.

[0003] At present, the mainstream method in the field of remote sensing image captioning is based on encoder-decoder, in which the encoder is used to extract image features and map them to a high-dimensional space. The text is also mapped to this high-dimensional space through a decoder, and then the sentence is generated according to the encoded image features. The most common encoder-decoder framework is to use the convolutional neural network CNN or ViT as the encoder to extract features of remote sensing images, and the long short-term memory network LSTM or Transformer as the decoder to generate caption descriptions. However, the encoder and decoder here cannot be trained at the same time. Usually, the image encoder pre-trained on other tasks and other datasets is used. Such encoders are limited by pre-defined labels. Directly using them to extract remote sensing image features will lead to the problem of ignoring the complex and diverse details in remote sensing images. In addition, during the training process, we found that the image features and text features are very different, that is, the image features and text features are not well aligned, which also leads to poor generated captions, such as incomplete extraction of ground objects and errors in key target counts.

[0004] The patent application document with publication number CN115035508A discloses a method for training a ViT image encoder using a remote sensing image scene classification pre-training task. This method uses the scene labels of the dataset itself to classify images, thereby alleviating the data mismatch problem mentioned above. However, this method still uses classification tasks for pre-training. Although some high-level semantic features can be learned, they are limited to high-level semantic features related to the scene.

[0005] Although Meng et al. also used a pre-trained image encoder in their paper "A Multiscale Grouping Transformer With CLIP Latents for Remote Sensing Image Captioning", the encoder was only pre-trained on natural images and the data mismatch problem mentioned above still exists.

[0006] Cao et al. used a multi-scale information fusion model in their paper "Aware-Transformer: A Novel Pure Transformer-Based Model for Remote Sensing Image Captioning". Before the image features are input into the decoder, the multi-scale features have been fused and extracted. However, this model cannot perform targeted image feature extraction based on text features, that is, it extracts low-level image features at the low-level text feature layer and extracts high-level image features at the high-level text feature layer.

[0007] Yang et al. proposed a cross-modal feature interaction module and cross-modal feature alignment loss to promote feature alignment in their paper HCNet: Hierarchical Feature Aggregation and Cross-Modal Feature Alignment for Remote Sensing Image Captioning. However, the authors used an LSTM (Long Short-Term Memory) decoder, which only promoted the alignment of a word and image features generated in the previous step. The solution of cross-modal feature alignment loss has limited effect.

[0008] According to the above analysis, the shortcomings of the existing technology are:

[0009] First: the image encoder used by the existing methods cannot extract high-level semantic features well when extracting image features, or there are large differences between the natural images and remote sensing images pre-trained by the image encoder. In short, the image encoder used by the target method is not very suitable for the task being performed. Second: the image multi-scale feature fusion method of the existing method only performs fusion, and is not well extracted in the decoders of different layers. Third: the existing method does not align the entire image features and the entire text features well in the multimodal feature alignment method. Summary of the invention

[0010] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, device and medium for generating remote sensing image description based on modal alignment and multi-scale extraction of remote sensing image and text contrast pre-trained features. By using a contrast pre-trained image encoder, a multi-scale fusion extraction module and a text encoding module, more semantic and multi-scale features can be extracted, and the generated description can contain more advanced and detailed semantic vocabulary.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A method for generating remote sensing image description based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features, comprising the following steps:

[0013] Step 1: Obtain the remote sensing caption generation dataset NWPU-Captions, including training set, validation set and test set.

[0014] Step 2: Preprocess the text and images of the training set, validation set, and test set of the remote sensing caption generation dataset NWPU-Captions obtained in step 1. Text preprocessing includes serialization, padding, and word list generation to obtain word lists and text features; image preprocessing includes scaling to a uniform size and normalization to obtain image features.

[0015] Step 3, build a remote sensing subtitle generation network based on the Transformer model, including: image encoder module, text encoder module and Transformer decoder module; and set the network's hyperparameters, including: the number of extracted feature layers of the image encoder module, the number of layers of the Transformer decoder module; the number of layers of the text encoder module, the number of heads of multi-head attention, the dimension of the hidden layer, and settings related to the learning process, including optimizer, learning rate, and number of warm-up steps.

[0016] In step 4, the image features and text features of the training set preprocessed in step 2 are input into the remote sensing subtitle generation network based on the Transformer model built in step 3, and the generated description is trained with cross entropy loss to obtain a trained remote sensing subtitle generation network based on the Transformer model. The validation set is then input into the model obtained in each round for verification, and the best model is selected based on its performance on the validation set.

[0017] Step 5: Input the image features in the test set preprocessed in step 2 into the model with the best effect in step 4 to obtain the captions corresponding to the image.

[0018] The text preprocessing in step 2 includes:

[0019] Step 2.1, sort all the words in the text of the training set, validation set and test set by frequency from high to low, replace all the words with a frequency lower than 1 to 10 with unknown words [UNK], and then encode the words from zero according to the frequency to generate a vocabulary.

[0020] Step 2.2, after the vocabulary is generated, the words in the original sentence are serialized according to the vocabulary, and then the length is unified, the long ones are directly truncated, and the short ones are filled to obtain the text features.

[0021] Image preprocessing includes: the images of the training set, validation set, and test set are uniformly scaled to 224×224, and the three channels RGB are normalized according to the mean [0.406, 0.423, 0.390] and the standard deviation [0.188, 0.175, 0.185] to obtain image features.

[0022] The remote sensing caption generation network based on the Transformer model in step 3 includes an image encoder module, a text encoder module and a Transformer decoder module;

[0023] Image encoder module: used to extract image features. The encoder uses the public RS5M remote sensing image dataset to fine-tune the ViT model that has been pre-trained on the natural image dataset. The image encoder module consists of 32 layers of Transformer sub-layers to extract multi-layer image features, using features at different levels. The feature dimension of each layer is 256×1280.

[0024] Text encoder module: used to encode text features and obtain context-related text features; the text encoder module includes a self-attention layer and two feedforward layers. The input is the text features preprocessed in step 2, and then passed through the word embedding (WordEmbed) layer and position encoding. In the self-attention layer, the text features are used as attention Q, K, V, and are learned according to the context to learn the relationship between different words. Two feedforward layers are introduced to first change the dimension to 2048 and then back to 512, so that the input and output dimensions of the entire module remain unchanged;

[0025] Each layer of the text encoder module is formulated as:

[0026]

[0027] Y = LayerNorm(X + T) (1.2)

[0028] Z=FFN(Y)=(YW1+b1)W2+b2 (1.3)

[0029] S = LayerNorm(Z+Y) (1.4)

[0030] Among them, T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, SelfAttn is the self-attention, FFN is the feedforward layer, and LayerNorm is the layer normalization;

[0031] Transformer decoder module: used for adaptive extraction of multi-scale features and fusion of text features; the Transformer decoder module consists of a self-attention layer, a multi-scale feature fusion extraction layer, a cross-attention layer and a feedforward layer, and the input is the features output by the text encoding module and the multi-layer image features extracted by the image encoder module; the self-attention layer, the feedforward layer and the text encoder module are consistent; the multi-scale feature fusion extraction layer includes two steps of fusion and extraction: the fusion step directly splices the extracted multi-layer image features in the channel dimension, and adjusts the dimension back to 512 dimensions through a linear layer; the extraction step is to perform channel attention and spatial attention on the fused features respectively, and obtain channel-weighted and spatial-weighted features respectively; in the cross-attention layer, the image features are used as K, V, and the text features are used as Q, and cross-attention is performed respectively. The obtained features are respectively passed through a linear layer and a Sigmoid function to calculate the corresponding weights, and the obtained features are weighted with their own weights to obtain fusion features of fused text and image; if the fused features have spatial dimensions and channel dimensions, these two features are added and then passed through the feedforward layer to obtain the output.

[0032] Each layer of the Transformer decoder module is expressed as:

[0033] X=SelfAttn(TW q ,TW k ,TW v ) (2.1)

[0034] Y=LayerNorm(X+T) (2.2)

[0035] I=Mullayer_Fusion(V1,V2,V3)=FFN(Concat(V1,V2,V3) (2.3)

[0036] F S =SpatialAttention(I),F c = ChannelAttention(I) (2.4)

[0037] SCA=CrossAttention(Y,F S ,F S ),CCA=CrossAttention(Y,F c ,F c)(2.5)

[0038] σ s =Sigmoid(FFN(Concat(Y,SCA))),σ c =Sigmoid(FFN(Concat(Y,CCA))) (2.6)

[0039] Fsion(SCA,CCA)=SCA⊙σ s +CCA⊙σ c (2.7)

[0040] H=LayerNorm(Fsion(SCA,CCA)+Y) (2.8)

[0041] Z=FFN(H+Y) (2.9)

[0042] S = LayerNorm(Z + H) (2.10)

[0043] Among them, T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, V1, V2, V3 are multi-layer image features extracted by image encoding, LayerNorm is layer normalization, SpatialAttention is spatial attention, ChannelAttention is channel attention, CrossAttention represents cross attention, FFN represents linear layer, Concat represents concatenation operation, "+" represents element-wise addition, and ⊙ represents element-wise multiplication.

[0044] The loss during the training process in step 4 is the cross entropy loss L XE , as follows:

[0045] Cross entropy loss L XE :

[0046]

[0047] In the formula, T represents the sentence length, Predicted word The true label of the first t-1 words at the time, θ represents the model parameters, p θ Predicted word The probability distribution of .

[0048] The present invention also provides a remote sensing image description generation system based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features, including:

[0049] The dataset acquisition module is used to obtain the remote sensing caption generation dataset NWPU-Captions, including training sets, validation sets, and test sets.

[0050] The dataset processing module is used to preprocess the text and images of the training set, validation set and test set of the acquired remote sensing caption generation dataset NWPU-Captions. Text preprocessing includes serialization, filling and generating vocabulary to obtain vocabulary and text features; image preprocessing includes scaling to a uniform size and normalization to obtain image features.

[0051] The remote sensing subtitle generation network building module based on the Transformer model is used to build a remote sensing subtitle generation network based on the Transformer model, including: image encoder module, text encoder module and Transformer decoder module; and set the network's hyperparameters, including: the number of extracted feature layers of the image encoder module, the number of layers of the Transformer decoder module; the number of layers of the text encoder module, the number of heads of multi-head attention, the dimension of the hidden layer, and settings related to the learning process, including optimizer, learning rate, and number of warm-up steps.

[0052] The model training module is used to input the preprocessed image features and text features of the training set into the remote sensing subtitle generation network based on the Transformer model, perform cross-entropy loss training on the generated description, and obtain a trained remote sensing subtitle generation network based on the Transformer model. The validation set is then input into the model obtained in each round for verification, and the best model is selected based on its performance on the validation set.

[0053] The result output module is used to input the preprocessed image features in the test set into the best model to obtain the captions corresponding to the image.

[0054] The present invention also provides a remote sensing image description generation device based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features, including:

[0055] Memory: A computer program storing the above-mentioned method for generating remote sensing image description based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image and text comparison, which is a computer-readable device.

[0056] Processor: used to implement the method for generating remote sensing image description based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text comparison when executing the computer program.

[0057] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for generating remote sensing image description based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image and text comparison.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. Compared with the existing method that uses a classification pre-trained image encoder, the present invention extracts image features by using a remote sensing image and text pre-trained image encoder, which can extract high-level semantic features and exceed the limitations of predefined labels, making it more suitable for remote sensing description generation tasks.

[0060] 2. In the existing methods, the fusion of multi-scale features does not take into account the different text information that different layers of the decoder focus on, and the fusion of multi-scale features is performed directly before the decoder. However, the present invention uses a multi-scale feature fusion extraction module to fuse and extract multi-scale features in different layers of the Transformer decoder module, and can adaptively learn to focus on specific areas and channels of the image.

[0061] 3. The present invention can alleviate the problem of poor description generation caused by the misalignment of image and text features through the proposed text encoder module. In the lower layer of the decoder, high-level image features and text features are directly fused, and the large difference between images and text leads to low utilization of image information. The text encoder proposed in the present invention directly solves this problem.

[0062] In summary, the remote sensing subtitle generation model of the Transformer based on CLIP image-text comparison pre-trained features proposed in the present invention fully considers the data and task differences between the image encoder used in the existing methods and the tasks performed, and is more suitable for remote sensing image subtitle tasks. At the same time, the proposed text encoder aligns the image and text features to a certain extent, making better use of image and text features. Finally, the proposed multi-scale fusion extraction model extracts semantic features of different levels at different layers of the Transformer for the first time in this field. Compared with the description generation effect of the ResNet+Transformer baseline model, on the NWPU-Captions dataset, the number of description scene description errors generated by the model proposed in the present invention dropped from 214 to 65, and the key target counting errors were also greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a flow chart of the implementation method of the present invention.

[0064] Figure 2 It is a network structure diagram proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0066] The invention discloses a remote sensing subtitle generation method based on a Transformer of CLIP image-text comparison pre-training features, which mainly solves three problems existing in the prior art when performing remote sensing image subtitle generation. First: the image encoder used does not match the task performed and the data does not match. In terms of tasks, most of the image encoders currently used are pre-trained by classification, which does not match the task of generating subtitles. In terms of data, most of the image encoders currently used are pre-trained on natural images, but the task of the invention is to work on remote sensing images, and there are differences in target scale, viewing angle, and spatial resolution between natural images and remote sensing images. The mismatch between the two will cause the extracted features to be unsuitable for the remote sensing subtitle generation task; second: the image features and text features are not aligned, which will result in low utilization of image features and poor generation effect; third: the current method uses the Transformer model to generate remote sensing image subtitles, usually only multi-scale features are fused, but the multi-scale features are not fully extracted in the decoder, which will result in missed detection of smaller and larger targets. The scheme is: use the encoder GeoRSCLIP pre-trained by CLIP with remote sensing image and text data to extract multi-layer image features, and then input the multi-layer image features into the improved model based on Transformerg proposed by the present invention for text generation. In this model, the present invention adds an additional text encoding layer in the decoder, so that the text features are fused with high-level visual features on the basis of learning global context and deep semantic information, which alleviates the problem of misalignment of image and text features; and proposes a multi-scale fusion extraction module, which adds channel attention and spatial attention to extract semantic information of different positions and channels at different decoder layers, so that the decoder is more suitable for remote sensing subtitle tasks. Compared with the prior art, the remote sensing subtitle generation model of the Transformer based on the image and text comparison pre-training features proposed by the present invention fully considers the data and task differences between the image encoder used in the existing method and the task performed, and is more suitable for remote sensing image subtitle tasks; at the same time, the proposed text encoder aligns the image and text features to a certain extent, making better use of image and text features; finally, the proposed multi-scale fusion extraction module extracts semantic features of different levels at different layers of the Transformer for the first time in this field.

[0067] like Figure 1 As shown, a method for generating remote sensing image description based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features includes the following steps:

[0068] Step 1: Obtain the remote sensing caption generation dataset NWPU-Captions from a public website. The dataset contains 31,500 256×256 images, each with 5 corresponding texts, which have been divided into training set, validation set, and test set in a ratio of 8:1:1.

[0069] Step 2: Preprocess the text and images of the training set, validation set, and test set of the remote sensing caption generation dataset NWPU-Captions obtained in step 1. Text preprocessing includes serialization, padding, and word list generation to obtain word lists and text features for subsequent direct use. Image preprocessing includes scaling to a uniform size and normalization to obtain image features because the image encoder used requires images of a fixed size, and normalization ensures a more even distribution of input data.

[0070] The text preprocessing in step 2 includes:

[0071] Step 2.1, sort all the words in the text of the training set, validation set, and test set by frequency from high to low, replace all words with a frequency lower than 5 with unknown words [UNK], and then encode the words from zero according to frequency to generate a vocabulary;

[0072] Step 2.2, after the vocabulary is generated, the words in the original sentence are serialized according to the vocabulary, and then the length is unified to 16. The long ones are directly truncated and the short ones are padded to 16 to obtain the text features;

[0073] Image preprocessing includes: the images of the training set, validation set, and test set are uniformly scaled to 224×224, and the three channels RGB are normalized according to the mean [0.406, 0.423, 0.390] and the standard deviation [0.188, 0.175, 0.185] to obtain image features.

[0074] like Figure 2 As shown in the figure, step 3 is to build a remote sensing caption generation network based on the Transformer model, including: image encoder module, text encoder module and Transformer decoder module; and set the network's hyperparameters, including: the number of extracted feature layers of the image encoder module, the number of layers of the Transformer decoder module; the number of layers of the text encoder module, the number of heads of the multi-head attention, the dimension of the hidden layer, and settings related to the learning process, including optimizer, learning rate, and number of warm-up steps.

[0075] The remote sensing caption generation network based on the Transformer model in step 3 includes an image encoder module, a text encoder module and a Transformer decoder module.

[0076] Image encoder module: used to extract image features. The encoder uses the CLIP image encoder fine-tuned on the RS5M remote sensing image and text dataset, specifically ViT-H, that is, the remote sensing image and text data comparison pre-training and fine-tuning are performed on the CLIP image encoder pre-trained for natural images. The feature dimension of an image output by the image encoder is (256, 1024), where 256 is the number of tokens in the image and 1024 is the channel dimension of the feature. The encoder has 32 layers. The present invention extracts features from multiple layers to obtain objects of different scales. After experiments, better results can be obtained by extracting features from 20, 25, and 32 layers and fusing them. The extracted multi-layer image features are spliced ​​in the channel dimension, and the dimension of the final image feature is (256, 3072).

[0077] The image encoder module is composed of 32 layers of Transformer sub-layers to extract multi-layer image features. The present invention utilizes the features of different levels of the 20th, 25th and 32nd layers, and the feature dimension of each layer is 256×1280; the present invention chooses to use an image encoder pre-trained on remote sensing graphic data to extract image features. Experiments have found that the image encoder used in the present invention has achieved good results.

[0078] Text encoder module: used to encode text features to obtain context-related text features; the text encoder module includes a self-attention layer and a two-layer feedforward layer. Inspired by CoCa, the original Transformer decoder fuses text and images from the beginning, but the text has not yet learned to obtain good context features. Therefore, the present invention draws on the structure of the CoCa model and designs a text encoder before fusing text and images.

[0079] The construction of the text encoder is the same as that of the Transformer encoder. The serialized text features are first passed through a word embedding layer to obtain the word vector corresponding to each word. The dimension of the word embedding layer is 512, that is, the word vector of a word is 512-dimensional, and then input into the text encoder. Each layer of the text encoder includes multi-head masked self-attention and two layers of feedforward layers. The number of heads of the self-attention is set to 8, and the dimensions of the feedforward layer are set to 2048 and 512 respectively to ensure that the input and output dimensions of the layer are consistent. It is worth noting that the self-attention calculation here should be masked, that is, the model cannot obtain information about ungenerated words when calculating attention. The text encoding layer is stacked with 4 layers in total to ensure that a good context vector is learned when fused with image features. The maximum length of all sentences in this step is set to 17. Sentences with a length less than 17 are filled with 0, and those with a length greater than 17 are directly truncated. The feature dimension obtained is (17, 512).

[0080] The input of the text encoder module is the text features preprocessed in step 2, and then passed through the word embedding (WordEmbed) layer and added with position encoding; self-attention needs to be masked because it is a generation task. When a word is generated, it cannot see the ungenerated words. In the self-attention layer, the text features are used as the Q, K, and V of attention. They are learned according to the context and the relationship between different words is learned to increase the nonlinearity of the features. Two feedforward layers are introduced to first change the dimension to 2048 and then back to 512, so that the input and output dimensions of the entire module remain unchanged; each layer of the text encoder module is expressed by the formula:

[0081]

[0082] Y = LayerNorm(X + T) (1.2)

[0083] Z=FFN(Y)=(YW1+b1)W2+b2 (1.3)

[0084] S = LayerNorm(Z+Y) (1.4)

[0085] Among them, T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, SelfAttn is the self-attention, FFN is the feedforward layer, and LayerNorm is the layer normalization;

[0086] Transformer decoder module: used for adaptive extraction of multi-scale features and fusion of text features; the Transformer decoder module consists of a self-attention layer, a multi-scale feature fusion extraction layer, a cross-attention layer, and a feedforward layer. In the original Transformer decoder, only the features of the top layer of the Transformer are used, but studies have shown that the image features of different sub-layer Transformers have information of different scales; and the words focused on by the text features in different sub-layers of Transformers are different, so the present invention utilizes the multi-layer features in the image encoder and performs different fusions in the decoders of different layers to dynamically extract the key information in the image according to the focus of the text features;

[0087] The input of the Transformer decoder module is the features output by the text encoding module and the multi-layer image features extracted by the image encoder module; the self-attention layer, feedforward layer and text encoder module are consistent, so they will not be repeated; the multi-scale feature fusion extraction layer includes two steps: fusion and extraction: the fusion step directly splices the extracted multi-layer image features in the channel dimension, adjusts the dimension back to 512 dimensions through a linear layer, and then adds them to the last layer of image features to ensure the stability of the gradient. Then the image features are separated for channel attention and spatial attention, and the weighted image features and spatial features are obtained respectively. Here, the linear layer of the spatial attention and channel attention is set to one layer, the input dimension and the output dimension are consistent, and the image dimension is (256, 512). The text features input to this layer will pass through a multi-head self-attention layer, which also contains a mask to prevent the model from seeing ungenerated words. The input and output dimensions remain unchanged. Then the text features are used as Q of multi-head cross attention, and the image features are used as K, V for feature fusion. The two fused features calculated are passed through a linear layer to calculate their corresponding weights, and the two features are weighted fused respectively. Finally, the dimensions of the two feed-forward layers are 2048 and 512 respectively. There are 4 layers of such decoder layers stacked in total.

[0088] Each layer of the Transformer decoder module is expressed as:

[0089] X=SelfAttn(TW q ,TW k ,TW v ) (2.1)

[0090] Y=LayerNorm(X+T) (2.2)

[0091] I=Mullayer_Fusion(V1,V2,V3)=FFN(Concat(V1,V2,V3) (2.3)

[0092] F S =SpatialAttention(I),F c = ChannelAttention(I) (2.4)

[0093] SCA=CrossAttention(Y,F S ,F S ),CCA=CrossAttention(Y,F c ,F c )(2.5)

[0094] σ s=Sigmoid(FFN(Concat(Y,SCA))),σ c =Sigmoid(FFN(Concat(Y,CCA))) (2.6)

[0095] Fsion(SCA,CCA)=SCA⊙σ s +CCA⊙σ c (2.7)

[0096] H=LayerNorm(Fsion(SCA,CCA)+Y) (2.8)

[0097] Z=FFN(H+Y) (2.9)

[0098] S = LayerNorm(Z + H) (2.10)

[0099] Among them, T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, V1, V2, V3 are multi-layer image features extracted by image encoding, LayerNorm is layer normalization, SpatialAttention is spatial attention, ChannelAttention is channel attention, CrossAttention represents cross attention, FFN represents linear layer, Concat represents concatenation operation, "+" represents element-wise addition, and ⊙ represents element-wise multiplication.

[0100] To generate fused features, the feature dimension output by the Transformer decoder module is (256, 512), which is mapped to the same dimension 2678 as the vocabulary through a linear layer, and then passed through a Softmax function to obtain the probability distribution of the generated vocabulary.

[0101] The hyperparameter settings of the network are: the number of layers of the Transformer decoder module is 4; the number of layers of the text encoder module is 4, the number of multi-head attention heads is 8, and the dimension of the hidden layer is 512.

[0102] Training parameter settings: The cross entropy loss is used as the loss function of the entire model, the noamopt optimizer is selected as the optimizer, the initial learning rate is set to 10E-5, the number of warm-up steps is set to 800, and the batch size is set to 150;

[0103] Step 4: Input the image features and text features of the training set preprocessed in step 2 into the remote sensing caption generation network based on the Transformer model built in step 3, perform cross entropy loss training on the generated descriptions, and use the 25,200 images in the training set of NWPU-Captions for the above model training. According to simulation experiments, the cross entropy loss is reduced to about 0.4 and the model converges. At this time, the best indicator can be obtained on the validation set, that is, the model with the best effect is selected;

[0104] The network loss during the training process is the cross entropy loss L XE , as follows:

[0105] Cross entropy loss L XE :

[0106]

[0107] In the formula, T represents the sentence length, Predicted word The true label of the first t-1 words at the time, θ represents the model parameters, p θ Predicted word The probability distribution of .

[0108] Step 5: Input the image features in the test set preprocessed in step 2 into the model with the best effect in step 4 to obtain the captions corresponding to the image.

[0109] The overall technical architecture problem to be solved by the present invention is the mismatch between the image encoder used in the prior art and the task performed when generating subtitles for remote sensing images. The problem lies in two aspects: first, in terms of tasks, most of the currently used image encoders are pre-trained for classification, which do not match the task of generating subtitles, resulting in the extracted features being unsuitable for the subtitle generation task; second, in terms of data, most of the currently used image encoders are pre-trained on natural images, but the task of the present invention is performed on remote sensing images, and there are differences between natural images and remote sensing images in terms of target scale, viewing angle, spatial resolution, etc.

[0110] The first specific technical problem to be solved by the present invention is that when using the Transformer model to generate remote sensing image subtitles, the image features and text features are not aligned, which will lead to low utilization of the image features and poor generation effect. In response to the first specific technical problem, the present invention proposes a text encoder to better encode the text. Compared with the original Transformer decoder, which directly starts to fuse image features after a layer of self-attention, the present invention adds an additional text encoding layer to the decoder, so that the text features learn the global context and deep semantic information, and then fuse with high-level visual features, alleviating the problem of misalignment of image and text features.

[0111] The second specific technical problem to be solved by the present invention is that when using the Transformer model to generate remote sensing image subtitles, usually only multi-scale features are fused, but the multi-scale features are not fully extracted in the decoder. In response to the second specific technical problem, the present invention proposes a multi-scale feature fusion extraction module, which fuses multi-scale features from different layers of the image encoder and extracts semantic features of different levels at different layers of the decoder.

[0112] In response to the overall technical architecture problem, the present invention uses a ViT (Vision Transforemr) that is pre-trained on natural images through image-text dataset comparison, and then fine-tunes the non-pre-trained image encoder on the remote sensing image-text dataset. This solves the problem of data mismatch and alleviates the problem of task mismatch, because studies have shown that models pre-trained through image-text comparison can extract features containing high-level semantics.

[0113] By combining the above methods, the remote sensing subtitle generation network based on the Transformer model of the present invention can realize the recognition of high-level semantic targets, alleviate the problem of misclassification of targets and scenes, and generate more accurate subtitles.

[0114] The key points and protection points of the present invention are:

[0115] 1. The present invention proposes to use an image encoder pre-trained on a remote sensing image and text dataset, which achieves data and task adaptation;

[0116] 2. The present invention designs a text encoder to better encode the text so that the image features and the text features are aligned;

[0117] 3. The present invention designs a multi-scale feature fusion extraction module, which utilizes the features of different layers of the image encoder and improves the ability to extract multi-scale features.

[0118] The effect of the present invention is further described below in conjunction with simulation experiments.

[0119] 1. Simulation experiment conditions:

[0120] The hardware test platform of the simulation experiment of the present invention is:

[0121] CPU: Xeon(R)CPU E5-2630v3@2.40GHz×16

[0122] Graphics card: NVIDIA Corporation GM200×2 24G

[0123] Operating system: Ubuntu 22.04.3LTS

[0124] Development environment, Python 3.9.13, cudann 8.9.1.23, PyTorch 2.1.0

[0125] 2. Simulation content and result analysis:

[0126] Table 1 is the detailed information of NWPU-Captions dataset;

[0127] Table 2 shows the ablation experiments of different modules on the NWPU-Captions dataset;

[0128] Table 3 is a comparison of the indicators of remote sensing image description generated by the method of the present invention and other methods;

[0129] The simulation experiment of the present invention is conducted on the dataset NWPU-Captions proposed by the Northwestern Polytechnical University team. Compared with other public datasets, this dataset has more images and richer descriptions. The detailed parameters of the data used in the present invention are shown in Table 1.

[0130] Table 1 Details of the NWPU-Captions dataset

[0131] Number of images 31500 Image size 256x256 Image spatial resolution 0.2～30m Number of descriptions per image 5 Different vocabulary 3149 Number of scenes 45

[0132] The simulation experiment of the present invention is carried out on the NWPU-Captions dataset, and the ablation experiments of different modules are shown in Table 2. In order to illustrate the effect of the image encoder used in the present invention, the present invention compares the commonly used image encoder ResNet-101 in this field, which is pre-trained on 1000 classifications on ImageNet. The effect of the image encoder proposed in the present invention can be seen from the first and second rows of Table 2. Compared with the commonly used ResNet-101, the image encoder of the present invention has been greatly improved in several indicators. Similarly, the comparison between the second and third rows of Table 2, and the comparison between the third and fourth rows, respectively show the improvement of the indicators of the text encoder and the multi-scale feature fusion extraction module proposed in the present invention.

[0133] Table 2 Ablation experiment results of different modules of the present invention on the NWPU-Captions dataset

[0134] method BLEU1 BLEU4 METEOR ROUGE CIDER ResNet-101+TransDec 89.0 70.5 46.2 78.4 203.3 GeoRSCLIP+TransDec 91.7 75.9 47.9 82.2 220.2 GeoRSCLIP+TextDec+TransDec 91.9 76.2 48.1 82.3 221.4 GeoRSCLIP+TextDec+MulSC+TransDec 92.7 77.2 48.2 83 225.4

[0135] TextDec: text decoder, MulSC: multi-scale feature fusion extraction layer,

[0136] TransDec: Transformer decoder

[0137] Invention simulation experiments are performed on the NWPU-Captions dataset. In order to illustrate the comparative effect between the method proposed in the present invention and other advanced methods in this field, comparative experiments are shown in Table 3.

[0138] Table 3 Comparison of the indicators of remote sensing image description generated by NWPU-Captions by the present invention and other advanced methods

[0139] method BLEU1 BLEU4 METEOR ROUGE CIDER MLCA-Net 74.5 47.8 33.7 60.1 126.4 BITA 88.54 67.6 45.2 78.5 197 HCNet 89.5 71.6 47.4 80.5 209.2 Ours 92.7 77.2 48.2 83 225.4

[0140] The present invention is compared with other advanced methods in this field. MLCA-Net: CNN image encoder + LSTM decoder, BITA: CLIP image encoder + LLM, HCNet: CNN image encoder + image-text alignment module + LSTM decoder. As can be seen from Table 3, the method of the present invention has achieved the best experimental results in all indicators on the NWPU-Captions dataset.

[0141] The present invention also provides a remote sensing image description generation system based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features, including:

[0142] The data set acquisition module is used to implement the remote sensing caption generation data set NWPU-Captions in step 1, including training set, validation set and test set;

[0143] The data set processing module is used to implement the preprocessing of the text and images of the training set, verification set and test set of the remote sensing caption generation data set NWPU-Captions obtained in step 1 in step 2. The text preprocessing includes serialization, filling and generating a vocabulary to obtain a vocabulary and text features; the image preprocessing includes scaling to a uniform size and normalization to obtain image features;

[0144] A remote sensing subtitle generation network construction module based on the Transformer model is used to implement the construction of a remote sensing subtitle generation network based on the Transformer model in step 3, including: an image encoder module, a text encoder module, and a Transformer decoder module; and set the network's hyperparameters, including: the number of extracted feature layers of the image encoder module, the number of layers of the Transformer decoder module; the number of layers of the text encoder module, the number of heads of the multi-head attention, the dimension of the hidden layer, and settings related to the learning process, including the optimizer, learning rate, and number of warm-up steps;

[0145] The model training module is used to implement step 4, inputting the image features and text features of the training set preprocessed in step 2 into the remote sensing subtitle generation network based on the Transformer model built in step 3, performing cross entropy loss training on the generated description, and obtaining a trained remote sensing subtitle generation network based on the Transformer model, and then inputting the verification set into the model obtained in each round for verification, and selecting the best model based on the performance on the verification set;

[0146] The result output module is used to implement step 5 in which the image features in the test set preprocessed in step 2 are input into the model with the best effect in step 4 to obtain the captions corresponding to the image.

[0147] The present invention also provides a remote sensing image description generation device based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features, including:

[0148] Memory: a computer program for storing the above-mentioned method for generating remote sensing image descriptions based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text comparison, which is a computer-readable device;

[0149] Processor: used to implement the method for generating remote sensing image description based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text comparison when executing the computer program.

[0150] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for generating remote sensing image description based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image and text comparison.

Claims

1. A method for generating remote sensing image descriptions based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features, characterized in that: The following steps are involved: Step 1, obtain the remote sensing caption generation dataset NWPU-Captions, including training set, validation set and test set; Step 2, preprocessing the text and images of the training set, validation set and test set of the remote sensing caption generation dataset NWPU-Captions obtained in step 1. The text preprocessing includes serialization, padding and generating a vocabulary to obtain a vocabulary and text features; the image preprocessing includes scaling to a uniform size and normalization to obtain image features; Step 3: Build a remote sensing caption generation network based on the Transformer model, including: image encoder module, text encoder module and Transformer decoder module; and set the network's hyperparameters, including: the number of extracted feature layers of the image encoder module, the number of layers of the Transformer decoder module; the number of layers of the text encoder module, the number of heads of the multi-head attention, the dimension of the hidden layer, and the settings related to the learning process, including the optimizer, learning rate, and the number of warm-up steps; Step 4: Input the image features and text features of the training set preprocessed in step 2 into the remote sensing subtitle generation network based on the Transformer model built in step 3, perform cross entropy loss training on the generated description, and obtain a trained remote sensing subtitle generation network based on the Transformer model. Then input the validation set into the model obtained in each round for verification, and select the model with the best effect based on its performance on the validation set. Step 5: Input the image features in the test set preprocessed in step 2 into the model with the best effect in step 4 to obtain the captions corresponding to the image.

2. According to claim 1, a method for generating remote sensing image description based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features is characterized in that: The text preprocessing in step 2 includes: Step 2.1, sort all the words in the text of the training set, validation set, and test set by frequency from high to low, replace all the words with a frequency lower than 1 to 10 with unknown words [UNK], and then encode the words from zero according to the frequency to generate a vocabulary; Step 2.2, after the vocabulary is generated, the words in the original sentence are serialized according to the vocabulary, and then the length is unified, the long ones are directly truncated, and the short ones are filled to obtain the text features; Image preprocessing includes: the images of the training set, validation set, and test set are uniformly scaled to 224×224, and the three channels RGB are normalized according to the mean [0.406, 0.423, 0.390] and the standard deviation [0.188, 0.175, 0.185] to obtain image features.

3. According to claim 1, a method for generating remote sensing image description based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features is characterized in that: The remote sensing caption generation network based on the Transformer model in step 3 includes an image encoder module, a text encoder module and a Transformer decoder module; Image encoder module: used to extract image features. The encoder uses the public RS5M remote sensing image dataset to fine-tune the ViT model pre-trained on the natural image dataset. The image encoder module consists of 32 layers of Transformer sublayers to extract multi-layer image features. The feature dimension of each layer is 256×1280. Text encoder module: used to encode text features to obtain context-related text features; the text encoder module includes a self-attention layer and a two-layer feedforward layer. The input is the text features preprocessed in step 2, and then passed through the word embedding (WordEmbed) layer and added with position encoding; In the self-attention layer, the text features are used as the Q, K, and V of the attention. The relationship between different words is learned according to the context. Two feed-forward layers are introduced to first change the dimension to 2048 and then back to 512, so that the input and output dimensions of the entire module remain unchanged; Each layer of the text encoder module is formulated as: Y = LayerNorm(X+T)(1.2) Z=FFN(Y)=(YW1+b1)W2+b2(1.3) S = LayerNorm(Z+Y)(1.4) Among them, T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, SelfAttn is the self-attention, FFN is the feedforward layer, and LayerNorm is the layer normalization; Transformer decoder module: used for adaptive extraction of multi-scale features and fusion of text features; the Transformer decoder module consists of a self-attention layer, a multi-scale feature fusion extraction layer, a cross-attention layer and a feedforward layer, and the input is the features output by the text encoding module and the multi-layer image features extracted by the image encoder module; the self-attention layer, the feedforward layer and the text encoder module are consistent; the multi-scale feature fusion extraction layer includes two steps of fusion and extraction: the fusion step directly splices the extracted multi-layer image features in the channel dimension, and adjusts the dimension back to 512 dimensions through a linear layer; the extraction step is to perform channel attention and spatial attention on the fused features respectively, and obtain channel-weighted and spatial-weighted features respectively; in the cross-attention layer, the image features are used as K, V, and the text features are used as Q, and cross-attention is performed respectively. The obtained features are respectively passed through a linear layer and a Sigmoid function to calculate the corresponding weights, and the obtained features are weighted with their own weights to obtain fusion features of fused text and image; if the fused features have spatial dimensions and channel dimensions, these two features are added and then passed through the feedforward layer to obtain the output; Each layer of the Transformer decoder module is expressed as: X=SelfAttn(TW q ,TW k ,TW v ) (2.1) Y=LayerNorm(X+T) (2.2) I=Mullayer_Fusion(V1,V2,V3)=FFN(Concat(V1,V2,V3) (2.3) F S =SpatialAttention(I),F c =ChannelAttention(I) (2.4) SCA=CrossAttention(Y,F S ,F S ),CCA=CrossAttention(Y,F c ,F c ) (2.5) σ s =Sigmoid(FFN(Concat(Y,SCA))),σ c =Sigmoid(FFN(Concat(Y,CCA))) (2.6) Fsion(SCA,CCA)=SCA⊙σ s +CCA⊙σ c (2.7) H=LayerNorm(Fsion(SCA,CCA)+Y) (2.8) Z=FFN(H+Y) (2.9) S = LayerNorm(Z + H) (2.10) Among them, T is the input text feature, W is the weight matrix, b is the bias, d is the scaling factor, V1, V2, V3 are multi-layer image features extracted by image encoding, LayerNorm is layer normalization, SpatialAttention is spatial attention, ChannelAttention is channel attention, CrossAttention represents cross attention, FFN represents linear layer, Concat represents concatenation operation, "+" represents element-wise addition, and ⊙ represents element-wise multiplication.

4. According to claim 1, a method for generating remote sensing image description based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features is characterized in that: The loss during the training process in step 4 is the cross entropy loss L XE , as follows: Cross entropy loss L XE : In the formula, T represents the sentence length, Represents the predicted word The true label of the first t-1 words at the time, θ represents the model parameters, p θ Represents the predicted word The probability distribution of .

5. A remote sensing image description generation system based on modality alignment and multi-scale extraction based on remote sensing image-text comparison pre-training features based on the method according to any one of claims 1 to 4, characterized in that: include: The dataset acquisition module is used to obtain the remote sensing caption generation dataset NWPU-Captions, including training set, validation set and test set; The dataset processing module is used to preprocess the text and images of the training set, validation set and test set of the acquired remote sensing caption generation dataset NWPU-Captions. The text preprocessing includes serialization, filling and generating vocabulary to obtain vocabulary and text features; the image preprocessing includes scaling to a uniform size and normalization to obtain image features; A remote sensing subtitle generation network building module based on the Transformer model, which is used to build a remote sensing subtitle generation network based on the Transformer model, including: image encoder module, text encoder module and Transformer decoder module; and set the network's hyperparameters, including: the number of extracted feature layers of the image encoder module, the number of layers of the Transformer decoder module; the number of layers of the text encoder module, the number of heads of the multi-head attention, the dimension of the hidden layer, and the settings related to the learning process, including the optimizer, learning rate, and the number of warm-up steps; The model training module is used to input the image features and text features of the preprocessed training set into the remote sensing subtitle generation network based on the Transformer model, perform cross entropy loss training on the generated description, and obtain the trained remote sensing subtitle generation network based on the Transformer model. Then, the validation set is input into the model obtained in each round for verification, and the best model is selected based on its performance on the validation set. The result output module is used to input the preprocessed image features in the test set into the best model to obtain the captions corresponding to the image.

6. A remote sensing image description generation device based on modality alignment and multi-scale extraction of remote sensing image text comparison pre-training features, characterized in that: include: Memory: a computer program storing a method for generating remote sensing image descriptions based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text comparison as described in any one of claims 1 to 4, which is a computer-readable device; Processor: used to implement the method for generating remote sensing image description based on modal alignment and multi-scale extraction of remote sensing image text comparison pre-training features as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a method for generating remote sensing image descriptions based on modal alignment and multi-scale extraction of pre-trained features of remote sensing image-text comparison as described in any one of claims 1 to 4.

Images