Image description generation method and system based on visual text alignment

Abstract

The application provides a kind of image description generation method and system based on visual text alignment, establishes remote sensing image description generation model, the model carries out remote sensing image high-level semantic feature extraction, multi-scale feature fusion, and adaptive decoding with gate mechanism, including extracting the multi-scale feature information of remote sensing image, adaptively fusing the multi-scale feature information extracted, guiding model to focus on relevant information, realizing the alignment of visual features of image and description sentence vocabulary;Train remote sensing image description generation model, input the remote sensing image to be described into the trained image description generation model, and obtain the corresponding image description sentence result.The remote sensing image automatic description generation scheme provided by the application can be used for massive multi-source heterogeneous remote sensing data, and the precision of image classification, target detection and other tasks is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image description generation, and more particularly, a method and system for image description generation based on visual text alignment are designed. BACKGROUND

[0002] With the rapid development of remote sensing technology, more and more types of remote sensing data are available, the types are more and more rich, and the resolution is higher and higher, and mankind has ushered in the era of big data. How to efficiently use massive multi-source remote sensing data is a major application requirement in the field of remote sensing. Massive multi-source heterogeneous remote sensing data provides rich data sources for various major applications, but also poses greater challenges to extracting and mining various information and knowledge hidden behind remote sensing big data. In addition, although some progress has been made in the field of remote sensing images, such as image classification and target detection, the description generation of remote sensing images not only needs to simulate human visual recognition of image semantic content by computer, but also needs to automatically convert the understanding of the image into accurate and natural description sentences, which is still a challenging task.

[0003] Traditional image description generation methods include template filling-based image description generation and retrieval-based image description generation methods. The template-based method uses traditional machine learning algorithms to extract image features, and then fills the extracted features into a predefined template to generate a description of the image. The generated description sentence can guarantee the accuracy of the syntax and the fluency of the semantics, but the limitation is that the use of fixed templates makes the sentence single, and the generated sentence lacks diversity and novelty. The retrieval-based method uses similarity measurement to retrieve similar images of the image to be described in the data set, and then obtains the description of the image to be described based on the description of the similar images. Compared with the template-based method, the generated description is relatively flexible in terms of sentence structure, but the quality of the description result is related to the retrieval performance. When the query image is not similar to the images in the data set, it cannot guarantee the correctness of the semantics.

[0004] With the rapid development of deep learning technology, models represented by the coding and decoding structure have been widely used in the field of remote sensing image description generation. Among them, due to the excellent feature extraction ability of convolutional neural networks for images, a CNN pre-trained on a large-scale data set is used to extract image features, and then a fixed-dimensional feature vector is sent to a recurrent neural network, long short-term memory network, and gated recurrent unit to generate sentence description. It is an effective method for remote sensing image description generation task, and this method currently achieves the best performance in terms of grammatical correctness, semantic fluency, and sentence diversity. However, how to accurately describe the visual features of complex scene remote sensing images and minimize semantic ambiguity between image regions and visual vocabulary remains a major factor limiting the accuracy of current remote sensing image description generation.

[0005] The patent document CN113420680A provides a remote sensing image region attention and text generation method based on GRU attention, which includes the following steps: step S1, extracting remote sensing image feature information in the encoder part, and outputting a feature map; step S2, the feature map and the text information in the training set are jointly used as the input of the decoder; the decoder includes a GRU attention network and an LSTM language network, the GRU attention network calculates a weight coefficient for the feature vector of the feature map and the word of the text information generated by the feature map before the word of the input feature map is generated, increases the selection and attention of the key area, and finally outputs the word and the state. The application can more accurately determine the key area in the preselected area of the remote sensing image to generate more accurate descriptions.

[0006] However, the above patent does not consider the multi-scale and background complexity characteristics of the remote sensing image, and in the decoding process, the relevance between the generated attention result and the query is not considered, and when predicting prepositions such as "and", "in", "of", etc., the description words can be predicted by the language model. SUMMARY

[0007] To solve this problem, the application provides a remote sensing image description generation model and system based on multi-scale features and visual-text alignment, which aims to improve the performance of the remote sensing image description generation model.

[0008] The application provides an image description generation method based on visual-text alignment, which performs the following processing,

[0009] A remote sensing image description generation model is established, which performs remote sensing image high-level semantic feature extraction, multi-scale feature fusion, and adaptive decoding with a gating mechanism;

[0010] Remote sensing image high-level semantic feature extraction includes extracting multi-scale feature information of the remote sensing image;

[0011] Multi-scale feature fusion includes adaptive fusion of the extracted multi-scale feature information;

[0012] Adaptive text generation decoding with a gating mechanism includes guiding the model to focus on relevant information to realize the alignment of the visual features of the image and the description sentence vocabulary;

[0013] The remote sensing image description generation model is trained, and the remote sensing image to be described is input into the trained image description generation model to obtain the corresponding image description sentence result.

[0014] Moreover, the visual features of the input image are obtained by using a model pre-trained on the ImageNet dataset.

[0015] Further, the visual features of the remote sensing image are extracted by using the VGG16.

[0016] Further, the implementation manner of the high-level semantic feature extraction of the remote sensing image is that image features of different scales are extracted by using different sizes of convolution kernels in the spatial relationship modeling module, then the channel relationship modeling module takes the output of the feature vector output by the previous group of the spatial relationship modeling module and the feature vector output by the current group of convolution as inputs, filters the redundant information, and obtains the optimized image feature representation.

[0017] Further, the implementation manner of the multi-scale feature fusion is that a multi-layer perception (MLP) is input for learning, and then a Sigmoid activation function is used to perform attention weighting on the features of multiple scales, so as to adaptively fuse the image feature information of different scales, thereby improving the understanding ability of the model to the remote sensing image features.

[0018] Further, the implementation manner of the adaptive text generation decoding of the gating mechanism is that an adaptive double-layer LSTM image text alignment module is used to decode the image features, wherein the first LSTM is a gated attention LSTM, denoted as Attention LSTM, and the second LSTM is an adaptive language LSTM, denoted as language LSTM; the extracted image multi-scale context features are input into the first LSTM in the decoding end for decoding to generate a description sentence of the image; the input vector input into the Attention LSTM at each time step t is the embedding vector of the current word, the average pooling features of the image, and the previous hidden state of the second LSTM; then, the specific position of the multi-scale features of the LSTM is guided according to the attention mechanism, the attention vector is optimized through the gating mechanism, next, the adaptive alignment of the visual features and the description sentence text information in the decoding process is realized through the semantic gate vector, and finally, the context vector generated through the gated attention and the hidden state of the Attention LSTM are input into the language LSTM to generate the description sentence.

[0019] Further, the remote sensing image description generation model is trained by using the sum of the negative log likelihoods of the correct description words generated at each time step.

[0020] On the other hand, the application provides an image description generation system based on visual text alignment, which is used to implement the image description generation method based on visual text alignment as described above.

[0021] Further, the application further provides a computer readable storage medium, which stores the program instructions of the image description generation method based on visual text alignment as described above.

[0022] Alternatively, a readable storage medium is included, and the readable storage medium stores a computer program, and the computer program implements the image description generation method based on visual text alignment when executed.

[0023] The present application extracts multi-scale information of remote sensing images through a multi-scale feature extraction module at the encoding end, further adaptively fuses image features of different scales through a multi-scale feature fusion module, uses an adaptive text model guided by a gating mechanism at the decoding end to guide the model to focus on relevant information, realizes alignment of visual features of images and vocabulary of description sentences, and finally trains the model by using cross-entropy loss to generate description sentences with correct grammar and fluent semantics, thereby improving the performance of the image description generation model. The remote sensing image automatic description generation scheme provided by the present application can be used for massive multi-source heterogeneous remote sensing data, and improves the accuracy of image classification, target detection and other applications. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A multi-scale feature and visual text alignment based image description generation model structure diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0026] The present application overcomes the problem of poor generalization ability of the existing remote sensing image description generation method, and generates more accurate remote sensing image description sentences by designing a multi-scale feature extraction module and a visual text alignment module, including:

[0027] A remote sensing image description generation model is established, which performs remote sensing image high-level semantic feature extraction, multi-scale feature fusion, and adaptive decoding with a gating mechanism;

[0028] Remote sensing image high-level semantic feature extraction includes extracting multi-scale feature information of remote sensing images;

[0029] Multi-scale feature fusion includes adaptively fusing the extracted multi-scale feature information;

[0030] Adaptive text generation decoding with a gating mechanism includes guiding the model to focus on relevant information to realize alignment of visual features of images and vocabulary of description sentences.

[0031] The remote sensing image description generation model is trained by using a large-scale image-text sample library as a training set of the model.

[0032] The remote sensing image to be described is input into the trained image description generation model to obtain a corresponding image description sentence result.

[0033] Referring to Figure 1 The remote sensing image description generation method provided in the embodiment of the present application is introduced below in combination with a specific process, and includes the following steps.

[0034] Step S1, a remote sensing image description generation model is first established, including a remote sensing image high-level semantic feature extraction module, a multi-scale feature fusion module and an adaptive decoding module with a gating mechanism.

[0035] The remote sensing image high-level semantic feature extraction module is used to extract multi-scale feature information of the remote sensing image.

[0036] The multi-scale feature fusion module is used to adaptively fuse the extracted multi-scale feature information.

[0037] The adaptive text generation decoding module with the gating mechanism is used to guide the model to focus on relevant information, so as to realize the alignment of the visual features of the image and the description sentence vocabulary.

[0038] In the embodiment, S1 is implemented through steps a and b.

[0039] Step a, construction of a complex remote sensing image semantic description module with multi-scale feature expression and aggregation.

[0040] 1) Basic deep convolutional network and multi-scale feature extraction:

[0041] It is further proposed in the present application that the remote sensing image high-level semantic feature extraction module is implemented by using a basic deep convolutional network, and is used to extract multi-scale feature information of the remote sensing image.

[0042] In specific implementation, any model (such as AlexNet, VGG, GoogleNet, etc.) pre-trained on a large-scale sample library ImageNet can be selected to extract image features.

[0043] The base deep convolutional network of the embodiment preferably adopts a pre-trained convolutional neural network VGG16 to extract features of the image, and then enters a multi-scale feature extraction module. In the multi-scale feature extraction module, first, in the spatial relationship modeling module, the input features are evenly divided into four groups along the channel dimension, and for each group of vectors, 1x1 and 3x3 convolution blocks are used for convolution calculation. Specifically, for the first group of input feature vectors, the output features are directly obtained by 1x1 convolution, and for the second, third and fourth groups of input feature vectors, the output features are obtained by 3x3 convolution together with the output of the previous group. Then, the channel relationship modeling module uses the output of the previous group of feature vectors output by the spatial relationship modeling module and the output of the current group of convolution feature vectors as input to filter redundant information and obtain an optimized image feature representation.

[0044] Specifically, under the implementation of the multi-scale feature extraction module:

[0045] (I) First, a spatial relationship modeling module is set, including dividing the input remote sensing image feature map X ∈ R H×W×C into four groups, each group being represented as i ∈ {1, 2, 3, 4}, H, W, C i respectively the height, width and number of channels of each group. In the branch of the multi-scale module, the first group of input X i is operated with a 1x1 convolution kernel, and the remaining each group of input X i is operated with a 3x3 convolution kernel. The 3x3 convolution operation can obtain a larger receptive field (1x1, 3x3, 5x5, 7x7 respectively).

[0046] The calculation formula is:

[0047] X = VGG(I) (1.1)

[0048]

[0049]

[0050] where I represents the input remote sensing image feature, VGG() represents the feature extraction of the input remote sensing image through the pre-trained VGG network, represents the convolution operation corresponding to each group of feature vectors, F i represents the output corresponding to the i-th group. By using multi-scale spatial features, the model can obtain more competitive feature representations.

[0051] (ii) Since the background of the remote sensing image is complex and has redundant background information, the channel relationship modeling module is used to enhance the recognition of different channel image features by the model and filter redundant image features. The implementation of the channel relationship modeling module is as follows,

[0052] First, the feature vectors of the current group and the optimized feature vectors are integrated using two 1x1 convolutions, and the integrated features of the i-th group are as follows:

[0053]

[0054] where, W SS represent the parameters of the two 1x1 convolutions in the module, represents the corresponding feature vector output of the i-1-th group.

[0055] Next, channel-level information is generated by using global average pooling:

[0056]

[0057] where, F gp (u ic ) represents global average pooling, u ic (m,n) represents the image feature output of the i-th group along the channel dimension, and m and n represent the height and width of the image.

[0058] In order to fully utilize the information aggregated in the global average pooling operation, the invention uses a fully connected layer and an activation function to extract useful channel information, and the i-th group of attention outputs is represented by α i , and W represents the weight of the fully connected layer, as shown in equation 1.6:

[0059] α i =(z i ,W)=σ(Wz i ) (1.6)

[0060] where, z i represents channel-level feature information, Wz i represents the weight, and σ() represents the activation function.

[0061] Finally, the spatial multi-scale feature output of F i is obtained:

[0062]

[0063] where, α isc represents the weight, F ic represents the feature in the channel dimension, and F weighted () represents the weighted operation.

[0064] Each group of optimized features F * is concatenated along the channel dimension:

[0065]

[0066] wherein,

[0067] wherein, denotes the image features of 4 scales, and concat() denotes concatenation, denotes the i-th group of features along the channel dimension, and C denotes the number of channels.

[0068] In summary, the multiscale feature extraction module of the present application first extracts image features of different scales through different size convolution kernels in the spatial relationship modeling module, and then enhances the recognition of different channel features by the channel relationship modeling module, removes redundant information, and finally obtains the multiscale features of the image.

[0069] 2) Then, the designed multiscale feature aggregation module is used to aggregate the extracted features. Specifically, a multi-layer perceptron (MLP) is input for learning, and then a Sigmoid activation function is used to perform attention weighting on the features of the four scales. The different scale image feature information extracted is adaptively fused, thereby improving the understanding ability of the model for remote sensing image features. Specifically, first, the multiscale remote sensing image features extracted are concatenated to obtain S. Next, the concatenated image features are reduced in dimension through an FC layer, the correlation between the multiscale features is learned through an MLP, and a score weight matrix W is obtained by calculating the scores of the reduced image features using a Sigmoid activation function. Then, the image feature maps of different scales are multiplied by the score weight matrix to obtain weighted image features. Wherein, W i is a weight coefficient, and S final denotes the image features obtained by attention and weighting.

[0070] The corresponding specific implementation in the embodiment is as follows:

[0071] First, the concatenated image features are reduced in dimension through an FC layer, the correlation between the multiscale features is learned through an MLP, and a score weight matrix W is obtained by calculating the scores of the reduced image features using a Sigmoid activation function.

[0072] W = Sigmoid(MLP(F * )) (1.9)

[0073] Wherein, Sigmoid() denotes a Sigmoid activation function, MLP() denotes a multi-layer perceptron, F * denotes the concatenated image features.

[0074] Then, the image feature maps of different scales are multiplied by the score weight matrix to obtain weighted image features.

[0075] wherein W i is a weight coefficient, S final represents the image feature obtained by attention post-weighting.

[0076] S final = FC(F * ) + FC(F * ) * W (1.10)

[0077] wherein FC() represents an FC layer, and * represents pixel multiplication.

[0078] Step b, construction of an adaptive text model with a gating mechanism. This step implements an adaptive decoding module with a gating mechanism.

[0079] Specifically, in the method of the present application, the first LSTM is a gated attention LSTM (denoted as AttentionLSTM), and the second LSTM is an adaptive language LSTM (denoted as languageLSTM), and the information of the language model is considered at the same time. The image multi-scale context features extracted by the encoding network are sent to the first LSTM (i.e., AttentionLSTM) in the decoding end for decoding to generate the description sentence of the image. The input vector input to the AttentionLSTM at each time step t is the embedding vector of the current word, the average pooling feature of the image, and the previous hidden state of the second LSTM. Then, according to the attention mechanism, the specific position of the LSTM multi-scale feature is guided, the attention vector is optimized through the gating mechanism, then the adaptive alignment of the visual feature and the description sentence text information is realized through the semantic gate vector, and finally, the context vector generated through the gated attention and the AttentionLSTM hidden state h are input into the languageLSTM to generate the description sentence.

[0080] In the embodiment, the image features are decoded by using an adaptive double-layer LSTM image text alignment module. The formula is:

[0081]

[0082]

[0083] A t = softmax(a t ) (1.13)

[0084]

[0085] wherein, denotes the hidden state of the Attention LSTM at time t, LSTM1() denotes the Attention LSTM, denotes the hidden state of the language LSTM at time t-1, denotes the hidden state of the Attention LSTM at time t-1, denotes the global average feature, W e is the word embedding matrix, x t is the one-hot encoding of the input word at time t, a j,t denotes the feature vector, j denotes the image region, W va , W ha , is a learnable parameter, a t denotes the vector of HxW attention weights corresponding to the HxW region feature vectors, A t = softmax(a t ) denotes the probability value. denotes the visual attention vector, V i denotes the feature vector.

[0086] The attention mechanism guides the decoding process to generate a weighted average feature vector at each time step, and the result of image description generation depends heavily on the result of attention. The present application first optimizes the attention vector, expands the existing attention mechanism, and combines the gating mechanism to make the decoder retain useful attention information during the decoding process, thereby obtaining an optimized attention vector

[0087]

[0088]

[0089]

[0090] wherein, denotes the optimized vector, g, i denote intermediate calculation results, b i , b g denote learnable parameters, denotes the weight, and denotes element-wise multiplication.

[0091] In order to effectively adjust the selection of visual information or sentence context information to generate the title, the present application introduces a semantic gate b t :

[0092]

[0093]

[0094]

[0095] where g t represents, represents the weight, represents the output of the t-th time step LSTM, represents the weight, represents the hidden layer state of the Attention LSTM at t-1 time, represents the memory unit, S t represents the visual vector, β t When the value of β t is 1, the process of generating words for the model utilizes the sentence context information, and when the value is 0, the input image information is utilized. At this time, c' t is the generated context vector.

[0096] In order to calculate β t , the present application additionally adds an element S t to the decoding model, which represents the degree of attention of the model to the sentence context, wherein the calculation formula of β t is as follows:

[0097]

[0098]

[0099] where z t represents the intermediate vector, w h represents the weight, V represents the feature vector of the image, w s and W h are model learnable weight parameters, represents the hidden layer state of the Attention LSTM at t time.

[0100] Next, the context vector c' t is sent into the language LSTM, and finally the sofmax layer is used to obtain the output description sentence.

[0101]

[0102]

[0103] where, represents the hidden layer state of the language LSTM at t time, LSTM2() represents the language LSTM, W0 represents the weight, and b0 represents the bias.

[0104] Step c, the image description generation model is trained by taking the large-scale remote sensing image-sentence dataset as the training set of the model.

[0105] The model is trained using the maximum likelihood estimation (MLE) loss, and the goal is to minimize the MLE loss. In each training step, the input x t and the previous hidden state h t-1 are combined to obtain the hidden state h t at this time. Then the probability distribution of the words in the sentence generation process is calculated by the softmax function, and the word with the highest probability is selected as the predicted word. Then the predicted word is the input of the next time step. The loss function of the model training is the sum of the negative log-likelihood of generating the correct description word in each time step:

[0106]

[0107] where θ is the parameter to be learned, L(θ) represents the loss function, represents the generated description sentence, p θ () represents the probability value, log() represents the logarithmic function, t represents the decoding time step, and T represents the time step of generating the last word.

[0108] Step d, input the picture to be described into the trained image description generation model, and then obtain the result of image description sentence generation.

[0109] In summary, the present application first extracts the multi-scale information of the remote sensing image through the multi-scale feature extraction module at the encoding end, further models the local and global multi-scale image features through the multi-scale feature fusion module, and uses the adaptive text model guided by the gating mechanism at the decoding end to guide the model to focus on relevant information, realizes the alignment of the visual features of the image and the vocabulary of the description sentence, and generates a description sentence with accurate grammar and fluent semantics.

[0110] Compared with patent document CN113420680A: the present application designs a multi-scale feature extraction module at the encoding end, which uses different sizes of convolution kernels to obtain different receptive fields to realize the extraction of multi-scale features of remote sensing images, and further proposes a feature aggregation module to adaptively fuse multi-scale features to realize feature optimization. The present application proposes a decoding network based on visual-text alignment at the decoding end, realizes the alignment of visual regions and description vocabulary during the decoding process, enhances the attention result through the gating mechanism, and then introduces the context vector to adaptively select visual features and sentence information to generate optimized description sentences. Therefore, the present application can more accurately express the features of remote sensing images and realize the adaptive alignment of different modal visual region features and description vocabulary, and automatically generate description sentences with accurate syntax and fluent semantics.

[0111] The effectiveness of the present application is verified below by using the NWPU-Captions dataset:

[0112] The NWPU-Captions dataset is annotated based on the NWPU-RESISC45 dataset, which includes 45 categories, 31500 images, and the pixel value of each image is 256*256, the resolution size is 0.228 meters, and the five description sentences of each image are completely annotated by different volunteers with professional backgrounds from Wuhan University and Huazhong University of Science and Technology, ensuring the diversity of the sentences. In addition, compared with other public datasets, the NWPU-Captions dataset is larger in size, contains more ground object categories, and better reflects the rich image changes of remote sensing images, as well as high intra-class diversity and high inter-class similarity. The relevant information of the NWPU-Captions dataset is shown in Table 1:

[0113] Table 1 Relevant information of the NWPU-Captions dataset

[0114]

[0115]

[0116] Evaluation indicators: BLEU, ROGUE, METEOR, CIDEr and SPICE. Specifically, the calculation methods of BLEU and ROUGE are relatively simple, and they focus on evaluating the fluency of the text, while MRTEOR and CIDEr comprehensively evaluate the generated sentences from the aspects of grammar and accuracy, which have a higher correlation with human judgment, and SPICE focuses on considering the relationship between the attributes of the description sentences and the target objects.

[0117] The feature extraction network of all models of the present application uniformly uses vgg16 as the backbone model, uses the deep learning Pytorch framework to build the network model, Pytorch is an open source machine learning framework, which has the advantages of high flexibility, can construct dynamic model graphs, and operates data when the model is executed. Adam is used as the optimizer, the initial learning rate of the encoder and the decoder is 1e-4 and 5e-4 respectively, the batchsize is set to 64, the dimension of the word embedding is set to 512, and the beam search is set to 5.

[0118] Firstly, the influence of the multi-scale feature extraction module and the multi-scale fusion module on the model performance is verified through ablation experiments. The ablation experiment results are shown in Table 2. By adding each sub-module, better results than the Baseline model are obtained, and the best image description accuracy is obtained by adding the multi-scale feature extraction module (MS) and the multi-scale feature fusion module (TR).

[0119] Table 2 shows the results of the ablation experiment of each module on the NWPU-Captions dataset

[0120] Method BLEU1 METEOR ROUGE_L CIDEr SPICE Baseline 0.731 0.334 0.581 1.092 0.274 Baseline+MS 0.738 0.344 0.596 1.125 0.286 Basline+TR 0.737 0.336 0.594 1.122 0.284 The method of the invention 0.741 0.626 0.611 1.159 0.289

[0121] The experiment also compares the image description generation method of the present application with other methods. The comparison methods are: (1) CSMLF is a retrieval-based method. The semantic embedding is learned by applying metric learning. The image features and sentence representations are projected into a common space. The distance between the test image and the description sentence is calculated. The nearest neighbor sentence is used as the description sentence of the test image. (2) Multimodal is a typical encoder-decoder structure. CNN is used as the encoder, and LSTM is used as the decoder to generate the description sentence. (3) SAT is the first time to introduce attention mechanism into the decoder of the encoder-decoder framework. It is used to give different weights to different regions of the image at each time step, guiding the model to dynamically focus on the regions of the image. (4) FC-ATT is a model based on attribute attention mechanism. High-level attribute features extracted in the full connection layer are used to guide the attention mechanism to apply high-level features of remote sensing images. The image description results of each method are shown in Table 3.

[0122] Table 3 shows the image description results on the NWPU-Captions dataset

[0123] Method BLEU1 METEOR ROUGE_L CIDEr SPICE CSMLF 0.715 0.318 0.576 1.063 0.263 Multimodal 0.726 0.339 0.589 1.078 0.281 SAT 0.734 0.337 0.601 1.109 0.284 FC-ATT 0.738 0.469 0.595 1.137 0.279 The method of the invention 0.741 0.626 0.611 1.159 0.289

[0124] As can be seen from Table 3, compared with other image description generation models, the method of the present application has a very competitive performance on five different indicators, which also proves that the method of the present application for extracting remote sensing image features by using multi-scale feature adaptive fusion is effective, and it is feasible to use an adaptive text model for decoding. In specific implementation, the method proposed in the technical solution of the present application can be automatically run by a computer software technology, and the system device of the method, such as a computer readable storage medium storing the corresponding computer program of the technical solution of the present application and a computer device including a computer device running the corresponding computer program, should also be within the protection scope of the present application.

[0125] In some possible embodiments, a visual-text alignment based image description generation system is provided, comprising a processor and a memory, the memory being configured to store program instructions, and the processor being configured to invoke the stored instructions in the memory to perform a visual-text alignment based image description generation method as described above.

[0126] In some possible embodiments, a visual-text alignment based image description generation system is provided, comprising a readable storage medium, and a computer program stored on the readable storage medium, the computer program being configured to implement a visual-text alignment based image description generation method as described above when executed.

[0127] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art can make various modifications or supplements to the specific embodiments described or adopt similar ways to replace them without departing from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. A method for image description generation based on visual text alignment, characterized in that: The following processing is performed, A remote sensing image description generation model is established, which performs remote sensing image high-level semantic feature extraction, multi-scale feature fusion, and adaptive decoding with a gating mechanism; The remote sensing image high-level semantic feature extraction includes extracting multi-scale feature information of the remote sensing image. The multi-scale feature fusion includes adaptively fusing the extracted multi-scale feature information; the multi-scale feature fusion is implemented by inputting a multi-layer perception (MLP) for learning, and then using a Sigmoid activation function to perform attention weighting on the features at multiple scales, to adaptively fuse the image feature information at different scales, thereby improving the understanding ability of the model for remote sensing image features. The adaptive text generation decoding with a gating mechanism includes guiding the model to focus on relevant information to realize alignment of the visual features of the image and the vocabulary of the description sentence; the adaptive text generation decoding with a gating mechanism is implemented by using an adaptive double-layer LSTM image-text alignment module to decode the image features, wherein the first LSTM is a gated attention LSTM, denoted as Attention LSTM, and the second LSTM is an adaptive language LSTM, denoted as language LSTM; the extracted multi-scale context features of the image are input into the first LSTM at the decoding end for decoding to generate a description sentence of the image; the input vector input into the Attention LSTM at each time step t is the embedding vector of the current word, the average-pooled features of the image, and the previous hidden state of the second LSTM; then, the specific position of the multi-scale features is guided by the attention mechanism, the attention vector is optimized through the gating mechanism, the adaptive alignment of the visual features and the text information of the description sentence is realized through the semantic gate vector in the subsequent decoding process, and finally, the context vector generated through the gated attention and the hidden state of the Attention LSTM are input into the language LSTM to generate the description sentence. The remote sensing image description generation model is trained, and the remote sensing image to be described is input into the trained image description generation model to obtain a corresponding image description sentence result. 2.The method of claim 1, wherein: The visual features of the input image are obtained by using a model pre-trained on the ImageNet dataset. 3.The method of claim 2, wherein: The visual features of the remote sensing image are extracted by using VGG16.

4. The method of claim 1 or 2 or 3, wherein: The implementation of the remote sensing image high-level semantic feature extraction is that different scale image features are extracted by different size convolution kernels in a spatial relationship modeling module, then the output of the previous group of feature vectors output by the spatial relationship modeling module and the current group of convolution features are used as inputs in a channel relationship modeling module to filter redundant information and obtain an optimized image feature representation.

5. The method of claim 1 or 2 or 3, wherein: The remote sensing image description generation model is trained using the sum of the negative log likelihoods of the correct description words generated at each time step. 6.A visual-text alignment based image description generation system, characterized in that: A method for implementing the image description generation based on visual-text alignment according to any one of claims 1-5.

7. The system of claim 6, wherein: The application discloses a computer readable storage medium, and the storage medium stores a computer program. 8.The image description generation system based on visual text alignment according to claim 6, characterized in that: The application discloses a computer readable storage medium, and the storage medium stores a computer program.

Images