Priori knowledge guidance Transform method and system for remote sensing image description

Abstract

The invention discloses a prior knowledge guidance Transform method and system for remote sensing image description. The method comprises the following steps: extracting target-level features by adopting a remote sensing rotating target detection model; constructing a target-target relationship in combination with a graph neural network and an attention mechanism; acquiring global representation of the remote sensing image; stacking prior knowledge to enhance attention and feed forward a neural network, guiding the model to pay attention to a target whose similarity with a scene dot product is greater than a set threshold, and capturing visual information of different levels; performing interaction of all levels of visual information and text information by adopting a net-shaped connection structure; word probability distribution of the vocabulary is obtained according to output of the text decoder, and words are generated step by step. According to the method, scene-target semantic association is fully established, similar targets are effectively distinguished by means of a graph structure, the description accuracy is enhanced, a prior scene-target knowledge guidance model is introduced to pay attention to targets related to a specific scene, and good semantic consistency is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image description generation, and in particular to a priori knowledge-guided Transformer method and system for remote sensing image description. Background Art

[0002] Image captioning is an important research direction in the intersection of computer vision and natural language processing. Its goal is to extract meaningful visual features from the input image through algorithms and generate natural language sentences describing the content of the image. Remote sensing image captioning aims to generate meaningful and grammatically accurate sentences for remote sensing images. In recent years, with the development of remote sensing technology, high-resolution remote sensing images have been widely used in many fields such as environmental monitoring, urban planning, and agricultural management. However, how to quickly and accurately understand and describe the content of these images has become an urgent problem to be solved. Compared with natural image captioning, most remote sensing images are taken from high altitude or satellites to capture geographical and environmental features, such as forests, rivers, urban buildings, farmland, etc. These targets are often very similar visually, with relatively simple colors and textures, and are numerous, covering a wide area, and containing a large number of different types of targets and complex geographic information, which can easily lead to confusion in the model during recognition and description, and it is difficult to determine the focus of the description, making remote sensing image captioning a specific challenging task.

[0003] Most of the existing image descriptions are based on deep learning methods. Vinyals et al. [O.Vinyals, A.Toshev, S.Bengio, and D.Erhan, "Show and tell: A neural image caption generator," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3156–3164.] first introduced an encoder-decoder framework, in which the encoder uses a convolutional neural network (CNN) to extract high-level visual features and the decoder uses a recurrent neural network (RNN) to generate descriptions. In addition, Xu et al. [K.Xu, J.Ba, and R.Kiros, "Show, attend and tell: Neural image caption generation with visual attention," in Proc. Int. Conf. Mach. Learn., 2015, pp. 2048–2057.] incorporated the attention mechanism into the framework to enhance the focus on different regions of the feature map in the image. Qu et al. [B.Qu, X.Li, D.Tao, and X.Lu, “Deep semantic understanding of high resolution remote sensing image,” in Proc. Int. Conf. Comput., Inf. Telecommun. Syst. (CITS), Jul. 2016, pp. 1–5, doi: 10.1109 / CITS.2016.7546397.] explored the problem of deep semantic understanding of high resolution remote sensing images and proposed a multimodal deep neural network that uses a convolutional neural network to extract image features and then combines them with the text description of the image through an RNN or LSTM. Vaswani et al. [A.Vaswani, N.Shazeer, and N.Parmar, "Attention is all you need," in Proc.Adv.Neural Inf.Process.Syst., 2017, pp.5998–6008.] introduced the Transformer model, replacing the traditional recurrent neural network (RNN) with the self-attention mechanism, which can process sequence data in parallel. Since then, most image description methods have adopted the attention mechanism. Although the above methods can generate grammatically correct descriptions, they usually cannot accurately capture the focus or theme of the image content and ignore the relationship between the target and the scene area. Summary of the invention

[0004] The purpose of the present invention is to provide a priori knowledge-guided Transformer method and system for remote sensing image description, which fully considers the influence of target-level features and scene-level features on remote sensing image description, effectively integrates the correlation and difference information between different targets or scene areas into their feature embedding, so that the model can distinguish visually similar targets, thereby providing a more accurate description of remote sensing images.

[0005] The technical solution to achieve the purpose of the present invention is: a priori knowledge-guided Transformer method for remote sensing image description, comprising the following steps:

[0006] In the first step, the remote sensing rotation target detection model ReDet is used in the feature extraction module to extract a specified number of regions of interest as target-level features and project them into the d-dimensional space through a fully connected layer;

[0007] In the second step, the features extracted by the feature extraction module in the first step are used as input, and the target-level features are enhanced using the feature enhancement module;

[0008] The third step is to insert the Class token V into the category cls As the concatenation of scene-level features and object-level features, it is used as the input of the visual encoder module;

[0009] In the fourth step, the visual encoder module stacks B encoding layers; each encoding layer uses the prior scene-target knowledge to guide the model to focus on targets whose dot product similarity with the scene is greater than the set threshold, and uses a feedforward neural network to implement nonlinear transformation and processing of the input vector, thereby enhancing the model's expressive power; the output of all encoding layers is used as the input of the visual encoder;

[0010] In the fifth step, the text embedding module uses word embedding and position encoding to obtain text features;

[0011] Step 6: The text decoder module stacks B decoding layers and receives the output of the visual encoder and text features. Each decoding layer first inputs the text features into the masked self-attention to ensure the causal relationship of sequence generation, then establishes multi-layer visual-text modality interaction through mesh cross attention, and finally uses the feedforward neural network to enhance the feature expression ability. The output of the last decoding layer is used as the output of the text decoder and input into the text generation module.

[0012] In the seventh step, the text generation module obtains the probability distribution of vocabulary words through the fully connected layer and the softmax layer, and gradually generates words.

[0013] A Transformer system guided by prior knowledge of remote sensing image description is used to implement the above method, and the system includes:

[0014] The feature extraction module uses the remote sensing rotation target detection model to extract target-level features;

[0015] The feature enhancement module combines graph neural networks and attention mechanisms to build target-target relationships and distinguish similar targets.

[0016] The category insertion module inserts the Class token as a scene-level feature before the object-level feature to obtain the global representation of the remote sensing image;

[0017] The visual encoder module stacks prior knowledge to enhance attention and feedforward neural networks, guides the model to focus on targets whose dot product similarity with the scene is greater than a set threshold, and captures different levels of visual information;

[0018] The text decoder module uses a mesh connection structure to interact with all levels of visual information and text information;

[0019] The text generation module obtains the word probability distribution of the vocabulary according to the output of the text decoder and generates words step by step.

[0020] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0021] Compared with the prior art, the present invention has the following significant features: (1) remote sensing rotating target detection is used to form a feature extraction module, which extracts richer features; (2) a combination of graph neural network and attention mechanism is used to refine the target-level features, making the features more accurate; (3) prior scene-target knowledge is calculated and integrated into the self-attention mechanism to guide the model to describe targets that are closely related to the scene, thereby generating a description that is more in line with the context.

[0022] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of the present invention.

[0024] Figure 2 This is the structural diagram of the encoder.

[0025] Figure 3 It is the structural diagram of the decoder. DETAILED DESCRIPTION

[0026] Combination Figure 1 to Figure 3 , the implementation process of the present invention is described in detail.

[0027] A priori knowledge-guided Transformer method for remote sensing image description, the steps are as follows:

[0028] In the first step, the feature extraction module uses the remote sensing rotation target detection model ReDet to extract a specified number of regions of interest as target-level features V o , and projected into d-dimensional space through the fully connected layer. The specific process is:

[0029] (1) Use the remote sensing rotation target detection model ReDet to generate a set of target regions, which are represented as target-level features through adaptive average pooling. in represents the characteristics of the i-th target, It means that the target-level feature matrix is ​​an M×D1 matrix, where M is the number of targets and D1 is the dimension of each target feature;.

[0030] (2) The target-level feature V o Projected into a d-dimensional space through a fully connected layer.

[0031] In the second step, the features extracted by the feature extraction module in the first step are used as input, and the target-level features are enhanced using the feature enhancement module (graph attention network). The specific process is as follows:

[0032] (1) For the target-level feature V extracted in the feature extraction module o , construct a fully connected graph G o =(V o ,E o ),in is a node set, E o is an edge set. ij Represents target-level features and The dot product similarity of Represents the normalized target-level features and The attention weight is:

[0033]

[0034]

[0035] in and is the learnable parameter matrix; d is the scaling factor, i.e., the spatial dimension scaled in the first step; N i is the neighborhood of node i in the graph.

[0036] (2) Compute enhanced object-level features based on other nodes and the edges connecting them The formula is as follows:

[0037]

[0038] in is the learnable parameter matrix.

[0039] (3) Use multi-head attention mechanism to capture multi-level information:

[0040]

[0041] in, represents the target-level features after being processed by the first attention head, are the corresponding learnable weight matrices, and d is the scaling factor, that is, the spatial dimension scaled in the first step.

[0042] (4) Concatenate all attention heads and project them into d-dimensional space through a fully connected layer to obtain the final target-level features Specifically expressed as:

[0043]

[0044] Among them, W r ∈R d×d is a learnable parameter matrix, and the result of the i-th attention head is

[0045] The third step is to insert the Class token V into the category cls It is used as the concatenation of scene-level features and object-level features and as the input of the visual encoder module. The specific process is as follows:

[0046] For the target-level features obtained by the feature enhancement module Add a Class token at the beginning of the feature sequence to form a new input sequence

[0047] In the fourth step, the visual encoder module stacks B encoding layers. Each encoding layer uses the prior scene-target knowledge to guide the model to focus on targets whose dot product similarity with the scene is greater than the set threshold, and uses the feedforward neural network to realize nonlinear transformation and processing of the input vector, thereby enhancing the model's expressive power. The output of all encoding layers is used as the input of the visual encoder. The specific process is:

[0048] (1) In each encoding layer, prior knowledge enhances attention through dot product similarity and softmax operation to obtain the prior knowledge matrix C∈R M×1 , and integrated into self-attention, the specific process is:

[0049]

[0050] and Q=XW q

[0051] K=(XW k )⊙(CW c )

[0052] V=(XW v )⊙(CW c )

[0053]

[0054] Among them, A represents the attention output, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and W q ∈R d×d ,W k ∈R d×d ,W v ∈R d×d ,W c ∈R 1×d is a learnable matrix; X represents the input of the current layer (i.e. the output of the previous layer), and the input of the first layer is the output of the feature enhancement module; each element in the matrix C represents the correlation between the scene and the target; the symbol ⊙ represents the Hadamard product. The feedforward neural network realizes nonlinear transformation and processing of the input vector through two linear transformations and a nonlinear activation function.

[0055] (2) After the residual connection and layer normalization operations, the result A of step (1) is input into a feedforward neural network (FFN), and the output of the current coding layer is obtained through residual connection and layer normalization (AddNorm). The details are as follows:

[0056] FFN(x)=max(0,xW1+b1)W2+b2

[0057] Among them, x represents the input feature vector, W1∈R d×d ,W2∈R d×d is a learnable matrix, b1∈R M×d ,b2∈R M ×d is bias.

[0058] (3) According to steps (1) and (2), the output of the visual encoder is obtained in is the output of the i-th encoding layer.

[0059] In the fifth step, the text embedding module uses word embedding and position encoding to obtain text features. The specific process is:

[0060] (1) Words are represented using onehot vectors and passed through a fully connected layer to obtain word embedding vectors.

[0061] (2) Use sine and cosine position encoding to help the model understand the position information of elements in the sequence. The details are as follows:

[0062]

[0063] Among them, PE represents the position encoding, pos represents the position of the current character in the input alphabet, i is the index in the embedding dimension, and d model is the dimension of the embedding vector.

[0064] (3) Add the word embedding vector from step (1) and the position encoding from step (2) to obtain the final text feature.

[0065] In the sixth step, the text decoder module stacks B decoding layers and receives the output of the visual encoder and text features. Each decoding layer first inputs the text features into the masked self-attention to ensure the causal relationship of sequence generation, then establishes multi-layer visual-text modality interaction through mesh cross attention, and finally uses the feedforward neural network to enhance the feature expression ability. The output of the last decoding layer is used as the output of the text decoder and input to the text generation module.

[0066] (1) In each decoding layer, the current layer first inputs the text features into the masked self-attention, as follows:

[0067]

[0068] andQ=mask(Y)W q

[0069] K=mask(Y)W k

[0070] V=mask(Y)W v

[0071] Among them, Y represents the input text sequence feature, Y mask represents the text sequence features after mask operation, W q ∈R d×d ,W k ∈R d×d ,W v ∈R d×d is a learnable matrix and mask(·) is a mask operation.

[0072] (2) Input the result of (1) into the mesh cross attention, as follows:

[0073]

[0074] Among them, S i Represents the output of the i-th encoding layer With the masked text sequence Y mask The joint representation of α i Represents the joint representation S i With the masked text sequence Y mask , B represents the number of encoding layers, and Z represents the mixed features of all cross-attention results. is a learnable matrix, σ is the sigmoid activation function, is the bias, and [~,~] is the concatenation operation.

[0075] (3) After the residual connection and layer normalization operation, the result Z of step (2) is input into a feedforward neural network (FFN), and the output of the current decoding layer is obtained after residual connection and layer normalization (AddNorm). The output of the last decoding layer is used as the output of the text decoder.

[0076] Step 7: The text generation module obtains the probability distribution of vocabulary words through the fully connected layer and the softmax layer, and gradually generates words. The details are as follows:

[0077] (1) Take the decoder output as input and connect it to the fully connected layer.

[0078] (2) After the output of (1), connect a Softmax classifier to calculate the word probability distribution and gradually generate words.

[0079] The present invention fully establishes the use of attention mechanism to enhance prior knowledge, adopts graph attention network and multi-head attention mechanism to refine and enhance the extracted features, further improves the expressive ability of features, effectively integrates target-level features and scene-level features, improves the richness and accuracy of feature expression, and promotes description generation by integrating into the Transformer structure. It can effectively handle the description generation problem of remote sensing image targets that are visually similar and contain complex and rich information, and improves the accuracy of the generated description context and the grammatical correctness.

[0080] The effects of the present invention are further described with reference to the embodiments:

[0081] Example

[0082] In the simulation experiment, this method is compared with the encoder-decoder network mRNN based on VGG-16+RNN, the VGG-16+LSTM encoder-decoder network Soft-attention based on the soft attention mechanism, and the M 2The transformer is evaluated on the same remote sensing image caption dataset, Sydney-Captions dataset. The Sydney-Captions dataset is collected and generated from the Sydney dataset of Google Earth. Each remote sensing image is cropped from a remote sensing image of Sydney, Australia with a resolution of 18,000*14,000 pixels. The cropped resolution is 500*500 pixels. It contains a total of 613 remote sensing images divided into 7 categories, such as airports, oceans, and factories. The dataset uses 237 different vocabularies to generate five different description tags for each remote sensing sample. The indicators for evaluating the model's predicted description include BLEU 1, BLEU 2, BLEU 3, BLEU 4, METEOR, ROUGE_L, and CIDEr. BLEU evaluates the degree of match between the sentences generated by the model and the reference sentences through the N-tuple matching rule, specifically analyzing the co-occurrence of vocabulary, word order, and sentence length. It is suitable for image description and machine translation tasks, and mainly measures accuracy. METEOR is an improvement on BLEU, using word alignment and considering the matching of synonyms, stems, etc., to make it more relevant to manual evaluation. ROUGE is a similarity measurement method based on recall, which calculates the co-occurrence probability of N-tuples in sentences generated by the model and reference sentences. ROUGE_L is mainly based on the precision and recall of the longest common subsequence, which is an effective indicator for evaluating machine translation and automatic summarization. CIDEr is designed for image annotation tasks and measures annotation consistency by reducing the weight of non-keywords. The better the model performs on these indicators, the higher its BLEU, METEOR, ROUGE_L and CIDEr scores.

[0083] Table 1 Test results of different methods on Sydney-Captions dataset

[0084]

[0085] From Table 1, we can see that the proposed method achieves better results than other methods. In general, the proposed method outperforms the current state-of-the-art methods M in terms of BLEU 1, BLEU 2, BLEU 3, BLEU 4, METEOR, ROUGE_L, and CIDEr, respectively. 2 transformer 1.90%, 2.82%, 3.44%, 3.84%, 4.46%, and 3.80% (relative improvement), which shows that the description quality of this method is better.

Claims

1. A Transformer method guided by prior knowledge of remote sensing image description, characterized in that The following steps are involved: In the first step, the remote sensing rotation target detection model ReDet is used in the feature extraction module to extract a specified number of regions of interest as target-level features and project them into the d-dimensional space through a fully connected layer; In the second step, the features extracted by the feature extraction module in the first step are used as input, and the target-level features are enhanced using the feature enhancement module; The third step is to insert the Class token V into the category cls As the concatenation of scene-level features and object-level features, it is used as the input of the visual encoder module; In the fourth step, the visual encoder module stacks B encoding layers; each encoding layer uses the prior scene-target knowledge to guide the model to focus on targets whose dot product similarity with the scene is greater than the set threshold, and uses a feedforward neural network to implement nonlinear transformation and processing of the input vector, thereby enhancing the model's expressive power; the output of all encoding layers is used as the input of the visual encoder; In the fifth step, the text embedding module uses word embedding and position encoding to obtain text features; Step 6: The text decoder module stacks B decoding layers and receives the output of the visual encoder and text features. Each decoding layer first inputs the text features into the masked self-attention to ensure the causal relationship of sequence generation, then establishes multi-layer visual-text modality interaction through mesh cross attention, and finally uses the feedforward neural network to enhance the feature expression ability. The output of the last decoding layer is used as the output of the text decoder and input into the text generation module. In the seventh step, the text generation module obtains the probability distribution of vocabulary words through the fully connected layer and the softmax layer, and gradually generates words.

2. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that: In the first step, the feature extraction module uses the remote sensing rotation target detection model ReDet to extract a specified number of regions of interest as target-level features V o , and projected into the d-dimensional space through the fully connected layer. The specific process is: (1) Use the remote sensing rotation target detection model ReDet to generate a set of target regions, which are represented as target-level features through adaptive average pooling. in represents the characteristics of the i-th target, It indicates that the target-level feature matrix is ​​an M×D1 matrix, where M is the number of targets and D1 is the dimension of each target feature; (2) The target-level feature V o Projected into a d-dimensional space through a fully connected layer.

3. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that: In the second step, the features extracted by the feature extraction module in the first step are used as input, and the target-level features are enhanced using the feature enhancement module. The specific process is as follows: (1) For the target-level feature V extracted in the feature extraction module o , construct a fully connected graph G o =(V o ,E o ),in is a node set, E o is an edge set; e ij Represents target-level features and The dot product similarity of Represents the normalized target-level features and The attention weight is: in and is a learnable parameter matrix; d is the scaling factor, i.e. the spatial dimension scaled to in the first step; N i is the neighborhood of node i in the graph; (2) Compute enhanced object-level features based on other nodes and the edges connecting them The formula is as follows: in is a learnable parameter matrix; (3) Use multi-head attention mechanism to capture multi-level information: in, represents the target-level features after being processed by the first attention head, are the corresponding learnable weight matrices; (4) Concatenate all attention heads and project them into d-dimensional space through a fully connected layer to obtain the final target-level features Specifically expressed as: Among them, W r ∈R d×d is a learnable parameter matrix, and the result of the i-th attention head is 4. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that: The third step is to insert the Class token V into the category cls As the concatenation of scene-level features and object-level features, as the input of the visual encoder module, the specific process is: For the target-level features obtained by the feature enhancement module Add a Class token at the beginning of the feature sequence to form a new input sequence 5. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that: In the fourth step, the visual encoder module stacks B encoding layers; each encoding layer uses the prior scene-target knowledge to guide the model to focus on targets whose dot product similarity with the scene is greater than the set threshold, and uses the feedforward neural network to realize nonlinear transformation and processing of the input vector, thereby enhancing the model's expressive power; the output of all encoding layers is used as the input of the visual encoder; the specific process is: (1) In each encoding layer, prior knowledge enhances attention through dot product similarity and softmax operation to obtain the prior knowledge matrix C∈R M×1 , and integrated into self-attention, the specific process is: and Q=XW q K=(XW k )⊙(CW c ) V=(XW v )⊙(CW c ) Among them, A represents the attention output, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and W q ∈R d×d ,W k ∈R d ×d ,W v ∈R d×d ,W c ∈R 1×d is a learnable matrix; X represents the input of the current layer, that is, the output of the previous layer, and the input of the first layer is the output of the feature enhancement module; each element in the matrix C represents the correlation between the scene and the target; the symbol ⊙ represents the Hadamard product; the feedforward neural network realizes the nonlinear transformation and processing of the input vector through two linear transformations and a nonlinear activation function; (2) After the residual connection and layer normalization operations, the result A of step (1) is input into a feed-forward neural network FFN, and the output of the current coding layer is obtained through residual connection and layer normalization, as follows: FFN(x)=max(0,xW1+b1)W2+b2 Among them, x represents the input feature vector, W1∈R d×d ,W2∈R d×d is a learnable matrix, b1∈R M×d ,b2∈R M×d is bias; (3) According to steps (1) and (2), the output of the visual encoder is obtained in is the output of the i-th encoding layer.

6. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that: In the fifth step, the text embedding module uses word embedding and position encoding to obtain text features. The specific process is as follows: (1) Words are represented using onehot vectors and passed through a fully connected layer to obtain word embedding vectors; (2) Sin-cosine position encoding is used to help the model understand the position information of elements in the sequence; the details are as follows: Among them, PE represents the position encoding, pos represents the position of the current character in the input alphabet, i is the index in the embedding dimension, and d model is the dimension of the embedding vector; (3) Add the word embedding vector from step (1) and the position encoding from step (2) to obtain the final text feature.

7. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that: In the sixth step, the text decoder module stacks B decoding layers and receives the output of the visual encoder and text features. Each decoding layer first inputs the text features into the masked self-attention to ensure the causal relationship of sequence generation, then establishes multi-layer visual-text modality interaction through mesh cross attention, and finally uses the feedforward neural network to enhance the feature expression ability. The output of the last decoding layer is used as the output of the text decoder and input into the text generation module. The details are as follows: (1) In each decoding layer, the current layer first inputs the text features into the masked self-attention, as follows: andQ=mask(Y)W q K=mask(Y)W k V=mask(Y)W v Among them, Y represents the input text sequence feature, Y mask represents the text sequence features after mask operation, W q ∈R d×d ,W k ∈R d×d ,W v ∈R d×d is a learnable matrix, mask(·) is a mask operation; (2) Input the result of (1) into the mesh cross attention, as follows: Among them, S i Represents the output of the i-th encoding layer With the masked text sequence Y mask The joint representation of α i Represents the joint representation S i With the masked text sequence Y mask , B represents the number of encoding layers, and Z represents the mixed features of all cross-attention results. is a learnable matrix, σ is the sigmoid activation function, is the bias, [·,·] is the concatenation operation; (3) After the residual connection and layer normalization operations, the result Z of step (2) is input into a feedforward neural network, and the output of the current decoding layer is obtained through residual connection and layer normalization; the output of the last decoding layer is used as the output of the text decoder.

8. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that: In the seventh step, the text generation module obtains the probability distribution of vocabulary words through the fully connected layer and the softmax layer, and gradually generates words, as follows: (1) Take the decoder output as input and connect it to the fully connected layer; (2) After the output of (1), connect a Softmax classifier to calculate the word probability distribution; gradually generate words.

9. A Transformer system guided by prior knowledge of remote sensing image description, characterized in that For implementing the method described in any one of claims 1 to 8, the system comprises: The feature extraction module uses the remote sensing rotation target detection model to extract target-level features; The feature enhancement module combines graph neural networks and attention mechanisms to build target-target relationships and distinguish similar targets. The category insertion module inserts the Class token as a scene-level feature before the object-level feature to obtain the global representation of the remote sensing image; The visual encoder module stacks prior knowledge to enhance attention and feedforward neural networks, guides the model to focus on targets whose dot product similarity with the scene is greater than a set threshold, and captures different levels of visual information; The text decoder module uses a mesh connection structure to interact with all levels of visual information and text information; The text generation module obtains the word probability distribution of the vocabulary according to the output of the text decoder and generates words step by step.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images