Prior knowledge guided transformer method and system for remote sensing image description

Abstract

This invention discloses a prior knowledge-guided Transformer method and system for remote sensing image description, comprising: extracting target-level features using a remote sensing rotating target detection model; constructing target-target relationships by combining graph neural networks and attention mechanisms; obtaining a global representation of the remote sensing image; stacking prior knowledge to enhance attention and feedforward neural networks, guiding the model to focus on targets with a dot product similarity to the scene greater than a set threshold, and capturing visual information at different levels; using a mesh connection structure to interact visual and textual information at all levels; and generating words step by step by obtaining the word probability distribution of the vocabulary based on the output of the text decoder. This invention fully establishes scene-target semantic associations, effectively distinguishes similar targets with the help of graph structures, enhances the accuracy of description, and introduces prior scene-target knowledge to guide the model to focus on targets related to specific scenes, exhibiting good semantic consistency.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image description generation technology, and in particular to a Transformer method and system for remote sensing image description guided by prior knowledge. Background Technology

[0002] Image captioning is an important research direction at the intersection of computer vision and natural language processing. Its goal is to extract meaningful visual features from input images using algorithms and generate natural language sentences that describe the content of those images. Remote sensing image captioning aims to generate meaningful and grammatically accurate sentences for remote sensing images. In recent years, with the development of remote sensing technology, high-resolution remote sensing images have been widely used in environmental monitoring, urban planning, agricultural management, and many other fields. However, how to quickly and accurately understand and describe the content of these images has become a pressing problem. Compared with natural image captioning, remote sensing images, mostly taken from high altitudes or by satellite, depict geographical and environmental features such as forests, rivers, urban buildings, and farmland. These targets are often visually very similar, with relatively uniform colors and textures, and are numerous, covering a wide area. They contain a large number of different types of targets and complex geographical information, which can easily lead to confusion in model recognition and description, making it difficult to determine the focus of the description. This makes remote sensing image captioning a particularly challenging task.

[0003] Most existing image captioning methods are based on deep learning. Vinyals et al. [O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3156–3164.] first introduced an encoder-decoder framework, in which the encoder uses a convolutional neural network (CNN) to extract high-level visual features, and the decoder uses a recurrent neural network (RNN) to generate the caption. Furthermore, Xu et al. [K. Xu, J. Ba, and R. Kiros, “Show, attend and tell: Neural image caption generation with visual attention,” in Proc. Int. Conf. Mach. Learn., 2015, pp. 2048–2057.] incorporated an attention mechanism into their framework to enhance attention to different regions of the feature map in the image. Qu et al. [B.Qu, X.Li, D.Tao, and X.Lu, “Deep semantic understanding of high resolution remote sensing image,” in Proc. Int. Conf. Comput., Inf. Telecommun. Syst. (CITS), Jul. 2016, pp. 1–5, doi:10.1109 / CITS.2016.7546397.] explored the problem of deep semantic understanding of high-resolution remote sensing images and proposed a multimodal deep neural network that uses convolutional neural networks to extract image features and then combines them with textual descriptions of the image through RNN or LSTM. Vaswani et al. [A. Vaswani, N. Shazeer, and N. Parmar, “Attention is all you need,” in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 5998–6008.] introduced the Transformer model, replacing the traditional recurrent neural network (RNN) with a self-attention mechanism. This enabled parallel processing of sequential data, and most subsequent image captioning methods have adopted the attention mechanism. While these methods can generate grammatically correct descriptions, they often fail to accurately capture the focus or subject of the image content, ignoring the relationship between the target and scene regions. Summary of the Invention

[0004] The purpose of this invention is to provide a prior knowledge-guided Transformer method and system for remote sensing image description, which fully considers the influence of target-level features and scene-level features on remote sensing image description, effectively integrates the correlation and difference information between different target or scene regions into its feature embedding, enabling the model to distinguish visually similar targets, thereby providing a more accurate description of remote sensing images.

[0005] The technical solution to achieve the purpose of this invention is: a Transformer method guided by prior knowledge for remote sensing image description, comprising the following steps:

[0006] The first step is to use the remote sensing rotating target detection model ReDet in the feature extraction module to extract a specified number of regions of interest as target-level features, and then project them onto the d-dimensional space through a fully connected layer;

[0007] The second step is to use the features extracted by the feature extraction module in the first step as input and enhance the target-level features using the feature enhancement module.

[0008] The third step is for the category insertion module to insert the Class tokenV. cls The scene-level features and target-level features are concatenated and used as input to the visual encoder module;

[0009] The fourth step involves stacking B encoding layers in the visual encoder module. Each encoding layer uses prior scene-target knowledge to guide the model to focus on targets whose dot product similarity to the scene is greater than a set threshold, and uses a feedforward neural network to perform nonlinear transformation and processing of the input vector, thereby enhancing the model's expressive power. The outputs of all encoding layers serve as the inputs to the visual encoder.

[0010] Fifth, the text embedding module uses word embedding and positional encoding to obtain text features;

[0011] In the sixth step, the text decoder module stacks B decoding layers, receiving the output of the visual encoder and text features. Each decoding layer first inputs the text features into a masked self-attention to ensure the causal relationship of the sequence generation, then establishes multi-layer visual-text modal interaction through mesh cross-attention, and finally uses a feedforward neural network to enhance the feature representation capability. The output of the last decoding layer is used as the output of the text decoder and input into the text generation module.

[0012] The seventh step is the text generation module, which obtains the probability distribution of words in the vocabulary through fully connected layers and softmax layers, and gradually generates words.

[0013] A prior knowledge-guided Transformer system for remote sensing image description, used to implement the above method, includes:

[0014] The feature extraction module uses a remote sensing rotating target detection model to extract target-level features;

[0015] The feature enhancement module combines graph neural networks and attention mechanisms to construct target-target relationships, which are used to distinguish similar targets;

[0016] The category insertion module inserts a class token as a scene-level feature before the target-level feature to obtain a global representation of the remote sensing image.

[0017] The visual encoder module stacks prior knowledge to enhance attention and feedforward neural networks, guiding the model to focus on targets with a dot product similarity to the scene greater than a set threshold, and capturing visual information at different levels.

[0018] The text decoder module uses a mesh connection structure to enable interaction between visual and text information at all levels.

[0019] The text generation module generates words step by step based on the probability distribution of words in the vocabulary obtained from the output of the text decoder.

[0020] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0021] Compared with the prior art, the significant features of this invention are: (1) a feature extraction module is constructed by remote sensing rotating target detection, which extracts richer features; (2) the target-level features are refined by combining graph neural networks and attention mechanisms, making the features more accurate; (3) prior scene-target knowledge is calculated and integrated into the self-attention mechanism to guide the model to describe targets closely related to the scene, thereby generating a description that is more in line with the context.

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0023] Figure 1 This is a structural diagram of the present invention.

[0024] Figure 2 This is a structural diagram of the encoder.

[0025] Figure 3 This is a structural diagram of the decoder. Detailed Implementation

[0026] Combination Figures 1-3 The implementation process of the present invention will be described in detail.

[0027] A prior knowledge-based approach to remote sensing image description guides the Transformer method, with the following steps:

[0028] The first step, in the feature extraction module, uses the remote sensing rotating target detection model ReDet to extract a specified number of regions of interest as target-level features V. o And projected onto d-dimensional space through a fully connected layer. The specific process is as follows:

[0029] (1) A set of target regions is generated using the remote sensing rotating target detection model ReDet, and then processed by adaptive average pooling to represent them as target-level features. in This represents the feature of the i-th target. This indicates that the target-level feature matrix is ​​an M×D1 matrix, where M is the number of targets and D1 is the dimension of each target feature.

[0030] (2) Target-level features V o Project it into a d-dimensional space through a fully connected layer.

[0031] The second step involves using the features extracted in the first step (feature extraction module) as input to enhance the target-level features using a feature enhancement module (graph attention network). The specific process is as follows:

[0032] (1) For the target-level features V extracted by the feature extraction module o Construct a fully connected graph G o =(V o E o ),in It is a set of nodes, E o It is an edge set. Edge e ij Representing target-level features and dot product similarity, Represents the target-level features after normalization and The attention weight is calculated using the following formula:

[0033]

[0034]

[0035] in and is a learnable parameter matrix; d is a scaling factor, i.e., the spatial dimension scaled to in the first step; N i It is the neighborhood of node i in the graph.

[0036] (2) Calculate enhanced target-level features based on other nodes and the edges connecting them. The formula is as follows:

[0037]

[0038] in It is a learnable parameter matrix.

[0039] (3) Utilize multi-head attention mechanisms to capture multi-level information:

[0040]

[0041] in, This represents the target-level features after processing by the first attention head. These are the corresponding learnable weight matrices, and d is the scaling factor, i.e., the spatial dimension scaled to in the first step.

[0042] (4) Concatenate all attention heads and project them onto d-dimensional space through a fully connected layer to obtain the final target-level features. Specifically, it is expressed as follows:

[0043]

[0044] Among them, W r ∈R d×d It is a learnable parameter matrix, and the result of the i-th attention head is...

[0045] The third step is for the category insertion module to insert the Class tokenV. cls The scene-level features and target-level features are concatenated and used as input to the visual encoder module. The specific process is as follows:

[0046] For the target-level features obtained by the feature enhancement module Add a class token to the beginning of the feature sequence to form a new input sequence.

[0047] The fourth step involves stacking B encoding layers in the visual encoder module. Each encoding layer utilizes prior scene-target knowledge to guide the model to focus on targets with a dot product similarity to the scene greater than a set threshold, and employs a feedforward neural network to perform non-linear transformations and processing of the input vector, thereby enhancing the model's expressive power. The outputs of all encoding layers serve as the input to the visual encoder. The specific process is as follows:

[0048] (1) In each encoding layer, prior knowledge enhancement attention is obtained by dot product similarity and softmax operation to obtain the prior knowledge matrix C∈R. M×1 And integrate it into self-attention, the specific process is as follows:

[0049]

[0050] and Q = XW q

[0051] K = (XW) k )⊙(CW c )

[0052] V=(XW v )⊙(CW c )

[0053]

[0054] Where A represents the attention output, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and W represents the value matrix. q ∈R d×d W k ∈R d×d W v ∈R d×d W c ∈R 1×d It is a learnable matrix; X represents the input of the current layer (i.e., the output of the previous layer), and the input of the first layer is the output of the feature enhancement module; each element in matrix C represents the correlation between the scene and the target; the symbol ⊙ represents the Hadamard product. The feedforward neural network achieves nonlinear transformation and processing of the input vector through two linear transformations and a nonlinear activation function.

[0055] (2) After residual connection and layer normalization operations, the result A from step (1) is input into a feedforward neural network (FFN), and after residual connection and layer normalization (AddNorm), the output of the current coding layer is obtained. Specifically:

[0056] FFN(x) = max(0, xW1+b1)W2+b2

[0057] Where x represents the input feature vector, W1∈R d×d ,W2∈R d×d It is a learnable matrix, b1∈R M×d b2∈R M ×d It is a bias.

[0058] (3) Based on steps (1) and (2), obtain the output of the visual encoder. in It is the output of the i-th coding layer.

[0059] The fifth step involves the text embedding module using word embedding and positional encoding to obtain text features. The specific process is as follows:

[0060] (1) Words are represented using one-hot vectors and passed through a fully connected layer to obtain word embedding vectors.

[0061] (2) Sine and cosine positional encoding is used to help the model understand the positional information of elements in the sequence. Specifically:

[0062]

[0063] Where PE represents position encoding, pos represents the position of the current character within the input letters, i is the index in the embedding dimension, and d... model It is the dimension of the embedded vector.

[0064] (3) Add the word embedding vector from step (1) and the position encoding from step (2) to obtain the final text features.

[0065] In the sixth step, the text decoder module stacks B decoding layers, receiving the output of the visual encoder and text features. Each decoding layer first inputs the text features into a masked self-attention mechanism to ensure the causal relationship of the sequence generation, then establishes multi-layered visual-text modal interactions through meshed cross-attention, and finally utilizes a feedforward neural network to enhance feature representation capabilities. The output of the last decoding layer serves as the output of the text decoder and is input into the text generation module.

[0066] (1) In each decoding layer, the text features of the current layer are first input into the mask self-attention, as follows:

[0067]

[0068] and Q = mask(Y)W q

[0069] K = mask(Y)W k

[0070] V = mask(Y)W v

[0071] Where Y represents the input text sequence features, Y mask W represents the text sequence features after masking. q ∈R d×d W k ∈R d×d W v ∈R d×d It is a learnable matrix, and mask(·) is the masking operation.

[0072] (2) Input the result of (1) into the mesh cross-attention, as follows:

[0073]

[0074] Among them, S i Indicates the output of the i-th coding layer With masked text sequence Y mask The joint representation of α i S represents the union. i With masked text sequence Y mask The degree of correlation, where B represents the number of encoding layers, and Z represents the mixed features that aggregate all cross-attention results. It is a learnable matrix, and σ is the sigmoid activation function. [~,~] is the offset, and [~,~] is the concatenation operation.

[0075] (3) After the result Z from step (2) is processed through residual connections and layer normalization, it is input into a feedforward neural network (FFN). After residual connections and layer normalization (AddNorm), the output of the current decoding layer is obtained. The output of the last decoding layer is used as the output of the text decoder.

[0076] Step 7: The text generation module uses fully connected layers and softmax layers to obtain the probability distribution of words in the vocabulary, gradually generating words. Details are as follows:

[0077] (1) Use the decoder output as input to connect to a fully connected layer.

[0078] (2) After the output of (1), connect a Softmax classifier to calculate the word probability distribution. Gradually generate words.

[0079] This invention fully utilizes attention mechanisms to enhance prior knowledge, employs graph attention networks and multi-head attention mechanisms to refine and enhance extracted features, further improving the expressive power of features. It effectively integrates target-level and scene-level features, improving the richness and accuracy of feature representation. By integrating it into the Transformer structure, it promotes description generation and can effectively handle the problem of generating descriptions of visually similar targets in remote sensing images with complex and rich information, improving the accuracy of generated description context and grammatical correctness.

[0080] The effects of the present invention will be further explained with reference to the embodiments:

[0081] Example

[0082] In simulation experiments, this method was compared with the VGG-16+RNN-based encoder-decoder network mRNN, the VGG-16+LSTM-based encoder-decoder network Soft-attention, and the MRNN that embeds prior knowledge of fused object relationships into the encoder. 2Transformer was evaluated on the same remote sensing image captioning dataset, Sydney-Captions. The Sydney-Captions dataset was collected and generated from the Google Earth Sydney dataset. Each remote sensing image was cropped from a 18,000*14,000 pixel remote sensing image of Sydney, Australia, resulting in a 500*500 pixel resolution image. It contains 613 images categorized into seven classes: airport, ocean, and factory. The dataset uses 237 different words to generate five distinct descriptive labels for each remote sensing sample. Evaluation metrics for the model's predicted descriptions include BLEU 1, BLEU 2, BLEU 3, BLEU 4, METEOR, ROUGE_L, and CIDEr. BLEU evaluates the matching degree between the model-generated sentences and reference sentences using N-tuple matching rules, specifically analyzing the co-occurrence of words, word order, and sentence length. It is suitable for image captioning and machine translation tasks, primarily measuring accuracy. METEOR improves upon BLEU by employing word alignment methods, considering synonym and stemming matching to enhance its relevance to human evaluation. ROUGE is a recall-based similarity metric that calculates the co-occurrence probability of N-tuples in the model-generated sentence and the reference sentence. ROUGE_L, primarily based on the precision and recall of the longest common subsequence, is an effective indicator for evaluating machine translation and automatic summarization. CIDEr is specifically designed for image annotation tasks, measuring annotation consistency by reducing the weight of non-keywords. The better the model performs on these metrics, the higher its scores in BLEU, METEOR, ROUGE_L, and CIDEr.

[0083] Table 1. Test results of different methods on the Sydney-Captions dataset.

[0084]

[0085] As can be seen from Table 1, the method proposed in this invention achieves better results compared to other methods. Overall, this invention outperforms the current state-of-the-art method M in terms of BLEU 1, BLEU 2, BLEU 3, BLEU 4, METEOR, ROUGE_L, and CIDEr. 2 The transformer percentages of 1.90%, 2.82%, 3.44%, 3.84%, 4.46%, and 3.80% (relative improvement) indicate that the description quality of this method is better.

Claims

1. A Transformer method guided by prior knowledge for remote sensing image description, characterized in that, Includes the following steps: The first step is to use the remote sensing rotating target detection model ReDet in the feature extraction module to extract a specified number of regions of interest as target-level features, and then project them onto the d-dimensional space through a fully connected layer; The second step is to use the features extracted by the feature extraction module in the first step as input and enhance the target-level features using the feature enhancement module. The third step is for the category insertion module to insert the Class token V. cls The scene-level features and target-level features are concatenated and used as input to the visual encoder module; The fourth step involves stacking B encoding layers in the visual encoder module. Each encoding layer uses prior scene-target knowledge to guide the model to focus on targets whose dot product similarity to the scene is greater than a set threshold, and uses a feedforward neural network to perform nonlinear transformation and processing of the input vector, thereby enhancing the model's expressive power. The outputs of all encoding layers serve as the inputs to the visual encoder. Fifth, the text embedding module uses word embedding and positional encoding to obtain text features; In the sixth step, the text decoder module stacks B decoding layers, receiving the output of the visual encoder and text features. Each decoding layer first inputs the text features into a masked self-attention to ensure the causal relationship of the sequence generation, then establishes multi-layer visual-text modal interaction through mesh cross-attention, and finally uses a feedforward neural network to enhance the feature representation capability. The output of the last decoding layer is used as the output of the text decoder and input into the text generation module. The seventh step is the text generation module, which obtains the probability distribution of words in the vocabulary through fully connected layers and softmax layers, and gradually generates words.

2. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that, The first step, in the feature extraction module, uses the remote sensing rotating target detection model ReDet to extract a specified number of regions of interest as target-level features V. o And projected onto d-dimensional space through a fully connected layer, the specific process is as follows: (1) A set of target regions is generated using the remote sensing rotating target detection model ReDet, and then processed by adaptive average pooling to represent them as target-level features. in The features of the i-th target are represented. This indicates that the target-level feature matrix is ​​an M×D1 matrix, where M is the number of targets and D1 is the dimension of each target feature; (2) Target-level features V o Project it into a d-dimensional space through a fully connected layer.

3. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that, The second step involves using the features extracted by the feature extraction module from the first step as input to enhance the target-level features using the feature enhancement module. The specific process is as follows: (1) For the target-level features V extracted by the feature extraction module o Construct a fully connected graph G o =(V o E o ),in It is a set of nodes, E o It is an edge set; e ij Representing target-level features and dot product similarity, Represents the target-level features after normalization and The attention weight is calculated using the following formula: in and It is a learnable parameter matrix; d is the scaling factor, i.e., the spatial dimension scaled to in the first step; N i It is the neighborhood of node i in the graph; (2) Calculate enhanced target-level features based on other nodes and the edges connecting them. The formula is as follows: in It is a learnable parameter matrix; (3) Utilize multi-head attention mechanisms to capture multi-level information: in, This represents the target-level features after processing by the first attention head. These are the corresponding learnable weight matrices; (4) Concatenate all attention heads and project them onto d-dimensional space through a fully connected layer to obtain the final target-level features. Specifically, it is expressed as follows: Among them, W r ∈R d×d It is a learnable parameter matrix, and the result of the i-th attention head is...

4. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that, The third step is for the category insertion module to insert the Class tokenV. cls The scene-level features and target-level features are concatenated and used as input to the visual encoder module. The specific process is as follows: For the target-level features obtained by the feature enhancement module Add a class token to the beginning of the feature sequence to form a new input sequence.

5. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that, The fourth step involves stacking B encoding layers in the visual encoder module. Each encoding layer utilizes prior scene-target knowledge to guide the model to focus on targets whose dot product similarity to the scene is greater than a set threshold. A feedforward neural network is used to perform non-linear transformations and processing of the input vector, thereby enhancing the model's expressive power. The outputs of all encoding layers serve as the input to the visual encoder. The specific process is as follows: (1) In each encoding layer, prior knowledge enhancement attention is obtained by dot product similarity and softmax operation to obtain the prior knowledge matrix C∈R. M×1 And integrate it into self-attention, the specific process is as follows: and Q=XW q K=(XW k )⊙(CW c ) V=(XW v )⊙(CW c ) Where A represents the attention output, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and W represents the value matrix. q ∈R d×d W k ∈R d ×d W v ∈R d×d W c ∈R 1×d It is a learnable matrix; X represents the input of the current layer, i.e. the output of the previous layer. The input of the first layer is the output of the feature enhancement module; each element in matrix C represents the correlation between the scene and the target; the symbol ⊙ represents the Hadamard product; the feedforward neural network realizes the nonlinear transformation and processing of the input vector through two linear transformations and a nonlinear activation function; (2) After residual connection and layer normalization operations are performed on the result A of step (1), it is input into a feedforward neural network FFN. After residual connection and layer normalization, the output of the current coding layer is obtained, as follows: FFN(x) = max(0, xW1+b1)W2+b2 Where x represents the input feature vector, W1∈R d×d ,W2∈R d×d It is a learnable matrix, b1∈R M×d b2∈R M×d It is a bias; (3) Based on steps (1) and (2), obtain the output of the visual encoder. in It is the output of the i-th coding layer.

6. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that, The fifth step involves the text embedding module using word embedding and positional encoding to obtain text features. The specific process is as follows: (1) Words are represented using one-hot vectors and passed through a fully connected layer to obtain word embedding vectors; (2) Sine and cosine position encoding is used to help the model understand the position information of elements in the sequence; the details are as follows: Where PE represents position encoding, pos represents the position of the current character within the input letters, i is the index in the embedding dimension, and d... model It is the dimension of the embedded vector; (3) Add the word embedding vector from step (1) and the position encoding from step (2) to obtain the final text features.

7. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that, Step 6: The text decoder module stacks B decoding layers, receiving the output of the visual encoder and text features. Each decoding layer first inputs the text features into a masked self-attention mechanism to ensure the causal relationship of the sequence generation, then establishes multi-layer visual-text modal interaction through meshed cross-attention, and finally uses a feedforward neural network to enhance the feature representation capability. The output of the last decoding layer is used as the output of the text decoder and input into the text generation module. Specifically: (1) In each decoding layer, the text features of the current layer are first input into the mask self-attention, as follows: andQ=mask(Y)W q K=mask(Y)W k V=mask(Y)W v Where Y represents the input text sequence features, Y mask W represents the text sequence features after masking. q ∈R d×d W k ∈R d×d W v ∈R d×d It is a learnable matrix, and mask(·) is the masking operation; (2) Input the result of (1) into the mesh cross-attention, as follows: Among them, S i Indicates the output of the i-th coding layer With masked text sequence Y mask The joint representation of α i S represents the union. i With masked text sequence Y mask The degree of correlation, where B represents the number of encoding layers, and Z represents the mixed features that aggregate all cross-attention results. It is a learnable matrix, and σ is the sigmoid activation function. It is an offset, and [·,·] is a concatenation operation; (3) After the result Z from step (2) is processed by residual connection and layer normalization, it is input into a feedforward neural network. After residual connection and layer normalization, the output of the current decoding layer is obtained; the output of the last decoding layer is used as the output of the text decoder.

8. The prior knowledge-guided Transformer method for remote sensing image description according to claim 1, characterized in that, Step 7, the text generation module, obtains the probability distribution of words in the vocabulary through fully connected layers and softmax layers, and gradually generates words, as follows: (1) Use the decoder output as input to connect to a fully connected layer; (2) After the output of (1), connect a Softmax classifier to calculate the word probability distribution; generate words step by step.

9. A Transformer system guided by prior knowledge for remote sensing image description, characterized in that, The system for implementing the method of any one of claims 1-8 comprises: The feature extraction module uses a remote sensing rotating target detection model to extract target-level features; The feature enhancement module combines graph neural networks and attention mechanisms to construct target-target relationships, which are used to distinguish similar targets; The category insertion module inserts a class token as a scene-level feature before the target-level feature to obtain a global representation of the remote sensing image. The visual encoder module stacks prior knowledge to enhance attention and feedforward neural networks, guiding the model to focus on targets with a dot product similarity to the scene greater than a set threshold, and capturing visual information at different levels. The text decoder module uses a mesh connection structure to enable interaction between visual and text information at all levels. The text generation module generates words step by step based on the probability distribution of words in the vocabulary obtained from the output of the text decoder.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images