A remote sensing image description generation method based on external knowledge retrieval enhancement

Abstract

The application relates to a remote sensing image description generation method based on external knowledge retrieval enhancement, which comprises the following steps: step 1, visual feature coding; step 2, external knowledge retrieval enhancement; step 3, cross-modal generation decoder; and step 4, multi-task joint optimization. The application has the beneficial effects that: the retrieval enhancement generation paradigm is used; through manifold alignment based on principal component analysis and a diversity perception reordering strategy, high-confidence semantic knowledge is efficiently extracted and fused from an external knowledge base, and the limitation of a parameterized model in fine-grained semantic cognition is effectively made up. Meanwhile, a cross-modal attention entropy regularization mechanism is adopted, the cross-modal alignment quality in the generation stage is improved, the attention collapse phenomenon is inhibited, and the model is encouraged to balance attention to global information of an image. In addition, a joint loss function is used in combination with a focal loss and length normalization, and the long-tail vocabulary prediction and short sentence generation problems are systematically alleviated.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image description, and more specifically to a remote sensing image description generation method based on external knowledge retrieval enhancement. Background Technology

[0002] Remote sensing image description aims to automatically generate natural language descriptions that accurately reflect the content and semantic relationships of remote sensing scenes, serving as a bridge between remote sensing visual perception and human language understanding. Thanks to the rapid development of deep learning technologies such as convolutional neural networks and Transformers, feature alignment based on an "encoder-decoder" architecture and a visual-language model has become the mainstream trend. However, directly transferring these general-domain technologies to the remote sensing field still presents challenges, including the scarcity and long-tail distribution of domain semantic knowledge, difficulties in fine-grained cross-modal alignment, homogenization and generalization of generated descriptions, and the risk of overfitting due to data sparsity.

[0003] The retrieval-enhanced generation paradigm addresses core challenges in remote sensing image description tasks, such as domain knowledge scarcity and difficulties in cross-modal alignment. It utilizes non-parametric external knowledge bases as supplementary information sources to fill semantic gaps in parametric models and improves generation accuracy through structured constraint mechanisms. Summary of the Invention

[0004] To address the dual challenges of complex scene semantics and scarce training data in remote sensing image description tasks, this invention proposes a remote sensing image description generation method based on external knowledge retrieval enhancement. First, the input remote sensing image is encoded into a sequence of visual features highly aligned with the linguistic semantic space. Then, principal component analysis manifold alignment and diversity-aware reordering strategies are used to efficiently extract high-confidence and semantically complementary auxiliary text from an external knowledge base. Finally, the visual features are fused with the retrieved knowledge prefixes, and a fine-grained alignment between visual and linguistic modalities is established through a cross-attention mechanism, thereby generating a more comprehensive description.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A remote sensing image description generation method based on external knowledge retrieval enhancement, comprising the following steps:

[0006] Step 1: By comparing the language and image pre-trained models, the input remote sensing image is processed to obtain a visual feature sequence aligned with the language semantic space;

[0007] Step 2: Based on the visual feature sequence obtained in Step 1, obtain the prompt prefix through knowledge base retrieval and reordering;

[0008] Step 3: Based on the visual feature sequence obtained in Step 1 and the cue prefix obtained in Step 2, the preliminary descriptive text is generated by fusing the two-generation decoder through a generative pre-trained transformer.

[0009] Step 4: Based on the preliminary description of the text generation process in Step 3, the final accurate and comprehensive description text is obtained through end-to-end training of the joint loss function.

[0010] Based on the above technical solution, the present invention can be further improved as follows:

[0011] Furthermore, step 1 specifically involves:

[0012] Step 1.1: Process the input image using a contrastive language and image pre-trained image processor. Perform resizing and normalization preprocessing operations to ensure it fits the pre-trained model;

[0013] Step 1.2: The processed image is divided into fixed-size image blocks of 32×32 pixels, converted into serialized block embeddings by linear projection, and a learnable positional code is added to preserve the spatial structure information of the image.

[0014] Step 1.3: Input the embedded sequence obtained in Step 1.2 into the contrastive language and image pre-trained visual encoder; finally, extract the last hidden state of the encoder as the visual feature sequence, calculated using the following formula:

[0015] ;

[0016] Among them, F v I represents the obtained visual feature sequence, and I represents the embedded sequence of the image after preprocessing. This indicates the sequence length after image segmentation. The feature embedding dimension.

[0017] Furthermore, the contrastive language and image pre-trained visual encoder consists of a multi-layer deep learning model encoder based on an attention mechanism. Each layer contains a multi-head self-attention mechanism and a feedforward network, and training stability is ensured through residual connections and layer normalization.

[0018] Furthermore, step 2 specifically involves:

[0019] Step 2.1, Knowledge Base Construction and Feature Alignment: The CLIP text encoder is used to extract feature vectors from all descriptive statements in the remote sensing text knowledge base. Principal component analysis is then used to map these vectors to a low-dimensional manifold space, achieving feature alignment and dimensionality reduction. The calculation formula is as follows:

[0020] ;

[0021] Where T pca It is the text feature matrix after dimensionality reduction by PCA, and T is the set of all descriptive statements in the knowledge base. , where represents the feature dimension after dimensionality reduction.

[0022] Step 2.2, Efficient Semantic Retrieval: Encode the visual feature sequence obtained in Step 1 and reduce its dimensionality to obtain the query vector. A hierarchical navigation small-world indexing algorithm is used to perform approximate nearest neighbor retrieval in the knowledge base, and a preliminary set of candidate descriptions is selected. The calculation formula is as follows:

[0023] ;

[0024] in, This represents the optimal set of k descriptions retrieved, where k represents the number of nearest neighbors to be retrieved.

[0025] Step 2.3, Diversity-Aware Reordering and Prefix Construction: Through a comprehensive scoring function Reorder the candidate set and perform final selection. The optimal description is concatenated into a prompt prefix according to the language model input format [KNOWLEDGE] cap_1 || cap_2 || ... || cap_k [IMAGE];

[0026] Among them, S(c) in the comprehensive scoring function i ) represents the overall score of the i-th candidate set. For cosine similarity, For length-normalized fractions, For diversity scores based on word overlap, c i Here, q is the i-th candidate description, and q is the query vector obtained after encoding and dimensionality reduction of the visual feature sequence. These represent the weighting coefficients for each score.

[0027] Furthermore, step 3 specifically involves:

[0028] Step 3.1: Using the second generation of generative pre-trained transformers as the decoder architecture, the visual feature sequence obtained in step 1 is received simultaneously at each time step. The prompt prefix obtained in step 2 Cross-modal fusion is achieved through a cross-attention mechanism, which injects visual features into the generation process. The cross-attention calculation formula is as follows:

[0029] ;

[0030] in , , , This indicates the hidden state of the parent layer. These are the query, key, and value projection matrices, respectively, F.v This is the visual feature sequence obtained in step 1, where Q, K, and V are the query, key, and value matrices, respectively, and d k It refers to the key and query dimensions in the attention mechanism.

[0031] Step 3.2 introduces an attention entropy regularization mechanism to suppress attention collapse, the calculation formula of which is as follows:

[0032] ;

[0033] ;

[0034] in, Indicates the first The attention distribution entropy of an attention head at time step t. For balance coefficient, It is the attention entropy regularization loss, where H is the total number of attention heads, T is the number of time steps, L is the length of the attention distribution, and A is the number of attention heads. h,t A is the attention distribution of the h-th attention head at time step t. h,t,l It is the attention weight of the h-th attention head on the l-th visual feature at time step t. It is a small constant used to prevent the logarithm from being negative infinity;

[0035] Step 3.3, the inference stage uses a cluster search algorithm, combined with repetition penalty and length penalty techniques, to generate preliminary descriptive text. The output probability distribution calculation formula is as follows:

[0036] ;

[0037] Furthermore, step 4 specifically involves:

[0038] Construct a loss function that includes the focus loss function, the length normalized loss function, and the loss function obtained in step 3.2. A joint optimization framework using three loss functions is employed to perform end-to-end collaborative training of the model, optimizing it at the lexical, sentence, and attention levels to ultimately obtain accurate and comprehensive descriptive text; among which:

[0039] The focus loss function is used to address the long-tail vocabulary distribution problem, and its calculation formula is as follows:

[0040] ;

[0041] in, It is a focal loss. This represents the model's predicted probability for the target word. For focusing parameters;

[0042] The length-normalized loss function is used to alleviate the problem of generating short sentences, and its calculation formula is as follows:

[0043] ;

[0044] in, For length normalization loss, These are the weighting coefficients. Indicates the effective length of the generated sequence. The preset target length;

[0045] The results obtained through the focus loss function, the length normalized loss function, and step 3.2 are as follows: The total loss function is as follows:

[0046] ;

[0047] in, For the total loss, It is a focal loss. For length normalization loss, It is the attention entropy regularization loss.

[0048] The beneficial effects of this invention are as follows: Utilizing a retrieval-enhanced generation paradigm, and employing a manifold alignment and diversity-aware reordering strategy based on principal component analysis, it efficiently extracts and integrates high-confidence semantic knowledge from an external knowledge base, effectively compensating for the limitations of parametric models in fine-grained semantic cognition. Simultaneously, a cross-modal attention entropy regularization mechanism is employed to improve the quality of cross-modal alignment during the generation stage, suppressing attention collapse and encouraging the model to focus on global image information in a balanced manner. Furthermore, the use of a joint loss function combined with focus loss and length normalization systematically alleviates the problems of long-tail vocabulary prediction and short sentence generation. Attached Figure Description

[0049] Figure 1 This is a flowchart of the present invention;

[0050] Figure 2 This is a flowchart of step 1 of the present invention;

[0051] Figure 3 This is a flowchart of step 2 of the present invention;

[0052] Figure 4 This is a flowchart of step 3 of the present invention. Detailed Implementation

[0053] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0054] like Figure 1As shown, a remote sensing image description generation method based on external knowledge retrieval enhancement includes the following steps:

[0055] Step 1, Visual Feature Encoding

[0056] like Figure 2 As shown, the CLIP-ViT (contrastive language and image pre-training) model is used to process the input remote sensing image to obtain a visual feature sequence aligned with the linguistic semantic space. Visual feature encoding is used to transform the raw remote sensing pixel information into semantically rich deep features. The CLIP-ViT model, pre-trained on large-scale internet image-text pairs, is used as the visual backbone network. Its core advantage lies in the natural alignment between the visual features obtained through contrastive learning and the linguistic semantic space. Specifically:

[0057] Step 1.1: Use CLIP (Contrastive Language and Image Pre-training) image processor to process the input image. Perform preprocessing operations such as resizing and normalization to ensure that the input format is compatible with the pre-trained model.

[0058] Step 1.2: The processed image is segmented into fixed-size 32×32 pixel image blocks, converted into serialized block embeddings by linear projection, and learnable positional codes are added to preserve the spatial structure information of the image.

[0059] Step 1.3: Input the embedding sequence obtained in Step 1.2 into the CLIP-ViT-B / 32 (comparative language and image pre-training) visual encoder (composed of a multi-layer Transformer encoder, each layer containing a multi-head self-attention mechanism and a feedforward network, and training stability is ensured through residual connections and layer normalization; Transformer is a deep learning model based on an attention mechanism); finally, extract the hidden state of the last layer of the encoder as the visual feature sequence, calculated using the following formula:

[0060] ;

[0061] Among them, F v I represents the obtained visual feature sequence, and I represents the embedded sequence of the image after preprocessing. This indicates the length of the sequence after image segmentation (determined by the resolution of the input image). The feature embedding dimension is (768). These feature sequences will then be mapped to the cross-attention layer of the decoder as key-value pairs to guide text generation.

[0062] The CLIP-ViT model is pre-trained on large-scale internet image-text pairs. Through contrastive learning, it naturally aligns visual features with the semantic space of language, transforming raw remote sensing pixel information into semantically rich deep features. A hierarchical training strategy avoids overfitting on limited remote sensing data while allowing the model to adapt to the unique perspective and terrain features of remote sensing images. Considering the domain characteristics of remote sensing data and the limited training samples, this invention employs a hierarchical training strategy, freezing the low-level parameters of the CLIP model to preserve the general visual knowledge learned on large-scale data, and only fine-tuning the top-level Transformer module. This strategy effectively avoids the risk of overfitting on limited remote sensing data while enabling the model to adapt to the unique perspective and terrain features of remote sensing images.

[0063] Step 2, Enhanced External Knowledge Retrieval

[0064] like Figure 3 As shown, based on the visual feature sequence obtained in step 1, a cue prefix is ​​obtained through knowledge base retrieval and reordering. Considering the complex backgrounds and small targets in remote sensing images, this invention designs an ARG module to introduce non-parametric external knowledge, compensating for the limitations of parametric models in specific semantic cognition within the remote sensing domain. Through an efficient cross-modal retrieval mechanism, high-confidence and semantically complementary auxiliary text is obtained for the input image. Specifically:

[0065] Step 2.1, Knowledge Base Construction and Feature Alignment: The CLIP text encoder is used to extract feature vectors from all descriptive statements in the remote sensing text knowledge base. Considering the potential distribution differences between visual and textual features in the original CLIP semantic space, and to improve retrieval efficiency, Step 2.1 introduces Principal Component Analysis (PCA) as a linear projection layer to map textual features to a low-dimensional manifold space to achieve feature alignment and dimensionality reduction. The calculation formula is as follows:

[0066] ;

[0067] Where T pca It is the text feature matrix after dimensionality reduction by PCA, and T is the set of all descriptive statements in the knowledge base. , where represents the feature dimension after dimensionality reduction. This operation reduces retrieval complexity while preserving the main semantic information.

[0068] Step 2.2, Efficient Semantic Retrieval: In the retrieval stage, the visual feature sequence obtained in Step 1 is encoded and its dimensionality reduced to obtain the query vector. The Hierarchical Navigation Small World (HNSW) indexing algorithm is used to perform approximate nearest neighbor retrieval in the knowledge base, and a preliminary set of candidate descriptions is selected. The calculation formula is as follows:

[0069] ;

[0070] in, This represents the optimal set of k descriptions retrieved, where k represents the number of nearest neighbors to be retrieved.

[0071] Step 2.3, Diversity-Aware Reordering and Prefix Construction: Through a comprehensive scoring function Reorder the candidate set and perform final selection. The optimal description is concatenated into a cue prefix according to the [KNOWLEDGE] cap_1 || cap_2 || ... || cap_k [IMAGE] language model input format and used as an enhanced input for the decoder.

[0072] Among them, in the comprehensive scoring function S(c i ) represents the overall score of the i-th candidate set. For cosine similarity, For length-normalized fractions, For diversity scores based on word overlap, c i Here, q is the i-th candidate description, and q is the query vector obtained after encoding and dimensionality reduction of the visual feature sequence. These represent the weighting coefficients for each score. Specifically, Dynamically calculate the current candidate and the set of selected results. Jaccard distance:

[0073] .

[0074] To avoid search results that are not only semantically simplistic but also redundant, we designed a comprehensive scoring function. Reorder the candidate set. For the... The final score of each candidate is defined as a weighted sum of semantic similarity, length reward, and diversity score.

[0075] PCA dimensionality reduction reduces retrieval complexity while preserving key semantic information, addressing the issue of differences in the distribution of visual and textual features in the original semantic space. The HNSW algorithm achieves efficient near-nearest neighbor retrieval, quickly locating candidate texts related to visual features. Diversity-aware reordering avoids semantic monotony and redundancy in retrieval results by weighted summation of semantic similarity, length reward, and diversity score, ensuring the acquisition of highly confident and semantically complementary auxiliary texts.

[0076] Step 3, cross-modal generation decoder

[0077] like Figure 4As shown, based on the visual feature sequence obtained in step 1 and the cue prefix obtained in step 2, a preliminary descriptive text is generated by fusing the data through a GPT-2 (Generative Pre-trained Transformer Second Generation) decoder. The cross-modal generative decoder module is used to fuse visual semantics with external knowledge to generate accurate natural language descriptions. This module uses GPT-2 as its basic architecture and achieves deep interaction between visual and linguistic modalities through a carefully designed attention mechanism, ensuring that the generated description accurately reflects the semantic content of the remote sensing scene while maintaining fluency. Specifically:

[0078] Step 3.1, the cross-modal fusion mechanism is the core of the decoder design. At each time step, using GPT-2 as the decoder architecture, two key inputs are processed simultaneously: the visual feature sequence from the visual encoding module, i.e., the sequence obtained in step 1. and hint prefixes from the search enhancement module Visual features are injected into the generation process through a cross-attention mechanism, enabling the decoder to dynamically focus on relevant regions of the image. Specifically, for the decoder's... The formula for calculating cross-attention in layers is as follows:

[0079] ;

[0080] in , , , This indicates the hidden state of the parent layer. These are the query, key, and value projection matrices, respectively, F. v This is the visual feature sequence obtained in step 1, where Q, K, and V are the query, key, and value matrices, respectively, and d k It refers to the key and query dimensions in the attention mechanism.

[0081] Step 3.2, the attention entropy regularization mechanism, is a significant innovation for remote sensing image description tasks. Remote sensing scenes typically contain multiple salient objects, and traditional attention mechanisms are prone to "attention collapse," where the model overemphasizes a few salient regions while ignoring other important information, leading to incomplete descriptions or alignment errors. Therefore, this step introduces a regularization term based on entropy maximization: its calculation formula is as follows:

[0082] ;

[0083] ;

[0084] in, It is the attention entropy regularization loss. The balance coefficient is H, where H is the total number of attention heads and T is the number of time steps. Indicates the first The attention distribution entropy of an attention head at time step t, where L is the length of the attention distribution, and A is the attention distribution entropy. h,t It is the attention distribution of the h-th attention head at time step t. It is the attention weight of the h-th attention head on the l-th visual feature at time step t. It is a small constant used to prevent the logarithm from being negative infinity. This regularization term encourages the decoder to pay balanced attention to various semantic regions of the image during the generation process by maximizing the entropy of the attention distribution, thereby generating a more comprehensive and detailed description.

[0085] Step 3.3 employs a beam search algorithm during the inference phase. By balancing search width and generation quality, it ensures the fluency and accuracy of the output description. Simultaneously, techniques such as repetition penalties and length penalties are combined to effectively avoid duplicate segments and abnormally short sentences during the generation process. The final output probability distribution of the decoder is jointly determined by the visual context and textual cues. The formula for calculating the output probability distribution is as follows:

[0086] ;

[0087] in, The probability distribution F at time step t v It is the visual feature sequence obtained in step 1, P is the cue prefix obtained in step 2, and t is the current time step. These are the first t-1 words that have already been generated.

[0088] The cross-attention mechanism enables the decoder to adaptively focus on relevant regions of the image when generating each word, achieving deep interaction between visual and linguistic modalities; attention entropy regularization suppresses the "attention collapse" phenomenon and encourages the model to pay balanced attention to various semantic regions of the image; bundle search and penalty techniques balance the fluency and accuracy of generation, avoiding the problems of repeated fragments and abnormal short sentences.

[0089] Step 4, Multi-task joint optimization

[0090] Based on the initial description of the text generation process in step 3, the final accurate and comprehensive description text is obtained through end-to-end training using a joint loss function. To optimize syntactic structure while ensuring generation quality, the model employs a multi-task joint optimization module for end-to-end training. This module integrates three complementary optimization objectives to jointly guide the model's synergistic improvement in vocabulary prediction, generation quality, and attention distribution. Details are as follows:

[0091] Construct a loss function that includes the focus loss function, the length normalized loss function, and the attention entropy regularization term from step 3.2. A joint optimization framework using three loss functions enables end-to-end collaborative training of the model, optimizing it at the lexical, sentence, and attention levels. This ultimately yields accurate and comprehensive descriptive text, effectively enhancing the model's descriptive capabilities in complex remote sensing scenarios and generating accurate and comprehensive natural language descriptions. Specifically:

[0092] Focus Loss Function: This addresses the long-tailed vocabulary distribution problem commonly found in remote sensing descriptive data. Traditional cross-entropy loss, when dealing with class imbalance, is prone to being dominated by high-frequency words and neglecting rare feature concepts. Therefore, we introduce the classic focus loss mechanism from object detection, which enhances the model's ability to learn sparse semantics by adjusting the weights of difficult samples. The calculation formula is as follows:

[0093] ;

[0094] in, It is a focal loss. This represents the model's predicted probability for the target word. For focusing parameters.

[0095] Length-normalized loss function: Generally, when training data is limited, models tend to generate short sentences that are safe but lack sufficient information. Therefore, this invention designs a penalty mechanism based on the target length, calculated as follows:

[0096] ;

[0097] in, For length normalization loss, These are the weighting coefficients. Indicates the effective length of the generated sequence. The preset target length.

[0098] Total loss function: Combining the focus loss function, the length normalized loss function, and the attention entropy regularization term from step 3.2. (By maximizing the entropy of the attention distribution to prevent the "attention collapse" phenomenon), this constitutes the complete multi-task optimization framework, namely the following total loss function:

[0099] ;

[0100] in, For the total loss, It is a focal loss. For length normalization loss, It is attention entropy regularization loss

[0101] Focus loss addresses the problem of high-frequency words dominating training due to the distribution of long-tail words in remote sensing description data by adjusting the weights of difficult samples, thus strengthening the model's learning of rare land cover concepts. The length normalization loss function alleviates the problem of insufficient information in short sentences generated by the model through mean square error constraints. The attention entropy regularization term ensures the quality of cross-modal alignment. The three functions work together to constrain the model from three dimensions: vocabulary level, sentence level, and attention level, thereby improving the accuracy and comprehensiveness of descriptions in complex remote sensing scenarios.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing image description generation method based on external knowledge retrieval enhancement, characterized in that, The steps include the following: Step 1: By comparing the language and image pre-trained models, the input remote sensing image is processed to obtain a visual feature sequence aligned with the language semantic space; Step 2: Based on the visual feature sequence obtained in Step 1, obtain the prompt prefix through knowledge base retrieval and reordering; Step 3: Based on the visual feature sequence obtained in Step 1 and the cue prefix obtained in Step 2, the preliminary descriptive text is generated by fusing the two-generation decoder through a generative pre-trained transformer. Step 4: Based on the preliminary description of the text generation process in Step 3, the final accurate and comprehensive description text is obtained through end-to-end training of the joint loss function.

2. The remote sensing image description generation method based on external knowledge retrieval enhancement according to claim 1, characterized in that, Step 1 is as follows: Step 1.1: Process the input image using a contrastive language and image pre-trained image processor. Perform resizing and normalization preprocessing operations to ensure it fits the pre-trained model; Step 1.2: The processed image is divided into fixed-size image blocks of 32×32 pixels, converted into serialized block embeddings by linear projection, and a learnable positional code is added to preserve the spatial structure information of the image. Step 1.3: Input the embedded sequence obtained in Step 1.2 into the contrastive language and image pre-trained visual encoder; finally, extract the last hidden state of the encoder as the visual feature sequence, calculated using the following formula: ； Among them, F v I represents the obtained visual feature sequence, and I represents the embedded sequence of the image after preprocessing. This indicates the sequence length after image segmentation. The feature embedding dimension.

3. The remote sensing image description generation method based on external knowledge retrieval enhancement according to claim 2, characterized in that, The contrastive language and image pre-trained visual encoder consists of a multi-layer deep learning model encoder based on an attention mechanism. Each layer contains a multi-head self-attention mechanism and a feedforward network, and training stability is ensured through residual connections and layer normalization.

4. The remote sensing image description generation method based on external knowledge retrieval enhancement according to claim 1, characterized in that, Step 2 is as follows: Step 2.1, Knowledge Base Construction and Feature Alignment: The CLIP text encoder is used to extract feature vectors from all descriptive statements in the remote sensing text knowledge base. Principal component analysis is then used to map these vectors to a low-dimensional manifold space, achieving feature alignment and dimensionality reduction. The calculation formula is as follows: ； Where T pca It is the text feature matrix after dimensionality reduction by PCA, and T is the set of all descriptive statements in the knowledge base. , representing the feature dimension after dimensionality reduction; Step 2.2, Efficient Semantic Retrieval: Encode the visual feature sequence obtained in Step 1 and reduce its dimensionality to obtain the query vector. A hierarchical navigation small-world indexing algorithm is used to perform approximate nearest neighbor retrieval in the knowledge base, and a preliminary set of candidate descriptions is selected. The calculation formula is as follows: ； in, This represents the optimal set of k descriptions retrieved, where k represents the number of nearest neighbors to be retrieved. Step 2.3, Diversity-Aware Reordering and Prefix Construction: Through a comprehensive scoring function Reorder the candidate set and perform final selection. The optimal description is concatenated into a prompt prefix according to the language model input format [KNOWLEDGE] cap_1 || cap_2 || ... || cap_k [IMAGE]; Among them, S(c) in the comprehensive scoring function i ) represents the overall score of the i-th candidate set. For cosine similarity, For length-normalized fractions, For diversity scores based on word overlap, c i Here, q is the i-th candidate description, and q is the query vector obtained after encoding and dimensionality reduction of the visual feature sequence. These represent the weighting coefficients for each score.

5. The remote sensing image description generation method based on external knowledge retrieval enhancement according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Using the second generation of generative pre-trained transformers as the decoder architecture, the visual feature sequence obtained in step 1 is received simultaneously at each time step. The prompt prefix obtained in step 2 Cross-modal fusion is achieved through a cross-attention mechanism, which injects visual features into the generation process. The cross-attention calculation formula is as follows: ； in , , , This indicates the hidden state of the parent layer. These are the query, key, and value projection matrices, respectively, F. v This is the visual feature sequence obtained in step 1, where Q, K, and V are the query, key, and value matrices, respectively, and d k These are the key and query dimensions in the attention mechanism; Step 3.2 introduces an attention entropy regularization mechanism to suppress attention collapse, the calculation formula of which is as follows: ； ； in, It is the attention entropy regularization loss. The balance coefficient is H, where H is the total number of attention heads and T is the number of time steps. Indicates the first The attention distribution entropy of an attention head at time step t, where L is the length of the attention distribution, and A is the attention distribution entropy. h,t It is the attention distribution of the h-th attention head at time step t. It is the attention weight of the h-th attention head on the l-th visual feature at time step t. It is a small constant used to prevent the logarithm from being negative infinity; Step 3.3, the inference stage uses a cluster search algorithm, combined with repetition penalty and length penalty techniques, to generate preliminary descriptive text. The output probability distribution calculation formula is as follows: ； in, The probability distribution F at time step t v It is the visual feature sequence obtained in step 1, P is the cue prefix obtained in step 2, and t is the current time step. These are the first t-1 words that have already been generated.

6. The remote sensing image description generation method based on external knowledge retrieval enhancement according to claim 5, characterized in that, Step 4 is as follows: Construct a loss function that includes the focus loss function, the length normalized loss function, and the loss function obtained in step 3.

2. A joint optimization framework using three loss functions is employed to perform end-to-end collaborative training of the model, optimizing it at the lexical, sentence, and attention levels to ultimately obtain accurate and comprehensive descriptive text; among which: The focus loss function is used to address the long-tail vocabulary distribution problem, and its calculation formula is as follows: ； in, It is a focal loss. This represents the model's predicted probability for the target word. For focusing parameters; The length-normalized loss function is used to alleviate the problem of generating short sentences, and its calculation formula is as follows: ； in, For length normalization loss, These are the weighting coefficients. Indicates the effective length of the generated sequence. The preset target length; The results obtained through the focus loss function, the length normalized loss function, and step 3.2 are as follows: The total loss function is as follows: ； in, For the total loss, It is a focal loss. For length normalization loss, It is the attention entropy regularization loss.

Images