Scene-guided zero-shot remote sensing image text description generation method and system

Abstract

The application discloses a scene guidance-based zero-sample remote sensing image text description generation method and system, and the method comprises the following steps: performing end-to-end training on a text description generation network; extracting text features of a text description sentence; implementing Gaussian random disturbance enhancement on the text features, and then expanding the text features into serialized semantic representations through a prefix mapping network; generating scene prior embedding for the text features; fusing the serialized semantic representations and the scene prior embedding, and then generating a spliced embedding sequence from the original text description sentence; inputting the spliced embedding sequence into a language model decoder to generate a text description sentence conforming to a remote sensing image scene, and updating network parameters of the text description generation network by using an autoregressive language modeling loss optimization. The application aims to improve the semantic alignment stability of the model under the condition of zero remote sensing image and text labeling, and realizes high-precision, high-robustness and high-scene matching degree zero-sample remote sensing image text description generation.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image recognition and understanding technology, and specifically to a scene-guided zero-sample remote sensing image text description generation method and system. Background Technology

[0002] With the continuous upgrading of aerospace technology and Earth observation sensors, massive amounts of remote sensing image data can now be acquired efficiently. Remote sensing image captioning (RSIC) technology can transform the land cover categories, spatial structures, scene layouts, and semantic content of remote sensing images into natural language descriptions, possessing high application value in fields such as land resource surveys, urban planning, disaster emergency monitoring, military reconnaissance, and intelligent remote sensing interpretation. In previous research, remote sensing image captioning methods have mostly focused on entity semantic modeling and global-local visual feature modeling, achieving certain performance improvements under supervised learning paradigms. However, these methods heavily rely on large-scale, high-quality, professionally labeled remote sensing image-text pairing data. Remote sensing image interpretation is highly specialized, with high labeling costs, long cycles, and significant challenges, severely limiting the deployment and promotion of models in large-scale real-world scenarios. To overcome the dependence on labeled image-text pairs, zero-shot image captioning has become a research hotspot. This type of method relies on pre-trained visual-language models such as CLIP to construct a unified cross-modal semantic space. During the training phase, it uses only text data to learn and generate mappings, and during the inference phase, it directly projects the images onto the same space to complete the description generation. Here, zero-shot means that no additional remote sensing image-text pairing and annotation data is needed for fine-tuning in the downstream remote sensing image description task. However, existing zero-shot image description methods have significant drawbacks when directly transferred to remote sensing scenes. Firstly, the modal alignment granularity is insufficient. Visual features and textual semantics naturally differ in modality, and relying solely on coarse-grained mapping of the global embedding space cannot effectively capture the complex spatial details, dense feature distribution, and strong contextual relationships of remote sensing images. Secondly, general entity retrieval strategies are difficult to adapt to the remote sensing domain. Mainstream CLIP-based retrieval methods are affected by the small scale of remote sensing targets, dense feature distribution, and high background coupling, resulting in a significant decrease in entity discrimination accuracy and the introduction of semantic noise, further leading to problems such as descriptive illusion, semantic shift, and content distortion. Thirdly, there is a lack of high-level scene semantic constraints. Existing methods often perform direct visual-text alignment without scene limitations, ignoring the global scene attributes and overall semantic logic of remote sensing images, easily causing fragmented descriptive semantics, scene matching mismatches, and chaotic expression logic. Therefore, how to achieve stable, accurate, and remote sensing-adaptive zero-shot image description generation without relying on remote sensing image-text pairing annotation data has become a key technical problem urgently needing to be solved in the field of remote sensing image interpretation. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method and system for generating zero-shot remote sensing image text descriptions based on scene guidance, which addresses the above-mentioned problems in the prior art. This invention aims to improve the semantic alignment stability of the model under zero-sense remote sensing image annotation conditions and achieve high-precision, high-robustness, and high-scene-matching zero-shot remote sensing image text description generation.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A scene-guided zero-shot remote sensing image text description generation method includes end-to-end training of a text description generation network consisting of a visual-language model, a prefix mapping network, a word embedding layer, and a language model: S101, extracting text features of text description sentences based on the pre-trained visual-language model; S102, applying Gaussian random perturbation to the text features for enhancement, and then expanding them into a serialized semantic representation through the prefix mapping network; S103, constructing a scene semantic prior, and obtaining the optimal scene category by classifying the text features into a text scene through text template matching; S104, based on the optimal scene... Scene category constructs scene template text, and generates scene prior embeddings through word embedding layer; S105, the serialized semantic representation and scene prior embeddings are concatenated and fused in sequence dimension to obtain fused semantic input sequence; S106, the fused semantic input sequence and the text embedding sequence obtained by embedding the original text description statement are concatenated into a concatenated embedding sequence, and the concatenated embedding sequence is input into language model decoder to obtain text description statements that conform to remote sensing image scene, and the network parameters of prefix mapping network and language model decoder in text description generation network are optimized and updated using autoregressive language modeling loss.

[0005] Optionally, the function expression for applying Gaussian random perturbation enhancement to the text features in step S102 is as follows: ; in, To implement Gaussian random perturbation-enhanced text features, For normalization operations, These are the original text features. It is zero-mean Gaussian noise. , The Gaussian distribution is for zero-mean Gaussian noise. For the disturbance intensity, express An identity matrix of dimension 1.

[0006] Optionally, step S103 includes: S201, for a preset set of scene categories For each scene category, the scene category name is combined with a preset description template to construct template text, and the template text is encoded into a scene category embedding through a text encoder of a visual-language model; S202, calculate text features and a preset set of scene categories respectively. The cosine similarity of the scene category embeddings corresponding to each scene category in the dataset; S203, classify scenes based on the cosine similarity between text features and the scene category embeddings of each scene category; ; in, for Textual features of time Belongs to the scene category The probability distribution, Scene category Scene embedding, A collection of scene categories The size of the set, Scene category Scene embedding, This is the cosine similarity calculation function. For temperature parameters; S204: The scene category with the highest probability distribution is taken as the optimal scene category obtained from text scene classification.

[0007] Optionally, the functional expression for the autoregressive language modeling loss in step S106 is: ; in, For autoregressive language modeling loss, The number of training samples. For actual text description statements Length, Generate predicted text description statements for the large language model decoder. each word element The predicted probability, To predict the first text description statement each word element All previous morphemes, This is for splicing embedded sequences.

[0008] Optionally, it also includes testing or inference on the trained text description generation network: S301, Extracting image features from input remote sensing images based on a pre-trained vision-language model; S302, the image features are extended into a serialized semantic representation through a trained prefix mapping network; S303 achieves optimal scene category by fusing text similarity and visual prototype modeling to classify image scenes. S304, construct scene template text based on the optimal scene category, and generate scene prior embedding through word embedding layer; S305, the serialized semantic representation and the scene prior are embedded and spliced ​​together in the sequence dimension to obtain the fused semantic input sequence; S306 integrates the semantic input sequence into the trained language model decoder, generates text in an autoregressive manner, and completes decoding using a bundle search strategy, outputting text descriptions that conform to the characteristics of the remote sensing scene.

[0009] Optionally, step S303 includes: S401, calculate image features and a preset set of scene categories respectively. The cosine similarity of the scene category embeddings corresponding to each scene category is used as semantic similarity: ; in, Image features With scene category semantic similarity, For cosine similarity calculation function, scene category For a set of preset scene categories The first in There are several scene categories; the scene category embedding involves combining the scene category with a preset text description to construct template text, and then obtaining the template text through a CLIP text encoder. S402, for each scene category Each scene category is selected from image features based on its cosine similarity, which is greater than a preset threshold. The samples are used as a subset of the samples. ; S403, for sample subsets The image features within the image are aggregated by mean to obtain the initial visual prototypes for each scene category: ; in, Scene category The initial visual prototype, For sample subset Size, For sample subset The samples in; S404, normalizes the initial visual prototypes of each scene category to obtain normalized visual prototype representations: ; in, Scene category Normalized visual prototype representation, This is the normalization function.

[0010] S405, calculate the cosine similarity between the image features and the normalized visual prototype representation of each scene category as the image features. With scene category Prototype similarity: ; in, Image features With scene category Prototype similarity, This is the function for calculating cosine similarity. S406, Image Features Image features are obtained by fusing semantic similarity and prototype similarity with various scene categories. Scene discrimination scores for each scene category: ; in, Image features With scene category Scene discrimination score, These are the weighting coefficients; S407: The scene category with the highest scene discrimination score is taken as the optimal scene category obtained from image scene classification.

[0011] Optionally, a preset threshold is used in step S402. The expression for the computation function is: ; in, This is a preset ratio coefficient used to control the strictness of sample screening; For the image features of the th Sample Scene categories Cosine similarity between them.

[0012] Furthermore, the present invention also provides a scene-guided zero-shot remote sensing image text description generation system, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the scene-guided zero-shot remote sensing image text description generation method.

[0013] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the scene-guided zero-sample remote sensing image text description generation method by a processor.

[0014] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the scene-guided zero-sample remote sensing image text description generation method via a processor.

[0015] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: 1. The present invention can construct a shared visual-language semantic space through a pre-trained visual-language model to realize cross-modal unified semantic representation learning. Combined with Gaussian perturbation and prefix mapping mechanisms, it effectively alleviates the modal distribution difference between training text features and test image features under zero-shot conditions, enhances the robustness of the model to cross-modal distribution shifts, and achieves more stable and generalized semantic alignment; 2. The scene semantic prior guidance in the method of the present invention uses global scene semantics as a high-level constraint to replace the traditional unreliable entity retrieval strategy, suppressing semantic noise interference caused by false entity detection in remote sensing scenes from the root, solving the problems of semantic fragmentation, logical confusion, and inconsistency with actual scenes in zero-shot descriptions, and significantly improving the scene consistency and semantic accuracy of description statements; 3. The visual prototype optimization mechanism proposed in the present invention adaptively constructs visual prototypes by clustering image features of the same category, and completes scene classification by fusing text semantic similarity and prototype similarity, effectively correcting the modal shift between images and text, greatly improving the accuracy of remote sensing scene discrimination, and providing more reliable and stable high-level semantic anchors for description generation.

[0016] By combining the above-mentioned technical means, this invention can improve the semantic alignment stability of the model under zero remote sensing image annotation conditions, and achieve high-precision, high-robustness, and high-scene-matching zero-sample remote sensing image description generation. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the end-to-end training process of the text description generation network in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the principle of end-to-end training of the text description generation network in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the testing or inference process of the text description generation network in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram illustrating the principle of text description generation network testing or inference in an embodiment of the present invention.

[0021] Figure 5 This is a schematic diagram comparing the representation spaces of the visual-language model CLIP and the visual prototype in an embodiment of the present invention, where (a) is the representation space of the visual-language model CLIP and (b) is the representation space of the visual prototype. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0023] like Figure 1 and Figure 2 As shown, this embodiment of the scene-guided zero-shot remote sensing image text description generation method includes end-to-end training of a text description generation network composed of a visual-language model, a prefix mapping network, a word embedding layer, and a language model: S101, extracting text features of text description sentences based on the pre-trained visual-language model; S102, applying Gaussian random perturbation to the text features, and then expanding them into a serialized semantic representation through the prefix mapping network; S103, constructing scene semantic priors, and obtaining the optimal scene category by classifying the text features into text scenes through text template matching; S104, according to... The optimal scene category is used to construct scene template text, and the scene template text is used to generate scene prior embedding through word embedding layer; S105, the serialized semantic representation and scene prior embedding are concatenated and fused in the sequence dimension to obtain fused semantic input sequence; S106, the fused semantic input sequence and the text embedding sequence obtained by embedding the original text description statement are concatenated into a concatenated embedding sequence, and the concatenated embedding sequence is input into the language model decoder to obtain text description statements that conform to the remote sensing image scene, and the network parameters of the prefix mapping network and the language model decoder in the text description generation network are optimized and updated using autoregressive language modeling loss.

[0024] It should be noted that the pre-trained visual-language model mainly consists of a visual encoder and a text encoder. A unified cross-modal semantic space can be constructed using the visual encoder and text encoder to extract visual features from the input remote sensing image and text features from the text corpus. The pre-trained visual-language model can be selected according to needs; for example, this embodiment uses the pre-trained visual-language model CLIP. Introducing the pre-trained visual-language model CLIP, by constructing a unified cross-modal semantic space and extracting features from both images and text, can achieve preliminary alignment of global semantics, thus providing a pre-aligned cross-modal semantic representation. Image features are only used in the testing and inference phases. The input remote sensing image... Global visual features are obtained by inputting the visual encoder of the pre-trained vision-language model CLIP. and text description statements The text semantic features are obtained by inputting the pre-trained visual-language model CLIP into its text encoder. It can be represented as: ; ; in, The visual encoder for the pre-trained vision-language model CLIP. A text encoder for the pre-trained visual-language model CLIP. The output feature dimension of the pre-trained visual-language model CLIP is given. The visual encoder and text encoder of the visual-language model CLIP have the same output feature dimension to achieve the initial alignment of the visual features of the remote sensing image with the text features of the text corpus.

[0025] Although the pre-trained visual-language model CLIP achieves preliminary alignment between visual and text at the global semantic level, this alignment is mainly reflected in coarse-grained matching of semantic relevance, making it difficult to directly support the fine-grained semantic modeling and generation tasks required for image description. Especially under the setting of training solely based on text, the pre-trained visual-language model CLIP cannot explicitly observe the distribution of visual features, resulting in a distribution shift between the text representation learned during training and the image representation input during testing. To alleviate the aforementioned training-inference modality discrepancy problem, this embodiment introduces a random perturbation mechanism in the text feature space. Considering that the features of the pre-trained visual-language model CLIP are usually located in a normalized semantic space, this embodiment first normalizes the text features and injects Gaussian random perturbations into the unit hyperspherical neighborhood to expand its local semantic neighborhood range, thereby enhancing the model's robustness to potential cross-modal distribution shifts. Specifically, the function expression for implementing Gaussian random perturbation enhancement of text features in step S102 of this embodiment is: ; in, To implement Gaussian random perturbation-enhanced text features, For normalization operations, These are the original text features. It is zero-mean Gaussian noise. , The Gaussian distribution is for zero-mean Gaussian noise. For the disturbance intensity, express A 3D identity matrix. This design ensures that the perturbed features are still distributed on the unit hypersphere, thus maintaining consistency with the cosine similarity metric used by the pre-trained visual-language model CLIP.

[0026] This embodiment introduces a prefix mapping network to serialize and reconstruct the global semantic representation extracted by the pre-trained visual-language model CLIP. The necessity of this improvement lies mainly in two aspects: First, the output of the pre-trained visual-language model CLIP is a single global vector, which is a highly compressed semantic representation lacking structured semantic information. Meanwhile, the autoregressive language model decoder relies on serialized input to complete contextual modeling. The two representations are inherently mismatched; therefore, the global features need to be expanded into semantic word sequences to adapt to the input mechanism of the language model decoder. Second, in zero-shot application scenarios, model training relies on text features, while the inference stage uses image features as input. There are significant distributional differences between cross-modal features. Introducing a parameterized mapping function to complete feature reconstruction can effectively enhance the model's adaptability to cross-modal distribution shifts. The text features are then enhanced with Gaussian random perturbation. Extended to a fixed length via a prefix mapping network Serialization semantic representation The input adapted to the language model decoder can be represented as: ; in For prefix mapping networks, This represents the input feature dimension of the language model decoder. In this embodiment, the prefix mapping network, for example, can be the prefix mapping network used in the Clipcap model. The prefix mapping network converts the global vector into a serialized representation to adapt to the input structure of the language model decoder.

[0027] In this embodiment, step S103 includes constructing a scene semantic prior as a high-level constraint; achieving scene classification based on text templates and semantic matching during the training phase; proposing a scene classification method combining text semantic matching and visual prototype fusion during the testing phase to alleviate modality shift and improve scene discrimination accuracy; and converting the optimal scene category into a scene embedding to provide a stable semantic anchor point for generation. Specifically, step S103 in this embodiment includes: S201, for a preset set of scene categories For each scene category, the scene category name is combined with a preset description template to construct template text, and the template text is encoded into a scene category embedding through a text encoder of a visual-language model; set up A collection of scene categories Combined with preset description templates (e.g., "This is a remote sensing image about {}"), template text is constructed for each category. Scene embedding is obtained through CLIP text encoder. : ; in, A text encoder for the pre-trained visual-language model CLIP; S202, calculate text features and a preset set of scene categories respectively. The cosine similarity of the scene category embeddings corresponding to each scene category in the dataset; S203, due to the input text and category template text Both belong to the text modality, and corresponding features are extracted through the CLIP text encoder. Therefore, scene classification can be performed directly by calculating the cosine similarity between the two features, that is: scene classification is performed based on the cosine similarity between the text features and the scene category embedding of each scene category. ; in, for Textual features of time Belongs to the scene category The probability distribution, Scene category Scene embedding, A collection of scene categories The size of the set, Scene category Scene embedding, This is the cosine similarity calculation function. For temperature parameters; S204, select the scene category with the highest probability distribution as the optimal scene category obtained from text scene classification. .

[0028] In step S104, according to the optimal scene category Build scene template text , scene template text Generate scene prior embeddings through word embedding layer , can be represented as: ; in, For the word embedding layer, the desired word embedding layer can be selected as needed, such as the word embedding layer of the GPT-2 language model. In step S105, the serialized semantic representation and the scene prior embedding are concatenated and fused along the sequence dimension to obtain the fused semantic input sequence. , can be represented as: ; Acquired fusion semantic input sequence It can be used as input to the language model decoder to generate image description text. To improve training stability and convergence speed, a teacher-forced strategy is adopted during the training phase. Specifically, in step S106, the semantic input sequence and the text embedding sequence obtained by embedding the original text description sentences are fused. The concatenation of embedded sequences can be represented as: ; The above approach allows the text description generation network to predict the output at all time steps in parallel during training, effectively mitigating the error accumulation problem. Finally, the concatenated embedded sequence is input to the language model decoder for autoregressive language modeling, and the network parameters of the text description generation network are optimized and updated using autoregressive language modeling loss. The language model decoder can be any type required, such as the GPT-2 language model decoder.

[0029] The functional expression for the autoregressive language modeling loss in step S106 of this embodiment is: ; in, For autoregressive language modeling loss, The number of training samples. For actual text description statements Length, Generate predicted text description statements for the large language model decoder. each word element The predicted probability, To predict the first text description statement each word element All previous morphemes, To concatenate the embedded sequences, during the training phase, the text description generation network relies solely on remote sensing text corpora for parameter optimization, without introducing any image input, thus avoiding dependence on paired image-text annotation data. To preserve the cross-modal semantic alignment capability of the pre-trained model, the CLIP parameters of the vision-language model are frozen, and only the parameters in the prefix mapping network and GPT-2 decoder are learnable.

[0030] like Figure 3 and Figure 4 As shown, it also includes testing or inference on the trained text description generation network: S301, Extracting image features from input remote sensing images based on a pre-trained vision-language model; S302, the image features are extended into a serialized semantic representation through a trained prefix mapping network; S303 achieves optimal scene category by fusing text similarity and visual prototype modeling to classify image scenes. S304, construct scene template text based on the optimal scene category, and generate scene prior embedding through word embedding layer; S305, the serialized semantic representation and the scene prior are embedded and spliced ​​together in the sequence dimension to obtain the fused semantic input sequence; S306 integrates the semantic input sequence into the trained language model decoder, generates text in an autoregressive manner, and completes decoding using a bundle search strategy, outputting text descriptions that conform to the characteristics of the remote sensing scene.

[0031] During the testing phase, as the input changes from text to image features, the performance will significantly decrease due to modal differences. If the scene classification strategy based on text matching from the training phase is directly applied, the performance will be significantly reduced. This is because, although image features and text features have some alignment capability in the CLIP semantic space, their distribution patterns still differ. This leads to unstable scene discrimination results when relying solely on text embeddings for matching, and even cross-class misclassification. Therefore, this embodiment proposes a scene classification method that integrates text semantic matching and visual prototype modeling to improve robustness and discrimination ability during the testing phase. Specifically, step S303 in this embodiment includes: S401, for a preset set of scene categories Scene category embedding for each scene category Calculate image features and a preset set of scene categories respectively. The cosine similarity of the scene category embeddings corresponding to each scene category is used as semantic similarity: ; in, Image features With scene category semantic similarity, For cosine similarity calculation function, scene category For a set of preset scene categories The first in There are several scene categories; the scene category embedding involves combining the scene category with a preset text description to construct template text, and then obtaining the template text through a CLIP text encoder. S402, for each scene category Each scene category is selected from image features based on its cosine similarity, which is greater than a preset threshold. The samples are used as a subset of the samples. ; S403, for sample subsets The image features within the image are aggregated by mean to obtain the initial visual prototypes for each scene category: ; in, Scene category The initial visual prototype, For sample subset Size, For sample subset The samples in; S404, normalizes the initial visual prototypes of each scene category to obtain normalized visual prototype representations: ; in, Scene category Normalized visual prototype representation, This is the normalization function.

[0032] S405, calculate the cosine similarity between the image features and the normalized visual prototype representation of each scene category as the image features. With scene category Prototype similarity: ; in, Image features With scene category Prototype similarity, This is the function for calculating cosine similarity. S406, Image Features Image features are obtained by fusing semantic similarity and prototype similarity with various scene categories. Scene discrimination scores for each scene category: ; in, Image features With scene category Scene discrimination score, These are the weighting coefficients; S407: The scene category with the highest scene discrimination score is taken as the optimal scene category obtained from image scene classification.

[0033] The input image features are obtained in step S401. Based on the similarity between the embeddings of each category, in order to alleviate the problem of cross-modal distribution shift, steps S402 to S405 further introduce a dynamic prototype modeling mechanism based on image features.

[0034] Figure 5 This is a schematic diagram comparing the representation spaces of the visual-language model CLIP and the visual prototype in an embodiment of the present invention, where (a) is the representation space of the visual-language model CLIP and (b) is the representation space of the visual prototype. Figure 5As shown in (a), in the representation space of the visual-language model CLIP, there are misclassified image features, such as... Figure 5 As shown in (b), after transformation to the representation space of the visual prototype, visual prototype modeling constructs a visual prototype representation that better fits the image distribution by aggregating image features of the same category. This effectively corrects the cross-modal bias when relying solely on text embedding, and misclassified image features can achieve accurate scene classification. Based on the above prototype representation, the similarity between image features and visual prototypes can be further calculated. Finally, the text semantic matching score and prototype similarity are fused to obtain the final scene discrimination score. By introducing a dynamic prototype modeling mechanism, this embodiment adaptively adjusts the category representation using the distribution information of the image features themselves during the testing phase. This effectively alleviates the distribution offset problem between images and text while maintaining cross-modal semantic consistency, significantly improving the stability and accuracy of scene classification.

[0035] In step S402 of this embodiment, a preset threshold is used. The expression for the computation function is: ; in, This is a preset ratio coefficient used to control the strictness of sample screening; For the image features of the th Sample Scene categories Cosine similarity between them.

[0036] In step S303, the optimal scene category is obtained by fusing text similarity and visual prototype modeling to classify the image scene. Step S304: Based on the optimal scene category Construct scene template text and generate scene prior embeddings through word embedding layers. Step S305 serializes the semantic representation. With scene prior embedding The fused semantic input sequence is obtained by concatenating and fusing the sequences along the sequence dimension. , can be represented as: ; Acquired fusion semantic input sequence It can be used as input to a language model decoder to generate image description text.

[0037] During the inference phase, due to the lack of textual supervision, the model employs a standard autoregressive generative strategy. Specifically, given an input remote sensing image I, its cross-modal feature representation is first extracted using the CLIP visual encoder. Then, on the one hand, a pre-trained prefix mapping network is used to obtain the serialized semantic representation. On the other hand, a combination of text-visual similarity matching and visual prototype modeling is used to perform scene discrimination on images and obtain high-level semantic prior information. Furthermore, scene text templates are constructed using the predicted scene category names, and corresponding scene embedding representations are obtained based on GPT-2. Z and After fusion, the final conditional input representation is formed, namely: the fused semantic input sequence. .

[0038] Step S306 involves inputting the fused semantic input sequence into the GPT-2 language model decoder. Through autoregressive generation and bundle search strategies, guided by global cross-modal representations and explicit scene priors, autoregressive generation of remote sensing image descriptions is completed, outputting textual descriptions that conform to the characteristics of the remote sensing scene, until an end marker is generated or the maximum length is reached. It should be noted that autoregressive generation and bundle search strategies are well-known methods for language model string generation; therefore, their implementation details will not be elaborated here.

[0039] To verify the effectiveness and advancement of the scene-guided zero-shot remote sensing image text description generation method in this embodiment, experiments were conducted on the mainstream benchmark dataset RSICD in the field of remote sensing image description. Representative state-of-the-art methods were selected for comparison, including Qwen2-VL, GPT-4V, Echo, and VieCap. The experiments employed standard evaluation metrics commonly used in the field of image description to measure the quality of the results generated by different methods from multiple dimensions, specifically including BLEU-N (N=1,2,3,4), METEOR, ROUGE-L, and CIDEr-D. Among them, BLEU-N is an n-gram-based accuracy metric that measures the lexical overlap and word order fluency between the generated text and the reference description from single characters to four-character phrases, including four specific metrics from BLEU-1 to BLEU-4; METEOR introduces synonyms and semantic associations on the basis of lexical matching, which is more in line with human semantic understanding; ROUGE-L focuses on recall, paying attention to the degree of coverage of the reference description by the generated text and the similarity of sentence structure; CIDEr-D is designed for image description scenarios, and uses TF-IDF weighted n-gram matching to more accurately evaluate the overall semantic relevance of the sentence, which is one of the core evaluation metrics in this field. The experimental results are shown in Table 1.

[0040] Table 1: Comparison of experimental results of the method in this embodiment with other advanced algorithms on the RSICD dataset.

[0041] As shown in Table 1, the method in this embodiment achieves the best performance on the RSICD dataset compared to other advanced algorithms. All evaluation metrics are superior to the comparative methods, demonstrating the effectiveness and advancement of the proposed method. Specifically, the SG-ZSCap method proposed in this embodiment achieves significant improvements in key metrics, with the BLEU-4 metric reaching 35.90 and the CIDEr-D metric reaching 201.50, representing improvements of 3.60 and 19.60 respectively compared to the second-best performing VieCap method. This fully demonstrates the advantages of this method in semantic consistency, fine-grained expression, and overall generation quality.

[0042] Further analysis reveals that the cross-modal unified semantic representation learning method in this embodiment constructs a shared visual-linguistic semantic space through a pre-trained CLIP model. Combined with Gaussian perturbation and prefix mapping mechanisms, it effectively alleviates the modal distribution differences between training text features and test image features under zero-shot conditions, enhances the model's robustness to cross-modal distribution shifts, and achieves more stable and generalized semantic alignment. Scene semantic prior guidance uses global scene semantics as a high-level constraint, replacing the unreliable entity retrieval strategy in traditional methods. This fundamentally suppresses semantic noise interference caused by false entity detection in remote sensing scenarios, solves the problems of semantic fragmentation and inconsistency with actual scenarios in zero-shot descriptions, and significantly improves the scene consistency and semantic accuracy of descriptive statements. The dynamic visual prototype optimization mechanism adaptively constructs visual prototypes by clustering image features of the same category, and completes scene classification by fusing text semantic similarity and prototype similarity. This effectively corrects the modal shift between images and text, significantly improves the accuracy of remote sensing scene discrimination, and provides a more reliable and stable high-level semantic anchor for description generation. The overall architecture of the method in this embodiment is designed for the characteristics of remote sensing images, such as dense semantics, high coupling of ground objects, and strong scene context association. All components work together to achieve cross-modal semantic alignment, scene-level constraint guidance, and adaptive feature fusion. Without relying on remote sensing image-text pairing and annotation data, it achieves high-precision, high-fluency, and high-robust zero-shot remote sensing image description generation. The overall model has strong generalization and wide applicability.

[0043] Furthermore, this embodiment also provides a scene-guided zero-shot remote sensing image text description generation system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the scene-guided zero-shot remote sensing image text description generation method. This embodiment also provides a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the scene-guided zero-shot remote sensing image text description generation method via a processor. This embodiment also provides a computer program product, including a computer program or instructions programmed or configured to execute the scene-guided zero-shot remote sensing image text description generation method via a processor.

[0044] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0045] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A scene-guided zero-shot remote sensing image text description generation method, characterized in that, The process includes end-to-end training of a text description generation network consisting of a visual-language model, a prefix mapping network, a word embedding layer, and a language model decoder: S101, extracting text features from the text description sentences based on the pre-trained visual-language model; S102, applying Gaussian random perturbation to the text features, then expanding them into a serialized semantic representation through the prefix mapping network; S103, constructing a scene semantic prior, and classifying the text features into the optimal scene category through text template matching; S104, constructing scene template text based on the optimal scene category, and generating scene prior embeddings from the scene template text through the word embedding layer; S105, concatenating and fusing the serialized semantic representation and the scene prior embeddings along the sequence dimension to obtain a fused semantic input sequence; S106, concatenating the fused semantic input sequence and the text embedding sequence obtained from the original text description sentence embeddings into a concatenated embedding sequence, inputting the concatenated embedding sequence into the language model decoder to obtain text description sentences that conform to the remote sensing image scene, and employing autoregressive language processing. The zero-shot remote sensing image text description generation method further includes testing or inference on the trained text description generation network: S301, extracting image features of the input remote sensing image based on a pre-trained visual-language model; S302, expanding the image features into a serialized semantic representation through the trained prefix mapping network; S303, achieving image scene classification by fusing text similarity and visual prototype modeling to obtain the optimal scene category; S304, constructing scene template text based on the optimal scene category, and generating scene prior embeddings through a word embedding layer; S305, concatenating and fusing the serialized semantic representation and scene prior embeddings along the sequence dimension to obtain a fused semantic input sequence; S306, inputting the fused semantic input sequence into the trained language model decoder, generating text in an autoregressive manner, and completing decoding using a bundle search strategy to output text description statements that conform to the remote sensing scene features; Step S303 includes: S401, calculate image features and a preset set of scene categories respectively. The cosine similarity of the scene category embeddings corresponding to each scene category is used as semantic similarity: ； in, Image features With scene category semantic similarity, For cosine similarity calculation function, scene category For a set of preset scene categories The first in There are several scene categories; the scene category embedding involves combining the scene category with a preset text description to construct template text, and then obtaining the template text through a CLIP text encoder. S402, for each scene category Each scene category is selected from image features based on its cosine similarity, which is greater than a preset threshold. The samples are used as a subset of the samples. ; S403, for sample subsets The image features within the image are aggregated by mean to obtain the initial visual prototypes for each scene category: ； in, Scene category The initial visual prototype, For sample subset Size, For sample subset The samples in; S404, normalizes the initial visual prototypes of each scene category to obtain normalized visual prototype representations: ； in, Scene category Normalized visual prototype representation, This is the normalization function; S405, calculate the cosine similarity between the image features and the normalized visual prototype representation of each scene category as the image features. With scene category Prototype similarity: ； in, Image features With scene category Prototype similarity, This is the function for calculating cosine similarity. S406, Image Features Image features are obtained by fusing semantic similarity and prototype similarity with various scene categories. Scene discrimination scores for each scene category: ； in, Image features With scene category Scene discrimination score, These are the weighting coefficients; S407: The scene category with the highest scene discrimination score is taken as the optimal scene category obtained from image scene classification.

2. The method for generating text descriptions of zero-shot remote sensing images based on scene guidance according to claim 1, characterized in that, The function expression for applying Gaussian random perturbation enhancement to text features in step S102 is as follows: ； in, To implement Gaussian random perturbation-enhanced text features, For normalization operations, These are the original text features. It is zero-mean Gaussian noise. , A Gaussian distribution with zero-mean Gaussian noise. For the disturbance intensity, express An identity matrix of dimension 1.

3. The method for generating text descriptions of zero-shot remote sensing images based on scene guidance according to claim 1, characterized in that, Step S103 includes: S201, for a preset set of scene categories For each scene category, the scene category name is combined with a preset description template to construct template text, and the template text is encoded into a scene category embedding through a text encoder of a visual-language model; S202, calculate text features and a preset set of scene categories respectively. The cosine similarity of the scene category embeddings corresponding to each scene category in the dataset; S203, classify scenes based on the cosine similarity between text features and the scene category embeddings of each scene category; ； in, for Textual features of time Belongs to the scene category The probability distribution, Scene category Scene embedding, A collection of scene categories The size of the set, Scene category Scene embedding, This is the cosine similarity calculation function. For temperature parameters; S204: The scene category with the highest probability distribution is taken as the optimal scene category obtained from text scene classification.

4. The method for generating text descriptions of zero-shot remote sensing images based on scene guidance according to claim 1, characterized in that, The functional expression for the autoregressive language modeling loss in step S106 is: ； in, For autoregressive language modeling loss, The number of training samples. For actual text description statements Length, Generate predicted text description statements for the large language model decoder. each word element The predicted probability, To predict the first text description statement each word element All previous morphemes, This is for splicing embedded sequences.

5. The method for generating text descriptions of zero-shot remote sensing images based on scene guidance according to claim 1, characterized in that, Preset threshold in step S402 The expression for the computation function is: ； in, This is a preset ratio coefficient used to control the strictness of sample screening; For the image features of the th Sample Scene categories Cosine similarity between them.

6. A scene-guided zero-shot remote sensing image text description generation system, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the scene-guided zero-sample remote sensing image text description generation method according to any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the scene-guided zero-sample remote sensing image text description generation method according to any one of claims 1 to 5.

8. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the scene-guided zero-sample remote sensing image text description generation method according to any one of claims 1 to 5.

Images