Remote sensing image oriented cross-language description generation method
By jointly training English and Chinese remote sensing datasets using a two-layer recurrent neural network, and utilizing an attention mechanism to generate cross-lingual descriptions of remote sensing images, this approach solves the problem of inaccurate Chinese description generation in existing technologies and achieves high-quality multilingual description generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2024-06-26
- Publication Date
- 2026-05-15
AI Technical Summary
Existing remote sensing image description generation algorithms cannot effectively generate Chinese descriptions. Furthermore, due to the differences between Chinese and English in language structure, character form, and grammatical rules, problems such as polysemous words and grammatical ambiguity arise during the translation process.
A two-layer recurrent neural network is used for cross-language description generation. By jointly training English and Chinese remote sensing datasets, and utilizing the differential representations of Chinese and English, an attention mechanism is introduced to guide the model to focus on important visual features based on the model's current output and empirical knowledge, thereby generating accurate Chinese and English descriptions.
It improves the accuracy of remote sensing image description generation and multilingual support capabilities, meeting practical application needs, especially significantly improving accuracy when generating Chinese descriptions.
Smart Images
Figure CN118711080B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing technology, specifically relating to a method for generating cross-language descriptions for remote sensing images. Background Technology
[0002] Image description generation is a cross-modal task encompassing computer vision, natural language processing, and deep learning. Its aim is to identify the semantic information of a given image, which typically includes objects within the image, their attributes, and the hierarchical relationships between them. Descriptive text is then generated by combining computer vision and natural language processing techniques. Remote sensing images refer to images containing geographic information captured from high altitudes by satellites, aircraft, drones, etc. Remote sensing image description generation combines the two concepts mentioned above, namely, generating natural language text descriptions for remote sensing images, describing the scenes and ground objects within the images, as well as their attributes and relationships. This research direction has significant potential for applications, such as generating real-time text or speech descriptions for photos taken by drones in war, reconnaissance, traffic control, and rescue scenarios, and providing reference text descriptions for monitoring changes in the Earth's surface environment and forest and grassland cover.
[0003] While many methods from natural image description generation research can be applied to remote sensing image description generation, this field faces several unique challenges. Remote sensing images taken from the sky lack a fixed viewing direction, and rotation does not affect the viewing experience; this phenomenon is known as rotational blurring in remote sensing images. The scale of objects in remote sensing images varies greatly, causing the same object to appear completely different in different images. For example, a patch of green vegetation might appear as a lawn in a smaller image and as a grassland in a larger one; this phenomenon is known as size blurring in remote sensing images. A single remote sensing image may contain many complex elements. For instance, a scene near an airport might include marker lines, cars, terminals, and airplanes; it cannot be simply described as an airport.
[0004] Common remote sensing image caption generation methods follow an encoder-decoder architecture, where the encoder extracts image features, and the decoder establishes the relationship between image features and text to generate a text description. Wang et al., in their paper "Wang Q, Huang W, Zhang X, et al. Word-Sentence Framework for Remote Sensing Image Captioning[J].IEEE Transactions on Geoscience and Remote Sensing, 2021:10532-10543," proposed a word-sentence framework, including a word extractor and a sentence generator, to improve the interpretability of remote sensing image caption generation methods. Sumbul G et al., in their paper "Sumbul G, Nayak S, Demir B. SD-RSIC: Summarization-Driven Deep Remote Sensing Image Captioning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021:6922-6934," proposed a novel summary-driven remote sensing image caption generation method. This method first pre-trains a pointer generator network for summarization, and then combines standard text with the summary text to generate a comprehensive description of the image. Li et al. proposed a recurrent attention mechanism in the paper "Li Y, Zhang X, Gu J, et al. RecurrentAttention and Semantic Gate for Remote Sensing Image Captioning[J].IEEE Transactions on Geoscience and Remote Sensing, 2022:1-16.", which encodes the input image into a context-aware feature representation.
[0005] The datasets used in these studies were all described in English, which prevented the algorithms from generating Chinese descriptions, limiting the application of remote sensing image description generation algorithms in China. Furthermore, Chinese and English differ significantly in language structure, character forms, and grammatical rules. Word order in Chinese is more flexible, while in English, word order has a greater impact on semantics. Chinese lacks explicit word inflections to indicate tense, number, or gender; in English annotation, the singular and plural forms of the same word can express different semantic information. For example, "building" describes houses or buildings, while "buildings" describes residential areas. These differences can be used as supplementary information to improve algorithm performance. Although translation software can be used to translate English descriptions into Chinese, or the English output of the model can be translated into Chinese, both methods often result in ambiguity or even errors due to polysemous words and grammatical structures in practice. Summary of the Invention
[0006] To overcome the shortcomings of existing technologies, this invention provides a cross-lingual description generation method for remote sensing images. It simultaneously inputs English and Chinese remote sensing datasets and jointly trains the model using the differentiated representations of Chinese and English, thereby improving the accuracy of remote sensing image description generation. Furthermore, by jointly training with the English and Chinese datasets, the model can generate both English and Chinese descriptions simultaneously, meeting practical application needs. This invention guides the model to focus on visually important regions based on its current output and empirical knowledge, effectively improving the accuracy of image description.
[0007] The technical solution adopted by this invention to solve its technical problem is as follows:
[0008] Step 1: Constructing the Chinese dataset;
[0009] Obtain publicly available remote sensing image datasets with English text descriptions, use the English descriptions as a reference, and manually annotate the remote sensing images according to the requirements of Chinese annotation;
[0010] Step 2: Visual feature extraction;
[0011] Visual features of remote sensing images were extracted using a convolutional neural network pre-trained on the ImageNet dataset.
[0012] Step 3: Natural Language Decoding;
[0013] A mapping and decoding between visual features and text features of remote sensing images is established through a two-layer recurrent neural network, and an attention mechanism is introduced in the process.
[0014] First, each visual feature is weighted and input into the initial state. The model's existing experience and knowledge guide the allocation of attention, and the importance of information is judged based on the attention, with priority given to information with high importance. Then, at each time step, the model generates a probability distribution to represent the next word or word sequence to be generated at the current time step. The generation process is iterative until a special end symbol is generated or the preset maximum generation length is reached.
[0015] Preferably, in step 1, during manual annotation:
[0016] 1) Clearly define the target, scene, and subordinate relationships in the remote sensing image, and ensure that this information is included in the annotation text when annotating;
[0017] 2) The labeled text conforms to written language conventions, ensures the integrity of the Chinese description text components, and avoids inversion and component omission;
[0018] 3) Avoid redundancy;
[0019] 4) Ensure that the vocabulary used is accurate, consistent, and standardized;
[0020] 5) A descriptive text must contain at least six words;
[0021] 6) Check and correct the annotated text.
[0022] Preferably, step 2 specifically comprises:
[0023] Step 2-1: Remove the last average pooling layer and fully connected layer from the pre-trained convolutional neural network;
[0024] Step 2-2: Freeze all parameters of the first three stages of the pre-trained model, and fine-tune the parameters of the last two stages during training.
[0025] Steps 2-3: Obtain remote sensing image features v from different stages in encoder E n The fusion is performed to obtain a visual fusion feature v that has global and local semantic information;
[0026] Given a remote sensing image input I∈C×H×W, use a pre-trained encoder E to extract visual features V of the remote sensing image at different scales. n :
[0027] v n =E n (I,W fine-tune W frozen )
[0028] Where n represents the stage of the encoder, W fine-tuneThese are parameters in the encoder that need fine-tuning, updated by the gradient descent algorithm, W. frozen These are the parameters frozen in the encoder and will not be updated during training; the encoder E consists of multiple stages, each layer is composed of a varying number of residual blocks, and each residual block is composed of a convolutional layer and skip connections;
[0029] Obtain visual features at different scales v n Next, the features are fused. To meet the model dimensionality requirements, the obtained features need to be downsampled, and then the sampled features are fused.
[0030]
[0031] Downsampling is a downsampling operation, and Concat represents a tensor concatenation operation.
[0032] Preferably, step 3 specifically comprises:
[0033] For the visual fusion feature v extracted in step 2, a two-layer LSTM decoder is used to decode it into English and Chinese natural language descriptions of the remote sensing image;
[0034] Step 3-1: Process the English and Chinese text annotations to generate a vocabulary list Σ, and encode the text using one-hot encoding; introduce an attention mechanism into the first LSTM layer to store the hidden state of the second LSTM layer from the previous time step. The image fusion feature v is concatenated with the encoded vector of the previously generated text and used as the input to the first LSTM layer. The attention weights are calculated as follows:
[0035]
[0036] α t =softmax(a t )
[0037] Among them W e ∈R E×|Σ| It is a word embedding of the vocabulary Σ, ∏ t W is the one-hot encoding of the input word at time t. va ∈R H ×V W ha ∈R H×M and ω a These are learnable parameters.
[0038] Step 3-2: Calculate attention features based on attention weights Will and After connection, the input is fed into the second-layer LSTM for decoding, using the symbol y.1:T To represent the word sequence (y1, y2, ..., y T At each time t, the conditional probability distribution of the possible output terms is given by the following equation:
[0039]
[0040] Among them W p ∈R |∑|×M and b p ∈R |∑| These are learnable parameters and bias terms;
[0041] Step 3-3: Given a target true label sequence Given an image description generation model with parameter θ, minimize the following cross-entropy loss function during training:
[0042]
[0043] in, This represents the t-th label in the actual label sequence. Let p0 represent the sequence of the first t-1 labels, where T represents the length of the actual labels, and p0 represents the sequence of the first t-1 labels. Given the condition, the prediction result is The conditional probability.
[0044] The beneficial effects of this invention are as follows:
[0045] The proposed cross-lingual description generation method for remote sensing images fully leverages the shallow and deep features extracted by pre-trained convolutional neural networks, thereby improving the quality of the model's generated descriptions of remote sensing images. Joint training with Chinese and English datasets allows the model to access richer semantic information due to their differences. An attention mechanism is introduced into the encoder, guiding the model to focus on visually important regions based on its current output and empirical knowledge, thus effectively improving the accuracy of image descriptions. Attached Figure Description
[0046] Figure 1 This is an overall structural diagram of the method of the present invention. Detailed Implementation
[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0048] This invention discloses a cross-lingual description generation method for remote sensing images. The method aims to improve the accuracy of remote sensing image description generation by simultaneously inputting English and Chinese remote sensing datasets and jointly training the model using the differential representations of Chinese and English. Furthermore, by jointly training with the English and Chinese datasets, the model can generate both English and Chinese descriptions simultaneously, meeting the needs of practical applications.
[0049] This invention proposes a cross-lingual description generation method for remote sensing images, aiming to automatically generate correct Chinese and English descriptions for remote sensing images. The technical solution includes the following steps:
[0050] 1. Construction of the Chinese Dataset. A publicly available dataset of remote sensing images with English text descriptions was obtained. Using the English descriptions as a reference, the remote sensing images were manually annotated according to the requirements of Chinese annotation, and the descriptions were further enriched based on the image content.
[0051] 2. Visual Feature Extraction. Visual feature extraction is the process of extracting features from input remote sensing images using a pre-trained network. Image feature representation is crucial for the quality of generated descriptions. Given the large amount of data in the ImageNet dataset, convolutional neural networks pre-trained on the ImageNet dataset can be used to extract visual features from remote sensing images. As the number of convolutional layers increases, pooling layers filter out some small targets in the image, deeper feature maps can obtain high-level semantic information of salient targets, and shallower feature maps can extract fine-grained feature information. The features from each layer are downsampled and then fused. The fused features contain both global and local information, which can better guide the generation of image descriptions.
[0052] 3. Natural Language Decoding. Utilizing the fused visual features extracted in step two above, a two-layer recurrent neural network is used to establish a mapping between visual features and text features for decoding, introducing an attention mechanism in this process. First, each feature is weighted and input into the initial state. The model's existing experience and knowledge guide the allocation of attention, determining which information is important and prioritizing its processing. Subsequently, at each time step, the model generates a probability distribution to represent the next word or word sequence to be generated at the current time step. The generation process iterates until a specific end symbol is generated or the preset maximum generation length is reached.
[0053] Example:
[0054] This invention proposes a method for generating cross-lingual descriptions of remote sensing images. (Refer to...) Figure 1 The specific implementation steps of this invention are as follows:
[0055] Step 1: Constructing the Chinese dataset.
[0056] This study collected publicly available datasets of remote sensing images with English descriptions. Using these English descriptions as a reference, the images were manually annotated according to Chinese annotation requirements, and the descriptions were further enriched based on the image content. Specifically, the following strategies and steps were followed when annotating each image: 1) Clearly define the targets, scenes, and hierarchical relationships in the remote sensing image. This information should be the focus of the annotation and ensured to be included in the annotation text; 2) The annotation text should conform to formal written language, ensuring the completeness of the Chinese description text components and avoiding inversions, omissions, etc.; 3) Use concise descriptions and avoid redundancy; 4) Avoid overly subjective or ambiguous language and ensure accurate, consistent, and standardized vocabulary; 5) Each description text should contain at least six characters; 6) Check and correct the completed annotation text.
[0057] Step 2, visual feature extraction.
[0058] A convolutional neural network (CNN) pre-trained on the ImageNet dataset was used as the encoder to extract visual features from remote sensing images. Since classification was not required, the final average pooling and fully connected layers of the pre-trained CNN were removed. To better adapt the encoder to the remote sensing images, all parameters of the first three stages of the pre-trained model were frozen, while the parameters of the last two stages were fine-tuned during training. The remote sensing image features v from different stages in encoder E were then obtained. n The features are fused to obtain a fused feature v that contains both global and local semantic information.
[0059] Specifically, given a remote sensing image input I∈C×H×W, a pre-trained encoder E is used to extract visual features V of the remote sensing image at different scales. n :
[0060] v n =E n (I,W fine-tune W frozen )
[0061] Where n represents the stage of the encoder, W fine-tune These are parameters in the encoder that need fine-tuning, updated by the gradient descent algorithm, W. frozen These are parameters frozen in the encoder and are not updated during training. The encoder E consists of multiple stages, each layer of which is composed of a varying number of residual blocks, and each residual block is composed of a convolutional layer and skip connections.
[0062] Obtain visual features at different scales v n Next, the features are fused. To meet the model dimensionality requirements, the obtained features need to be downsampled, and then the sampled features are fused.
[0063]
[0064] Downsampling is a downsampling operation, and Concat represents a tensor concatenation operation.
[0065] Step 3, Natural Language Decoding.
[0066] For the visual fusion feature v extracted in step 2, a two-layer LSTM decoder is used to decode it into English and Chinese natural language descriptions of the remote sensing image. First, the English and Chinese text annotations are processed to generate a vocabulary Σ, and the text is encoded using one-hot encoding. An attention mechanism is introduced into the first layer LSTM to store the hidden state of the second layer LSTM from the previous time step. The image fusion feature v is concatenated with the encoded vector of the previously generated text and used as the input to the first LSTM layer. The attention weights are calculated as follows:
[0067]
[0068] α t =softmax(a t )
[0069] Among them W e ∈R E×|Σ| It is a word embedding of the vocabulary Σ, ∏ t W is the one-hot encoding of the input word at time t. va ∈R H ×V W ha ∈R H×M and ω a These are learnable parameters.
[0070] Calculate attention features based on attention weights Will and After connection, the input is fed into the second-layer LSTM for decoding, using the symbol y. 1:T To represent the word sequence (y1, y2, ..., y T At each time t, the conditional probability distribution of the possible output words is given by the following equation:
[0071]
[0072] Among them W p ∈R |∑|×M and b p ∈R |∑| These are learnable parameters and bias terms.
[0073] Given a target true label sequence Given an image description generation model with parameter θ, minimize the following cross-entropy loss function during training:
[0074]
[0075] The effects of this invention can be further illustrated by the following experiments.
[0076] 1. Experimental conditions;
[0077] This invention is based on a simulation experiment conducted using Python under a Linux version 5.0.0-23-generic (buildd@lgw01-amd64-030) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) system environment and an NVIDIA GeForce RTX3090 graphics card.
[0078] The Chinese dataset used in the experiment was constructed based on the publicly available English remote sensing image description dataset RSICD. During encoder fine-tuning training, the initial learning rate was 4e-4, and the initial learning rate during decoder training was also 4e-4. The Adam optimizer was used, and a total of 50 training epochs were conducted with a batch size of 128.
[0079] 2. Experiment Content
[0080] This invention uses BLEU (Bilingual Evaluation Understudy) to evaluate the accuracy of the algorithm-generated descriptions. The core of this algorithm is to calculate the accuracy of a sentence by comparing the number of evaluation sentences and reference sentences (N-grams), combined with a BP penalty factor, calculated as follows:
[0081]
[0082] Among them, P N It is the precision of each order of N-gram, h k (c i ) indicates that the k-th N-gram appears in sentence c to be evaluated. i The number of times in, hk(s) ij ) represents the number of times the k-th N-gram appears in the standard reference sentence, h k (s ij ) represents the number of times a certain N-gram appears in multiple standard reference sentences, w n It is the precision weight of each N-gram, l c The length c of the sentence to be evaluated i , l s It is the length of the comment sentence.
[0083] Depending on the value of N, BLEU-1, BLEU-2, BLEU-3, and BLEU-4 can be defined respectively. The experimental results on the remote sensing image dataset are as follows: English generation results are shown in Table 1, and Chinese generation results are shown in Table 2.
[0084] Table 1. Experimental Results (in English)
[0085]
[0086] Table 2 Chinese Experimental Results
[0087]
[0088] As shown in Tables 1 and 2, the present invention outperforms the BUTD method in BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores, especially in generating Chinese description evaluation results, demonstrating the effectiveness of the present invention. Overall, the present invention achieves cross-language description generation of remote sensing images (English and Chinese), improves the model's ability to extract remote sensing image features, and thus enhances the accuracy of description generation.
Claims
1. A method for generating cross-lingual descriptions of remote sensing images, characterized in that, Includes the following steps: Step 1: Constructing the Chinese dataset; Obtain publicly available remote sensing image datasets with English text descriptions, use the English descriptions as a reference, and manually annotate the remote sensing images according to the requirements of Chinese annotation; Step 2: Visual feature extraction; Visual features of remote sensing images were extracted using a convolutional neural network pre-trained on the ImageNet dataset. Step 2 specifically involves: Step 2-1: Remove the last average pooling layer and fully connected layer from the pre-trained convolutional neural network; Step 2-2: Freeze all parameters of the first three stages of the pre-trained model, and fine-tune the parameters of the last two stages during training. Steps 2-3: Obtain remote sensing image features at different stages in encoder E By fusing the data, visual fusion features with global and local semantic information are obtained. ; Given a remote sensing image as input Visual features of remote sensing images at different scales are extracted using a pre-trained encoder E. : in, Indicates the stage of the encoder. These are parameters in the encoder that need fine-tuning and are updated by the gradient descent algorithm. These are the parameters frozen in the encoder and will not be updated during training; the encoder E consists of multiple stages, each layer is composed of a varying number of residual blocks, and each residual block is composed of a convolutional layer and skip connections. Obtain visual features at different scales Next, the features are fused. To meet the model dimensionality requirements, the obtained features need to be downsampled, and then the sampled features are fused. in, Downsampling It is a downsampling operation. This indicates a tensor concatenation operation; Step 3: Natural Language Decoding; A mapping and decoding between visual features and text features of remote sensing images is established through a two-layer recurrent neural network, and an attention mechanism is introduced in the process. First, each visual feature is weighted and input into the initial state. The model's existing experience and knowledge guide the allocation of attention, and the importance of information is judged based on the attention, with priority given to information with high importance. Then, at each time step, the model generates a probability distribution to represent the next word or word sequence to be generated at the current time step. The generation process is iterative until a special end symbol is generated or the preset maximum generation length is reached.
2. The method for generating cross-language descriptions of remote sensing images according to claim 1, characterized in that, In step 1, during manual annotation: 1) Clearly define the target, scene, and subordinate relationships in the remote sensing image, and ensure that this information is included in the annotation text when annotating; 2) The labeled text conforms to written language conventions, ensuring the integrity of the components of the Chinese descriptive text and avoiding inversion or omission of components; 3) Avoid redundancy; 4) Ensure that the vocabulary used is accurate, consistent, and standardized; 5) A descriptive text must contain at least six words; 6) Check and correct the annotated text.
3. The method for generating cross-language descriptions of remote sensing images according to claim 1, characterized in that, In step 1, step 3 specifically includes: For the visual fusion features extracted in step 2 The remote sensing image was decoded into English and Chinese natural language descriptions using a dual-layer LSTM decoder. Step 3-1: Process the English and Chinese text annotations to generate a vocabulary list. The text is encoded using one-hot encoding; an attention mechanism is introduced in the first LSTM layer to store the hidden state of the second LSTM layer from the previous time step. Image fusion features The encoded vector of the previously generated text is concatenated and used as the input to the first LSTM layer. The attention weights are calculated as follows: in It is a vocabulary list Word embedding, It is a moment One-hot encoding of the input word; These are learnable parameters; Step 3-2: Calculate attention features based on attention weights ,Will and After connection, the input is fed into the second-layer LSTM for decoding, using symbolic representation. To represent word sequence At every moment The conditional probability distribution of the possible output words is given by the following formula: in These are learnable parameters and bias terms; Step 3-3: Given a target true label sequence and a parameter is The image description generation model minimizes the following cross-entropy loss function during training: in, Represents the first in the real label sequence A tag, Indicates the preceding A sequence of labels, Indicates the length of the actual label. Indicated by Given the condition, the prediction result is The conditional probability.