Intelligent interactive remote sensing information extraction method combined with visual language large model
By combining the intelligent interactive method of visual language big model, remote sensing scenarios are extracted and described, the problem of simple description of traditional algorithm output and weak generalization capabilities is solved, detailed and intuitive scene description is achieved, and the ability of automated algorithms is enhanced.
Patent Information
- Application Number
- CN202411986770.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-06-03
AI Technical Summary
When traditional remote sensing algorithms automatically identify and interpret remote sensing scenarios, the output scene category information generalization ability is weak, the description is simple, and lacks intuitive and detailed description of the scene.
Using an intelligent interactive remote sensing information extraction method combining visual language big model, image features are extracted through the visual encoding module, the mapping module realizes the alignment of image and text features, and the language inference module generates detailed scene description text.
It realizes a detailed and intuitive description of remote sensing scenarios, enhances the generalization ability of scene category information, and the generated description text is accurate and automated, improving the efficiency of human-computer interaction.
Smart Images
Figure CN120088784A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent extraction of remote sensing information, and particularly to a method for comprehensively understanding and extracting remote sensing visual and language multimodal information by a large model in human-computer interaction. Specifically, it is an intelligent interactive remote sensing information extraction method combining a vision-language large model. Background Technique
[0002] A vision-language large model is a multimodal model that combines computer vision and natural language processing. It can learn from both images and texts simultaneously to handle various tasks. Vision-language large models are defined as multimodal models that can learn from images and texts. They usually have good zero-shot capabilities and generalization capabilities and can process various types of images, including documents, web pages, etc. They mainly consist of an image encoder, an embedding projector, and a text decoder. The image encoder converts the input image into a fixed-size feature vector. The embedding projector is used to align the image and text representations, and the text decoder generates the final text output based on these representations.
[0003] In the process of automatic recognition and interpretation of remote sensing scenes, on the one hand, traditional algorithms overly focus on key hot regions of images and output specific scene category information. This category information highly depends on the initial category types set during the training of the algorithm model and lacks the generalization ability of scene category information. On the other hand, the amount of information described in the output of traditional algorithms for image scenes is limited, lacking detailed and intuitive descriptions, and often losing important information in the image that is either less important in quantity or occupies a relatively small area but is still important. Therefore, there is an urgent need for a method with a friendly human-computer interaction experience and automated algorithm implementation to achieve a detailed interpretation of remote sensing scene information in the visual and language modalities. Summary of the Invention
[0004] The present invention provides an intelligent interactive remote sensing information extraction method combining a vision-language large model, which can effectively solve the drawbacks of weak generalization ability and simple description in the output of scene category information by traditional algorithms in the current process of automatic recognition and interpretation of remote sensing scenes, lacking intuitive and direct descriptions of scenes and requiring additional information subsequently.
[0005] To achieve the above object, the present invention provides the following technical solution: An intelligent interactive remote sensing information extraction method combining a vision-language large model for text description of remote sensing scenes, including the following steps:
[0006] Step 1, extract the remote sensing scene slice to be processed, with a slice size of 224×224, and currently only the RGB mode is supported for the image band.
[0007] Step 2, use the visual coding module to extract image features.
[0008] Step 3: Achieve feature alignment from images to text through a mapping module;
[0009] Step 4: Use a language inference module to generate scene description text.
[0010] According to the above technical solution, in Step 1, when processing the remote sensing scene slices, the scene area to be processed is sliced. The size of the image needs to be restricted to 224×224, and it is extracted as a true color image and stretched. The numerical type of the original image is processed into 8-bit unsigned integer type.
[0011] According to the above technical solution, in Step 1, specifically, when generating remote sensing scene slices, first intercept the remote sensing image of the target area from the target layer and process it. When processing, slice the larger image according to the requirements of the model. Through resampling, cropping, and padding, a slice set with an image pixel length and width of 224×224 is generated to facilitate the extraction of a fixed number of image feature numbers. At the same time, stretch the image and convert the pixel numerical type to 8-bit unsigned integer type.
[0012] According to the above technical solution, in Step 2, when using the visual coding module to extract image features, it specifically includes the following steps:
[0013] 1). The image slices need to be divided into blocks to form a picture sequence block, that is, an image block, and then flattened into a one-dimensional vector after dimension mapping to complete the patch embedding operation;
[0014] 2). After patch embedding, the output tokens need to be embedded with position encoding to retain the position information of the image blocks on the original image;
[0015] 3). Feature extraction is performed through the transformer encoder, and the self-attention mechanism is used to focus on the importance of each image block in different dimensions to obtain the final image coding result.
[0016] According to the above technical solution, in Step 2, the visual coding module is used as an image encoder to extract image features and convert the image into a vector representation. The image encoder is ViT. The structure of this image encoder mainly includes an upsampling layer, a Patch feature embedding layer, and a global attention module. The Transformer Encoder module is the global attention module. The output of the image encoder is image feature embedding information, which is used to input the image decoder.
[0017] According to the above technical solution, in step 2, in the process of extracting image features, the ViT architecture is used to divide the 224×224 image into 16×16 pixel blocks, and a total of 14×14 pixel blocks are divided into patches. Because the image is limited to 3 bands, each pixel patch actually includes 16×16×3, a total of 768 pixels. Therefore, for the entire image, a feature layer with a size of (14, 14, 768) can be obtained. The feature layers are combined into a sequence, that is, the two-dimensional pixel matrix is expanded to obtain a feature layer of (196, 768), and the position information is added as required to complete the Patch+PositionEmbedding operation;
[0018] Secondly, the sequence information is passed into the Transformer Encoder for feature extraction. Through the Transformer's Multi-head Self-attention structure, attention is paid to the importance of each image block. After the construction of MultiHeadSelfAttention is completed, two full connections are added to build the entire TransformerBlock component. Through the combination of Patch+Position Embedding and multiple TransformerBlocks, image feature extraction is achieved.
[0019] According to the above technical solution, in the step 3, when the feature mapping module is used to implement the feature alignment processing from image to text, the function of the mapping module is to convert the image token into a natural language text token so that it is in the same feature space, and after the splicing operation, it is input into the large model for subsequent processing;
[0020] Among them, the essence of the mapping module is a matrix, which is a dimensionality reduction operation. It is responsible for aligning and transforming visual and language features so that they can be operated in a unified space. The mapping module maps the features (Xv) output by the image encoder to the same dimensional space (Hv) as the language model instruction tokens (Hq) to achieve the fusion of image and text. A simple linear layer matrix W is used to connect the features output by the image encoder with the language model instructions, and the image features are input into the language model.
[0021] According to the above technical solution, in step three, a simple linear layer is used to align the dimension of the image features and the text features, that is, the image encoding is transformed in dimension using a mapping matrix so that it has the same dimension as the text embedding space.
[0022] According to the above technical solution, in step four, after obtaining the image features, the language inference module is used to calculate the similarity between the image feature vector and all text feature vectors, and these similarities are converted into a probability distribution through the softmax function. The language inference module uses a language encoder, selects the Vicuna-7B large language model, and is fine-tuned on a specific instruction set to make it more suitable for the remote sensing scenario;
[0023] The goal of the model is to maximize the probability of the text that truly corresponds to the current image, so as to obtain the most matching descriptive text.
[0024] According to the above technical solution, in step four, specifically during the inference of the language inference module, the Vicuna-7B large language model is used to perform the inference task. Vicuna-7B is a model obtained by fine-tuning on the basis of LLaMa-7B using supervised data;
[0025] On this basis, semantic materials in the remote sensing scenario are used for fine-tuning to make it more suitable for text and dialogue inference in the remote sensing scenario. Vicuna adopts a Decoder-only architecture of Transformer, obtains the visual features of the image through a mapping module, and combines them with its own language features.
[0026] Compared with the prior art, the beneficial effects of the present invention are:
[0027] 1. By adopting a multi-modal method that fuses visual and text features, it can better achieve text description in the remote sensing scenario, and can more intuitively display the scene content. An automated algorithm is used to achieve an intuitive description of the remote sensing visual scene, which is convenient for manual understanding and use. At the same time, it can effectively extract the visual features of the remote sensing image scene, generate accurate natural language descriptions, and better achieve the function of generating descriptive text for the remote sensing image scene in an automated manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention.
[0029] In the drawings:
[0030] Figure 1 is a flowchart of the extraction method of the present invention;
[0031] Figure 2 is a schematic structural diagram of the ViT image encoder of the present invention;
[0032] Figure 3 is a schematic diagram of the network structure of the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0033] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustrating and explaining the present invention, and are not used to limit the present invention.
[0034] Embodiment: As Figures 1-3 shown, the present invention provides a technical solution, an intelligent interactive remote sensing information extraction method combining a vision-language large model, which is used to describe a remote sensing scene in text, and includes the following steps:
[0035] Step 1, extract the remote sensing scene slices to be processed, currently only RGB mode is supported;
[0036] Step 2, use the vision encoding module to extract image features;
[0037] Step 3, achieve feature alignment from image to text through the mapping module;
[0038] Step 4, use the language inference module to generate the scene description text.
[0039] Based on the above technical solution, in Step 1, when the remote sensing scene slices are used, during the specific processing, the scene area to be processed is sliced, the size of the image needs to be restricted, the true color image is extracted and the image is stretched, and the numerical type of the original image is processed into 8-bit unsigned integer type.
[0040] Based on the above technical solution, in Step 1, specifically, when generating the remote sensing scene slices, first intercept the remote sensing image of the target area from the target layer and process it. During the processing, slice the larger image according to the requirements of the model, and generate a slice set with the length and width of the image pixels being 224×224 through resampling, cropping and padding, so as to extract a fixed number of image feature numbers. At the same time, stretch the image and convert the pixel numerical type to 8-bit unsigned integer type.
[0041] Based on the above technical solution, in Step 2, when using the vision encoding module to extract image features, it specifically includes the following steps:
[0042] 1), the image slices need to be divided into blocks to form a picture sequence block, and then flattened into a one-dimensional vector after dimension mapping to complete the patch embedding operation;
[0043] 2), the tokens output after patch embedding need to be embedded with position encoding to retain the position information;
[0044] 3), perform feature extraction through the transformer encoder, use the self-attention mechanism to pay attention to the importance of each image block in different dimensions, and obtain the final image encoding result.
[0045] Based on the above technical solution, in step 2, a visual encoding module is used as an image encoder to extract image features and convert the image into a vector representation. The image encoder is a Vision Transformer (ViT). The image encoder structure mainly includes an upsampling layer, a Patch feature embedding layer, and a global attention module. The Transformer Encoder module is a global attention module, which can better capture the long-distance dependencies between image slices. The output of the image encoder is image feature embedding information, which is used to input the image decoder.
[0046] Based on the above technical solution, in step 2, in the process of extracting image features, the ViT architecture is used to divide the 224×224 image into 16×16 pixel blocks, and a total of 14×14 pixel blocks are divided into patches. Because the image is limited to 3 bands, each pixel patch actually includes 16×16×3, a total of 768 pixels. Therefore, for the entire image, a feature layer with a size of (14, 14, 768) can be obtained. The feature layers are combined into a sequence, that is, the two-dimensional pixel matrix is expanded to obtain a feature layer of (196, 768), and the position information is added as required to complete the Patch+PositionEmbedding operation;
[0047] Secondly, the sequence information is passed into the Transformer Encoder for feature extraction. Through the MultiheadSelfAattention structure of Transformer, attention is paid to the importance of each image block. After the construction of MultiHeadSelfAttention is completed, two full connections are added to build the entire TransformerBlock component. Through the combination of Patch+Position Embedding and multiple TransformerBlocks, the extraction of image features is realized.
[0048] Based on the above technical solution, in step 3, when using the feature mapping module to implement feature alignment processing from image to text, the function of the mapping module is to convert the image patch into a language token so that it is in the same feature space, and after the concatenation operation, it is handed over to the large model for subsequent processing;
[0049] Among them, the essence of the mapping module Projection is a matrix, which is a dimensionality reduction operation responsible for aligning or transforming visual and language features so as to operate in a unified space. The mapping module maps the feature Xv output by the image encoder to the same dimensional space Hv as the language model instruction tokens Hq, realizing the fusion of images and texts. A simple linear layer matrix W is used to connect the feature output by the image encoder with the language model instruction and input the image feature into the language model.
[0050] Based on the above technical solution, in step three, a simple linear layer is used to align the dimensions of the image feature and the text feature, that is, the mapping matrix is used to perform dimensionality transformation on the image encoding so that it has the same dimension as the text embedding space.
[0051] Based on the above technical solution, in step four, after obtaining the image feature, the language inference module is used to calculate the similarity between the image feature vector and all text feature vectors, and these similarities are converted into a probability distribution through the softmax function. The language inference module uses a language encoder, selects Vicuna-7B, and fine-tunes it on a specific instruction set to make it more suitable for the remote sensing scenario;
[0052] The goal of the model is to maximize the probability of the text that truly corresponds to the current image, so as to obtain the most matching descriptive text.
[0053] Based on the above technical solution, in step four, specifically during the inference of the language inference module, the Vicuna-7B large language model is used to perform the inference task. Vicuna-7B is a model obtained by fine-tuning on the basis of LLaMa-7B using supervised data;
[0054] On this basis, semantic materials in the remote sensing scenario are used for fine-tuning to make it more suitable for text and dialogue inference in the remote sensing scenario. Vicuna adopts the Decoder-only architecture of Transformer, obtains the visual features of the image through the mapping module Projection, and combines them with its own language features;
[0055] Moreover, by adopting the attention mechanism, the model can simultaneously focus on visual and text information and perform joint inference, answer natural language questions about images, and perform complex multi-step inferences.
[0056] Finally, it should be noted that the above are only preferred examples of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. An intelligent interactive remote sensing information extraction method combined with a large visual language model, characterized by: It is used to describe the remote sensing scene in words, including the following steps: Step 1: Extract the remote sensing scene slices to be processed. The slice size is 224×224. The image band currently only supports RGB mode. Step 2: Use the visual encoding module to extract image features; Step 3: Align features from image to text through a mapping module; Step 4: Use the language reasoning module to generate scene description text.
2. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 1 is characterized in that: In the step 1, when the remote sensing scene slices are used for specific processing, the scene area to be processed is sliced, the image size needs to be limited to 224×224, extracted as a true color image and stretched, and the numerical type of the original image is processed as an 8-bit unsigned integer.
3. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 2 is characterized in that: In the step 1, specifically, when generating remote sensing scene slices, firstly, the remote sensing image of the target area is intercepted from the target layer and processed. During the processing, the larger image is sliced according to the requirements of the model, and the image pixel length and width are generated by resampling, cropping and filling to generate a slice set of 224×224 in size, so as to extract a fixed number of image features. At the same time, the image is stretched and the pixel value type is converted to an 8-bit unsigned integer.
4. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 1 is characterized in that: In the step 2, when the visual coding module is used to extract image features, the following steps are specifically included: 1) The image slices need to be divided into blocks to form image sequence blocks, i.e. image blocks, which are then flattened into one-dimensional vectors after dimension mapping to complete the patch embedding operation; 2) After patch embedding, the output tokens need to be embedded in the position code to retain the position information of the image block on the original image; 3) Feature extraction is performed in the transformer encoder, and the self-attention mechanism is used to focus on the importance of each image block in different dimensions to obtain the final image encoding result.
5. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 4 is characterized in that: In the step 2, a visual coding module is used as an image encoder to extract image features and convert the image into a vector representation. The image encoder is ViT. The image encoder structure mainly includes an upsampling layer, a Patch feature embedding layer and a global attention module. The Transformer Encoder module is the global attention module. The output of the image encoder is image feature embedding information, which is used to input the image decoder.
6. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 4 is characterized in that: In step 2, in the process of extracting image features, the ViT architecture is used to divide the 224×224 image into 16×16 pixel blocks, and a total of 14×14 pixel patches. Because the image is limited to 3 bands, each pixel patch actually includes 16×16×3, a total of 768 pixels. Therefore, for the entire image, a feature layer of size (14, 14, 768) can be obtained. The feature layers are combined into a sequence, that is, the two-dimensional pixel matrix is expanded to obtain a feature layer of (196, 768), and the position information is added as required to complete the Patch+Position Embedding operation; Secondly, the sequence information is passed into the Transformer Encoder for feature extraction. Through the Transformer's Multi-head Self-attention structure, attention is paid to the importance of each image block. After the construction of MultiHeadSelfAttention is completed, two full connections are added to build the entire TransformerBlock component. Through the combination of Patch+Position Embedding and multiple TransformerBlocks, image feature extraction is achieved.
7. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 1 is characterized in that: In the step 3, when the feature mapping module is used to implement the feature alignment processing from image to text, the function of the mapping module is to convert the image token into a natural language text token so that it is in the same feature space, and after the splicing operation, it is input into the large model for subsequent processing; Among them, the essence of the mapping module is a matrix, which is a dimensionality reduction operation. It is responsible for aligning and transforming visual and language features so that they can be operated in a unified space. The mapping module maps the features (Xv) output by the image encoder to the same dimensional space (Hv) as the language model instruction tokens (Hq) to achieve the fusion of image and text. A simple linear layer matrix W is used to connect the features output by the image encoder with the language model instructions, and the image features are input into the language model.
8. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 7 is characterized in that: In step three, a simple linear layer is used to align the dimensions of the image features and text features, that is, the image encoding is transformed using a mapping matrix so that it has the same dimension as the text embedding space.
9. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 1 is characterized in that: In the step 4, after obtaining the image features, the similarity between the image feature vector and all text feature vectors is calculated through the language reasoning module, and these similarities are converted into probability distributions through the softmax function. The language reasoning module uses a language encoder, selects the Vicuna-7B large language model, and fine-tunes it in a specific instruction set to make it more suitable for remote sensing scenarios; The goal of the model is to maximize the probability of the text that actually corresponds to the current image, so as to obtain the most matching description text.
10. The intelligent interactive remote sensing information extraction method combined with a visual language large model according to claim 9, characterized in that: In the step 4, when the language reasoning module is specifically reasoning, the Vicuna-7B large language model is used to perform the reasoning task. Vicuna-7B is a model fine-tuned using supervised data based on LLaMa-7B; On this basis, we use semantic data in remote sensing scenarios to fine-tune it to make it more suitable for text and dialogue reasoning in remote sensing scenarios. Vicuna adopts the Transformer's Decoder-only architecture, obtains the visual features of the image through the mapping module, and combines it with its own language features.