Multi-modal large language model for realizing fine-grained visual perception of remote sensing image
By designing a multimodal large language model, using the shared visual coding module and the modal alignment mapping layer, the problem of insufficient fine-grained visual perception capabilities in the remote sensing field is solved, and the fine-grained understanding of remote sensing images and complex scene reasoning is realized, which improves the Model flexibility and responsiveness.
Patent Information
- Application Number
- CN202411112788.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
Smart Images

Figure CN119027960A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the intersection field of remote sensing and computer vision, and specifically relates to a multimodal large language model for realizing fine-grained visual perception of remote sensing images. Background Art
[0002] Fine-grained interpretation and more complex visual reasoning of remote sensing images are crucial for making reasonable and informed decisions in practical applications. At present, recent progress has been made in the visual cue model in the field of natural images, and users can interact with artificial intelligence (AI) tools through various visual cues (such as boxes, points, and free shapes) to achieve a deeper understanding of images. However, due to the significant differences between natural images and remote sensing images in terms of imaging conditions, target perspectives, and geographical breadth, existing natural domain visual cue models face challenges in remote sensing scenarios and cannot be directly migrated to the remote sensing field.
[0003] Although there have been works on multimodal large language models, such as EarthGPT and GeoChat, which aim to explore the application of multimodal large language models in remote sensing and unify multi-visual tasks, there are still some shortcomings. For example, the existing multimodal large models in the remote sensing field mainly focus on interpreting image-granular remote sensing data interpretation, lack finer-grained visual perception, and only support interaction with language instructions. The lack of support for visual instructions limits the flexibility of multimodal large models in the real world and affects more complex reasoning of remote sensing data. Obviously, the research and application of fine-grained multimodal large language models in the field of remote sensing is still in its infancy and lacks simple and feasible methods. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a multimodal large language model that realizes fine-grained visual perception of remote sensing images, so as to meet the research and application needs of fine-grained multimodal large language models in the field of remote sensing.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] The embodiment of the present invention provides a multimodal large language model for realizing fine-grained visual perception of remote sensing images, including:
[0007] The shared visual encoding module includes two functionally complementary visual encoders, which respectively receive downsampled images of different resolutions for encoding, convert the encoded image visual features into the same dimension, and then perform channel connection to obtain an integrated multi-scale feature map. At the same time, the visual cues are respectively input into the two functionally complementary visual encoders to obtain the encoded visual cues; the modality alignment mapping layer is used to convert the multi-scale feature map and the encoded visual cues into the language semantic space to obtain the mapping image token and the visual cue token; the text segmenter module is used to convert the text instructions of the image into a vector embedding to obtain the text instruction embedding; the large language model decoder is used to receive the mapping image token, the visual cue token and the text instruction embedding to generate a response sequence.
[0008] Optionally, a multimodal large language model for realizing fine-grained visual perception of remote sensing images completes model training through the following steps: obtaining a multi-domain dataset, including a natural scene dataset and a remote sensing visual instruction dataset; training a modal alignment mapping layer in the model according to the multi-domain dataset, updating the modal alignment mapping layer parameters, and obtaining a first training model; obtaining a target image set, and converting the target image set into a picture-visual instruction-text instruction following format; training a large language model decoder attention layer in the first training model according to the target image set converted into the picture-visual instruction-text instruction following format, updating the large language model decoder attention layer parameters, and obtaining a second training model; obtaining a remote sensing instruction dataset containing fine-grained specified object classification and regional caption data; inserting a learnable low-rank adaptation matrix into the Transformer layer of the large language model decoder in the second training model, and updating the Transformer layer parameters of the large language model decoder according to the remote sensing instruction dataset containing fine-grained specified object classification and regional caption data.
[0009] Optionally, the self-attention head of the attention layer of the large language model decoder consists of three parts: query Q, key K and value V. The implementation formula of the l-th layer self-attention head is:
[0010]
[0011] Where X represents the target image set input converted to the instruction-following format, Q l (X), K l (X), V l (X) represents the output of three transformations, is the query weight matrix of the self-attention head at layer l, is the query transformation parameter of the self-attention head at layer l, is the key weight matrix of the self-attention head at layer l, is the key transformation parameter of the self-attention head at layer l, is the value weight matrix of the self-attention head at layer l, is the value transformation parameter of the self-attention head at layer l;
[0012] The score of the single attention at layer l is calculated as:
[0013]
[0014] Among them, d k is the dimension of the key.
[0015] Optionally, inserting the learnable low-rank adaptation matrix into the Transformer layer of the large language model decoder in the second training model includes: for each self-attention head in the l-th layer Transformer, introducing four low-rank matrices The Transformer multi-attention output formula adapted for the lth layer is:
[0016]
[0017] Where H is the number of self-attention heads, The four low-rank matrices are adapted to the query Q, key K and value V of the self-attention head at the first layer and the linear layer O, Q h,l , K h,l , V h,l , O h,l are the query Q, key K and value V of the self-attention head at layer l and the parameters of the linear layer O, respectively. k is the dimension of the key.
[0018] Optionally, the two visual encoders are DINOv2-ViTL / 14 and CLIP-ConvNeXt, wherein DINOv2-ViT L / 14 is pre-trained on a VisionTransformer network framework, and CLIP-ConvNeXt is pre-trained on a convolutional neural network framework.
[0019] Optionally, two visual encoders receive downsampled images of different resolutions for encoding respectively, convert the encoded image visual features into the same dimension, perform channel connection, and obtain an integrated multi-scale feature map as follows:
[0020] V img =Concat(MoV(I i )),i=1,2,...,N
[0021] Among them, V img is a multi-scale feature map, I iTo downsample images of different resolutions, MoV is the encoding algorithm for two visual encoders, and Concat is the channel concatenation operation.
[0022] Optionally, the visual prompts are input into two visual encoders respectively to obtain encoded visual prompts, including:
[0023] The visual cue dimension is adjusted to the same dimension as the downsampled image and input into two visual encoders. The encoded visual cue V is obtained according to the following consensus prompt :
[0024] V prompt =MOV(P);
[0025] Among them, P is a visual cue that adjusts the dimension to the same dimension as the sampled image, and MOV is the encoding algorithm for encoding by two visual encoders.
[0026] Optionally, receiving a mapping image token, a visual cue token, and a text instruction embedding, generating a response sequence includes:
[0027] Y=LLM(Φ(V img ),Φ(V prompt ),X instruct );
[0028] Where Y is the response sequence, LLM represents the large language model decoder, Φ(V img ) is the mapping image token obtained after the multi-scale feature map is transformed by the modality alignment mapping layer, Φ(V prompt ) is the visual cue token obtained after the encoded visual cue is converted by the modality alignment mapping layer, X instruct To convert the text instructions of the image into a vector embedding, the text instruction embedding is obtained.
[0029] Optionally, the large language model decoder uses the Llama2 language model as a base model.
[0030] The present invention provides a multimodal large language model for realizing fine-grained visual perception of remote sensing images. The shared visual encoding module can simultaneously process images of different resolutions and capture multi-scale features from local details to global contexts, which helps to understand the image content more comprehensively. The visual cues (such as key points or regions) are input into the visual encoder, and the model can focus on specific parts of the image related to the task and enhance the perception of details. The modality alignment mapping layer ensures that the visual features and the encoded visual cues can be mapped to the same semantic space as the text information, which is convenient for subsequent language model decoder processing. The text segmenter module converts natural language instructions into vector embeddings, so that the language model can understand and process text information, and improves the model's ability to respond to instructions. The large language model decoder combines the visual features, visual cues and text instructions of the image to generate a response sequence, which enables the model to generate more accurate and relevant outputs. The model integrates information from both visual and language modalities, realizes effective information fusion through the modality alignment mapping layer, and improves the understanding of complex remote sensing scenes.
[0031] This embodiment provides a multimodal large language model for realizing fine-grained visual perception of remote sensing images. Through cross-domain phased training, EarthMarker has multimodal instruction capabilities of various granularities (such as image level, point level and region level) in the remote sensing domain. In the first multi-domain image-text alignment stage, the LLM is effectively converted into a multimodal large language model MLLM with image-level understanding capabilities. Subsequently, using the natural scene reference dataset, EarthMarker has the basic spatial perception ability of the image. This is conducive to the development of subsequent specified content understanding capabilities in the remote sensing field. In addition, by using the remote sensing visual cue dataset, EarthMarker is endowed with regional and point-level image understanding capabilities. It is worth noting that the use of datasets from different fields for training enhances the open vocabulary reasoning ability. It should be emphasized that throughout the training process, the updateable parameters are disjoint, preventing interference between image understanding and visual cue capabilities of different granularities.
[0032] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art may be taught from the practice of the present invention. The objectives and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:
[0034] Figure 1A specific schematic diagram of a multimodal large language model for realizing fine-grained visual perception of remote sensing images according to the present invention;
[0035] Figure 2 A schematic diagram of the training process of a multimodal large language model for realizing fine-grained visual perception of remote sensing images in the present invention;
[0036] Figure 3 The figure is a test result diagram of the EarthMarker model of the present invention and other multimodal large language models (MLLMs);
[0037] Figure 4 This is a comparison diagram of supervised learning results of the expert model and the EarthMarker model on the NWPU dataset of the present invention;
[0038] Figure 5 Schematic diagram of the responses of different multimodal large language models and EarthMarker provided by this solution to the same remote sensing image under the same user request;
[0039] Figure 6 This is a specific effect diagram of the case practice using EarthMarker in this invention. DETAILED DESCRIPTION
[0040] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0042] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0043] This embodiment provides a multimodal large language model that realizes fine-grained visual perception of remote sensing images. Figure 1 As shown, including:
[0044] The shared visual encoding module 101 includes two functionally complementary visual encoders, which respectively receive downsampled images of different resolutions for encoding, convert the encoded image visual features into the same dimension, and then perform channel connection to obtain an integrated multi-scale feature map. At the same time, the visual cues are respectively input into the two functionally complementary visual encoders to obtain the encoded visual cues;
[0045] The modality alignment mapping layer 102 is used to convert the multi-scale feature map and the encoded visual cue into a language semantic space to obtain a mapping image token and a visual cue token;
[0046] A text segmenter module 103, used to convert the text instructions of the image into vector embeddings to obtain text instruction embeddings;
[0047] The large language model decoder 104 is used to receive the mapped image token, the visual prompt token and the text instruction embedding to generate a response sequence.
[0048] Exemplarily, in this embodiment, the image and the corresponding visual cue share a visual encoding mechanism for feature sharing, so that the visual encoder can better understand and associate the relationship between the image and the visual cue. Specifically, the shared visual encoding module 101 adopts a hybrid visual expert MoV, which is designed to encode visual information. MoV integrates two visual encoders, DINOv2-ViTL / 14 and CLIP-ConvNeXt, where DINOv2-ViTL / 14 is pre-trained on the Vision Transformer network framework and CLIP-ConvNeXt is pre-trained on the convolutional neural network framework, thereby providing complementary visual semantics. In order to refine the robust multi-scale visual features, the input image I is downsampled to different resolutions, represented as I i , and then sent to the shared visual encoding module 101 for encoding by the hybrid visual expert MoV. The advantages of the two visual backbone networks are used to enhance visual perception and refine the key details in the image.
[0049] Subsequently, the encoded visual features are converted to the same dimension and channel-wise connected to obtain an integrated multi-scale feature map, denoted as V img . This process can be described as:
[0050] V img =Concat(MoV(I i )), i = 1, 2, ..., N, where V img is a multi-scale feature map, I iTo downsample images of different resolutions, MOV is the encoding algorithm for two visual encoders, and Concat is the channel concatenation operation.
[0051] The key step of the encoder sharing mechanism is to "treat visual cues as images". Visual cues can be parts of an image that are used to emphasize reminders and can exist in the form of various visual tokens, such as boxes, dots, and free shapes. In particular, visual cues of dimension (H×W×1) are processed to the same dimension (H×W×3) as the input image (downsampled image). Then, the transformed visual cues P can also be input into MoV together with the image. The encoded visual cues are represented as V prompt , this process can be described as:
[0052] V prompt =MOV(P);
[0053] Among them, P is a visual cue that adjusts the dimension to the same dimension as the sampled image, and MOV is the encoding algorithm for encoding by two visual encoders.
[0054] The modality alignment mapping layer 102 converts the multi-scale feature map and the encoded visual cues into the language semantic space to obtain the mapped image token and the visual cue token. The purpose of the modality alignment mapping layer is to ensure that the features of different modalities such as the input image and various visual cues can be compared and interacted in a unified framework, thereby improving the performance of multimodal tasks.
[0055] For text instructions for images, such as Figure 1 As shown, this embodiment provides a text segmentation module 103, which converts the text instructions of the image into vector embeddings to obtain text instruction embeddings. After obtaining the mapped image token, visual prompt token and text instruction embedding, they are integrated into the entire multimodal input sequence.
[0056] The large language model decoder 104 receives the mapping image token and visual cue token output by the modality alignment mapping layer 102 and the text instruction embedding output by the text word segmenter module 103, and generates a response sequence Y according to the mapping image token, visual cue token and text instruction embedding. The specific process can be described as follows:
[0057] Y=LLM(Φ(V img ),Φ(V prompt ),X instruct );
[0058] Where Y is the response sequence, LLM is the large language model decoder, Φ(V img) is the mapping image token obtained after the multi-scale feature map is transformed by the modality alignment mapping layer, Φ(V prompt ) is the visual cue token obtained after the encoded visual cue is converted by the modality alignment mapping layer, X instruct To convert the text instructions of the image into a vector embedding, the text instruction embedding is obtained.
[0059] As an optional implementation, the large language model decoder uses the Llama2 language model as a base model. Llama2 is a Transformer-based decoder.
[0060] The above four modules together constitute a multimodal large language model for realizing fine-grained visual perception of remote sensing images. This model can be called EarthMarker. To simplify the description, the following multimodal large language model for realizing fine-grained visual perception of remote sensing images is called EarthMarker.
[0061] The embodiment provides a multimodal large language model for realizing fine-grained visual perception of remote sensing images. By sharing a visual encoding module, images of different resolutions can be processed simultaneously, and multi-scale features from local details to global contexts can be captured, which helps to understand the image content more comprehensively. Visual cues (such as key points or regions) are input into the visual encoder, and the model can focus on specific parts of the image related to the task, enhancing the perception of details. The modality alignment mapping layer ensures that the visual features and the encoded visual cues can be mapped to the same semantic space as the text information, which is convenient for subsequent language model decoder processing. The text segmenter module converts natural language instructions into vector embeddings, so that the language model can understand and process text information, and improves the model's ability to respond to instructions. The large language model decoder combines the visual features, visual cues and text instructions of the image to generate a response sequence, which enables the model to generate more accurate and relevant outputs. The model integrates information from both visual and language modalities, and achieves effective information fusion through the modality alignment mapping layer, thereby improving the understanding of complex remote sensing scenes.
[0062] As an optional implementation, EarthMarker completes model training through the following steps:
[0063] Acquire multi-domain datasets, including natural scene datasets and remote sensing visual command datasets;
[0064] Training a modality alignment mapping layer in the model according to the multi-domain data set, updating parameters of the modality alignment mapping layer, and obtaining a first training model;
[0065] Acquire a target image set, and convert the target image set into an image-visual instruction-text instruction following format;
[0066] Training the large language model decoder attention layer in the first training model according to the target image set converted into the picture-visual instruction-text instruction following format, updating the large language model decoder attention layer parameters, and obtaining a second training model;
[0067] Get a remote sensing command dataset containing fine-grained object classification and region caption data;
[0068] The learnable low-rank adaptation matrix is inserted into the Transformer layer of the large language model decoder in the second training model, and the Transformer layer parameters of the large language model decoder are updated according to the remote sensing instruction dataset containing fine-grained specified object classification and region caption data.
[0069] For example, in order to achieve basic image-level understanding, spatial perception, and regional / point-level remote sensing data interpretation capabilities, this embodiment designs a cross-domain phased training method. Figure 2 As shown, it is divided into three stages: multi-domain image-text alignment, spatial perception tuning, and remote sensing visual cue tuning. Throughout the training process, lightweight training is maintained to avoid expensive full parameter tuning. This embodiment proposes a disjoint parameter strategy, that is, different parameters are updated in each stage. This strategy is conducive to gradually and solidly understanding the image, and naturally solves the interference between image-text understanding, visual cue understanding, and fine-grained instruction following.
[0070] Specifically, the first stage adopts a multi-domain image-text alignment strategy. In this stage, image-level data from natural and remote sensing domains are used for pre-training to align visual and textual knowledge in a high-dimensional feature space. This strategy enables EarthMarker to deeply understand the overall semantics of the image. Specifically, first, multi-domain datasets are obtained, including natural scene datasets and remote sensing visual instruction datasets, for example, using the natural scene caption dataset COCO caption, and the remote sensing image caption and scene classification subsets in the newly constructed remote sensing visual instruction dataset. In this training stage, multi-scale visual features and language representations are integrated into the large language model decoder to develop image-level understanding capabilities. Within this stage, such as Figure 2 As shown, the shared visual encoding module (mixed visual expert MoV) and the large language model decoder (LLM) remain frozen throughout the training process in order to focus on refining robust visual features. Only the modality alignment mapping layer, which serves as a visual language connector, is updated with parameters to enhance the multimodal capabilities of the multimodal large language model for fine-grained visual perception of remote sensing images proposed in this embodiment and ensure seamless integration of visual and textual information.
[0071] The second stage uses spatial perception tuning. In the previous step, EarthMarker achieved image-level understanding capabilities. In this step, in order to obtain spatial perception and object-level understanding, the target image set is obtained and converted into an instruction-following format, such as using publicly available natural scene datasets RefCOCO and RefCOCO+, and converting the publicly available natural scene datasets RefCOCO and RefCOCO+ into image-visual instruction-text instruction-following format. During the entire training process, such as Figure 2 As shown in the figure, the shared visual encoding module (mixed visual expert MoV) and the modality alignment mapping layer (projection) remain frozen throughout the training process, and the attention layer of the large language model decoder (LLM) is unfrozen to align the spatial region features with the language embedding. Specifically, the key module self-attention head of the LLM consists of three parts: key K, query Q, and value V, which are transformed by multiple linear layers. Then, the implementation formula of the l-th layer self-attention head is:
[0072]
[0073] Where X represents the target image set input converted to the instruction-following format, Q l (X), K l (X), V l (X) represents the output of three transformations, is the query weight matrix of the self-attention head at layer l, is the query transformation parameter of the self-attention head at layer l, is the key weight matrix of the self-attention head at layer l, is the key transformation parameter of the self-attention head at layer l, is the value weight matrix of the self-attention head at layer l, is the value transformation parameter of the self-attention head at layer l;
[0074] The score of the single attention at layer l is calculated as:
[0075]
[0076] Among them, d k is the dimension of the key. Otherwise, the other modules remain frozen.
[0077] The third stage uses remote sensing visual prompt command tuning. The focus of the last stage is to accurately follow user instructions and complete complex region-level and point-level visual reasoning tasks. In this stage, Figure 2As shown, the shared visual encoding module (hybrid visual expert MoV), modality alignment mapping layer (projection), and large language model decoder (LLM) remain frozen during the entire training process. This embodiment uses LoRA technology for tuning, loads the weights trained in the previous stage, and continues to train EarthMarker on the region-text and point-text pairs of the remote sensing instruction dataset. First, obtain the data containing fine-grained specified object classification and regional caption data, and update the Transformer layer parameters of the large language model decoder according to the remote sensing instruction dataset containing fine-grained specified object classification and regional caption data. Specifically, the four learnable low-rank adaptation matrices Insert it into the Transformer layer of the large language model decoder LLM. The Transformer multi-attention output formula adapted for the lth layer is:
[0078]
[0079] Where H is the number of self-attention heads, The four low-rank matrices are adapted to the query Q, key K and value V of the self-attention head at the first layer and the linear layer O, Q h,l , K h,l , V h,l , O h,l are the query Q, key K and value V of the self-attention head at layer l and the parameters of the linear layer O, respectively. k is the dimension of the key.
[0080] This embodiment provides a multimodal large language model for realizing fine-grained visual perception of remote sensing images. Through cross-domain phased training, EarthMarker has multimodal instruction capabilities of various granularities (such as image level, point level and region level) in the remote sensing domain. In the first multi-domain image-text alignment stage, the LLM is effectively converted into a multimodal large language model MLLM with image-level understanding capabilities. Subsequently, using the natural scene reference dataset, EarthMarker has the basic spatial perception ability of the image. This is conducive to the development of subsequent specified content understanding capabilities in the remote sensing field. In addition, by using the remote sensing visual cue dataset, EarthMarker is endowed with regional and point-level image understanding capabilities. It is worth noting that the use of datasets from different fields for training enhances the open vocabulary reasoning ability. It should be emphasized that throughout the training process, the updateable parameters are disjoint, preventing interference between image understanding and visual cue capabilities of different granularities.
[0081] To illustrate the effect of this scheme, the test results of the EarthMarker model and other multimodal large language models (MLLMs) in the zero-shot comparison test on the UCMerced and AID datasets are given as follows: Figure 3 As shown in Figure 2, the EarthMarker model shows significant advantages. In addition, the supervised learning comparison results of the expert model and the EarthMarker model on the NWPU dataset are also given, as shown in Figure 2. Figure 4 As shown, it can be seen that from the perspective of BLEU, METEOR, ROUGE-L and SPICE, several different indicators for evaluating the quality of machine-generated text in the field of natural language processing (NLP), as well as CIder (0-5) for evaluating the performance of image description generation tasks, the EarthMarker model proposed in the embodiment of the present invention has more significant advantages overall.
[0082] In order to further illustrate the difference between this solution and other multimodal large language models (MLLMs), the responses of different multimodal large language models and EarthMarker provided by this solution to the same remote sensing image under the same user request are given, as shown in the figure below: Figure 5 As shown in the figure, it can be seen that for both images, EarthMarker answered correctly, while the other multimodal large language models were all wrong or partially wrong.
[0083] In order to further demonstrate the solution and effect of the embodiment of the present invention, the specific effect of using EarthMarker for case practice is given below. Figure 6 As shown, it contains two remote sensing images.
[0084] For the first image, for the content where the visual cue is a marker, such as Figure 6 The upper left corner shows:
[0085] When the user asks to identify the class of each marker in the image, EarthMarker responds with:
[0086] Mark 1: tennis court; Mark 2: swimming pool; Mark 3: basketball court; Mark 4: swimming pool; Mark 5: vehicle.
[0087] like Figure 6 The upper right corner shows that when the user asks for a brief description of each marker point, EarthMarker responds:
[0088] Marker 1: A tennis court is on the lower left, with a rectangular basketball court above it;
[0089] Marker 2: A swimming pool is at the bottom right, with a rectangular basketball court above it;
[0090] Marker 3: A rectangular basketball court at the top;
[0091] Marker 4: A vehicle is at the bottom right.
[0092] For the second image, EarthMarker supports region category recognition, and the visual prompt is the content of the tooltip, such as Figure 6 The center left shows:
[0093] When the user requested to distinguish the category of visual prompt box 4, EarthMarker responded: Visual prompt box 4: Industrial area.
[0094] When the user requests to identify the categories of all visual prompt boxes in visual prompt box 4, EarthMarker responds as follows: visual prompt box 1: stadium; visual prompt box 2: overpass; visual prompt box 3: highway toll station.
[0095] like Figure 6 The middle right side shows that the user requested a brief description of visual prompt box 4, and EarthMarker responded that visual prompt box 4 almost covers the entire image, which contains a highway toll booth, an overpass, a ground runway and a stadium.
[0096] The user requested a brief description of each visual prompt box in this image, and EarthMarker responded: Visual prompt box 1: The stadium is on the right; Visual prompt box 2: The overpass is at the bottom; Visual prompt box 3: The highway toll station is at the bottom right.
[0097] like Figure 6 The lower left corner shows that the user requested a summary description of all the visual prompt boxes in the image, and EarthMarker responded: In the satellite image, there is a stadium on the right (visual prompt box 1) and an adjacent highway toll station (visual prompt box 3), as well as a small overpass (visual prompt box 2).
[0098] The user requested a detailed description of visual prompt box 4, and EarthMarker responded: This photo may be an aerial or satellite image of an urban area. The area is densely populated with various buildings, roads, and green spaces. From the size and layout of the buildings, it can be seen that there are both residential and commercial buildings here.
[0099] like Figure 6The lower right corner shows that EarthMarker supports relationship analysis. The user requests to analyze the relationship between all visual prompt boxes in the image. EarthMarker responds as follows: (Visual prompt box 1) (Visual prompt box 2) (Visual prompt box 3) These three areas will be analyzed for their contribution to urban construction. They may be part of an urban transportation network, where the stadium serves as an event venue, the bridge facilitates waterway traffic, and the toll booth manages vehicle flow. They are spatially related because they are usually close to each other in an urban environment, and their functions are often interrelated. Bridges and toll booths may be connected to the stadium's infrastructure.
[0100] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.
Claims
1. A multimodal large language model for fine-grained visual perception of remote sensing images, It is characterized in that include: The shared visual encoding module includes two complementary visual encoders. The two visual encoders receive downsampled images of different resolutions for encoding, convert the encoded image visual features into the same dimension, and then connect the channels to obtain an integrated multi-scale feature map. At the same time, the visual cues are input into the two complementary visual encoders to obtain the encoded visual cues. The modality alignment mapping layer is used to convert the multi-scale feature maps and the encoded visual cues into the language semantic space to obtain the mapped image token and the visual cue token; The text tokenizer module is used to convert the text instructions of the image into vector embeddings to obtain text instruction embeddings; The large language model decoder receives the mapped image tokens, visual cue tokens, and textual command embeddings and generates a response sequence.
2. A multimodal large language model for realizing fine-grained visual perception of remote sensing images according to claim 1, It is characterized in that The model training is completed by following the steps below: Acquire multi-domain datasets, including natural scene datasets and remote sensing visual command datasets; Training a modality alignment mapping layer in the model according to the multi-domain data set, updating parameters of the modality alignment mapping layer, and obtaining a first training model; Obtain a target image set, and convert the target image set into a picture-visual instruction-text instruction following format; Training the large language model decoder attention layer in the first training model according to the target image set converted into the picture-visual instruction-text instruction following format, updating the large language model decoder attention layer parameters, and obtaining a second training model; Get a remote sensing command dataset containing fine-grained object classification and region caption data; The learnable low-rank adaptation matrix is inserted into the Transformer layer of the large language model decoder in the second training model, and the Transformer layer parameters of the large language model decoder are updated according to the remote sensing instruction dataset containing fine-grained specified object classification and region caption data.
3. A multimodal large language model for realizing fine-grained visual perception of remote sensing images according to claim 2, It is characterized in that The self-attention head of the attention layer of the large language model decoder consists of three parts: query Q, key K and value V. The implementation formula of the l-th layer self-attention head is: Where X represents the target image set input converted to the instruction-following format, Q l (X), K l (X), V l (X) represents the output of three transformations, is the query weight matrix of the self-attention head at layer l, is the query transformation parameter of the self-attention head at layer l, is the key weight matrix of the self-attention head at layer l, is the key transformation parameter of the self-attention head at layer l, is the value weight matrix of the self-attention head at layer l, is the value transformation parameter of the self-attention head at layer l; The score of the single attention at layer l is calculated as: Among them, d k is the dimension of the key.
4. According to claim 2, a multimodal large language model for realizing fine-grained visual perception of remote sensing images, It is characterized in that Insert the learnable low-rank adaptation matrix into the Transformer layer of the large language model decoder in the second training model, including: For each self-attention head in the l-th layer Transformer, four low-rank matrices are introduced The Transformer multi-attention output formula adapted for the lth layer is: Where H is the number of self-attention heads, The four low-rank matrices are adapted to the query Q, key K and value V of the self-attention head at the first layer and the linear layer O, Q h,l , K h,l , V h,l , O h,l are the query Q, key K and value V of the self-attention head at layer l and the parameters of the linear layer O, respectively. k is the dimension of the key.
5. According to claim 1, a multimodal large language model for realizing fine-grained visual perception of remote sensing images, It is characterized in that The two visual encoders are DINOv2-ViTL / 14 and CLIP-ConvNeXt, where DINOv2-ViTL / 14 is pre-trained on the VisionTransformer network framework and CLIP-ConvNeXt is pre-trained on the convolutional neural network framework.
6. A multimodal large language model for realizing fine-grained visual perception of remote sensing images according to claim 5, It is characterized in that The two visual encoders receive downsampled images of different resolutions for encoding, convert the encoded image visual features into the same dimension, connect the channels, and obtain the integrated multi-scale feature map: V img =Concat(MoV(I i )),i=1,2,...,N Among them, V img is a multi-scale feature map, I i To downsample images of different resolutions, MoV is the encoding algorithm for two visual encoders, and Concat is the channel concatenation operation.
7. The multimodal large language model for realizing fine-grained visual perception of remote sensing images according to claim 5, It is characterized in that The visual cues are input into two visual encoders respectively to obtain encoded visual cues, including: The visual cue dimension is adjusted to the same dimension as the downsampled image and input into two visual encoders. The encoded visual cue V is obtained according to the following consensus prompt : In prompt =MOV(P); Among them, P is a visual cue that adjusts the dimension to the same dimension as the sampled image, and MOV is the encoding algorithm for encoding by two visual encoders.
8. The multimodal large language model for realizing fine-grained visual perception of remote sensing images according to claim 6, It is characterized in that Receiving the mapped image token, visual cue token, and text instruction embedding, the generated response sequence includes: Y=LLM(Φ(V img ),Φ(V prompt ),X instruct ); Where Y is the response sequence, LLM represents the large language model decoder, Φ(V img ) is the mapping image token obtained after the multi-scale feature map is transformed by the modality alignment mapping layer, Φ(V prompt ) is the visual cue token obtained after the encoded visual cue is converted by the modality alignment mapping layer, X instruct To convert the text instructions of the image into a vector embedding, the obtained text instruction embedding.
9. The multimodal large language model for realizing fine-grained visual perception of remote sensing images according to claim 1, It is characterized in that The large language model decoder uses the Llama2 language model as the base model.
Citation Information
Patent Citations
Visual big language model construction method for multi-source ship remote sensing image interpretation
CN117994677A
Multi-modal large language model for realizing multi-sensor remote sensing image interpretation
CN118115847A
Cited By
Vision and language fused multi-modal large model system
CN119227744A
Universal cognitive architecture based on remote sensing multi-modal data
CN119295882A
Agricultural pest control method fusing computer vision and language model
CN119443286A
Crop leaf disease diagnosis method and system driven by large language model
CN119785223A
Visual language processing method and system of remote sensing image based on expert hybrid model
CN119904742A
