Visual language processing method and system of remote sensing image based on expert hybrid model

Abstract

The invention discloses a visual language processing method and system for a remote sensing image based on an expert hybrid model, and relates to the data processing and artificial intelligence technology, and the method comprises the steps: obtaining remote sensing image data; a descriptive text is generated based on the remote sensing image data by utilizing a visual language model RS-MoE combined with an expert hybrid framework, the RS-MoE comprises an image encoder, a VLM encoder and an expert hybrid module (MoE Block), and the MoE Block comprises an instruction router for dynamically generating task prompts and a plurality of large language models (LLMs). Based on the visual language model provided by the embodiment of the invention, detailed description conforming to the context is provided for the input remote sensing image, and a remote sensing scene is comprehensively reflected.

Description

Technical Field

[0001] The present application relates to the fields of data processing and artificial intelligence technology, and in particular to a visual language processing method and system for remote sensing images based on an expert mixture model. Background Art

[0002] Early RSIC research focused on traditional methods, including template-based methods and retrieval-based methods. However, these methods are unable to generate rich and diverse descriptive sentences. Recent studies generally adopt an encoder-decoder architecture, which divides the RSIC task into an image encoding stage (extracting semantic features from the input image) and a sequence modeling stage (generating text and sentences using the extracted features). Depending on the specific model used, encoder-decoder methods can be further divided into models based on CNN encoders and RNN / LSTM decoders, and models based on CNN encoders and Transformer decoders. Although these methods perform satisfactorily in the RSIC task, they can usually only generate one or two simple descriptions, limiting their practicality. This limitation usually stems from two main reasons: the sentences in the dataset used to train the model are simple and repetitive; and the ability of these models to extract semantic features and generate complex descriptions is relatively limited.

[0003] Large Language Models (LLMs) and Visual Language Models (VLMs) have recently achieved remarkable success in multiple fields, including computer vision, natural language processing, and robotics. In particular, VLMs have effectively narrowed the gap between visual images and natural language by improving the understanding of cross-modal relations, achieving a level of visual understanding comparable to that of humans. Therefore, some researchers have recently shifted their research focus to applying VLMs to remote sensing visual interpretation tasks. Specifically for the RSIC task, RSGPT introduced the manually annotated RSICap dataset, which provides high-quality and detailed descriptions of remote sensing images. However, RSGPT only fine-tuned the existing VLM model on this dataset, without proposing a new model to deeply explore and utilize these detailed descriptions. Summary of the invention

[0004] The embodiments of the present application provide a method and system for visual language processing of remote sensing images based on an expert mixture model, which provides a detailed and contextual description of the input remote sensing image and comprehensively reflects the remote sensing scene.

[0005] The present application embodiment provides a method for visual language processing of remote sensing images based on an expert mixture model, including:

[0006] Acquisition of remote sensing image data;

[0007] A visual language model RS-MoE combined with an expert mixture framework is used to generate descriptive text based on the remote sensing image data. The RS-MoE includes an image encoder (Image Encoder), a VLM encoder (VLM Encoder) and an expert mixture module (MoE Block), wherein the MoE Block includes an instruction router (InstructionRouter) for dynamically generating task prompts and multiple large language models (LLMs). The training of the RS-MoE is divided into two stages, wherein the first stage is configured to fine-tune the VLM Encoder and the LLM to initialize the model weights suitable for the remote sensing image description RSIC task, and the second stage is configured to fine-tune the MoE Block to generate the desired remote sensing image description text.

[0008] Optionally, the image encoder is ViT-G / 14, so as to utilize the RS-MoE to extract visual features of the remote sensing image data.

[0009] Optionally, the VLM encoder is specifically configured to extract visual features matching a given instruction T, wherein:

[0010] The visual features extracted by the image encoder and the input description are input to the self-attention layer of the VLM encoder to obtain an intermediate feature set F SA ;

[0011] The intermediate feature set F SA The features in the image encoder are combined with the visual features extracted by the image encoder, and a cross-attention layer is used to generate a visual feature F that illustrates perception. CA ;

[0012] The generated caption-aware visual features F CA Through the feedforward network and the fully connected layer, the output description perception feature F VLM .

[0013] Optionally, the MoE Block includes an instruction router (InstructionRouter) for dynamically generating task prompts and three large language model LLM blocks, wherein:

[0014] The description router is configured to generate N hints {P1,P2,..,P N}, and based on prompt learning, it is explained that the router adapts to the characteristics of the input image to generate prompts related to the visual features, and the prompts related to the visual features are used to guide the expert model to generate a target description;

[0015] Any LLM block is configured as a corresponding subtask.

[0016] Optionally, RS-MoE may be trained in the following manner:

[0017] In the first stage of training, the weights of the image encoder are kept, and the weights of the VLM encoder and LLM blocks are trained using the LoRA method to perform model optimization through the key projection layers in the self-attention layer and the cross-attention layer.

[0018] Optionally, RS-MoE training may also include:

[0019] In the second stage of training, the image encoder and the VLM encoder are kept, and the weights of the mixture of experts module (MoEBlock) are tuned;

[0020] The weights of the MoE Block are as follows:

[0021] The description is converted into concrete prompts by the description router and combined with the visual features extracted by the VLM encoder to input into each LLM block to optimize the generation task.

[0022] An embodiment of the present application also proposes a visual language processing system for remote sensing images based on an expert mixture model, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the visual language processing method for remote sensing images based on an expert mixture model as described above are implemented.

[0023] The embodiment of the present application provides a visual language processing method for remote sensing images based on an expert mixture model. The visual language model based on the embodiment of the present application provides a detailed and contextual description of the input remote sensing image to fully reflect the remote sensing scene.

[0024] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0026] Figure 1 The following is a basic flow chart of the visual language processing method of remote sensing images based on the expert mixture model according to the embodiment of the present application;

[0027] Figure 2 The following is a schematic diagram of the architecture of the visual language model RS-MoE combined with the expert mixture framework in the embodiment of the present application;

[0028] Figures 3A-3C Schematic diagram of remote sensing images of three different scenes: urban streets, industrial areas and rural areas in the application example of this application. DETAILED DESCRIPTION

[0029] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0030] Image description integrates computer vision and natural language processing technology, and can automatically generate descriptive text for images, effectively bridging the information gap between the visual and language fields. Unlike the description of common natural images, remote sensing image description (RSIC) faces greater challenges due to the inherent characteristics of remote sensing images. Remote sensing images cover larger geographical areas and contain more diverse geographical objects. Therefore, the RSIC task requires not only information about describing geographical objects, but also a comprehensive understanding of the relationships and interactions between various geographical objects. The present application embodiment provides a visual language processing method for remote sensing images based on an expert mixture model, such as Figure 1 As shown, including:

[0031] Acquisition of remote sensing image data;

[0032] A visual language model RS-MoE combined with an expert mixture framework is used to generate descriptive text based on the remote sensing image data. The RS-MoE includes an image encoder (Image Encoder), a VLM encoder (VLM Encoder) and an expert mixture module (MoE Block), wherein the MoE Block includes an instruction router (InstructionRouter) for dynamically generating task prompts and multiple large language models (LLMs). The training of the RS-MoE is divided into two stages, wherein the first stage is configured to fine-tune the VLM Encoder and the LLM to initialize the model weights suitable for the remote sensing image description RSIC task, and the second stage is configured to fine-tune the MoE Block to generate the desired remote sensing image description text.

[0033] In a specific example, I∈R is defined in the embodiment of the present application. W×H×3is a remote sensing image, where W and H represent the width and height of the image respectively, and 3 represents the RGB color channels. The goal of the Remote Sensing Image Description Task (RSIC) is to generate a descriptive text S = {s1, ···, sn}, where each s i is an effective summary of the image content. In addition, let O = {o1, o2, ..., om} be the set of visible geographic objects in the image, and each oj represents an identified geographic object. The relationship between objects is defined as R, where A set of tuples (oi,ok) representing important interactions or spatial relations between objects.

[0034] like Figure 2 As shown, the RS-MoE model overview of the embodiment of the present application includes three key components: image encoder (Image Encoder), VLM encoder (VLM Encoder) and expert mixture module (MoE Block). MoE Block contains an instruction router (Instruction Router) that dynamically generates task-specific prompts and three lightweight large language models (LLMs), which focus on different aspects of the description task. In the description example generated in the upper right corner, different colors correspond to different description dimensions: orange represents the overall theme, purple represents the specific object, and green represents the relationship between objects.

[0035] The RS-MoE of the embodiment of the present application adopts a special two-stage training strategy to handle the remote sensing image description task. The first stage (a) fine-tunes the VLM encoder and LLM block to initialize the model weights suitable for the RSIC task; the second stage (b) fine-tunes the MoE Block to generate more detailed remote sensing image description text.

[0036] In some embodiments, the image encoder is ViT-G / 14, so as to utilize the RS-MoE to extract visual features of the remote sensing image data. In a specific example, for a given remote sensing image I, firstly, the visual features F are extracted by the image encoder fI. I In the specific example, ViT-G / 14 can be used as the image encoder. ViT-G / 14 is a large vision transformer with nearly 2 billion parameters. During the entire model training process, the visual backbone network is kept frozen.

[0037] The VLM encoder (f VLM) is specifically used to extract visual features that match a given instruction T. In a specific example, a series of learnable query embeddings and corresponding instructions are input into the self-attention layer of the VLM encoder to generate an intermediate feature set FSA. In some specific embodiments, the VLM encoder is specifically configured to extract visual features that match a given instruction T, wherein:

[0038] The visual features extracted by the image encoder and the input description are input to the self-attention layer of the VLM encoder to obtain an intermediate feature set F SA .

[0039] The intermediate feature set F SA The features in the image are combined with the visual features F extracted by the image encoder I , using the cross-attention layer, to generate visual features F that illustrate perception CA .

[0040] The generated caption-aware visual features F CA Through the feedforward network and the fully connected layer, the output description perception feature F VLM .

[0041] The learned description perception feature F VLM The corresponding description T is fed into the LLM block fl as input prompts to generate a detailed and coherent natural language description S. This module is used to initialize the LLM weights in the MoE module in the first stage of model training.

[0042] In some embodiments, the MoE Block includes an instruction router for dynamically generating task prompts and three large language model LLM blocks, represented as {f l1 ,f l2 ,...,f lN},in:

[0043] The description router is configured to generate N hints {P1,P2,..,P N}, and based on prompt learning, it is explained that the router adapts to the characteristics of the input image to generate prompts related to the visual features, and the prompts related to the visual features are used to guide the expert model to generate a target description;

[0044] Each LLM block is configured to a corresponding subtask, such as topic interpretation, object recognition, and object relationship analysis. This modular design effectively decomposes the complexity of the task and reduces the risk of generalization errors. Dynamically adjusted hints ensure that each LLM uses the most relevant information, thereby enhancing the overall performance.

[0045] Due to the complexity and diversity of remote sensing image information, directly training the model may lead to performance degradation due to sparsity. Therefore, in some embodiments, RS-MoE is trained in the following manner:

[0046] In the first stage of training, the weights of the image encoder are maintained, and the weights of the VLM encoder and LLM blocks are trained using the LoRA method to optimize the model through the key projection layer in the self-attention layer and the cross-attention layer. In the specific example, this stage mainly optimizes the weights of the VLM encoder and LLM blocks and initializes the model weights for the RSIC task. After freezing the weights of the image encoder, the VLM encoder is trained to associate visual features and caption information to enhance its ability to generate context-aware features. The LoRA technique is used to reduce the trainable parameters, and the model is fully optimized through the key projection layer in the self-attention layer and the cross-attention layer.

[0047] The trained LLM block generates the final remote sensing image description text, converting complex visual patterns into accurate language representations.

[0048] In some embodiments, training the RS-MoE further comprises:

[0049] In the second stage of training, the image encoder and the VLM encoder are kept, and the weights of the mixture of experts module (MoEBlock) are tuned;

[0050] The weights of the MoE Block are as follows:

[0051] The description is converted into concrete prompts by the description router and combined with the visual features extracted by the VLM encoder to input into each LLM block to optimize the generation task.

[0052] This stage focuses on adjusting the weights of the MoE Block to generate more detailed remote sensing image description text. The description router converts the description into specific prompts and combines the visual features extracted by the VLM encoder into each LLM to further optimize the generation task.

[0053] By reducing the number of parameters through LoRA, we can ensure efficiency while increasing the model's attention to input data, ultimately significantly improving the accuracy and responsiveness of the system.

[0054] The embodiment of the present application provides a visual language processing method for remote sensing images based on an expert mixture model. The visual language model based on the embodiment of the present application provides a detailed and contextual description of the input remote sensing image to fully reflect the remote sensing scene.

[0055] The present application embodiment also proposes an implementation case of a visual language processing method for remote sensing images based on an expert mixture model, comprising the following steps:

[0056] Dataset Description

[0057] 1.RSICap

[0058] This dataset is built on the DOTA object detection dataset and contains 2,585 pairs of high-quality remote sensing image-text paired data for training the performance of the visual-language model (VLM) on remote sensing images. Images are randomly selected from the DOTA dataset and resized to 512×512 pixels. Text annotations are manually annotated by five experts in the remote sensing field. In addition, to evaluate the performance of the VLM model, the authors constructed an evaluation set called RSIEval, which contains 1,000 512×512 pixel remote sensing images and their manual annotations.

[0059] 2.UCM-Captions

[0060] This dataset is derived from the UC Merced Land Use Dataset and contains 2,100 remote sensing images divided into 21 land use categories, each containing 100 images. Each image is 256×256 pixels in size and has a pixel resolution of 0.3048 meters. Image descriptions are manually extracted from large-scale images provided by USGS National Map Urban Area Images. Each image is accompanied by a description of five sentences, which have relatively fixed structures and simple descriptions.

[0061] 3.Sydney-Captions

[0062] This dataset is derived from Sydney Data Set and contains 613 remote sensing images divided into 7 land use categories. Each image is 500×500 pixels in size and has a pixel resolution of 0.5 meters. Each image is accompanied by a description of five sentences.

[0063] 4.RSICD

[0064] The dataset contains 10,921 remote sensing images from four platforms: Google Earth, Baidu Maps, MapABC, and Tiandi Map. These images were uniformly resized to 224×224 pixels, although the original resolutions varied. The text descriptions of the images were created by volunteers with annotation experience and remote sensing knowledge. The number of original description sentences for each image ranged from 1 to 5. In order to enrich the description content, the authors duplicated the descriptions with less than five sentences to ensure that each image had five different descriptions.

[0065] Evaluation Metrics

[0066] 1.BLEU(BiLingual Evaluation Understudy)

[0067] BLEU compares the text generated by the model with the reference text, checks the existence and frequency of n-grams (consecutive n words) that appear in the generated text, and calculates the score based on the precision of these n-grams. The BLEU scores of 1 to 4 words were tested during the evaluation.

[0068] 2. ROUGE-L

[0069] This metric aims to evaluate the quality of summarization by calculating the longest common subsequence (LCS) between the generated text and the reference text. In the image description task, ROUGE-L measures the degree of content overlap between the generated description and the reference description. ROUGE-L is a recall-oriented metric that focuses on the coverage of the reference description by the generated description.

[0070] 3.METEOR (Metric for Evaluation of Translation with Explicit Ordering)

[0071] METEOR not only considers the precision of matching words in the generated text and the reference text, but also considers the recall rate, thus striking a balance between accuracy and coverage. In addition, METEOR also considers synonyms and stemming, which can compare text content in more detail.

[0072] 4.CIDEr(Consensus-based Image Description Evaluation)

[0073] CIDEr is developed specifically for image description scoring and measures the consistency between candidate descriptions by comparing them with human-written reference descriptions. This metric evaluates the similarity of n-grams in the generated description and the reference description, and gives higher weights to more frequent n-grams to reflect their importance in the description.

[0074] This application example trained the model proposed in the embodiment of the present application on 3 NVIDIAA100 GPUs, each GPU has 80GB of memory. Three different versions of LLM (large language model) were selected for the MoE model: Llama-3.2-1B, Llama-3.2-3B and Vicuna-7B. In the MoE module, each LLM was fine-tuned for only 5 epochs. During the training process, the learning rate was initially set to 1e-4, and a warm-up phase of the learning rate was applied in the first epoch. In the specific example, the AdamW optimizer was used, the parameters were set to β1=0.9, β2=0.999, the weight decay was 0.05, and the cosine decay strategy was used to adjust the learning rate.

[0075] The results are as follows:

[0076] Performance in remote sensing image description tasks

[0077] Since these vision-language models (VLMs) were originally designed for natural images, this example fine-tunes each model following the training strategies specified in their respective papers to ensure that they are optimally adapted to remote sensing tasks. To ensure a fair comparison, all models are fine-tuned for 5 epochs, which is consistent with our RS-MoE training strategy. Subsequently, this example tests all models on the RSIEval dataset, including three versions of RS-MoE, to evaluate their description accuracy. The results are shown in Table 1, where the RS-MoE-7B model outperforms other models on all seven evaluation metrics, including four BLEU score variants, as well as METEOR, ROUGE-L, and CIDEr scores.

[0078] Table 1

[0079]

[0080] According to the results, the two lightweight variants of RS-MoE proposed in this application, RS-MoE-1B and RS-MoE-3B, even surpass the larger 7B and 13B state-of-the-art models in some indicators. For example, RS-MoE-1B surpasses BLIP2-13B in BLEU-1, METEOR, ROUGE-L, and CIDEr scores. In addition, RS-MoE-1B not only surpasses BLIP2-13B, but also surpasses MiniGPT4-13B and InstructBLIP-13B in the METEOR indicator. These findings highlight the superior efficiency of our smaller models, with RS-MoE-1B and RS-MoE-3B providing performance comparable to larger models while requiring far fewer parameters than their opponents. Such compact architectures are particularly beneficial for resource-constrained environments because they provide lower computational overhead and are well suited for deployment in real-world applications with limited memory and processing power.

[0081] The RS-MoE model of the embodiment of the present application demonstrates excellent efficiency and accuracy in solving RSIC tasks. By leveraging its innovative architecture, including the MoE framework and specialized training strategies, the model is able to effectively capture and interpret the complex visual information inherent in remote sensing images, thereby generating accurate, contextually relevant descriptions. Therefore, the RS-MoE of the embodiment of the present application not only excels in performance, but is also very practical for real-world applications, especially in environments with limited computing resources.

[0082] In order to more intuitively compare the description performance of the RS-MoE model proposed in this application with other models on a given remote sensing image, this example further randomly selects three images from the RSIEval dataset and lists the descriptions generated by each model ( Figures 3A-3C ). The three images represent three different scenes: city streets, industrial areas, and rural areas. The corresponding results are shown below:

[0083] for Figure 3A The comparison results are:

[0084] BLIP2: A satellite image showing a city street with cars and buildings.

[0085] MiniGPT4: This is a satellite image of a city block in San Diego, California, USA. The block consists of single-family homes with swimming pools in the front yards. The houses are surrounded by trees and greenery. The street is lined with palm trees and there are some vehicles parked on the street.

[0086] InstructBLIP: This is an image showing a residential area with many houses arranged in a grid pattern. Swimming pools can be seen in the yards of several houses and several cars are parked on the street. The road runs through the middle of the residential area. The houses are surrounded by trees and bushes, and some palm trees can be seen in the background.

[0087] RSGPT: This is an image showing a residential area with many houses and swimming pools. Nine swimming pools are visible in the image. The houses are well maintained and the swimming pools are clean and clearly visible. There are also several trees in the residential area, some of which are located near the swimming pools. The road runs through the middle of the community.

[0088] MOE-RSIC: This is an image of a well-maintained residential community with many houses and swimming pools. Almost all of the houses in the community are equipped with swimming pools, with a total of twelve swimming pools visible in the image. In addition to the swimming pools, there are multiple cars parked in the driveways of the houses. A road runs through the middle of the community with power lines running through it. There are also multiple trees scattered throughout the community. Figure 3B The comparison results are:

[0089] BLIP2: A satellite image showing an industrial area with many trees.

[0090] MiniGPT4: This image shows a large industrial complex with several large storage tanks and other buildings. The tanks are white cylindrical in shape, and some smaller tanks are also located within the complex.

[0091] InstructBLIP: This is a satellite image showing an industrial area. There are several white storage tanks in the center of the image, surrounded by trees and other buildings. There is a road on the left and a smaller road on the right. In the foreground, there is a wooded area.

[0092] RSGPT: This is a satellite image showing an industrial area. There are some large white buildings in the image, a road at the bottom, and a smaller road on the left. A small body of water is visible in the upper right corner of the image. Overall, the image shows an industrial area with several large storage tanks and other buildings.

[0093] MOE-RSIC: This is a satellite image showing an industrial area. There are several large white tanks on the left side of the image, surrounded by trees and other buildings. There is a road on the left and a smaller road on the right. There is a forest on the right. There are some tanks on the left side of the forest. The road passes through the industrial area, and there are many small tanks above the road and a large tank below.

[0094] for Figure 3C The comparison results are:

[0095] BLIP2: An aerial image showing rural areas.

[0096] MiniGPT4: This image shows a satellite view of a farmland surrounded by many trees and with some houses nearby. There are some vehicles parked in a parking lot.

[0097] InstructBLIP: This image is an aerial view of a rural area, with a river in the center of the image, trees on both sides of the river, and some buildings in the distance. In the foreground is a farm field with crops. The road runs along the edge of the farm field, and there are some vehicles on the road. There is a parking lot with several cars in the lower left corner, and there is a building in the lower left corner. Overall, this image shows a peaceful rural landscape, including natural features and man-made structures.

[0098] RSGPT: This is an aerial image of a rural area with a river in the center and trees and shrubs growing on both sides. There is a road running parallel to the river on the left side of the image, with some buildings visible in the distance. There is a parking lot in the foreground with several cars parked. There are some electric poles and power lines above the road. Overall, this image shows a peaceful rural landscape that blends natural and man-made elements.

[0099] MOE-RSIC: This is an aerial image of a rural area with a river running through the center. There is a lot of greenery around it, including trees and grass. There is also a parking lot with several cars parked in it. There is a farm field in the upper right of the image with some kind of crop growing in it. There is a road next to the river, and there are several buildings on the left side of the road. On the right side is a farm surrounded by trees and crops in the field.

[0100] By comparing the descriptions generated by different models, several key conclusions can be drawn. The descriptions provided by the BLIP2 model are relatively simple and focus on the main objects in the remote sensing image. The MiniGPT4 and InstructBLIP models incorporate more details and contextual information into the descriptions, such as trees, swimming pools, and roads around the house. These details enrich the descriptions, but sometimes lead to information redundancy. The RSGPT model describes the image in a more structured way. However, some of its generated descriptions may lack accuracy, such as the number of swimming pools, and may contain some subjective or hallucinatory sentences. In contrast, the RS-MoE model of the embodiment of the present application clearly describes the image from three perspectives (subject, object, and relationship between objects). Obviously, the RS-MoE model provides a more accurate description of the type, quantity, and spatial relationship of objects.

[0101] Results on UCM-Captions

[0102] This example further verifies the performance of the model of the embodiment of the present application on the classic UCM-Captions dataset. In order to make a comprehensive comparison, this example selects sixteen deep learning models designed specifically for the RSIC task, representing the most advanced methods in the field of remote sensing image description. In addition, this example also evaluates the most recent advanced VLM models in the remote sensing field, including RSGPT, SkyEyeGPT, RS-LLaVA, and RS-CapRet. It is worth noting that this example evaluates the 7B version RS-MoE model trained on the RSICap dataset without additional fine-tuning on the UCM-Captions dataset. The comparison results between the baseline model and our model are shown in Table 2. It is worth mentioning that even without fine-tuning on the UCM-Captions dataset, the model proposed in this application still achieved the highest performance in multiple indicators, including BLEU-1, BLEU-2, METEOR, and CIDEr. Specifically, the model proposed in this application achieved a METEOR score of 66.97, far exceeding the second highest score of 49.21 of the RS-LLaVA-13B model, further demonstrating the robustness and generalization ability of RS-MoE on different datasets.

[0103] Table 2

[0104]

[0105] The embodiment of the present application proposes a new visual language model (VLM) based on the mixture of experts (MoE) framework, named RS-MoE, which is specially customized for the remote sensing image description (RSIC) task. This is the first time that the MoE framework has been applied to the remote sensing field. This application extends the core principles of MoE to the unique challenges of remote sensing images. By decomposing the RSIC task into specialized subtasks, different expert models are used to promote more accurate and context-aware description generation.

[0106] Specifically, the RS-MoE of the embodiment of the present application consists of three key components: an image encoder, a VLM encoder, and a MoE module. Unlike the traditional MoE module, the MoE module of the embodiment of the present application includes an innovative instruction router and multiple lightweight LLMs. The instruction router is designed to generate tailored prompts for each LLM.

[0107] In experiments, these prompts focus on topic understanding, object recognition, and relational reasoning. The instruction router dynamically adjusts prompts based on visual features and task instructions, providing more precise control, enabling the model to generate descriptions specifically for each description goal. In addition, unlike traditional MoE models that use feedforward networks (FFNs) as experts, lightweight LLMs are used as expert models. This design enables each LLM to efficiently process complex task-specific language information and generate detailed, context-aware descriptions while ensuring computational feasibility.

[0108] The present application also proposes a two-stage training strategy to fine-tune the MoE framework for the RSIC task, solving the problem of model degradation caused by sparsity when directly applying MoE to remote sensing images. This sparsity stems from the fact that the model pre-trained on natural images lacks the expression of remote sensing specific features, which leads to low efficiency in capturing the complex characteristics of remote sensing images, thus affecting the model performance.

[0109] A low-rank adaptation (LoRA) training strategy was introduced in the RS-MoE fine-tuning stage, significantly reducing the number of trainable parameters. After fine-tuning using only the RSICap dataset (about 2000 images), the applicant evaluated the model on the RSICap dataset and tested it directly on traditional datasets such as UCM-Captions, Sydney-Captions, and RSICD without additional fine-tuning on these datasets. Experimental results show that the RS-MoE-7B model of this application achieves state-of-the-art performance, while the RS-MoE-1B variant performs comparable to the 13B VLM in generating accurate and contextually relevant descriptions. In addition, the applicant extended the model to the Remote Sensing Visual Question Answering (RSVQA) task and demonstrated good generalization capabilities by continuing to achieve state-of-the-art performance on this task.

[0110] An embodiment of the present application also proposes a visual language processing system for remote sensing images based on an expert mixture model, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the visual language processing method for remote sensing images based on an expert mixture model as described above are implemented.

[0111] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0112] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0114] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A visual language processing method for remote sensing images based on an expert mixture model, characterized in that: include: Acquisition of remote sensing image data; A visual language model RS-MoE combined with an expert mixture framework is used to generate descriptive text based on the remote sensing image data. The RS-MoE includes an image encoder (Image Encoder), a VLM encoder (VLM Encoder) and an expert mixture module (MoE Block), wherein the MoE Block includes an instruction router (InstructionRouter) for dynamically generating task prompts and multiple large language models (LLMs). The training of the RS-MoE is divided into two stages, wherein the first stage is configured to fine-tune the VLM Encoder and the LLM to initialize the model weights suitable for the remote sensing image description RSIC task, and the second stage is configured to fine-tune the MoE Block to generate the desired remote sensing image description text.

2. The visual language processing method of remote sensing images based on an expert mixture model according to claim 1, characterized in that: The image encoder is ViT-G / 14, so as to utilize the RS-MoE to extract the visual features of the remote sensing image data.

3. The visual language processing method of remote sensing images based on expert mixture model according to claim 2, characterized in that: The VLM encoder is specifically configured to extract visual features matching a given instruction T, wherein: The visual features extracted by the image encoder and the input description are input to the self-attention layer of the VLM encoder to obtain an intermediate feature set F SA ; The intermediate feature set F SA The features in the image encoder are combined with the visual features extracted by the image encoder, and a cross-attention layer is used to generate a visual feature F that illustrates perception. CA ; The generated caption-aware visual features F CA Through the feedforward network and the fully connected layer, the output description perception feature F VLM .

4. The visual language processing method of remote sensing images based on an expert mixture model as claimed in claim 3, characterized in that: The MoE Block includes an instruction router that dynamically generates task prompts and three large language model (LLM) blocks, where: The description router is configured to generate N hints {P1,P2,..,P N }, and based on prompt learning, it is explained that the router adapts to the characteristics of the input image to generate prompts related to the visual features, and the prompts related to the visual features are used to guide the expert model to generate a target description; Any LLM block is configured as a corresponding subtask.

5. The visual language processing method of remote sensing images based on expert mixture model according to claim 1, characterized in that: It also includes training RS-MoE in the following way: In the first stage of training, the weights of the image encoder are kept, and the weights of the VLM encoder and LLM blocks are trained using the LoRA method to perform model optimization through the key projection layers in the self-attention layer and the cross-attention layer.

6. The method for visual language processing of remote sensing images based on an expert mixture model as claimed in claim 5, characterized in that: Training RS-MoE also includes: In the second stage of training, the image encoder and the VLM encoder are kept, and the weights of the mixture of experts module (MoEBlock) are tuned; The weights of the MoE Block are as follows: The description is converted into concrete prompts by the description router and combined with the visual features extracted by the VLM encoder to input into each LLM block to optimize the generation task.

7. A visual language processing system for remote sensing images based on an expert mixture model, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the visual language processing method of remote sensing images based on an expert mixture model as described in any one of claims 1 to 6 are implemented.

Images