A remote sensing multi-modal reasoning method based on a hybrid expert mechanism

Abstract

The application is suitable for the technical field of remote sensing, and provides a remote sensing multi-modal reasoning method based on a mixed expert mechanism, comprising the following steps: constructing a mixed expert enhanced geospatial perception visual language basic model; constructing a remote sensing data set for model training, including two stages: stage I forms a basic data set by aggregating existing remote sensing data sets across tasks, and stage II improves the diversity of the data set by using a data enhancement technology; a two-stage training strategy is adopted, combining pre-training initialization and sparse architecture optimization, and a total loss function containing self-recurrence loss and auxiliary loss is designed to train the model. The application adapts to task requirements through a mixed expert mechanism, fuses multi-modal features, captures long-range dependencies in sequences, optimizes performance, constructs a diversified remote sensing data set, and prompts the model to distinguish between global and details. While improving the performance of remote sensing perception tasks, the application maintains strong performance on the reasoning task.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing, and particularly relates to a remote sensing multi-modal reasoning method based on a hybrid expert mechanism. BACKGROUND

[0002] In the field of remote sensing, the interpretation of remote sensing images is a key link to achieve information extraction and application. However, remote sensing images often have complex background environment and dense small targets, which brings great challenges to the interpretation work. At present, remote sensing image interpretation mainly relies on two ways. On the one hand, manual interpretation needs professional personnel to operate with rich experience and skills, but this way is not only inefficient and costly, but also extremely difficult to operate when facing large-scale remote sensing data. On the other hand, the old system based on deep learning realizes automatic interpretation to a certain extent, but its adaptability and accuracy in complex scenes are insufficient, which cannot meet the requirements of modern remote sensing applications for efficient and accurate interpretation.

[0003] In recent years, visual language model (VLM) as a new technology brings new hope to improve the efficiency and security of remote sensing image interpretation. It can realize automatic understanding and interpretation of image content by combining visual information and natural language description. However, the application of existing visual language model in the field of remote sensing still faces many challenges. First, the features in remote sensing images are complex and diverse, including different ground object types, textures and spectral features, which makes it difficult for the model to accurately extract and identify targets. Second, the existing remote sensing data set has limitations in scale and diversity, which makes it difficult for the model to fully learn the features in various scenes during training, thereby affecting its performance in basic perception tasks such as target counting and detection. This phenomenon is called the Moravec paradox, which means that the model performs well in high-level cognitive tasks but has obvious shortcomings in basic perception tasks. In addition, the existing technology has a serious imbalance between perception tasks and reasoning tasks, and the lack of perception accuracy limits the application of the model in complex scenes, making it difficult to meet the demand for accurate interpretation and efficient processing.

[0004] The main reason for the above problems is the sharing of model parameters, which makes it difficult to balance the requirements of global features (used for reasoning) and local details (used for perception). In addition, the training data set of the existing remote sensing visual language model lacks feature decoupling design, and fails to fully train the model to distinguish the ability of background and target. These defects and deficiencies seriously limit the further development and application of visual language model in the field of remote sensing, therefore, an innovative solution is urgently needed to overcome these defects and meet the actual needs, in order to effectively solve the imbalance between perception tasks and reasoning tasks, and improve the perception accuracy and adaptability of remote sensing model in complex scenes. Therefore, the present application proposes a remote sensing multi-modal reasoning method based on a hybrid expert mechanism. SUMMARY

[0005] The present application aims to provide a remote sensing multi-modal reasoning method based on a hybrid expert mechanism, aiming to solve the problems raised in the background art.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] A remote sensing multi-modal reasoning method based on a hybrid expert mechanism, comprising the following steps:

[0008] Model construction: a hybrid expert enhanced geospatial perception visual language base model is constructed, which contains a visual encoder, a visual projection layer, a word embedding layer and a large language model based on a hybrid expert module, the large language model based on the hybrid expert module is composed of a plurality of stacked LLM blocks and a hybrid expert layer; wherein the visual encoder is used to process the input image and generate a visual token sequence; the visual projection layer is used to align the dimension of the visual token with the hidden size of the LLM; the word embedding layer is used to generate a text token sequence, realizing multi-modal feature fusion; the large language model based on the hybrid expert module processes the spliced visual token sequence and text token sequence through the hybrid expert layer to enhance the adaptability and computational efficiency of the model;

[0009] Training data set construction: a remote sensing data set for model training is constructed, including two stages: stage I forms a basic data set by aggregating existing remote sensing data sets across tasks, and stage II uses data enhancement techniques to improve the diversity of the data set;

[0010] Model training: a two-stage training strategy is adopted, combining pre-training initialization and sparse architecture optimization, and a total loss function containing autoregressive loss and auxiliary loss is designed to train the model.

[0011] Further, the visual encoder is constructed based on a pre-trained CLIP-ViT(L-14) backbone network, supports an input image size of 504x504, divides the image into 1296 patches, and the generated visual token sequence is represented as:

[0012] ;

[0013] In the formula, is the visual token sequence; is a single visual token; is a real number field; is the sequence length of the visual token; is the feature dimension of each visual token.

[0014] Further, the visual projection layer is a multi-layer perceptron adapter, and the visual projection layer is a function f mapping visual tokens from the dimension to the hidden size of the LLM D so that the visual tokens are in the same feature space as the text tokens.

[0015] Further, the word embedding layer processes the text token sequence representation generated after processing the text as:

[0016] token ;

[0017] wherein token is the text token sequence; is the visual token for text; is the real number field; is the sequence length of the text token; is the hidden layer size of the LLM.

[0018] Further, in the large language model based on the hybrid expert module, the visual token and the text token are spliced into a combined sequence and then input into the LLM block, and the LLM is composed of stacked multi-head self-attention and feedforward neural network blocks; each LLM block contains layer normalization and residual connection; the hybrid expert layer allocates tokens to different experts for processing through a router, and the probability distribution generated by the router is represented as:

[0019] ;

[0020] wherein is the probability distribution of each token allocated to each expert; is the input unnormalized score on the first expert; is the input unnormalized score on the first expert; is the index variable in the summation symbol, indicating the numbering of all experts 1, 2,..., K; E summed in turn; is the number of experts; each token is processed by the first expert with the highest probability, and the final output is:

[0021] ;

[0022] wherein is the final output of the MoE module to the input ; the number of actually selected experts; an index variable; the first expert network outputs the input .

[0023] Further, the data enhancement technique includes count change clipping and attribute editing; the count change clipping generates count difference samples by randomly masking part of the object and updating the annotation; the attribute editing forces visual-linguistic alignment through position operation and color operation.

[0024] Further, in the two-stage training strategy, stage I initializes the network using pre-trained weights and fine-tunes the matrix of the LLM using the LoRA technology; stage II introduces a hybrid expert architecture to convert the model into a sparse visual language model, and the hybrid expert layer and the standard MLP layer are arranged alternately, specifically, every other layer is replaced with a MoE layer instead of a standard MLP layer.

[0025] The training target includes an autoregressive loss and an auxiliary loss, and the total loss function is:

[0026] ;

[0027] In the formula, is the total loss function; is the autoregressive loss, which optimizes the generation capability by constraining the model to generate an output sequence in order; is the auxiliary loss, which optimizes resource utilization efficiency by balancing the token distribution between experts.

[0028] Compared with the prior art, the beneficial effects of the present application are:

[0029] 1、The visual encoder based on the pre-training CLIP-ViT(L-14) backbone network, the visual projection layer and the word embedding layer are used as the core modules. The visual encoder supports the input size of 504*504 through the interpolation position coding, divides the image into 1296 patches, and enhances the capture ability of complex details and small objects in the remote sensing image; the visual projection layer maps the visual token to the feature space aligned with the hidden size of the large language model, ensuring the compatibility of visual and text features; the word embedding layer generates a text token sequence, providing efficient representation for multi-modal feature fusion. Through the deep fusion of visual and text features, the understanding ability of complex scenes is significantly improved, while the efficiency and universality are maintained.

[0030] 2, The application adopts multi-head self-attention (MSA) and feed-forward neural network (FFN) as the basic module of the large language model, and introduces a mixed expert (MoE) layer to enhance adaptability. After the visual token and the text token are spliced, they are input into the LLM block composed of MSA and FFN stacks, each block contains layer normalization (LN) and residual connection, which improves the complex pattern recognition and feature extraction capability; the MoE layer predicts the probability of token allocation to the expert through the router, and the top k experts with the highest probability are processed and the weighted sum is output. This design not only captures the long-range dependence of the sequence through MSA and FFN, but also enhances the flexibility and computational efficiency of the model through the parallel experts of the MoE layer, avoiding the limitations of single FFN processing multi-modal data, and finally outputting high-quality features to improve the processing capability of visual and text joint tasks.

[0031] 3, The application adopts a pre-trained model initialization and a sparse MoE architecture as the core training strategy. In stage I, the model is initialized by CLIP-ViT (L-14) encoder, pre-trained MLP adapter and Vicuna-v1.5, and the remote sensing image is adapted by LoRA fine-tuning, while the key modules are frozen to maintain stability; in stage II, the MoE architecture is introduced, and efficient calculation and task adaptation are realized through dynamic sparsity and expert parallelism. This strategy not only enhances the inference ability of the model in complex visual tasks, but also reduces the consumption of computing resources through sparsification, improving the applicability and controllability in the field of remote sensing.

[0032] 4, The application adopts an autoregressive loss and an auxiliary loss to form a total loss function. The autoregressive loss optimizes the model generation capability, enabling it to sequentially generate high-quality output sequences; the auxiliary loss imposes a load balancing constraint on the MoE layer, promoting experts to balance token processing and avoiding performance bottlenecks caused by uneven resource allocation. The combination of the two significantly improves the training effect and inference performance of the model.

[0033] 5, The application adopts count change clipping and attribute editing as data enhancement methods. Count change clipping generates count difference samples by randomly masking part of the objects and updating the annotations to enhance the model's adaptability to different object densities; attribute editing forces visual-language alignment through position and color operations, ensuring pixel-level fidelity and visual consistency while improving the ability to understand complex scenes. The two methods together improve the diversity of the dataset and optimize the performance of the model in multi-modal tasks. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The architecture diagram of the visual language base model enhanced by the mixed expert for geographic spatial perception.

[0035] Figure 2 The two-stage training schematic diagram of the visual language base model enhanced by the mixed expert for geographic spatial perception.

[0036] Figure 3 For data enhancement processing method; wherein (a) is the count change shear; (b) is the attribute editing.

[0037] Figure 4 For the result graph of qualitative comparison in example 1.

[0038] Figure 5 For the radar chart result graph of quantitative comparison in example 1. DETAILED DESCRIPTION

[0039] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present application, the technical solutions of the present application will be described in detail below, but it cannot be understood as limiting the scope of the implementation of the present application.

[0040] The present application provides a remote sensing multi-modal reasoning method based on a mixed expert mechanism, comprising the following steps:

[0041] I. Model construction;

[0042] A mixed expert enhanced geospatial perception visual language base model (hereinafter referred to as RemoteReason) is constructed. The model architecture includes a visual encoder, a visual projection layer (a multi-layer perception machine MLP adapter), a word embedding layer, and a large language model based on a mixed expert layer (see Figure 1 ). Among them, the large language model is composed of multiple stacked LLM blocks and MoE layers. In order to comprehensively evaluate the multi-modal understanding ability of the model, five types of query set tasks are designed, including scene classification (SC), visual question answering (VQA), image description (IC), visual positioning (VG) and target counting (OC). Through these query set tasks, the model can identify image scenes (such as "navy shipyard"), answer visual question answering problems (such as "is the aircraft carrier now on the sea or on the pier?"), generate image descriptions (such as "two aircraft carriers parked side by side"), mark target positions, and count the number of targets (such as 2 aircraft carriers in the figure), so as to systematically evaluate the visual language reasoning and geospatial perception ability of RemoteReason.

[0043] (1) Visual encoder, used for processing input images and generating visual token sequences, which is based on the pre-trained CLIP-ViT(L-14) backbone network. When the input image resolution is 336×336, the input image is divided into 576 patches; through interpolation position encoding, this encoder supports 504×504 input image size, doubling the number of patches of each image to 1296. The visual token sequence generated by the visual encoder is represented as:

[0044] ;

[0045] wherein, is a sequence of visual tokens; is a single visual token; is a real number field; is the length of the sequence of visual tokens; is the feature dimension of each visual token.

[0046] This design enhances the ability to capture complex details and small objects in remote sensing images by supporting larger input sizes and more patch numbers.

[0047] (2) Visual projection layer (MLP adapter) is used to align the visual token with the hidden size of LLM, ensuring that the visual features are compatible with the text features in the same feature space. Let the visual projection layer be a function f , which can map the visual token from dimension (the dimension of the visual token) to the hidden size D of the LLM, ensuring that the visual token and the text token are in the same feature space.

[0048] (3) Word embedding layer is used to generate a sequence of text tokens, providing efficient representation for multi-modal feature fusion. After the text part is processed by the word embedding layer g, the generated text token sequence is represented as:

[0049] token ;

[0050] wherein, token is a sequence of text tokens; is a text visual token; is a real number field; is the length of the sequence of text tokens; is the hidden layer size of LLM.

[0051] (4) Large language model based on hybrid expert module, by introducing multiple feedforward neural networks (FFN) to enhance the adaptability of the model, used to process the spliced visual token sequence and text token sequence.

[0052] Visual token and text token are first spliced into a unified combined sequence , which is then input into LLM blocks, which are composed of stacked multi-head self-attention (MSA) and feed-forward network (FFN) blocks. Each LLM block contains layer normalization (LN) and residual connections to enhance the model's ability to recognize complex patterns and extract features. The forward propagation of LLM can be mathematically represented as:

[0053] ;

[0054] ;

[0055] where, is the output of the l -th layer after MSA + residual connection; is the final output of the l -th layer; is the layer normalization operation on -th layer; is the layer normalization operation on -th layer; is the output (hidden state) of the l -th layer, which serves as the input of the l -th layer; is the index of the current layer, ranging from 1 to the total number of layers L ; is the number of layers in LLM. The final output of the model is obtained by layer normalization on the output of the last layer:

[0056] ;

[0057] where, is the final output feature of the large language model; is the output of the L -th layer of LLM.

[0058] MoE layers enhance model flexibility and computational efficiency through the expert parallel mechanism, the core of which is to assign tokens to different experts for processing through a router. The FFN in the MoE layer replication stage II (here, stage II refers to the stage II training defined in the subsequent model training part, i.e., the sparse optimization stage of introducing a mixed expert architecture) is copied to form an expert set , where is an expert, i.e., a parallel multiple feed-forward network in the MoE layer. The router (which is a linear layer) predicts the probability of each token being assigned to each expert, and the probability distribution is represented as:

[0059] ;

[0060] where, is the probability distribution of each token being assigned to each expert; is the input unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; E unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert;

[0061] unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert;

[0062] unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert; unnormalized score on the i-th expert;

[0063] unnormalized score on the i-th expert; unnormalized score on the i-th expert;

[0064] unnormalized score on the i-th expert; unnormalized score on the i-th expert;

[0065] unnormalized score on the i-th expert; unnormalized score on the i-th expert;

[0066] (1) Count change cut (see Figure 3 (a) in the training set DOTAv2 training subset P2739.jpg, for example, by randomly masking part of the object and updating the annotation to generate samples with count difference, so as to enhance the adaptability of the model to different object density (while keeping the background semantics unchanged). The specific steps are as follows:

[0067] First, sample a ratio , which obeys a uniform distribution (that is randomly selected between 0.15 and 0.30); Then calculate the number of objects to be masked as , where is the original count (forced to satisfy ). Select m objects for zero padding (i.e. the pixel value of the pixel point where the object is located is set to 0, denoted as , where x, y is the pixel coordinate), and update the object count in the annotation to . Four variants are generated for each image, and the count difference of each variant is . Finally, the newly generated P2739_1.jpg to P2739_N.jpg (N=4) are stored in the DOTAv2 training subset.

[0068] (2) Attribute editing (see Figure 3 (b) in the training set): Through position operation and color operation, forced visual-linguistic alignment can improve the model's understanding of complex scenes while ensuring pixel-level fidelity and visual consistency.

[0069] For position operation, given a pointer expression (such as "bottom car"), first parse and replace the position phrase in the pointer expression (such as "bottom" → "top right"), and then map the modified position phrase to the target area of the 3x3 image grid. The corresponding object is repositioned to a random valid position within the target sector, satisfying two constraints: maintaining a 5% margin from the sector boundary, and maintaining an intersection-over-union with other objects. Apply the Poisson blending technique to maintain visual consistency, and then update all affected object bounding box annotations.

[0070] For color operation, replace the color term in the pointer expression (such as "green" → "red"), and perform the corresponding palette transfer on the image object, changing the hue value while maintaining the material reflectance property.

[0071] The application adopts count change clipping and attribute editing as a data enhancement method of the model, and the significance lies in that: the count change clipping enhances the adaptability of the model to different object density scenes by generating samples with different object quantities; the attribute editing improves the understanding and matching ability of the model to complex semantic description through the cross-modal alignment operation of position and color; the two methods jointly improve the diversity of the data set and optimize the generalization performance of the model in the multi-modal task.

[0072] III. Model training

[0073] The model adopts a two-stage training strategy, combines pre-training initialization and sparsification architecture optimization, and designs a total loss function containing autoregressive loss and auxiliary loss to improve the adaptability, efficiency and multi-modal understanding performance of the model in the remote sensing task.

[0074] (1) Two-stage training strategy (including stage I and stage II, see Figure 2 );

[0075] Stage I training uses pre-training weight to initialize the network, and fine-tunes specific parts for remote sensing related tasks. Specifically, the pre-training CLIP-ViT(L-14) encoder trained on a large amount of text and visual data, the pre-training MLP adapter trained on the 558K subset of LAION-CC-SBU dataset by BLIP caption, and Vicuna-v1.5 are used to initialize the model. To adapt to remote sensing images, LoRA (Low-Rank Adaptation) technology is used to fine-tune the query (Q) and value (V) matrices of LLM, and the specified rank is set to 64; at the same time, the parameters of the MLP adapter and the CLIP-ViT(L-14) encoder are frozen during training to maintain stability. Throughout the process, the model is always trained at an image resolution of 504x504 to match the input size supported by the visual encoder.

[0076] Stage II training converts the model into a sparse VLM by introducing a mixed expert (MoE) architecture. MoE is initialized by copying the FFN multiple times; MoE layers and standard MLP layers are arranged alternately, specifically, every other layer is replaced by a MoE layer. When the visual token and the text token When inputting the MoE layer, the router calculates the matching weight (probability distribution) between each token and each expert; each token is processed by the top 2 experts in weight, and the weighted sum of the processing results of the two experts is output; the unselected experts remain in an inactive state, and the expert parallel size is set to 1 to support the parallel execution of experts in a distributed environment. Through dynamic sparsity and expert parallelism, efficient calculation and task adaptation are achieved in this stage. Not only does it enhance the model's reasoning ability in complex visual tasks, but it also significantly reduces computational resource consumption through sparsification, improving the model's applicability and controllability in the field of remote sensing.

[0077] (2) Training target;

[0078] The model training target includes an autoregressive loss and an auxiliary loss to optimize the generation ability of the model and the allocation efficiency of the expert resources. The total loss function is composed of an autoregressive loss and an auxiliary loss , and the auxiliary loss is scaled by a balance coefficient :

[0079] ;

[0080] The autoregressive loss is used to optimize the generation ability of the LLM, and it is used to constrain the model to generate a high-quality output sequence in order. Given a visual token and a text token , the remote large model generates an output sequence by sequentially generating each element, where is is a real field, K = P + N represents the length of the output sequence, P is the length of the conditional input sequence, N is the length of the sequence generated by the model, D is the hidden size of the LLM. The loss formula is:

[0081] ;

[0082] In the formula, N is the target sequence length generated by the model; i is the index of the current generated token; is a conditional probability distribution representing the model parameter θ , where represents the trainable parameters, and the loss is only calculated for the generated text; is the token to be predicted by the model; is the visual input feature; to generate a history text of the sequence to the i -1 token.

[0083] Auxiliary loss is used to apply load balancing constraints to the MoE layer, which promotes balanced processing of tokens among experts and avoids performance bottlenecks caused by uneven resource allocation. A differentiable load balancing loss is integrated in each MoE layer to promote balanced processing of tokens among experts. The loss formula is:

[0084] ;

[0085] In the formula, is the total number of elements considered in the auxiliary loss; i is the index of the current element in the auxiliary loss; is the predicted value or model output of the i th element in the auxiliary loss; is the average routing probability of , which is as follows:

[0086] ;

[0087] ;

[0088] In the formula, is the proportion of tokens processed by each expert; is the sequence length or the total number of tokens; is an indicator function to determine whether is true; is the predicted probability distribution of the model for the input ; is the probability that the input belongs to the th class; is the average of each element of the probability distribution , resulting in a normalized scalar value used in the auxiliary loss or regularization;

[0089] Autoregressive loss can improve the performance of the model in the generation task, and auxiliary loss optimizes the efficiency of expert resource allocation. The combination of the two significantly improves the training effect and inference performance of the model.

[0090] The two-stage training strategy and loss function design of the present application have clear targeting: stage I initializes the pre-trained model and fine-tunes LoRA, which quickly adapts to the remote sensing scene while preserving the general visual-linguistic ability, and freezes key modules to ensure stability; for example Figure 2 ​As shown, the reference answer text is "This is a satellite aerial view of a city, with a large ring-shaped traffic hub in the center, and multiple wide roads radiating out from the ring. Above and to the left is a large playground and runway, next to a sports field or sports facility; below and to the right are densely packed high-rise buildings…", and the training uses the autoregressive loss. Phase II introduces the MoE architecture, which improves the model's reasoning ability in complex visual tasks while reducing computational consumption through dynamic sparsity (only activating part of the experts) and expert parallelism; the autoregressive loss combined with the auxiliary loss optimizes the model's generation ability and balances the allocation of expert resources, ultimately improving the model's training effect and inference performance. As shown in Figure 2 As shown, the output is "This photo shows a city center. In the center of the picture is a huge ring-shaped intersection; above the image, a standard sports field can be seen at coordinates [15, 13, 33, 25 | 2]; to the right of the area, there are neat rows of buildings, which seem to belong to a large school or research institution." The corresponding reference answer is "This is a satellite aerial view of a city, with a large ring-shaped traffic hub in the center, and multiple wide roads radiating out from the ring. Above and to the left is a large playground and runway, next to a sports field or sports facility; below and to the right are densely packed high-rise buildings…", and the training uses the autoregressive loss + auxiliary loss, which optimizes the generation ability and balances the allocation of expert resources.

[0091] The specific implementation of the present application is described in detail in conjunction with specific embodiments.

[0092] Embodiment 1;

[0093] 1. Qualitative comparison;

[0094] In this study, the performance of the remote large model, GeoChat, and VHM was qualitatively evaluated, and the results are as follows Figure 4As shown, through different remote sensing image understanding tasks, the performance differences of the model in image understanding, scene recognition and object positioning are compared. In the baseball field number recognition task, facing the same remote sensing image of the scene, the remote dream large model accurately identifies 4 baseball fields, GeoChat identifies 2, and VHM identifies 3, which reflects that the remote dream large model is more accurate in judging the number of target objects. When analyzing the basketball court remote sensing image, the remote dream large model focuses on the basketball court in the center of the image, which is simple and accurate to capture the core scene; GeoChat supplements the identification of the tennis court to show the multi-element analysis capability, but the remote dream large model is more direct and efficient in core feature extraction; VHM is only described as a bird's eye view of the basketball court, and the details of the scene are not fully excavated. When judging the scene to which the image belongs, the remote dream large model and GeoChat both stably identify as a bridge, and VHM misjudges as a highway, indicating that the remote dream large model has high stability in identifying bridge-type scenes and is comparable to GeoChat. In the largest basketball court positioning task in the westernmost part, the remote dream large model's marked area is closer to "GroundTruth", and in complex remote sensing images, it combines the understanding of spatial relationships and target features more accurately, and has higher positioning accuracy.

[0095] As can be seen, the remote dream large model has significant advantages in remote sensing image understanding tasks: it can effectively combine visual information and language description, accurately identify the number of target objects, directly and efficiently extract core scene features, has high positioning accuracy in complex scenes, and performs better than GeoChat and VHM in scene recognition and object positioning. It has good consistency in different scenes and outstanding generalization ability. However, the model may have some errors in some complex scenes, which may be related to image resolution, lighting conditions, or object occlusion. Future research can focus on fine-tuning the model and multi-modal information fusion to further improve the performance of the model.

[0096] 2. Quantitative comparison;

[0097] To verify the performance of the model, the present application carries out systematic comparison with 10 representative large models on 19 public remote sensing benchmarks: 4 general multi-modal large models (MiniGPT-v2, LLaVA-1.5, Qwen2.5-VL, DeepSeek-VL) and 6 remote sensing special large models (GeoChat, VHM, SkySenseGPT, LHRS-Bot, RSUniVLM, EarthDial) in Figure 5 Only the best and second-best results of these comparison models are shown in the table. The tasks cover four categories: vision grounding (VG), image captioning (IC), object counting (OC), and vision question answering (VQA).

[0098] Figure 5 The radar chart uses a blue circle to mark the optimal (State-of-the-Art, SOTA) result of the contrast model, and a green circle marks the second optimal result. The RemoteReason is located in the outermost ring on all test sets, obviously surrounding the blue and green circles, showing overall leadership.

[0099] In the visual question answering task, the RemoteReason, with excellent image semantic understanding and complex reasoning ability, achieved the highest accuracy on RSVQA-LR, RSVQA-HR and Eval. In visual positioning, DIOR-RSVG, AVVG, RRSIS-D, RSVG, VRSBench and Eval all use Pr@0.5 as the index, and the RemoteReason is better than the blue circle, and the spatial target distribution modeling and position matching ability are outstanding. The image description task covers UCM-Captions, RSICD, NWPU-Captions, Sydney-Captions and Eval, and the RemoteReason leads in the METEOR index, and the consistency and richness of the generated are significantly better than existing models. The target counting task includes RSOD, NWPU-VHR, DOTAv2, VisionDrone, Eval and other multi-scale dense scenes, and the RemoteReason still occupies the outermost ring, showing excellent fine-grained target recognition and quantity estimation ability.

[0100] Overall, the RemoteReason not only achieves overall performance improvement in multi-task in the field of remote sensing, but also performs much better than the current mainstream general large model and remote sensing special model. This result fully proves the importance and feasibility of building a large model system with unified tasks, strong expression ability and excellent generalization performance in the remote sensing scene, and also shows the potential and value of the RemoteReason as a unified multi-modal basic model in remote sensing intelligent understanding tasks.

[0101] The above is only the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the concept of the present application, can make several modifications and improvements, these should be considered as the protection scope of the present application, these will not affect the effect and practicality of the patent.

Claims

1. A remote sensing multimodal reasoning method based on a hybrid expert mechanism, characterized in that, Includes the following steps: Model Construction: A hybrid expert-enhanced geospatial perception visual language foundation model is constructed. This foundation model comprises a visual encoder, a visual projection layer, a word embedding layer, and a large language model based on hybrid expert modules. The large language model based on hybrid expert modules consists of multiple stacked LLM blocks and hybrid expert layers. The visual encoder processes the input image and generates a sequence of visual tokens. The visual projection layer aligns the dimensions of the visual tokens with the hidden dimensions of the LLM. The word embedding layer generates a sequence of text tokens, achieving multimodal feature fusion. The large language model based on hybrid expert modules processes the concatenated visual and text token sequences through the hybrid expert layers to enhance the model's adaptability and computational efficiency. Training dataset construction: Constructing a remote sensing dataset for model training includes two phases: Phase I forms a base dataset by aggregating existing remote sensing datasets across tasks, and Phase II uses data augmentation techniques to enhance dataset diversity. Model training: A two-stage training strategy is adopted, which combines pre-training initialization and sparse architecture optimization, and a total loss function including autoregressive loss and auxiliary loss is designed to train the model. In the two-stage training strategy, Stage I initializes the network with pre-trained weights and fine-tunes the LLM matrix using LoRA technology; Stage II introduces a hybrid expert architecture to convert the model into a sparse visual language model, with hybrid expert layers and standard MLP layers interleaved, specifically replacing the standard MLP layer with a MoE layer every other layer. The training objectives include autoregressive loss and auxiliary loss, and the total loss function is: ； In the formula, This is the total loss function; The autoregressive loss is used to optimize the generation capability by constraining the model to generate output sequences sequentially. To mitigate losses, the auxiliary loss mechanism optimizes resource utilization efficiency by balancing token allocation among experts.

2. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The visual encoder is built on a pre-trained CLIP-ViT-L-14 backbone network, supports input image sizes of 504×504, segments the image into 1296 patches, and generates a visual token sequence as follows: ； In the formula, A visual token sequence; For a single visual token; For the real number field; The length of the visual marker sequence; The feature dimensions for each visual token.

3. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The visual projection layer is a multilayer perceptron adapter. Let the visual projection layer be a function f, which projects the visual token from dimension... The hidden size D mapped to the LLM ensures that the visual token and the text token reside in the same feature space.

4. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The text token sequence generated after the word embedding layer processes the text is represented as follows: token ; In the formula, token A sequence of text tokens; For text-based visual tokens; For the real number field; The length of the text token sequence; This represents the hidden layer size of the LLM.

5. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, In the large language model based on hybrid expert modules, visual tokens and text tokens are concatenated into a combined sequence. The input is then fed into an LLM block, which consists of stacked multi-head self-attention and feedforward neural network blocks; each LLM block contains layer normalization and residual connections; the hybrid expert layer distributes tokens to different expert processes via a router, and the probability distribution generated by the router is represented as follows: ； In the formula, The probability distribution assigned to each expert for each label; For input In the Unnormalized scores on individual experts; For input In the Unnormalized scores on individual experts; The index variable in the summation symbol indicates that the summation is performed sequentially over all expert numbers 1, 2, ..., E; The number of experts; each token is selected by the highest probability. After processing by several experts, the final output is: ； In the formula, For the MoE module to input The final output; The actual number of experts selected; For index variables; For the first A network of experts on the input The output.

6. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The data augmentation techniques include count change clipping and attribute editing; count change clipping generates count difference samples by randomly masking parts of the object and updating annotations; attribute editing forces visual-linguistic alignment through position and color operations.

Images