Remote sensing multi-modal reasoning method based on hybrid expert mechanism

Abstract

The invention is applicable to the technical field of remote sensing, and provides a remote sensing multi-modal reasoning method based on a hybrid expert mechanism, which comprises the following steps of: constructing a hybrid expert enhanced geographic space perception visual language basic model; constructing a remote sensing data set for model training, which comprises two stages: in the stage I, forming a basic data set by aggregating existing remote sensing data sets across tasks, and in the stage II, improving the diversity of the data set by adopting a data enhancement technology; a double-stage training strategy is adopted, pre-training initialization and sparse architecture optimization are combined, and meanwhile, a total loss function containing autoregression loss and auxiliary loss is designed to train the model. According to the method, task requirements are adapted through a mixed expert mechanism, multi-modal features are fused, a sequence long-range dependency relationship is captured, performance is optimized, a diversified remote sensing data set is constructed, and a model is promoted to distinguish the global situation and details. According to the method, the performance of the remote sensing perception task is improved, and the powerful performance of the reasoning task is maintained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing, and particularly relates to a remote sensing multi-modal reasoning method based on a hybrid expert mechanism. BACKGROUND

[0002] In the field of remote sensing, the interpretation of remote sensing images is a key link to achieve information extraction and application. However, remote sensing images often have complex background environment and dense small targets, which brings great challenges to the interpretation work. At present, remote sensing image interpretation mainly relies on two ways. On the one hand, manual interpretation needs professional personnel to operate with rich experience and skills, but this way is not only low in efficiency and high in cost, but also difficult to operate when facing large-scale remote sensing data. On the other hand, the old system based on deep learning realizes automatic interpretation to a certain extent, but its adaptability and accuracy in complex scenes are insufficient, which cannot meet the requirements of modern remote sensing applications for efficient and accurate interpretation.

[0003] In recent years, visual language model (VLM) as a new technology brings new hope to improve the efficiency and security of remote sensing image interpretation. It can realize automatic understanding and interpretation of image content by combining visual information and natural language description. However, the application of existing visual language model in the field of remote sensing still faces many challenges. First, the features in remote sensing images are complex and diverse, including different ground object types, textures and spectral features, which makes it difficult for the model to accurately extract and identify targets. Second, the existing remote sensing data set has limitations in scale and diversity, which makes it difficult for the model to fully learn the features in various scenes during training, thereby affecting its performance in basic perception tasks such as target counting and detection. This phenomenon is called the Moravec paradox, which means that the model performs well in high-level cognitive tasks but has obvious shortcomings in basic perception tasks. In addition, the existing technology has a serious imbalance between perception tasks and reasoning tasks, and the lack of perception accuracy limits the application of the model in complex scenes, making it difficult to meet the demand for accurate interpretation and efficient processing.

[0004] The main reason for the above problems is the sharing of model parameters, which makes it difficult to balance the requirements of global features (used for reasoning) and local details (used for perception). In addition, the training data set of the existing remote sensing visual language model lacks feature decoupling design, and fails to fully train the model to distinguish the ability of background and target. These defects and deficiencies seriously limit the further development and application of visual language model in the field of remote sensing, therefore, an innovative solution is urgently needed to overcome these defects and meet the actual needs, in order to effectively solve the problem of imbalance between perception tasks and reasoning tasks, and improve the perception accuracy and adaptability of remote sensing model in complex scenes. Therefore, the present application proposes a remote sensing multi-modal reasoning method based on a hybrid expert mechanism. SUMMARY

[0005] The present application aims to provide a remote sensing multi-modal reasoning method based on a hybrid expert mechanism, aiming to solve the problems raised in the background art.

[0006] The purpose of the present application is achieved by the following technical solutions: A remote sensing multi-modal reasoning method based on a hybrid expert mechanism, comprising the following steps: Model construction: a hybrid expert enhanced geospatial perception visual language base model is constructed, the visual language base model comprising a visual encoder, a visual projection layer, a word embedding layer and a large language model based on a hybrid expert module, the large language model based on a hybrid expert module being composed of a plurality of stacked LLM blocks and a hybrid expert layer; wherein the visual encoder is used to process an input image and generate a visual token sequence; the visual projection layer is used to align the dimension of the visual token with the hidden size of the LLM; the word embedding layer is used to generate a text token sequence, realizing multi-modal feature fusion; the large language model based on the hybrid expert module processes the spliced visual token sequence and text token sequence through the hybrid expert layer to enhance the adaptability and computational efficiency of the model; Training data set construction: a remote sensing data set for model training is constructed, including two stages: stage I forms a base data set by aggregating existing remote sensing data sets across tasks, and stage II uses data enhancement techniques to improve the diversity of the data set; Model training: a two-stage training strategy is adopted, combining pre-training initialization and sparsification architecture optimization, and a total loss function containing autoregressive loss and auxiliary loss is designed to train the model.

[0007] Further, the visual encoder is constructed based on a pre-trained CLIP-ViT(L-14) backbone network, supports an input image size of 504x504, divides the image into 1296 patches, and the generated visual token sequence is represented as: ; In the formula, is the visual token sequence; is a single visual token; is a real number field; is the sequence length of the visual token; is the feature dimension of each visual token.

[0008] Further, the visual projection layer is a multi-layer perception adapter, and the visual projection layer is a function f , which maps the visual token from a dimension to the hidden size of the LLM Dto make the visual token and the text token in the same feature space.

[0009] Further, the word embedding layer processes the text token sequence generated after the text to represent: token ; In the formula, token is a text token sequence; is a text visual token; is a real number field; is the sequence length of the text token; is the hidden layer size of the LLM.

[0010] Further, in the large language model based on the mixed expert module, the visual token and the text token are spliced into a combined sequence and then input into the LLM block, the LLM is composed of stacked multi-head self-attention and feed-forward neural network blocks; each LLM block contains layer normalization and residual connection; the mixed expert layer allocates tokens to different experts through a router, and the probability distribution generated by the router is represented as: ; In the formula, is the probability distribution of each token allocated to each expert; is the input unnormalized score on the first expert; is the input unnormalized score on the first expert; is the index variable in the summation symbol, indicating the numbering of all experts 1, 2,..., K; E summed in turn; K is the number of experts; each token is processed by the first expert with the highest probability, and the final output is: ; In the formula, is the final output of the MoE module to the input ; is the actual number of selected experts; is the index variable; is the output of the first expert network to the input .

[0011] Further, the data augmentation techniques include count change cropping and attribute editing; count change cropping generates count difference samples by randomly masking part of the object and updating the annotations; attribute editing enforces visual-linguistic alignment through position manipulation and color manipulation.

[0012] Further, in the two-stage training strategy, stage I uses pre-training weights to initialize the network and adopts LoRA technology to fine-tune the matrix of the LLM; stage II introduces a hybrid expert architecture to convert the model into a sparse visual language model, and the hybrid expert layer and the standard MLP layer are arranged alternately, specifically, every other layer is replaced by a MoE layer instead of a standard MLP layer. The training target includes an autoregressive loss and an auxiliary loss, and the total loss function is: ; In the formula, is the total loss function; is the autoregressive loss, which optimizes the generation capability by constraining the model to generate an output sequence in order; is the auxiliary loss, which optimizes the resource utilization efficiency by balancing the token distribution between experts.

[0013] Compared with the prior art, the beneficial effects of the present application are: 1、The present application adopts a visual encoder based on a pre-training CLIP-ViT(L-14) backbone network, a visual projection layer and a word embedding layer as core modules. The visual encoder supports an input size of 504x504 through interpolation position encoding, divides an image into 1296 patches, and enhances the capture ability of complex details and small objects in remote sensing images; the visual projection layer maps the visual token to a feature space aligned with the hidden size of the large language model, ensuring the compatibility of visual and text features; the word embedding layer generates a text token sequence, providing efficient representation for multi-modal feature fusion. Through the deep fusion of visual and text features, the understanding ability of complex scenes is significantly improved, while the efficiency and universality are maintained.

[0014] 2, The application adopts multi-head self-attention (MSA) and feed-forward neural network (FFN) as the basic module of the large language model, and introduces a mixed expert (MoE) layer to enhance adaptability. After the visual token and the text token are spliced, they are input into the LLM block composed of MSA and FFN stacking, each block contains layer normalization (LN) and residual connection, which improves the complex pattern recognition and feature extraction capability; the MoE layer predicts the probability of token allocation to the expert through the router, and the top k experts with the highest probability are processed and the weighted sum is output. This design not only captures the long-range dependence of the sequence through MSA and FFN, but also enhances the flexibility and computational efficiency of the model through the parallel experts of the MoE layer, avoiding the limitations of single FFN processing multi-modal data, and finally outputting high-quality features to improve the processing capability of visual and text joint tasks.

[0015] 3, The application adopts a pre-trained model initialization and a sparse MoE architecture as the core training strategy. In stage I, the model is initialized by CLIP-ViT (L-14) encoder, pre-trained MLP adapter and Vicuna-v1.5, and the remote sensing image is adapted by LoRA fine-tuning, while the key modules are frozen to maintain stability; in stage II, the MoE architecture is introduced, and efficient calculation and task adaptation are realized through dynamic sparsity and expert parallelism. This strategy not only enhances the inference ability of the model in complex visual tasks, but also reduces the consumption of computing resources through sparsification, improving the applicability and controllability in the field of remote sensing.

[0016] 4, The application adopts an autoregressive loss and an auxiliary loss to form a total loss function. The autoregressive loss optimizes the model generation capability, enabling it to sequentially generate high-quality output sequences; the auxiliary loss imposes load balancing constraints on the MoE layer, promoting experts to balance token processing and avoiding performance bottlenecks caused by uneven resource allocation. The combination of the two significantly improves the training effect and inference performance of the model.

[0017] 5, The application adopts count change clipping and attribute editing as data enhancement methods. Count change clipping generates count difference samples by randomly masking part of the objects and updating the annotations to enhance the model's adaptability to different object densities; attribute editing forces visual-language alignment through position and color operations, ensuring pixel-level fidelity and visual consistency while improving the ability to understand complex scenes. The two methods together improve the diversity of the dataset and optimize the performance of the model in multi-modal tasks. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The architecture diagram of the visual language base model enhanced by the mixed expert for geographic spatial perception.

[0019] Figure 2 The two-stage training schematic diagram of the visual language base model enhanced by the mixed expert for geographic spatial perception.

[0020] Figure 3 For data enhancement processing method; wherein (a) is the count change shear; (b) is the attribute editing.

[0021] Figure 4 For the result graph of qualitative comparison in example 1.

[0022] Figure 5 For the radar chart result graph of quantitative comparison in example 1. DETAILED DESCRIPTION

[0023] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present application, the technical solutions of the present application will be described in detail below, but it cannot be understood as limiting the scope of the implementation of the present application.

[0024] The present application provides a remote sensing multi-modal reasoning method based on a mixed expert mechanism, comprising the following steps: I. Model construction; A mixed expert enhanced geospatial perception visual language base model (hereinafter referred to as RemoteReason) is constructed. The model architecture includes a visual encoder, a visual projection layer (a multi-layer perception machine MLP adapter), a word embedding layer, and a large language model based on a mixed expert layer (see Figure 1 ). Among them, the large language model is composed of multiple stacked LLM blocks and MoE layers. In order to comprehensively evaluate the multi-modal understanding ability of the model, five types of query set tasks are designed, including scene classification (SC), visual question answering (VQA), image description (IC), visual positioning (VG) and target counting (OC). Through these query set tasks, the model can identify image scenes (such as "navy shipyard"), answer visual question answering problems (such as "is the aircraft carrier now on the sea or on the pier?"), generate image descriptions (such as "two aircraft carriers parked side by side"), mark target positions, and count the number of targets (such as 2 aircraft carriers in the figure), so as to systematically evaluate the visual language reasoning and geospatial perception ability of the RemoteReason model.

[0025] (1) Visual encoder, used for processing input images and generating visual token sequences, which is based on the pre-trained CLIP-ViT(L-14) backbone network. When the input image resolution is 336×336, the input image is divided into 576 patches; through interpolation position encoding, this encoder supports 504×504 input image size, doubling the number of patches of each image to 1296. The visual token sequence generated by the visual encoder is represented as: ; In the formula, is the visual token sequence; is a single visual token; is a real number field; is the sequence length of visual tokens; is the feature dimension of each visual token.

[0026] This design enhances the ability to capture complex details and small objects in remote sensing images by supporting larger input sizes and more patches.

[0027] (2) Visual projection layer (MLP adapter) is used to align the visual token with the hidden size of LLM, ensuring that visual features are compatible with text features in the same feature space. Let the visual projection layer be a function f , which maps the visual token from dimension (the dimension of the visual token) to the hidden size of LLM D , ensuring that the visual token and the text token are in the same feature space.

[0028] (3) Word embedding layer is used to generate text token sequence, providing efficient representation for multi-modal feature fusion. After processing by the word embedding layer g, the generated text token sequence is represented as: token ; In the formula, token is the text token sequence; is the text visual token; is a real number field; is the sequence length of text tokens; is the hidden layer size of LLM.

[0029] (4) Large language model based on hybrid expert module, by introducing multiple feed-forward neural networks (FFN) to enhance the adaptability of the model, used to process the spliced visual token sequence and text token sequence.

[0030] Visual token and text token are first spliced into a unified combined sequence , which is then input into the LLM block. LLM consists of stacked multi-head self-attention (MSA) and feed-forward neural network (FFN) blocks. Each LLM block contains layer normalization (LN) and residual connection to enhance the model's ability to recognize complex patterns and feature extraction. The forward propagation of LLM can be mathematically represented as: ; ; In the formula, is the output of the first layer l after MSA+ residual connection; is the output of the first layer l after MSA+ residual connection; is the output of the first layer after MSA+ residual connection; is the output of the first layer after MSA+ residual connection; is the output of the first layer l -1, as the input of the first layer l ; is the index of the current layer, from 1 to the total number of layers L ; is the number of layers in LLM. The final output of the model is obtained by layer normalization on the output of the last layer: ; wherein, is the final output feature of the large language model; is the output of the first layer L of the LLM.

[0031] MoE layer enhances the flexibility and computational efficiency of the model through the expert parallel mechanism. The core is to assign tokens to different experts for processing through the router. The FFN in the MoE layer replication stage II (stage II here refers to the stage II training defined in the subsequent model training part, that is, the sparse optimization stage of introducing a mixed expert architecture) is copied to form an expert set , wherein is an expert, that is, a plurality of parallel feedforward networks in the MoE layer. The router (which is a linear layer) predicts the probability of each token being assigned to each expert, and the probability distribution is represented as: ; wherein, is the probability distribution of each token being assigned to each expert; is the input unnormalized score on the first expert; is the input unnormalized score on the first expert; is the index variable in the summation symbol, indicating the summation of all expert numbers 1, 2,..., K sequentially; E ; is the number of experts; the router generates weights , which are normalized by the softmax function, wherein represents a lightweight training parameter, represents the number of experts, denotes the real number field, denotes the hidden layer size of LLM. Each token is processed by the top experts with the final output being the weighted sum of these expert outputs: ; where, is the final output of MoE module for input ; is the number of experts actually selected; is the index variable; is the output of the expert network for input .

[0032] The model takes a visual encoder, a visual projection layer, and a word embedding layer as core modules, effectively fusing visual and textual features through depth, improving the understanding of complex scenes. At the same time, MSA and FFN are used as the basic modules of the large language model, and MoE layers are introduced to enhance adaptability. This design not only effectively captures long-range dependencies in sequences through the combination of MSA and FFN, but also enhances the flexibility and computational efficiency of the model through the expert parallel mechanism of the MoE layer, while avoiding the limitations of single FFN when processing multi-modal data. Ultimately, the large language model outputs high-quality features through layer normalization, significantly improving the processing capability of visual and textual joint tasks.

[0033] II. Construction of training data set; The remote sensing data set for model training is divided into two stages: stage I aggregates existing remote sensing data sets across various tasks to form a basic data set; stage II uses data augmentation techniques (including count change clipping and attribute editing) to improve data set diversity for fine-tuning of the mixed expert component.

[0034] (1) Count change clipping (see Figure 3 (a)): Taking the DOTAv2 training subset P2739.jpg in the training set as an example, samples with count differences are generated by randomly masking part of the objects and updating the annotations, thereby enhancing the model's adaptability to different object densities (while maintaining the background semantics unchanged). The specific steps are as follows: First, sample a ratio , which is uniformly distributed (i.e. randomly between 0.15 and 0.30); then calculate the number of objects to be masked as , where is the original count (forced to satisfy ). Select m objects for zero padding (i.e. set the pixel value of the object's pixel point to 0, denoted as where x, y is the pixel coordinate), while updating the object count in the annotation as Four variants are generated for each image, and the count difference of each variant is The newly generated P2739_1.jpg to P2739_N.jpg (N=4) are stored in the DOTAv2 training subset.

[0035] (2) Attribute editing (see Figure 3 (b) in the above): Forced visual-linguistic alignment through position manipulation and color manipulation to improve the model's understanding of complex scenes while ensuring pixel-level fidelity and visual consistency.

[0036] For position manipulation, given a pointer expression (e.g., "car at the bottom"), first parse and replace the position phrase in the pointer expression (e.g., "bottom" -> "top right"), then map the modified position phrase to the target region of the 3x3 image grid. The corresponding object is repositioned to a random valid position within the target sector, satisfying two constraints: maintaining a 5% margin from the sector boundary, and maintaining the intersection-over-union ratio with other objects. Apply the Poisson blending technique to maintain visual consistency, then update all affected object bounding box annotations.

[0037] For color manipulation, replace the color term in the pointer expression (e.g., "green" -> "red"), and perform the corresponding color palette transfer on the image object, changing the hue value while maintaining the material reflectance property.

[0038] The present application adopts count change cutting and attribute editing as the data augmentation method of the model, which has the significance that: count change cutting enhances the model's adaptability to different object density scenes by generating samples with different object quantities; attribute editing improves the model's understanding and matching ability for complex semantic descriptions through cross-modal alignment operations of position and color; both methods improve the diversity of the dataset and optimize the model's generalization performance in multi-modal tasks.

[0039] III. Model training The present model adopts a two-stage training strategy, combining pre-training initialization and sparsification architecture optimization, while designing a total loss function containing autoregressive loss and auxiliary loss to improve the model's adaptability, efficiency and multi-modal understanding performance in remote sensing tasks.

[0040] (1) Two-stage training strategy (including stage I and stage II, see Figure 2 ); Stage I training initializes the network with pre-trained weights and fine-tunes specific parts for remote sensing related tasks. Specifically, the model is initialized with a pre-trained CLIP-ViT(L-14) encoder trained on a large amount of text and visual data, a pre-trained MLP adapter trained on a 558K subset of the LAION-CC-SBU dataset via BLIP captioning, and Vicuna-v1.5. To adapt to remote sensing images, LoRA (Low-Rank Adaptation) technique is used to fine-tune the query (Q) and value (V) matrices of the LLM, setting the specified rank to 64; meanwhile, the parameters of the MLP adapter and CLIP-ViT(L-14) encoder are frozen during training to maintain stability. Throughout the process, the model is always trained at an image resolution of 504x504 to match the input size supported by the visual encoder.

[0041] Stage II training converts the model into a sparse VLM by introducing a MoE (Mixture-of-Experts) architecture. The MoE is initialized by replicating the FFN multiple times; MoE layers are interleaved with standard MLP layers, specifically replacing a standard MLP layer with a MoE layer every other layer. When visual tokens and text tokens are input into the MoE layer, the router calculates the matching weights (probability distribution) between each token and the experts; each token is processed by the top 2 experts in terms of weights, and the weighted sum of the outputs of these two experts is output; the unselected experts remain inactive, and the expert parallelism size is set to 1 to support expert parallel execution in a distributed environment. This stage achieves efficient computation and task adaptation through dynamic sparsity and expert parallelism. Not only does it enhance the model's reasoning ability in complex visual tasks, but it also significantly reduces computational resource consumption through sparsification, improving the model's applicability and controllability in the remote sensing field.

[0042] (2) Training objectives; The model training objectives include autoregressive loss and auxiliary loss to optimize the model's generation ability and the allocation efficiency of expert resources. The total loss function is composed of autoregressive loss and auxiliary loss , and the auxiliary loss is scaled by a balance coefficient : ; The autoregressive loss is used to optimize the LLM's generation ability by constraining the model to generate high-quality output sequences in order. Given visual tokens and text tokens , the model generates an output sequence by sequentially generating each element where is, is a real field, K = P + N denotes the length of the output sequence, P is the conditional input sequence length, N is the sequence length generated by the model, D is the hidden size of the LLM. The loss formula is: ; where, N is the target sequence length generated by the model; i is the index of the token currently generated; is the conditional probability distribution indicating that the model parameter is θ , where represents trainable parameters, and the loss is only calculated for the generated text; is the token that the model currently wants to predict; is the visual input feature; is the history text of the generated sequence to the i −1th token.

[0043] Auxiliary loss is used to impose a load balancing constraint on the MoE layer, promoting balanced processing of tokens among experts to avoid performance bottlenecks caused by uneven resource allocation. A differentiable load balancing loss is integrated in each MoE layer to promote balanced processing of tokens among experts. The loss formula is: ; where, is the total number of elements considered in the auxiliary loss; i is the index of the current element in the auxiliary loss; is the predicted value or model output of the i th element in the auxiliary loss; is the average routing probability of , which is as follows: ; ; where, is the proportion of tokens processed by each expert ; is the sequence length or total token number; is an indicator function to determine whether it is true; is the prediction probability distribution of the model for the input ; is the probability distribution each element of the is averaged to obtain a normalized scalar value for the auxiliary loss or regularization; for input the probability of belonging to the class.

[0044] The autoregressive loss can improve the performance of the model in the generation task, and the auxiliary loss optimizes the allocation efficiency of the expert resources, and the combination of the two can significantly improve the training effect and inference performance of the model.

[0045] The two-stage training strategy and loss function design of the application have clear directionality: stage I initializes the pre-training model and fine-tunes LoRA, which quickly adapts to the remote sensing scene while retaining the general visual-linguistic ability, and freezes the key modules to ensure stability; as shown in the reference answer text "This is a satellite aerial photo of a city, the center of the picture is a large ring-shaped traffic hub, and many wide roads radiate from the ring island. There is a large playground and runway on the upper left, and a sports field or sports facility next to it; the lower and right sides are densely distributed high-rise buildings…", the training adopts autoregressive loss. Stage II introduces MoE architecture, which improves the inference ability of the model in complex visual tasks while reducing computational consumption through dynamic sparsity (only activating part of the experts) and expert parallelism; the combination of autoregressive loss and auxiliary loss optimizes the generation ability of the model and balances the allocation of expert resources, ultimately improving the training effect and inference performance of the model. As shown in the reference answer text "This picture shows a city center. The picture in the middle is a huge ring-shaped intersection; above the image, a standard sports field can be seen, with coordinates [15, 13, 33, 25|2]; the right side of the area is lined with buildings, which seems to belong to a large school or research institution.", the output is "This is a satellite aerial photo of a city, the center of the picture is a large ring-shaped traffic hub, and many wide roads radiate from the ring island. There is a large playground and runway on the upper left, and a sports field or sports facility next to it; the lower and right sides are densely distributed high-rise buildings…", the training adopts autoregressive loss+auxiliary loss, which optimizes the generation ability and balances the allocation of expert resources. Figure 2 Figure 2

[0046] The specific implementation of the application is described in detail in conjunction with specific embodiments.

[0047] Embodiment 1; 1. Qualitative comparison; In this study, the performance of the remote model, GeoChat, and VHM was qualitatively evaluated, and the results are shown in Figure 4 ​​As shown, through different remote sensing image understanding tasks, the performance differences of the model in image understanding, scene recognition and object positioning are compared. In the baseball field number recognition task, facing the same remote sensing image of the scene, the remote dream large model accurately identifies 4 baseball fields, GeoChat identifies 2, and VHM identifies 3, which reflects that the remote dream large model is more accurate in judging the number of target objects. When analyzing the basketball court remote sensing image, the remote dream large model focuses on the basketball court in the center of the image, which is simple and accurate to capture the core scene; GeoChat supplements the identification of the tennis court to show the multi-element analysis capability, but the remote dream large model is more direct and efficient in core feature extraction; VHM is only described as a bird's eye view of the basketball court, and the details of the scene are not fully excavated. When judging the scene to which the image belongs, the remote dream large model and GeoChat both stably identify as a bridge, and VHM misjudges as a highway, indicating that the remote dream large model has high stability in identifying bridge-type scenes and is comparable to GeoChat. In the largest basketball court positioning task in the westernmost part, the remote dream large model's marked area is closer to "GroundTruth", and in complex remote sensing images, it combines the understanding of spatial relationships and target features more accurately, and has higher positioning accuracy.

[0048] As can be seen, the remote dream large model has significant advantages in remote sensing image understanding tasks: it can effectively combine visual information and language description, accurately identify the number of target objects, directly and efficiently extract core scene features, has high positioning accuracy in complex scenes, and performs better than GeoChat and VHM in scene recognition and object positioning. It has good consistency in different scenes and outstanding generalization ability. However, the model may have some errors in some complex scenes, which may be related to image resolution, lighting conditions, or object occlusion. Future research can focus on fine-tuning the model and multi-modal information fusion to further improve the performance of the model.

[0049] 2. Quantitative comparison; To verify the performance of the model, the present application compares it with 10 representative large models on 19 public remote sensing benchmarks: 4 general multi-modal large models (MiniGPT-v2, LLaVA-1.5, Qwen2.5-VL, DeepSeek-VL) and 6 remote sensing special large models (GeoChat, VHM, SkySenseGPT, LHRS-Bot, RSUniVLM, EarthDial) in Figure 5 Only the best and worst results of these comparison models are shown in the table. The tasks cover four categories: vision grounding (VG), image captioning (IC), object counting (OC), and vision question answering (VQA).

[0050] Figure 5 The radar chart of the RemoteReason model is marked by a blue circle, and the State-of-the-Art (SOTA) result of the contrast model is marked by a green circle. The RemoteReason model is located in the outermost ring in all test sets, and obviously surrounds the blue circle and the green circle, showing overall leadership.

[0051] In the visual question answering task, the RemoteReason model has excellent image semantic understanding and complex reasoning ability, and achieves the highest accuracy in RSVQA-LR, RSVQA-HR and Eval. In the visual positioning aspect, DIOR-RSVG, AVVG, RRSIS-D, RSVG, VRSBench and Eval all use Pr@0.5 as the index, and the RemoteReason model is better than the blue circle, and the spatial target distribution modeling and position matching ability are outstanding. The image description task covers UCM-Captions, RSICD, NWPU-Captions, Sydney-Captions and Eval, and the RemoteReason model leads in the METEOR index, and the consistency and richness of the generated are significantly better than existing models. The target counting task includes RSOD, NWPU-VHR, DOTAv2, VisionDrone, Eval and other multi-scale dense scenes, and the RemoteReason model still occupies the outermost ring, showing excellent fine-grained target recognition and quantity estimation ability.

[0052] Overall, the RemoteReason model not only achieves overall performance improvement in the multi-task of remote sensing, but also performs much better than the current mainstream general large model and remote sensing special model. This result fully proves the importance and feasibility of building a task-unified, strong expression and excellent generalization performance large model system in the remote sensing scene, and also shows the potential and value of the RemoteReason model as a unified multi-modal basic model in the remote sensing intelligent understanding task.

[0053] The above is only a preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which should be considered as the protection scope of the present application, these will not affect the effect and practicality of the patent.

Claims

1. A remote sensing multimodal reasoning method based on a hybrid expert mechanism, characterized in that, Includes the following steps: Model Construction: A hybrid expert-enhanced geospatial perception visual language foundation model is constructed. This foundation model comprises a visual encoder, a visual projection layer, a word embedding layer, and a large language model based on hybrid expert modules. The large language model based on hybrid expert modules consists of multiple stacked LLM blocks and hybrid expert layers. The visual encoder processes the input image and generates a sequence of visual tokens. The visual projection layer aligns the dimensions of the visual tokens with the hidden dimensions of the LLM. The word embedding layer generates a sequence of text tokens, achieving multimodal feature fusion. The large language model based on hybrid expert modules processes the concatenated visual and text token sequences through the hybrid expert layers to enhance the model's adaptability and computational efficiency. Training dataset construction: Constructing a remote sensing dataset for model training includes two phases: Phase I forms a base dataset by aggregating existing remote sensing datasets across tasks, and Phase II uses data augmentation techniques to enhance dataset diversity. Model training: A two-stage training strategy is adopted, which combines pre-training initialization and sparse architecture optimization, and a total loss function including autoregressive loss and auxiliary loss is designed to train the model.

2. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The visual encoder is built on a pre-trained CLIP-ViT(L-14) backbone network, supports input image sizes of 504×504, segments the image into 1296 patches, and generates a visual token sequence as follows: ； In the formula, A visual token sequence; For a single visual token; For the real number field; The length of the visual marker sequence; The feature dimensions for each visual token.

3. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The visual projection layer is a multilayer perceptron adapter; let the visual projection layer be a function. f , transforming visual tokens from dimensions Hidden dimensions mapped to LLM D This allows visual tokens and text tokens to reside in the same feature space.

4. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The text token sequence generated after the word embedding layer processes the text is represented as follows: token ; In the formula, token A sequence of text tokens; For text-based visual tokens; For the real number field; The length of the text token sequence; This represents the hidden layer size of the LLM.

5. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, In the large language model based on hybrid expert modules, visual tokens and text tokens are concatenated into a combined sequence. The input is then fed into an LLM block, which consists of stacked multi-head self-attention and feedforward neural network blocks; each LLM block contains layer normalization and residual connections; the hybrid expert layer distributes tokens to different expert processes via a router, and the probability distribution generated by the router is represented as follows: ； In the formula, The probability distribution assigned to each expert for each label; For input In the Unnormalized scores on individual experts; For input In the Unnormalized scores on individual experts; The index variable in the summation symbol represents the summation over all expert numbers 1, 2, ... E Summing in sequence; The number of experts; each token is selected by the highest probability. After processing by several experts, the final output is: ； In the formula, For the MoE module to input The final output; The actual number of experts selected; For index variables; For the first A network of experts on the input The output.

6. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, The data augmentation techniques include count change clipping and attribute editing; count change clipping generates count difference samples by randomly masking parts of the object and updating annotations; attribute editing forces visual-linguistic alignment through position and color operations.

7. The remote sensing multimodal reasoning method based on a hybrid expert mechanism according to claim 1, characterized in that, In the two-stage training strategy, Stage I initializes the network with pre-trained weights and fine-tunes the LLM matrix using LoRA technology; Stage II introduces a hybrid expert architecture to convert the model into a sparse visual language model, with hybrid expert layers and standard MLP layers interleaved, specifically replacing the standard MLP layer with a MoE layer every other layer. The training objectives include autoregressive loss and auxiliary loss, and the total loss function is: ； In the formula, This is the total loss function; The autoregressive loss is used to optimize the generation capability by constraining the model to generate output sequences sequentially. To mitigate losses, the auxiliary loss mechanism optimizes resource utilization efficiency by balancing token allocation among experts.

Images