Remote sensing large model performance improvement method based on multi-modal multi-scale adaptive data fusion

Abstract

The invention provides a remote sensing large model performance improvement method based on multi-modal multi-scale adaptive data fusion, and relates to the technical field of remote sensing image large language models. According to the method, a network structure is rewritten based on an open-source visual universal language model, an MFAM-SPP layer used for achieving multi-modal fusion and multi-scale self-adaption is added, a remote sensing large model with improved performance is obtained, data flow and dimension conversion logic among ViT, MFAM-SPP, BERT and Q-Former are defined, and under the condition that the speed is not excessively reduced, the remote sensing large model with improved performance is obtained. The multi-modal processing capability and the multi-scale target detection capability are improved, and the precision of the whole model is greatly improved.

Description

Technical Field

[0001] This application relates to the field of large language model technology for remote sensing images, and in particular to a method for improving the performance of large remote sensing models based on multimodal and multiscale adaptive data fusion. Background Technology

[0002] In recent years, the rapid development of LLM (Large Language Model) has injected new vitality into the field of artificial intelligence, giving rise to LLM-based AI such as Visual ChatGPT (Visual Chat Generative Pre-trained Transformer), WorldGPT (Worldwide Generative Pre-trained Transformer), HuggingGPT (Hugging Face Generative Pre-trained Transformer), and TreeGPT (Tree-based Thinking Generative Pre-trained Transformer). This type of AI, with LLM as its core, can accurately understand user input, formulate reasonable plans and solutions, and efficiently complete various complex tasks with the help of toolchains. For example, Visual ChatGPT successfully implemented visual question answering functionality through structured prompting interaction with ChatGPT (Chat Generative Pre-trained Transformer) and collaborative cooperation among multiple models.

[0003] However, existing methods in the field of remote sensing data interpretation have many shortcomings. Traditional remote sensing algorithms are usually designed for specific application scenarios, and their deployment and operation often rely on the operation of professional technicians. This makes it difficult for ordinary users to interpret remote sensing data independently, which greatly limits the widespread application of remote sensing data.

[0004] Therefore, in recent years, many LLM and VLM (Visual Language Model) models have emerged, demonstrating significant application potential in remote sensing tasks. Combining these two models forms the large-scale remote sensing model. This large-scale model possesses the ability to integrate multiple high-performance remote sensing image processing models, enabling it to flexibly select appropriate tools based on different problems and effectively address diverse task requirements. Its multi-tool integration characteristic provides richer means for solving complex remote sensing problems, greatly enhancing the possibility of handling complex tasks. Remote sensing AI also supports multi-turn dialogue and can understand the context of the dialogue. This interactive advantage helps to better understand user intent, provide services that better meet user needs, make the remote sensing interpretation process more intelligent and human-centered, improve the interaction efficiency between users and the remote sensing interpretation system, enhance user experience, and lay the foundation for widespread application.

[0005] However, current large-scale remote sensing models, such as the classic SkyEyeGPT model and EarthGPT model, still have the following problems: Insufficient multimodal processing capabilities: Existing large remote sensing models are mostly designed for single-modal data (such as processing only optical images), and cannot efficiently fuse multimodal data such as optical, SAR (Synthetic Aperture Radar), and hyperspectral data. The polarization characteristics of SAR data and the spectral characteristics of hyperspectral data are difficult to coordinate with the spatial characteristics of optical data, resulting in low accuracy in cross-modal tasks (such as "using SAR data to verify the building damage areas identified by optical data").

[0006] Low accuracy in multi-scale target recognition: Remote sensing images contain targets with extremely large scale differences (from meter-level utility poles to kilometer-level lakes). Existing models use fixed-scale feature extraction, such as the traditional SPP (Spatial Pyramid Pooling) with fixed pooling of 1×1, 2×2, and 4×4. Small targets (such as 5×5 pixel boats) are prone to losing details due to pooling, while large targets (such as 100×100 pixel farmland) are prone to semantic ambiguity due to local feature redundancy, ultimately affecting the accuracy of detection and classification.

[0007] Therefore, the aforementioned technical problems urgently need to be solved. Summary of the Invention

[0008] In view of the above problems, this application is made to provide a method, apparatus, and related products for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion, which overcomes or at least partially solves the above problems. The technical solution is as follows: Firstly, a method for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion is provided, the method comprising: Based on an open-source visual universal language model, an initial remote sensing large model is constructed to adapt to multimodal remote sensing image data and text data. The architecture of the initial remote sensing large model includes a visual encoder ViT, a text editor BERT, a query converter Q-Former layer, a low-rank cyclic inference computation layer, and a dynamic threshold filtering system. By adding a multimodal adaptive multiscale spatial pyramid pooling (MFAM-SPP) layer to the initial large remote sensing model to achieve multimodal fusion and multiscale adaptation, a large remote sensing model with improved performance is obtained. In response to a user's input of a question containing multimodal remote sensing image data and text data, the multimodal remote sensing image data in the question is preprocessed to obtain a multimodal preprocessing matrix; The multimodal preprocessing matrix is ​​input into the ViT in the performance-enhanced remote sensing large model, and the steps of input adaptation, hierarchical coding, interference suppression, and cross-modal connection are performed to output a visual feature vector. The visual feature vector is input into the MFAM-SPP layer, and the steps of multimodal fusion, dynamic scale adaptive pooling, and feature aggregation are performed to output the MFAM-SPP feature matrix. The text data from the question is input into BERT in the performance-enhanced remote sensing big data model, and the output is a text feature vector. The MFAM-SPP feature matrix and text feature vector are input into the Q-Former layer of the large remote sensing model to improve performance. The steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion are performed to output a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the low-rank cyclic inference computation layer of the performance-enhanced remote sensing large model. The steps of low-rank matrix decomposition and cyclic inference are performed to output the inference feature matrix. Then, based on the inference feature matrix, the preliminary inference results containing confidence and intersection-union ratio (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system in the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results. Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the MFAM-SPP layer, the parameters of the dynamic scale adaptive pooling step are adjusted, and the MFAM-SPP layer and the subsequent Q-Former layer and low-rank cyclic inference calculation layer process are re-executed.

[0009] In one possible implementation, the multimodal remote sensing image data includes optical data, synthetic aperture radar (SAR) data, and hyperspectral data; The visual feature vector is a three-channel feature vector including spatial channel, spectral channel and polarization channel. The spatial channel corresponds to the spatial structure features of optical data or SAR data, the spectral channel corresponds to the band information of hyperspectral data, and the polarization channel corresponds to the polarization parameters of SAR data. The visual feature vector is input into the MFAM-SPP layer, which performs multimodal fusion, dynamic scale adaptive pooling, and feature aggregation to output the MFAM-SPP feature matrix, including: The three-channel feature vector is input into the MFAM-SPP layer to perform a multimodal fusion step. Differential pooling is performed on each channel of the three-channel feature vector to obtain the three-channel pooling result. A cross-modal attention layer is introduced to calculate the similarity of the three-channel pooling results. Based on the similarity of the three-channel pooling results, the attention weights of each pair of channels are calculated. Then, feature interaction is performed on each channel based on the attention weights of each pair of channels. The interacted channels are concatenated according to the channel dimension to generate multimodal fusion features. The process involves performing dynamic scale adaptive pooling, determining the dominant scale type and pooling scale combination based on multimodal fusion features, dynamically adjusting the pooling window and step size based on the preset scale type and scale dynamic adjustment relationship, and learning scale weights through an attention mechanism to generate multi-scale pooling features. The feature aggregation step is performed to aggregate multimodal fusion features and multi-scale pooling features, and output the MFAM-SPP feature matrix.

[0010] In one possible implementation, differential pooling is performed on each channel of the three-channel feature vector to obtain the three-channel pooling result, including: For the spatial channels of the three-channel feature vector, max pooling is used to preserve the spatial features of the target edge and contour, and the spatial channel pooling result is obtained. For the spectral channels of the three-channel feature vector, average pooling is used to smooth spectral noise and preserve the continuous features between bands, resulting in the spectral channel pooling results. For the polarization channels of the three-channel feature vector, a hybrid pooling method of max and average is used to preserve the key extreme value features of polarization entropy and avoid extreme value anomalies caused by speckle noise, thus obtaining the polarization channel pooling results.

[0011] In one possible implementation, a cross-modal attention layer is introduced to calculate the similarity of the three-channel pooling results. Based on this similarity, attention weights are calculated for each pair of channels. Then, feature interactions are performed on each channel based on these pairwise attention weights, including: A cross-modal attention layer is introduced to calculate the cosine similarity of the three-channel pooling results; The attention weights for each pair of channels are calculated based on the similarity of the three-channel pooling results as follows: ; in, S It is the result of spatial channel pooling; Sp This is the result of spectral channel pooling; P This is the result of polarization channel pooling; A, B Take respectively S, Sp, P; yes A, B Attention weights for each pair of channels; The feature interactions between channels are performed based on the attention weights of each pair of channels as follows: ; ; ; in, S' It is the result of spatial channel pooling after interaction; Sp' This is the result of the interactive spectral channel pooling. P' It is the result of polarization channel pooling after interaction.

[0012] In one possible implementation, the dominant scale type and pooling scale combination are determined based on multimodal fusion features, including: A pre-defined lightweight convolutional neural network is used, with input being multimodal fusion features, including metadata such as resolution and sensor type, and output being the dominant scale type; The pooling scale combination is determined based on the dominant scale type.

[0013] In one possible implementation, the preset scale types include a first scale, a second scale, and a third scale; the dominant scale type includes one or more of the first scale, the second scale, and the third scale; and the pooling scale includes a first type of granularity, a second type of granularity, and a third type of granularity. First type of particle size F Second type of particle size M and third type of granularity C The weight formulas are as follows: ; ; ; in, It is the percentage of pixels at the first scale; It is the percentage of pixels at the second scale; It is the percentage of pixels at the third scale.

[0014] Secondly, a device for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion is provided, the device comprising: The building unit is used to build an initial large remote sensing model based on an open-source visual general language model, adapted to multimodal remote sensing image data and text data. The architecture of the initial large remote sensing model includes a visual encoder ViT, a text editor BERT, a query converter Q-Former layer, a low-rank loop inference computation layer, and a dynamic threshold filtering system. The boosting unit is used to add a multimodal adaptive multiscale spatial pyramid pooling (MFAM-SPP) layer to the initial large remote sensing model to achieve multimodal fusion and multiscale adaptation, resulting in a large remote sensing model with improved performance. The inference unit, in response to a user-input query containing multimodal remote sensing image data and text data, preprocesses the multimodal remote sensing image data in the query to obtain a multimodal preprocessing matrix; and The multimodal preprocessing matrix is ​​input into the ViT module of a performance-enhanced remote sensing model, performing input adaptation, hierarchical coding, interference suppression, and cross-modal concatenation steps to output a visual feature vector; and The visual feature vectors are input into the MFAM-SPP layer, where multimodal fusion, dynamic scale adaptive pooling, and feature aggregation are performed to output the MFAM-SPP feature matrix; and The text data from the question is input into BERT, a performance-enhancing remote sensing model, to output a text feature vector; and The MFAM-SPP feature matrix and text feature vectors are input into the Q-Former layer of a performance-enhanced remote sensing model. This layer performs query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, outputting a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the low-rank recurrent inference computation layer of the performance-enhanced remote sensing large model. Low-rank matrix decomposition and recurrent inference steps are performed, outputting the inference feature matrix. Based on this matrix, preliminary inference results including confidence scores and Intersection over Union (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system within the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results; and Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; and If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the MFAM-SPP layer, the parameters of the dynamic scale adaptive pooling step are adjusted, and the MFAM-SPP layer and the subsequent Q-Former layer and low-rank cyclic inference calculation layer process are re-executed.

[0015] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the remote sensing large model performance improvement method based on multimodal multiscale adaptive data fusion as described in any of the preceding claims.

[0016] Fourthly, a storage medium is provided, the storage medium storing a computer program, wherein the computer program is configured to execute the remote sensing large model performance improvement method based on multimodal multiscale adaptive data fusion as described above during runtime.

[0017] Fifthly, a computer program product is provided, including a computer program configured to execute, at runtime, the remote sensing large model performance improvement method based on multimodal multiscale adaptive data fusion as described above.

[0018] Using the above technical solutions, the remote sensing large model performance improvement method, device, and related products based on multimodal and multiscale adaptive data fusion provided in this application embodiment are based on an open-source visual universal language model. The method rewrites the network structure and adds an MFAM-SPP (Multimodal Fusion and Adaptive Multi-scale Spatial Pyramid Pooling) layer to achieve multimodal fusion and multi-scale adaptation. Ultimately, through multimodal processing and dynamic scale adaptation, the following technical effects are achieved: Significantly enhanced multimodal fusion capability: By combining "modal-specific pooling" with "cross-modal attention interaction" in the MFAM-SPP layer, feature synergy of multimodal remote sensing image data (such as optical data, SAR data, and hyperspectral data) is achieved, solving the problems of dimensional imbalance and poor information complementarity of multimodal data. The cross-modal target detection accuracy is improved compared with existing models (such as SkyEyeGPT).

[0019] Multi-scale target adaptability optimization: The MFAM-SPP layer introduces "dynamic scale prediction" combined with "adaptive pooling", which automatically adjusts the pooling scale according to remote sensing image metadata (resolution, sensor type) and preliminary features, greatly improving the retention rate of small target features and improving the global semantic extraction accuracy of large targets.

[0020] Balancing computational efficiency and accuracy: By employing ViT hierarchical encoding and low-rank decomposition techniques with cyclic inference, accuracy is improved while model inference latency is reduced and hardware resource consumption is decreased (e.g., GPU memory consumption is reduced from 16GB to 12GB), thus meeting the requirements of real-time remote sensing interpretation.

[0021] Strong module connectivity: Clearly define the data flow and dimension transformation logic between ViT, MFAM-SPP, BERT, and Q-Former. For example, ViT output features are mapped to the MFAM-SPP adapted dimension via a linear layer, and BERT text features and visual features are unified to 768 dimensions, ensuring lossless feature transfer across modules. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0023] Figure 1a The flowchart illustrates a method for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion, as provided in an embodiment of this application. Figure 1b This document illustrates a flowchart illustrating the performance enhancements provided by the embodiments of this application in enabling the large remote sensing model to complete question-and-answer sessions. Figure 2 This illustration shows a schematic diagram of the specific preprocessing framework provided in an embodiment of this application; Figure 3 This paper illustrates a schematic diagram of the specific framework of the ViT layer provided in an embodiment of this application; Figure 4 This paper illustrates a schematic diagram of the specific framework of the MFAM-SPP layer provided in an embodiment of this application. Figure 5 This paper illustrates a schematic diagram of the specific framework of the Q-Former layer provided in an embodiment of this application. Figure 6 This paper illustrates a schematic diagram of the specific framework of the filtering output layer provided in an embodiment of this application. Figure 7 The diagram shows the structure of the remote sensing large model performance enhancement device based on multimodal and multi-scale adaptive data fusion provided in this application embodiment; Figure 8 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0024] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0026] To address the aforementioned technical problems, embodiments of this application provide a method for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion, such as... Figure 1a As shown, the method for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion may include the following steps S101 to S112: Step S101: Based on the open-source visual general language model, adapt to multimodal remote sensing image data and text data, and construct an initial remote sensing large model. The architecture of the initial remote sensing large model includes a visual encoder ViT, a text editor BERT, a query converter Q-Former layer, a low-rank cyclic inference calculation layer, and a dynamic threshold filtering system.

[0027] In this step, the visual general language model can be an open-source model such as VisualGLM (Visual General Language Model)-6B, and this embodiment does not limit it.

[0028] ViT (Vision Transformer, visual encoder) extracts visual features from images and merges similar features; BERT (Bidirectional Encoder Representations from Transformers) generates text feature vectors; Q-Former (Querying Transformer) connects visual models and language models.

[0029] Multimodal remote sensing image data may include optical data, SAR data, hyperspectral data, etc., and this embodiment does not limit this.

[0030] Step S102: Add a multimodal adaptive multiscale spatial pyramid pooling (MFAM-SPP) layer to the initial large remote sensing model to achieve multimodal fusion and multiscale adaptation, resulting in a large remote sensing model with improved performance.

[0031] Step S103: In response to the user's input of a question containing multimodal remote sensing image data and text data, the multimodal remote sensing image data in the question is preprocessed to obtain a multimodal preprocessing matrix.

[0032] Step S104: Input the multimodal preprocessing matrix into the ViT in the performance-enhanced remote sensing large model, perform input adaptation, hierarchical coding, interference suppression, and cross-modal connection steps, and output visual feature vectors.

[0033] Step S105: Input the visual feature vector into the MFAM-SPP layer, perform multimodal fusion, dynamic scale adaptive pooling, and feature aggregation, and output the MFAM-SPP feature matrix.

[0034] Step S106: Input the text data from the question into BERT in the performance-enhanced remote sensing big data model, and output the text feature vector.

[0035] Step S107: Input the MFAM-SPP feature matrix and text feature vector into the Q-Former layer of the performance-enhanced remote sensing large model, perform query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, and output the cross-modal feature matrix.

[0036] Step S108: Input the cross-modal feature matrix into the low-rank cyclic inference computation layer of the performance-enhanced remote sensing large model, perform low-rank matrix decomposition and cyclic inference steps, output the inference feature matrix, and then obtain the preliminary inference result containing confidence and intersection-union ratio (IOU) based on the inference feature matrix.

[0037] In this step, IOU (Intersection Over Union) is a metric used in object detection tasks to measure the degree of overlap between the predicted bounding box and the ground truth bounding box.

[0038] Step S109: Input the preliminary inference results into the dynamic threshold screening system in the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results.

[0039] In this step, the target task corresponding to the preliminary inference result can be determined by the text feature vector.

[0040] Step S110: Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task.

[0041] Step S111: If the preliminary inference result meets the threshold requirements of the target task, then perform post-processing operations and output the final inference result containing the target coordinates, target category, and confidence level.

[0042] Step S112: If the preliminary inference result does not meet the threshold requirements of the target task, the preliminary inference result is sent back to the MFAM-SPP layer, the parameters of the dynamic scale adaptive pooling step are adjusted, and the MFAM-SPP layer and the subsequent Q-Former layer and low-rank cyclic inference calculation layer process are re-executed.

[0043] This embodiment rewrites the network structure based on the open-source visual general language model. By adding the MFAM-SPP layer to achieve multimodal fusion and multi-scale adaptation, the following technical effects are achieved through multimodal processing and dynamic scale adaptation: Significantly enhanced multimodal fusion capability: By combining "modal-specific pooling" with "cross-modal attention interaction" in the MFAM-SPP layer, feature synergy of multimodal remote sensing image data (such as optical data, SAR data, and hyperspectral data) is achieved, solving the problems of dimensional imbalance and poor information complementarity of multimodal data. The cross-modal target detection accuracy is improved compared with existing models (such as SkyEyeGPT).

[0044] Multi-scale target adaptability optimization: The MFAM-SPP layer introduces "dynamic scale prediction" combined with "adaptive pooling", which automatically adjusts the pooling scale according to remote sensing image metadata (resolution, sensor type) and preliminary features, greatly improving the retention rate of small target features and improving the global semantic extraction accuracy of large targets.

[0045] Balancing computational efficiency and accuracy: By employing ViT hierarchical encoding and low-rank decomposition techniques with cyclic inference, accuracy is improved while model inference latency is reduced and hardware resource consumption is decreased (e.g., GPU memory consumption is reduced from 16GB to 12GB), thus meeting the requirements of real-time remote sensing interpretation.

[0046] Strong module connectivity: Clearly define the data flow and dimension transformation logic between ViT, MFAM-SPP, BERT, and Q-Former. For example, ViT output features are mapped to the MFAM-SPP adapted dimension via a linear layer, and BERT text features and visual features are unified to 768 dimensions, ensuring lossless feature transfer across modules.

[0047] Figure 1b This document illustrates a flowchart illustrating the performance improvement of the large remote sensing model provided in this application, which completes a question-and-answer process. Figure 1b The process includes input image data and preprocessing, ViT visual feature extraction, input to the MFAM-SPP layer, input text and BERT text editing, aligning the output of the MFAM-SPP layer with the output of the BERT text editing input to the Q-Former layer for multimodal feature calculation, iterative inference calculation, matching score filtering. If the matching score is greater than the threshold, the result is post-processed and output; if the matching score is less than or equal to the threshold, it is fed back to the MFAM-SPP layer, the parameters of the dynamic scale adaptive pooling step are adjusted, and the MFAM-SPP layer and subsequent Q-Former layer and low-rank iterative inference calculation layer process are re-executed.

[0048] This application provides a possible implementation method. The multimodal remote sensing image data may include optical data, SAR data, and hyperspectral data. The visual feature vector is a three-channel feature vector including a spatial channel, a spectral channel, and a polarization channel. The spatial channel corresponds to the spatial structure features of the optical data or SAR data, the spectral channel corresponds to the band information of the hyperspectral data, and the polarization channel corresponds to the polarization parameters of the SAR data. In step S105 above, the visual feature vector is input into the MFAM-SPP layer, and multimodal fusion, dynamic scale adaptive pooling, and feature aggregation are performed to output the MFAM-SPP feature matrix. Specifically, this may include the following steps A1 to A3: Step A1: Input the three-channel feature vector into the MFAM-SPP layer and perform multimodal fusion. Perform differential pooling operation on each channel of the three-channel feature vector to obtain the three-channel pooling result. Introduce a cross-modal attention layer to calculate the similarity of the three-channel pooling results. Calculate the attention weights of each pair of channels based on the similarity of the three-channel pooling results, and then perform feature interaction on each channel based on the attention weights of each pair of channels. Concatenate the interacted channels according to the channel dimension to generate multimodal fusion features. Step A2 involves performing dynamic scale adaptive pooling, determining the dominant scale type and pooling scale combination based on multimodal fusion features; determining the dynamic adjustment pooling window and step size for the dominant scale type based on the pre-defined correspondence between the scale type and the scale and the pooling scale combination; and learning scale weights through an attention mechanism to generate multi-scale pooling features. Step A3 involves performing feature aggregation, which aggregates multimodal fusion features and multi-scale pooling features to output the MFAM-SPP feature matrix.

[0049] This embodiment achieves feature collaboration among multimodal remote sensing image data (such as optical data, SAR data, and hyperspectral data) by combining "modal-specific pooling" with "cross-modal attention interaction" in the MFAM-SPP layer. This addresses the issues of dimensionality imbalance and poor information complementarity among multimodal data, resulting in improved cross-modal target detection accuracy compared to existing models (such as SkyEyeGPT). Furthermore, the MFAM-SPP layer introduces "dynamic scale prediction" combined with "adaptive pooling," automatically adjusting the pooling scale based on remote sensing image metadata (resolution, sensor type) and preliminary features. This significantly improves the retention rate of small target features and enhances the global semantic extraction accuracy of large targets.

[0050] This application provides a possible implementation method in which differential pooling is performed on each channel of the three-channel feature vector in step A1 above to obtain the three-channel pooling result. Specifically, it may include the following steps A1-1 to A1-3: Step A1-1: Max pooling is applied to the spatial channels of the three-channel feature vectors to preserve the spatial features of the target edges and contours, resulting in spatial channel pooling results. Step A1-2: For the spectral channels of the three-channel feature vector, average pooling is applied to smooth spectral noise and preserve the continuous features between bands to obtain the spectral channel pooling results. Step A1-3: For the polarization channels of the three-channel feature vector, a hybrid pooling of max and average is used to preserve the key extreme value features of the polarization entropy value and avoid extreme value anomalies caused by speckle noise, thus obtaining the polarization channel pooling result.

[0051] This application provides a possible implementation method in which a cross-modal attention layer is introduced in step A1 above, the similarity of the three-channel pooling results is calculated, the attention weights of each pair of channels are calculated based on the similarity of the three-channel pooling results, and then feature interaction is performed on each channel based on the attention weights of each pair of channels. Specifically, it may include the following steps A1-4 to A1-6: Steps A1-4 introduce a cross-modal attention layer and calculate the cosine similarity of the three-channel pooling results; Steps A1-5: Based on the similarity of the three-channel pooling results, calculate the attention weights for each pair of channels as follows: ; in, S It is the result of spatial channel pooling; Sp This is the result of spectral channel pooling; P This is the result of polarization channel pooling; A, B Take respectively S, Sp, P ; yes A, B Attention weights for each pair of channels; Steps A1-6 involve performing feature interactions on each channel based on the attention weights of each pair of channels, as follows: ; ; ; in, S' It is the result of spatial channel pooling after interaction; Sp' This is the result of the interactive spectral channel pooling. P' It is the result of polarization channel pooling after interaction.

[0052] This application provides a possible implementation method in which the dominant scale type and pooling scale combination are determined based on the multimodal fusion features in step A2 above, which may specifically include the following steps A2-1 and A2-2: Step A2-1 uses a pre-defined lightweight convolutional neural network. The input is multimodal fusion features, including metadata such as resolution and sensor type, and the output is the dominant scale type.

[0053] In this step, the preset lightweight convolutional neural network can be a lightweight network such as MobileNetV3, and this embodiment does not limit it.

[0054] Step A2-2: Determine the pooling scale combination based on the dominant scale type.

[0055] This application provides a possible implementation method, wherein the preset scale type may include a first scale, a second scale, and a third scale; the dominant scale type may include one or more of the first scale, the second scale, and the third scale; and the pooling scale includes a first type of granularity, a second type of granularity, and a third type of granularity. First type of particle size F Second type of particle size M and third type of granularity C The weight formulas are as follows: ; ; ; in, It is the percentage of pixels at the first scale; It is the percentage of pixels at the second scale; It is the percentage of pixels at the third scale.

[0056] In this embodiment, the first scale, the second scale, and the third scale can be small target, medium target, and large target, respectively; the first type of granularity, the second type of granularity, and the third type of granularity can be fine granularity, medium granularity, and coarse granularity, respectively.

[0057] The above introduces Figure 1a and Figure 1b The embodiments shown have various implementation methods for each step. The following will further explain the method for improving the performance of remote sensing large models based on multimodal and multi-scale adaptive data fusion in this application through specific embodiments.

[0058] In a specific embodiment, the Visual General Language Model is the VisualGLM-6B open-source model. Based on the architecture of the VisualGLM-6B open-source model, it is adapted to multimodal remote sensing image data and text data to construct an initial large-scale remote sensing model. The architecture of the initial large-scale remote sensing model includes ViT, BERT, Q-Former layers, low-rank cyclic inference computation layers, and a dynamic threshold filtering system. An MFAM-SPP layer is added to the initial large-scale remote sensing model to achieve multimodal fusion and multi-scale adaptation, resulting in a performance-enhanced large-scale remote sensing model. This improves the multimodal processing capability and multi-scale target detection capability without significantly reducing the speed, and greatly improves the overall model accuracy.

[0059] The specific implementation plan is as follows: 1. Image input preprocessing (i.e., multimodal differential processing) See Figure 2 For the three core remote sensing modes—optical data, SAR data, and hyperspectral data—specific preprocessing and unified format conversion are performed to ensure the consistency and validity of input data.

[0060] 1.1 Modal-specific preprocessing Optical images (such as satellite RGB): Radiometric correction: The FLAASH (Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes) atmospheric correction module of ENVI (Environment for Visualizing Image, a software widely used in remote sensing image processing and analysis) is used to eliminate atmospheric scattering and aerosol effects. Geometric correction: Based on ground control points (GCPs), a quadratic polynomial transformation is used to align the image to the WGS84 (World Geodetic Coordinate System 1984) coordinate system, with the error controlled within 1 pixel; Contrast enhancement: CLAHE (Contrast Limit Adaptive Histogram Equalization) is used, with clipLimit (Contrast Limit Threshold) set to 2.0 to enhance the details of ground features at the edges.

[0061] SAR data (such as C-band fully polarimetric SAR data): Speckle noise reduction: Lee filtering (a classic adaptive filtering algorithm with a window size of 5×5) is used to suppress speckle noise; Polarization decomposition: The polarization data is converted into three-channel feature maps, namely "HH+VV, HH-VV, and HV", through Pauli decomposition (a basic polarimetric SAR target decomposition). Here, HH+VV represents the sum of the echoes from horizontal transmission and horizontal reception and vertical transmission and vertical reception; HH-VV represents the difference between the echoes from horizontal transmission and horizontal reception and vertical transmission and vertical reception; and HV represents the echoes from horizontal transmission and vertical reception. Radiation calibration: converting radar echo intensity into backscattering coefficient The formula is: Where K is the calibration coefficient (provided by sensor parameters), DN is a digital value, the original brightness value of the pixel in the SAR image, representing the intensity of the echo.

[0062] Hyperspectral data (such as Hyperion satellite data): Defective pixel repair: The neighborhood mean fill method is used to repair pixels whose gray values ​​deviate from the mean by 3 times the standard deviation within the band. Band selection: Select 20-30 sensitive bands related to the target by using the maximum information coefficient (MIC) (e.g., select NDVI-related bands for vegetation monitoring); here, NDVI is the normalized vegetation index, which is calculated by the reflectance of the near-infrared band and the red light band, and is often used to characterize the vegetation growth status. Dimensionality reduction: PCA (Principal Component Analysis) was used to reduce the screened bands to 10 dimensions, retaining more than 95% of the information.

[0063] 1.2 Unified Format Conversion Adaptive scaling: A bilinear interpolation algorithm is used to uniformly scale all modal data to 512×512 pixels. This is just an illustration; other pixels may also be used. Pixel normalization: Min-max normalization is used, and the formula is: This maps all pixel values ​​to the [0,1] range, eliminating brightness and intensity differences between different modalities; among them, It is the normalized value, where x is the pixel value. It is the minimum pixel value. It is the maximum pixel value; 3D matrix construction: The preprocessed data is converted into a 3D matrix (height H × width W × number of channels C), where: H=512, W=512, number of channels C=3 for optical data, number of channels C=3 for polarization data, and number of channels C=10 for hyperspectral data, which will be used as input for subsequent ViT.

[0064] See ViT visual encoder (feature extraction and interference suppression) Figure 3 It receives a multimodal preprocessing matrix and outputs a high-quality visual feature vector through four steps: input adaptation, hierarchical coding, interference suppression, and cross-modal connection.

[0065] 2.1 Remote Sensing Dedicated Input Adaptation Dynamic image block segmentation: dynamically adjusts block size based on modal characteristics and performs dynamic completion. Optical data or SAR data: 8×8 pixels / block (adapted to small target details); Hyperspectral data: 16×16 pixels / block (balancing spectral continuity and computational efficiency).

[0066] Edge completion: If the image size (512×512) is not an integer multiple of the block size, remote sensing feature continuity interpolation (based on the feature type of adjacent blocks to avoid black border filling) is used to complete the edge blocks, ultimately generating 8×8 image blocks of 64×64, or 16×16 image blocks of 32×32.

[0067] Geo-enhanced location coding: Extract latitude and longitude information from image metadata and calculate the relative coordinates of the center of each image patch. , ),in, , These are the coordinates of the image metadata. It is the minimum value of the x-axis coordinate. It is the maximum value of the x-axis coordinate. It is the minimum value of the y-axis coordinate. It is the maximum value of the y-axis coordinate.

[0068] The design employs a "spatial-geographic dual-dimensional encoding": the spatial dimension uses sinusoidal encoding (preserving the relative positions between blocks), while the geographic dimension uses linear encoding (incorporating latitude and longitude scales), ultimately generating a 768-dimensional location vector, which is added to the image block features (768-dimensional, mapped by a 1×1 convolution). The core purpose of this design is to address the issues of "lack of spatial correlation" and "geographic scale mismatch" in traditional ViT location encoding in remote sensing scenes, ensuring that the ViT visual encoder can accurately capture the "local spatial structure" and "global geographic attributes" of remote sensing images. This provides a feature foundation that combines "spatial details" and "geographic semantics" for subsequent multimodal fusion and multi-scale target recognition.

[0069] 2.2 Hierarchical Transformer Encoding (Multi-scale Target Adaptation) The Transformer encoder is divided into 3 levels (12 levels in total), with each level using a different attention window size: Shallow layer: 4 layers, attention window size is 4×4 image block, extract local details of small targets, adapted to 5-20 pixels (telephone pole, boat). Middle layer: 5 layers, attention window size is 8×8 image block, captures the structural features of the target in the middle, adapted to 20-50 pixels (building, farmland blocks). Deep layer: 3 layers, attention window size is 16×16 image patch, extract global semantics of large targets, adapted to larger than 50 pixels (lakes, mountains). Cross-level feature fusion: The shallow layer output is upsampled by 2 times and the deep layer output is downsampled by 0.5 times. Then, it is fused with the middle layer output through attention weighting (shallow layer weight 0.3, middle layer weight 0.5, deep layer weight 0.2) to generate a 64×64×768 multi-scale feature matrix.

[0070] 2.3 Interference Suppression Optimization Interference region detection: A lightweight U-Net network (input is ViT shallow features, output is a 512×512 binary mask) is used to detect cloud, noise and shadow regions (mask value 1 is interference, 0 is normal). Disturbance-aware attention: In Transformer attention calculation, the image patch with interference is multiplied by an interference suppression coefficient α (α = 0.2-0.4, the higher the interference intensity, the smaller α), as shown in the formula: ;in, , These are the attention levels before and after the interference. It is an interference value; Feature cleanup: For image blocks with interference accounting for more than 50%, the mean of normal features in a 3×3 neighborhood is used to replace them to ensure that the features are not distorted.

[0071] 2.4 Cross-modal transition Through a dynamic linear mapping layer, the multi-scale feature matrix (64×64×768) is converted into a three-channel feature (512×512×256) adapted to MFAM-SPP, consisting of spatial, spectral, and polarimetric channels. The spatial channel corresponds to the spatial structure features of optical or SAR data, the spectral channel corresponds to the band information of hyperspectral data, and the polarimetric channel corresponds to the polarization parameters of SAR data. Spatial channel (optical data or SAR data): 768 dimensions → 256 dimensions (1×1 convolution, 256 convolution kernels); Spectral channels (hyperspectral data): 768 dimensions → 256 dimensions (1×1 convolution + BN layer); Here, BN stands for batch normalization, a common technique in deep learning that accelerates model training and improves stability by standardizing each batch of training data (adjusting the mean to 0 and the variance to 1). Polarization channel (SAR data): 768-dimensional → 256-dimensional (1×1 convolution + GELU activation). Here, GELU stands for Gaussian error linear unit, an activation function that takes into account the probability distribution (normal distribution) of the input values ​​during activation. It is commonly used in modern neural network architectures such as Transformer.

[0072] 3. MFAM-SPP layer (multimodal fusion and multi-scale adaptation) See Figure 4 It receives the three-channel features output by ViT, and solves the problems of multimodal imbalance and multi-scale adaptation by modal fusion, dynamic scale pooling and final feature aggregation. 3.1 Multimodal Feature Adaptation (Core of Modality Fusion) 3.1.1 Modal Channel Separation and Alignment The three-channel features are split according to modality: spatial channel ( S ), spectral channels ( Sp ), polarization channel ( P By sharing a 1×1 convolutional layer (1×1 kernel size, 1 stride, padding=0), the dimensions of each channel are unified to 256 dimensions, avoiding pooling weight imbalance caused by dimensional differences between modalities.

[0073] 3.1.2 Modality-Specific Pooling Strategy Differential pooling is designed based on the characteristics of each modality: Spatial channel (optical data or SAR data): Max pooling is used, with a pooling window size of 3×3, to preserve spatial features such as target edges and contours; Spectral channels (hyperspectral data): average pooling is used with a pooling window size of 5×5 to smooth spectral noise and preserve continuous features between bands; Polarization channel (SAR data): Max-average hybrid pooling is used, with a weight ratio of 0.6:0.4 between the two. The pooling window is selected to be 4×4, which preserves key extreme value features such as polarization entropy value, while avoiding extreme value anomalies caused by speckle noise.

[0074] The core value of optical and SAR data in spatial channels lies in the spatial morphological information of ground features (such as building edges in optical images and coastline outlines in SAR images). These features are typically presented as "local extrema" (e.g., edge pixel grayscale values ​​differ significantly from their surroundings). Max pooling, by selecting the maximum pixel value within a 3×3 window, can directly highlight the spatial structure corresponding to local extrema (e.g., high grayscale values ​​at building corners and strong echo values ​​of ship outlines), avoiding the "smoothing effect" of average pooling on extrema that leads to edge blurring.

[0075] The core value of hyperspectral data lies in the continuous spectral curves between bands (such as the "red edge effect" of vegetation and the spectral absorption peaks of water bodies). The key to these characteristics is the "continuity of pixel values ​​within a band," rather than local extrema. Furthermore, single-band pixels are easily affected by spectral noise (such as random fluctuations caused by instrument errors). Average pooling, by calculating the pixel mean within a 5×5 window, can effectively smooth out random noise in a single band while preserving the overall continuous trend of pixel values ​​within a band (such as the increasing mean value from the red band to the near-infrared band in vegetation areas), avoiding the distortion of spectral curves caused by the "amplification" of noise extrema by max pooling.

[0076] The core value of SAR data in polarimetric channels lies in polarimetric scattering characteristics (such as bistatic scattering of buildings and volume scattering of vegetation). These characteristics must simultaneously consider "key extreme features" (such as polarimetric entropy values ​​and extreme parameters corresponding to scattering angles) and "resistance to speckle noise" (the inherent speckle in SAR data can cause extreme value anomalies). A 0.6-weighted max pooling method is used to preserve polarimetric extremes (such as high polarimetric entropy values ​​reflecting building scattering characteristics), while a 0.4-weighted average pooling method is used to suppress "false extremes" caused by speckle noise (such as abnormally high values ​​generated by random noise). The combination of these two methods achieves a balance between extreme value preservation and noise suppression.

[0077] 3.1.3 Cross-modal attention interaction A cross-modal attention layer is introduced to calculate the cosine similarity of the three-channel pooling results; The attention weights for each pair of channels are calculated based on the similarity of the three-channel pooling results as follows: ; in, S It is the result of spatial channel pooling; Sp This is the result of spectral channel pooling; P This is the result of polarization channel pooling; A, B Take respectively S, Sp, P ; yes A, B Attention weights for each pair of channels; The feature interactions between channels are performed based on the attention weights of each pair of channels as follows: ; ; ; in, S' It is the result of spatial channel pooling after interaction; Sp' This is the result of the interactive spectral channel pooling. P' It is the result of polarization channel pooling after interaction.

[0078] Feature stacking: After interaction S', ​​Sp', P' By splicing along the channel dimension, a 512×512×768 multimodal fusion feature is generated, providing a richer multimodal feature foundation for the subsequent "spatiotemporal fusion module".

[0079] 3.2 Dynamic Scale Adaptive Pooling (Core of Multi-Scale Adaptation) 3.2.1 Scale Prediction Module The MobileNetV3 lightweight network is used, with multimodal fusion features and metadata (resolution, sensor type) as input, and the dominant target scale type (small, medium, large) as output: The resolution is 0.5-2 meters, the sensor type is optical, the dominant scale is small + medium, and the multi-scale pooling scale combination is 4×4 (fine) + 8×8 (medium). The resolution is 5-10 meters, the sensor type is optical or SAR, the dominant scale is medium, and the multi-scale pooling scale is 6×6 (medium). For sensors with a resolution of 10-30 meters, the sensor type is SAR or hyperspectral, the dominant scale is medium + large, and the multi-scale pooling scale combination is 8×8 (medium) + 16×16 (coarse).

[0080] Fine-grained pooling: For small targets (such as a 2-meter-long utility pole, which is 2×2 pixels at 1-meter resolution), small window pooling can preserve its fine-grained features such as edges and top structure, avoiding the large window from completely covering the small target and causing the features to be diluted by average.

[0081] Medium-grained pooling: For medium-sized targets (such as a small building with an actual size of 10 meters, which is 10×10 pixels at a resolution of 1 meter), window pooling can capture its overall outline (such as the four corner features of a rectangular building) while avoiding feature fragmentation caused by small windows (which require multiple stitching to restore the whole building).

[0082] Coarse-grained pooling: For large targets (such as a 300-meter actual size, which is 30×30 pixels at a 10-meter resolution), extract their global semantic features (such as the spectral mean of the entire vegetation area and the trend of the river's direction), avoiding the fragmentation of large target features caused by small window pooling (which requires a large number of windows to be stitched together to restore the global picture, wasting computing resources).

[0083] 3.2.2 Adaptive Pooling Window Adjustment Dynamically adjust the pooling window and step size based on the predicted scale: For small targets, a 3×3 pooling window with a stride of 1 is used to reduce the loss of detail by using a small window. For medium targets, a 5×5 pooling window with a step size of 2 is used to balance detail and efficiency. For large targets, a 7×7 pooling window with a stride of 3 is used, the purpose of which is to improve the efficiency of global feature extraction with a large window.

[0084] In the MFAM-SPP layer, the functions of "multi-scale pooling scale combination" and "adaptive pooling window adjustment" are different. The former is macro-scale planning, which determines which scales of features to extract, while the latter is micro-execution optimization, which determines which window and step size to use for extracting features of each scale. Although both revolve around "scale", they have different positions and ultimately form a collaborative relationship between planning and execution.

[0085] The previous step, "multi-scale pooling scale combination" (such as 4×4 fine + 8×8 medium), is the selection of the macro scale range. The core is to determine "which levels of scale features to extract" and solve the problem of "which types of targets, small, medium and large, should be covered for the current resolution or sensor".

[0086] For example: High-resolution data of 0.5-2 meters needs to cover "small + medium" targets, so a combination of 4×4 (fine, corresponding to small targets) + 8×8 (medium, corresponding to medium targets) is selected, and it is explicitly stated that "16×16 coarse-grained (large target) features will not be extracted" (because there are few large targets at high resolution, extraction is meaningless); Low-resolution data of 10-30 meters needs to cover "medium + large" targets, so 8×8 (medium) + 16×16 (coarse, corresponding to large targets) is selected, and it is explicitly stated that "4×4 fine-grained features will not be extracted" (because small targets cannot be identified at low resolution).

[0087] It does not involve "what specific window size to use to extract a certain type of scale", but only defines the "scale range boundary".

[0088] The current step, "Adaptive Pooling Window Adjustment" (3×3, 5×5, 7×7), is a micro-level execution parameter optimization. Its core is to determine "what window and stride to use for feature extraction at each scale" and solve the problem of "what parameters can be used to extract features most efficiently for a certain type of target (small / medium / large)".

[0089] For example, after determining that "small target features" need to be extracted, it is necessary to further determine "using a 3×3 window with a step size of 1" (instead of 2×2 or 4×4), because 3×3 can balance "avoiding single-point noise" and "preserving details"; after determining that "large target features" need to be extracted, it is necessary to determine "using a 7×7 window with a step size of 3" (instead of 6×6 or 8×8), because 7×7 can cover a sufficiently large area, and a step size of 3 can reduce the amount of computation.

[0090] It optimizes the specific execution details of each scale in the "macroscale combination" without changing the scale type to be extracted.

[0091] 3.2.3 Dynamic Allocation of Scale Weights Learning scale weights through attention mechanisms: setting fine-grained (F ), medium granularity ( M ), and coarse granularity ( C ) pooling results. The weight formula is: ; ; ; Among them, is the pixel ratio of the first scale; is the pixel ratio of the second scale; is the pixel ratio of the third scale; ensure that the weight of the dominant scale feature is the highest (such as in the small target scenario = 0.6, = 0.3, <00​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Base model: bert-base-chinese-remote-sensing (fine-tuned on the remote sensing text dataset RSICD); Feature generation: After the text is encoded by BERT, the output of the [CLS] token is taken as the text feature vector, with a dimension of 768. Semantic enhancement: By embedding remote sensing knowledge graphs (encoding the relationships such as "farmland → vegetation" and "SAR → microwave remote sensing" into vectors) and weighting them with BERT features (with a weight ratio of 0.8:0.2), the semantic relevance of the text is enhanced.

[0096] 5. Q-Former layer (cross-modal feature alignment) See Figure 5 It receives MFAM-SPP features (512×512×768) and BERT text features (768-dimensional), performs query vector interaction, organizes multiple rounds of alignment, and finally fuses features to achieve visual and textual semantic unity. 5.1 Learnable Query Vector Design Initialize 32 learnable query vectors ( Q The query vector has 768 dimensions, consistent with the text feature dimensions. It is optimized through remote sensing scene pre-training (input is a "remote sensing image-text" pair) to ensure that it has the ability to associate vision with text.

[0097] 5.2 Multi-round cross-attention alignment Perform three rounds of cross-attention interaction, each round consisting of two steps: visual guidance text and text-guided visual interaction. Visual guiding text: Visual features are incorporated into the query vector; among them, It is the text feature of the current round. It is a visual feature. Attention () represents cross attention. It is a feature of MFAM-SPP; Text-guided visuals: This maps textual semantics to visual features. Alignment loss calculation: using contrastive learning loss This yields the minimum visual-text feature distance, where These are the textual features of each round.

[0098] 5.3 Cross-modal feature fusion The aligned visual features (512×512×768) and text features (768 dimensions) are fused through dimensional expansion and element-wise multiplication: the text features are expanded to 512×512×768 and then multiplied with the visual features to generate a 512×512×768 cross-modal feature matrix, which is then input into the recurrent inference layer.

[0099] 6. Low-rank cyclic inference computation (efficiency optimization) For cross-modal feature matrices, low-rank decomposition and multi-round inference are used to reduce computational complexity while maintaining accuracy. 6.1 Low-rank matrix decomposition For the inference layer weight matrix (e.g., 1024×1024) after the cross-modal feature matrix input, SVD (Singular Value Decomposition) is used to decompose it into two low-rank matrices: W=W_1×W_2, for example W_1∈R^(1024×64), W_2∈R^(64×1024). In this way, the number of parameters is reduced from 1,048,576 to 131,072 (a reduction of 88%).

[0100] 6.2 Circular Reasoning Mechanism Perform 3-5 rounds of inference iterations, with each round's process as follows: Feature update: F_{t+1}=F_t×W_1×W_2+Residual(F_t) (residual connections preserve the original features); Accuracy verification: Calculate the MAE (mean absolute error) of features in two adjacent rounds. If the MAE < 0.01 or the number of iterations reaches 5, terminate the loop. Output optimization: The final round of features is normalized by LayerNorm to generate a 512×512×768 inference feature matrix.

[0101] 7. Dynamic threshold filtering and result output See Figure 6 The filtering rules are adaptively adjusted according to the type of remote sensing task to ensure the accuracy of the output results. 7.1 Task Scenario Threshold Mapping In one embodiment, a mapping table is established from task type to threshold, wherein: The confidence threshold for the land classification task is ≥0.85, with no IOU threshold, and the core requirement is high boundary accuracy; The confidence threshold for disaster monitoring tasks is ≥0.7, and the IOU threshold is ≥0.5. The core requirement is high recall to avoid missed detections. The confidence threshold for maritime target detection tasks is ≥0.8, and the IOU threshold is ≥0.6. The core requirement is high accuracy in detecting small targets.

[0102] In another embodiment, a general threshold can be preset, such as a confidence threshold of ≥0.8 and an IOU threshold of ≥0.6. This embodiment does not impose any restrictions on this.

[0103] 7.2 Result Screening and Secondary Optimization Valid results: If the inference result meets the threshold requirements of the corresponding task, perform post-processing (such as non-maximum suppression (NMS) to remove duplicate detection boxes and morphological dilation to optimize segmentation boundaries) to generate standardized output (such as target coordinates, category, and confidence score). Invalid result: If the result is below the threshold, the result is sent back to the scale prediction module of the MFAM-SPP layer, the pooling scale is readjusted (e.g., increase fine-grained weights for small target scenes), and the MFAM-SPP layer and subsequent Q-Former layer and loop inference layer processes are re-executed.

[0104] 7.3 Output Format Spatial information: Target coordinates are in WGS84 latitude and longitude (e.g., "116.39°E, 39.91°N"). Attribute information: Encapsulated in JSON format (e.g., {"Target Category":"Farmland","Damage Level":"Moderate","Confidence Level":0.88}); Visualization: Generates a PNG image containing the original image, target annotations, and confidence scores, supporting API (Application Programming Interface) calls and file export.

[0105] This completes the input of image information and text questions.

[0106] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.

[0107] Based on the methods for improving the performance of large remote sensing models based on multimodal and multiscale adaptive data fusion provided in the above embodiments, and based on the same inventive concept, this application also provides a device for improving the performance of large remote sensing models based on multimodal and multiscale adaptive data fusion.

[0108] Figure 7 This is a structural diagram of the remote sensing large model performance enhancement device based on multimodal and multi-scale adaptive data fusion provided in the embodiments of this application. Figure 7 As shown, the remote sensing large model performance enhancement device based on multimodal and multiscale adaptive data fusion may specifically include a construction unit 710, an enhancement unit 720, and an inference unit 730.

[0109] Building unit 710 is used to build an initial large remote sensing model based on an open-source visual general language model, adapted to multimodal remote sensing image data and text data. The architecture of the initial large remote sensing model includes a visual encoder ViT, a text editor BERT, a query converter Q-Former layer, a low-rank cyclic inference computation layer, and a dynamic threshold filtering system. The lifting unit 720 is used to add a multimodal adaptive multiscale spatial pyramid pooling (MFAM-SPP) layer to the initial large remote sensing model to achieve multimodal fusion and multiscale adaptation, resulting in a large remote sensing model with improved performance. Inference unit 730 is configured to respond to a user-input query containing multimodal remote sensing image data and text data, preprocess the multimodal remote sensing image data in the query to obtain a multimodal preprocessing matrix; and The multimodal preprocessing matrix is ​​input into the ViT module of a performance-enhanced remote sensing model, performing input adaptation, hierarchical coding, interference suppression, and cross-modal concatenation steps to output a visual feature vector; and The visual feature vectors are input into the MFAM-SPP layer, where multimodal fusion, dynamic scale adaptive pooling, and feature aggregation are performed to output the MFAM-SPP feature matrix; and The text data from the question is input into BERT, a performance-enhancing remote sensing model, to output a text feature vector; and The MFAM-SPP feature matrix and text feature vectors are input into the Q-Former layer of a performance-enhanced remote sensing model. This layer performs query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, outputting a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the low-rank recurrent inference computation layer of the performance-enhanced remote sensing large model. Low-rank matrix decomposition and recurrent inference steps are performed, outputting the inference feature matrix. Based on this matrix, preliminary inference results including confidence scores and Intersection over Union (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system within the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results; and Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; and If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the MFAM-SPP layer, the parameters of the dynamic scale adaptive pooling step are adjusted, and the MFAM-SPP layer and the subsequent Q-Former layer and low-rank cyclic inference calculation layer process are re-executed.

[0110] This application provides a possible implementation method in which multimodal remote sensing image data includes optical data, synthetic aperture radar (SAR) data, and hyperspectral data; The visual feature vector is a three-channel feature vector including spatial channel, spectral channel and polarization channel. The spatial channel corresponds to the spatial structure features of optical data or SAR data, the spectral channel corresponds to the band information of hyperspectral data, and the polarization channel corresponds to the polarization parameters of SAR data. The inference unit 730 is also used for: The three-channel feature vector is input into the MFAM-SPP layer to perform a multimodal fusion step. Differential pooling is performed on each channel of the three-channel feature vector to obtain the three-channel pooling result. A cross-modal attention layer is introduced to calculate the similarity of the three-channel pooling results. Based on the similarity of the three-channel pooling results, the attention weights of each pair of channels are calculated. Then, feature interaction is performed on each channel based on the attention weights of each pair of channels. The interacted channels are concatenated according to the channel dimension to generate multimodal fusion features. The process involves performing dynamic scale adaptive pooling, determining the dominant scale type and pooling scale combination based on multimodal fusion features, dynamically adjusting the pooling window and step size based on the preset scale type and scale dynamic adjustment relationship, and learning scale weights through an attention mechanism to generate multi-scale pooling features. The feature aggregation step is performed to aggregate multimodal fusion features and multi-scale pooling features, and output the MFAM-SPP feature matrix.

[0111] This application embodiment provides a possible implementation, wherein the inference unit 730 is further configured to: For the spatial channels of the three-channel feature vector, max pooling is used to preserve the spatial features of the target edge and contour, and the spatial channel pooling result is obtained. For the spectral channels of the three-channel feature vector, average pooling is used to smooth spectral noise and preserve the continuous features between bands, resulting in the spectral channel pooling results. For the polarization channels of the three-channel feature vector, a hybrid pooling method of max and average is used to preserve the key extreme value features of polarization entropy and avoid extreme value anomalies caused by speckle noise, thus obtaining the polarization channel pooling results.

[0112] This application embodiment provides a possible implementation, wherein the inference unit 730 is further configured to: A cross-modal attention layer is introduced to calculate the cosine similarity of the three-channel pooling results; The attention weights for each pair of channels are calculated based on the similarity of the three-channel pooling results as follows: ; in, S It is the result of spatial channel pooling; Sp This is the result of spectral channel pooling; P This is the result of polarization channel pooling; A, B Take respectively S, Sp, P ; yes A, B Attention weights for each pair of channels; The feature interactions between channels are performed based on the attention weights of each pair of channels as follows: ; ; ; in, S' It is the result of spatial channel pooling after interaction; Sp' This is the result of the interactive spectral channel pooling. P' It is the result of polarization channel pooling after interaction.

[0113] This application provides a possible implementation method, wherein the preset scale type includes a first scale, a second scale, and a third scale; the dominant scale type includes one or more of the first scale, the second scale, and the third scale; and the pooling scale includes a first type of granularity, a second type of granularity, and a third type of granularity. First type of particle size F Second type of particle size M and third type of granularity C The weight formulas are as follows: ; ; ; in, It is the percentage of pixels at the first scale; It is the percentage of pixels at the second scale; It is the percentage of pixels at the third scale.

[0114] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the remote sensing large model performance improvement method based on multimodal multiscale adaptive data fusion of any of the above embodiments.

[0115] In an exemplary embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8The illustrated electronic device 800 includes a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, for example, via a bus 802. Optionally, the electronic device 800 may also include a transceiver 804. It should be noted that in practical applications, the transceiver 804 is not limited to one type, and the structure of this electronic device 800 does not constitute a limitation on the embodiments of this application.

[0116] Processor 801 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 801 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0117] Bus 802 may include a pathway for transmitting information between the aforementioned components. Bus 802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 802 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0118] The memory 803 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0119] The memory 803 stores computer program code that executes the scheme of this application, and its execution is controlled by the processor 801. The processor 801 executes the computer program code stored in the memory 803 to implement the content shown in the foregoing method embodiments.

[0120] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0121] Based on the same inventive concept, this application also provides a storage medium storing a computer program, wherein the computer program is configured to execute the remote sensing large model performance improvement method based on multimodal multi-scale adaptive data fusion of any of the above embodiments at runtime.

[0122] Based on the same inventive concept, this application also provides a computer program product, including a computer program configured to execute the remote sensing large model performance improvement method based on multimodal and multiscale adaptive data fusion of any of the above embodiments at runtime.

[0123] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0124] Those skilled in the art will understand that the technical solution of this application, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0125] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.

[0126] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.

Claims

1. A method for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion, characterized in that, The method includes: Based on an open-source visual universal language model, an initial remote sensing large model is constructed to adapt to multimodal remote sensing image data and text data. The architecture of the initial remote sensing large model includes a visual encoder ViT, a text editor BERT, a query converter Q-Former layer, a low-rank cyclic inference computation layer, and a dynamic threshold filtering system. By adding a multimodal adaptive multiscale spatial pyramid pooling (MFAM-SPP) layer to the initial large remote sensing model to achieve multimodal fusion and multiscale adaptation, a large remote sensing model with improved performance is obtained. In response to a user's input of a question containing multimodal remote sensing image data and text data, the multimodal remote sensing image data in the question is preprocessed to obtain a multimodal preprocessing matrix; The multimodal preprocessing matrix is ​​input into the ViT in the performance-enhanced remote sensing large model, and the steps of input adaptation, hierarchical coding, interference suppression, and cross-modal connection are performed to output a visual feature vector. The visual feature vector is input into the MFAM-SPP layer, and the steps of multimodal fusion, dynamic scale adaptive pooling, and feature aggregation are performed to output the MFAM-SPP feature matrix. The text data from the question is input into BERT in the performance-enhanced remote sensing big data model, and the output is a text feature vector. The MFAM-SPP feature matrix and text feature vector are input into the Q-Former layer of the large remote sensing model to improve performance. The steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion are performed to output a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the low-rank cyclic inference computation layer of the performance-enhanced remote sensing large model. The steps of low-rank matrix decomposition and cyclic inference are performed to output the inference feature matrix. Then, based on the inference feature matrix, the preliminary inference results containing confidence and intersection-union ratio (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system in the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results. Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the MFAM-SPP layer, the parameters of the dynamic scale adaptive pooling step are adjusted, and the MFAM-SPP layer and the subsequent Q-Former layer and low-rank cyclic inference calculation layer process are re-executed.

2. The method according to claim 1, characterized in that, Multimodal remote sensing image data includes optical data, synthetic aperture radar (SAR) data, and hyperspectral data; The visual feature vector is a three-channel feature vector including spatial channel, spectral channel and polarization channel. The spatial channel corresponds to the spatial structure features of optical data or SAR data, the spectral channel corresponds to the band information of hyperspectral data, and the polarization channel corresponds to the polarization parameters of SAR data. The visual feature vector is input into the MFAM-SPP layer, which performs multimodal fusion, dynamic scale adaptive pooling, and feature aggregation to output the MFAM-SPP feature matrix, including: The three-channel feature vector is input into the MFAM-SPP layer to perform a multimodal fusion step. Differential pooling is performed on each channel of the three-channel feature vector to obtain the three-channel pooling result. A cross-modal attention layer is introduced to calculate the similarity of the three-channel pooling results. Based on the similarity of the three-channel pooling results, the attention weights of each pair of channels are calculated. Then, feature interaction is performed on each channel based on the attention weights of each pair of channels. The interacted channels are concatenated according to the channel dimension to generate multimodal fusion features. The process involves performing dynamic scale adaptive pooling, determining the dominant scale type and pooling scale combination based on multimodal fusion features, dynamically adjusting the pooling window and step size based on the preset scale type and scale dynamic adjustment relationship, and learning scale weights through an attention mechanism to generate multi-scale pooling features. The feature aggregation step is performed to aggregate multimodal fusion features and multi-scale pooling features, and output the MFAM-SPP feature matrix.

3. The method according to claim 2, characterized in that, Differential pooling is performed on each channel of the three-channel feature vector to obtain the three-channel pooling result, including: For the spatial channels of the three-channel feature vector, max pooling is used to preserve the spatial features of the target edge and contour, and the spatial channel pooling result is obtained. For the spectral channels of the three-channel feature vector, average pooling is used to smooth spectral noise and preserve the continuous features between bands, resulting in the spectral channel pooling results. For the polarization channels of the three-channel feature vector, a hybrid pooling method of max and average is used to preserve the key extreme value features of polarization entropy and avoid extreme value anomalies caused by speckle noise, thus obtaining the polarization channel pooling results.

4. The method according to claim 2, characterized in that, A cross-modal attention layer is introduced to calculate the similarity of the three-channel pooling results. Based on this similarity, attention weights are calculated for each pair of channels. Then, feature interactions are performed on each channel based on these pairwise attention weights, including: A cross-modal attention layer is introduced to calculate the cosine similarity of the three-channel pooling results; The attention weights for each pair of channels are calculated based on the similarity of the three-channel pooling results as follows: ； in, S It is the result of spatial channel pooling; Sp This is the result of spectral channel pooling; P This is the result of polarization channel pooling; A, B Take respectively S, Sp, P ; yes A, B Attention weights for each pair of channels; The feature interactions between channels are performed based on the attention weights of each pair of channels as follows: ； ； ； in, S' It is the result of spatial channel pooling after interaction; Sp' This is the result of the interactive spectral channel pooling. P' It is the result of polarization channel pooling after interaction.

5. The method according to claim 2, characterized in that, The dominant scale type and pooling scale combination are determined based on multimodal fusion characteristics, including: A pre-defined lightweight convolutional neural network is used, with input being multimodal fusion features, including metadata on resolution and sensor type, and output being the dominant scale type; The pooling scale combination is determined based on the dominant scale type.

6. The method according to claim 2, characterized in that, The preset scale types include the first scale, the second scale, and the third scale; the dominant scale type includes one or more of the first scale, the second scale, and the third scale. Pooling scales include first-class granularity, second-class granularity, and third-class granularity; First type of particle size F Second type of particle size M and third type of granularity C The weight formulas are as follows: ； ； ； in, It is the percentage of pixels at the first scale; It is the percentage of pixels at the second scale; It is the percentage of pixels at the third scale.

7. A device for improving the performance of large remote sensing models based on multimodal and multi-scale adaptive data fusion, characterized in that, The device includes: The building unit is used to build an initial large remote sensing model based on an open-source visual general language model, adapted to multimodal remote sensing image data and text data. The architecture of the initial large remote sensing model includes a visual encoder ViT, a text editor BERT, a query converter Q-Former layer, a low-rank loop inference computation layer, and a dynamic threshold filtering system. The boosting unit is used to add a multimodal adaptive multiscale spatial pyramid pooling (MFAM-SPP) layer to the initial large remote sensing model to achieve multimodal fusion and multiscale adaptation, resulting in a large remote sensing model with improved performance. The inference unit, in response to a user-input query containing multimodal remote sensing image data and text data, preprocesses the multimodal remote sensing image data in the query to obtain a multimodal preprocessing matrix; and The multimodal preprocessing matrix is ​​input into the ViT module of the improved remote sensing large-scale model, performing input adaptation, hierarchical coding, interference suppression, and cross-modal concatenation steps to output a visual feature vector; and The visual feature vectors are input into the MFAM-SPP layer, where multimodal fusion, dynamic scale adaptive pooling, and feature aggregation are performed to output the MFAM-SPP feature matrix; and The text data from the question is input into BERT, a performance-enhancing remote sensing model, to output a text feature vector; and The MFAM-SPP feature matrix and text feature vectors are input into the Q-Former layer of a performance-enhanced remote sensing model. This layer performs query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, outputting a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the low-rank recurrent inference computation layer of the performance-enhanced remote sensing large model. Low-rank matrix decomposition and recurrent inference steps are performed, outputting the inference feature matrix. Based on this matrix, preliminary inference results including confidence scores and Intersection over Union (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system within the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results; and Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; and If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the MFAM-SPP layer, the parameters of the dynamic scale adaptive pooling step are adjusted, and the MFAM-SPP layer and the subsequent Q-Former layer and low-rank cyclic inference calculation layer process are re-executed.

8. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the remote sensing large model performance improvement method based on multimodal multiscale adaptive data fusion as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute, at runtime, the remote sensing large model performance improvement method based on multimodal multiscale adaptive data fusion as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, The computer program is configured to execute the method for improving the performance of large remote sensing models based on multimodal and multiscale adaptive data fusion as described in any one of claims 1 to 6 at runtime.

Images