Method and device for improving performance of remote sensing large model based on composite visual coding

Abstract

The invention provides a remote sensing large model performance improvement method and device based on composite visual coding, and relates to the technical field of remote sensing image large language models. According to the method, a network structure is rewritten on the basis of an open-source visual universal language model, a composite visual coding ConvToMe layer used for extracting local details and capturing global semantics is added, the ConvToMe layer extracts the local details through a ConvNeXt layer, the global semantics are captured in combination with a ToMe ViT layer, a remote sensing large model with improved performance is obtained, and the performance of the remote sensing large model is improved. Through composite visual coding, the global context and local details are reserved at the same time, the robustness is improved, and the model precision is improved at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image large language model, and particularly relates to a remote sensing large model performance improvement method and device based on composite visual coding. BACKGROUND

[0002] In recent years, the vigorous development of LLM (Large Language Model) has injected new vitality into the field of artificial intelligence, giving birth to artificial intelligence based on LLM such as Visual ChatGPT (Visual ChatGPT), WorldGPT (WorldGPT), HuggingGPT (HuggingGPT) and TreeGPT (TreeGPT). This kind of artificial intelligence takes LLM as the core, can accurately understand user input, and then formulate reasonable plans and solutions, and can efficiently complete various complex tasks with the help of tool chains. For example, Visual ChatGPT successfully realizes the function of visual question and answer through structured prompt interaction with ChatGPT (ChatGPT) and collaborative cooperation between multiple models.

[0003] However, in the field of remote sensing data interpretation, the existing methods have many shortcomings. Traditional remote sensing algorithms are usually designed for specific application scenarios, and their deployment and operation often depend on the operation of professional technicians, which makes it difficult for ordinary users to independently interpret remote sensing data, greatly limiting the wide application of remote sensing data.

[0004] Therefore, in recent years, many LLMs and VLMs (Visual Language Model) have emerged in the field of remote sensing tasks, and the combination of the two forms a remote sensing large model. The remote sensing large model has the ability to integrate multiple high-performance remote sensing image processing models, which enables it to flexibly select appropriate tools according to different problems and effectively cope with diversified task requirements. Its multi-tool integration feature provides more abundant means for solving complex remote sensing problems, greatly improving the possibility of handling complex tasks. Remote sensing artificial intelligence also supports multi-round dialogue and can understand the dialogue context. This interactive advantage helps better understand user intent, provides more user demand-oriented services, makes the remote sensing interpretation process more intelligent and humanized, improves the interaction efficiency between users and remote sensing interpretation systems, enhances user experience, and lays a foundation for wide application.

[0005] In the field of remote sensing, SkyEyeGPT, EarthGPT and other remote sensing large models have solved the problem of "scene solidification and high operation threshold" of traditional algorithms, but still have two core defects, and the defects directly affect the practicality of remote sensing interpretation: Global context and local detail imbalance: Existing models mostly use a single visual encoder, such as pure ViT (Vision Transformer) or pure convolutional network. Pure ViT can capture global semantics (such as large-area vegetation distribution), but has weak ability to extract local details of small targets (such as 5x5 pixel power poles and small ships), and is prone to lose key features due to token merging. Pure convolutional networks, such as ResNet (Residual Network), can preserve local edges (such as building outlines), but have limited receptive fields and are difficult to capture global spatial relationships of large targets (such as 100x100 pixel lakes and mountains), leading to "large target semantic ambiguity and small target missed detection".

[0006] Precision and speed are difficult to balance: To improve precision, existing models often increase network layers or expand parameter quantity, such as ViT-L / 16, where L stands for Large, and 16 represents a patch size of 16x16 pixels, resulting in a high inference delay of 1.5-2 seconds per image (512x512 image), and GPU memory usage of over 16GB, which cannot meet the real-time requirements of disaster emergency monitoring and other real-time requirements. If the model is compressed, such as ViT-B / 16, where B stands for Base, and 16 represents a patch size of 16x16 pixels, the precision decreases by 8%-12%, which cannot meet the high-precision requirements of land classification and other tasks.

[0007] Therefore, the above technical problems need to be solved. SUMMARY

[0008] In view of the above problems, the present application is proposed to provide a remote sensing large model performance improvement method and device based on composite visual coding, related products, which overcome the above problems or at least partially solve the above problems. The technical solution is as follows: In a first aspect, a remote sensing large model performance improvement method based on composite visual coding is provided, the method comprising: Based on an open-source visual general language model, adapt multi-modal remote sensing image data and text data, and build a preliminary remote sensing large model, wherein the architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a recurrent inference calculation layer, and a dynamic threshold screening system; A composite visual coding ConvToMe layer for extracting local details and capturing global semantics is added to the preliminary remote sensing large model. The ConvToMe layer extracts local details through a convolutional neural network ConvNeXt layer, and captures global semantics in combination with a token merging visual encoder ToMe ViT layer, to obtain a performance improved remote sensing large model; In response to user queries that include multimodal remote sensing image data and text data, the multimodal remote sensing image data in the user query is preprocessed to obtain a multimodal preprocessing matrix; The multimodal preprocessing matrix is ​​divided according to a preset ratio to obtain a first multimodal preprocessing matrix and a second multimodal preprocessing matrix. The first multimodal preprocessing matrix is ​​input into the ConvNeXt layer in the performance-enhanced remote sensing large model to output a local detail feature matrix. The second multimodal preprocessing matrix is ​​input into the ToMe ViT layer in the performance-enhanced remote sensing large model to output a global semantic feature matrix. By fusing the local detail feature matrix and the global semantic feature matrix, a composite visual feature matrix is ​​output. The text data from user questions is input into the BERT layer of a performance-enhancing remote sensing model, and the output is a text feature vector. The composite visual feature matrix and text feature vector are input into the Q-Former layer of the performance-enhanced remote sensing model. The steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion are performed to output a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the recurrent inference computation layer of the performance-enhanced remote sensing large model, and the steps of low-rank matrix factorization and recurrent inference are performed to output the inference feature matrix. Then, based on the inference feature matrix, the preliminary inference results containing confidence and intersection-union ratio (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system in the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results. Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layer and loop inference calculation layer processes are re-executed.

[0009] In one possible implementation, the ConvNeXt layer adopts a four-level architecture consisting of a Stem layer, a deep convolutional layer, an inverted bottleneck structure, and a dimension mapping layer. The Stem layer is used to implement input adaptation and preliminary feature extraction, and outputs a preliminary feature matrix. Deep convolutional layers are used to enhance local details and expand the receptive field, outputting an enhanced local feature matrix; The inverted bottleneck structure achieves local feature dimension expansion and compression through channel dimension expansion, feature extraction, and channel dimension compression, outputting a highly expressive local feature matrix. The dimension mapping layer is used to unify feature dimensions and adapt the output, outputting a local detail feature matrix.

[0010] In one possible implementation, the ToMe ViT layer adopts a four-level architecture: image patch segmentation and projection, geo-enhanced location coding, multi-layer Transformer encoder, and global feature reconstruction. The multi-layer Transformer encoder is divided into multiple stages, and a token merging ToMe module is inserted after each stage to form a progressive global modeling process of encoding, merging, and re-encoding.

[0011] In one possible implementation, geo-enhanced location coding is achieved through the following steps: Geographic coordinate extraction and preprocessing specifically involves reading latitude and longitude information from the input remote sensing image metadata, obtaining the coordinates of the upper left corner of the image, and calculating the absolute geographic coordinates of the center of each image block. Coordinate transformation specifically converts absolute latitude and longitude into relative coordinates, and maps the relative coordinates to the [0,1] interval to eliminate the influence of geographical differences between different images; Spatial dimension encoding, specifically using sinusoidal position encoding, generates position vectors based on the pixel index and column index of the token in the image, preserving the pixel-level relative positional relationship between tokens; Geographic dimension encoding specifically employs linear encoding, mapping relative latitude and longitude to a preset dimension vector through a linear layer, and integrating it with the actual geographic scale correlation. Encoding fusion specifically involves adding the spatial dimension encoding and the geographic dimension encoding element by element to generate the final location encoding vector, ensuring that the token carries both pixel location and geographic scale information simultaneously. Encoding injection involves adding the location encoding vector to the corresponding token feature vector without changing the token dimension, only supplementing spatial geographic association information.

[0012] In one possible implementation, there are multiple stages, including a first stage, a second stage, and a third stage. The first stage focuses on local token associations to initially capture small-scale spatial dependencies. The second stage extends to mid-level associations to strengthen the semantics of land feature combinations. The third stage achieves global associations and extracts global semantics and scene category information for large targets.

[0013] In one possible implementation, the ToMe module achieves semantically similar token merging through the following steps: Token similarity calculation, specifically calculating the cosine similarity between all tokens in the current token sequence; Similarity threshold, specifically set the merging threshold; Similar token clustering specifically uses a preset greedy clustering algorithm to divide tokens with similarity greater than the merging threshold into a cluster, and selects the token with the highest mean similarity within each cluster as the center token. The merged feature calculation specifically involves the new token feature being the weighted sum of the features of all tokens within the cluster, with the weight being the similarity between the token and the center token. Residual preservation specifically involves adding the new token features to the original central token features after merging via residual joins, ensuring that core semantics are not lost.

[0014] Secondly, a device for improving the performance of large remote sensing models based on composite visual coding is provided, the device comprising: The building unit is used to build a preliminary remote sensing large model based on the open-source visual general language model, adapted to multimodal remote sensing image data and text data. The architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a loop inference calculation layer, and a dynamic threshold filtering system. The enhancement unit is used to add a composite visual encoder ConvToMe layer to the initial large remote sensing model for extracting local details and capturing global semantics. The ConvToMe layer extracts local details through the convolutional neural network ConvNeXt layer and captures global semantics by combining the token merging visual encoder ToMe ViT layer, resulting in a large remote sensing model with improved performance. The inference unit is used to respond to user queries that include multimodal remote sensing image data and text data, preprocess the multimodal remote sensing image data in the user query to obtain a multimodal preprocessing matrix; and The multimodal preprocessing matrix is ​​divided according to a preset ratio to obtain a first multimodal preprocessing matrix and a second multimodal preprocessing matrix. The first multimodal preprocessing matrix is ​​input into the ConvNeXt layer of the performance-enhanced remote sensing model to output a local detail feature matrix. The second multimodal preprocessing matrix is ​​input into the ToMe ViT layer of the performance-enhanced remote sensing model to output a global semantic feature matrix. By fusing the local detail feature matrix and the global semantic feature matrix, a composite visual feature matrix is ​​output. The text data from user queries is input into the BERT layer of a performance-enhancing remote sensing model, outputting text feature vectors; and The composite visual feature matrix and text feature vector are input into the Q-Former layer of the performance-enhanced remote sensing model. This layer performs query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, outputting a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the recurrent inference computation layer of the performance-enhanced remote sensing large model. Low-rank matrix factorization and recurrent inference are performed, outputting the inference feature matrix. Based on this matrix, preliminary inference results including confidence scores and Intersection over Union (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system within the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results; and Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; and If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layer and loop inference calculation layer processes are re-executed.

[0015] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the performance improvement method for remote sensing large models based on composite visual coding as described above.

[0016] Fourthly, a storage medium is provided that stores a computer program, wherein the computer program is configured to execute the above-described method for improving the performance of large remote sensing models based on composite visual coding at runtime.

[0017] Fifthly, a computer program product is provided, including a computer program configured to execute the above-described method for improving the performance of large remote sensing models based on composite visual coding at runtime.

[0018] Using the above technical solutions, the remote sensing large model performance improvement method, device, and related products based on composite visual coding provided in this application rewrite the network structure based on the open-source visual universal language model, adding a ConvToMe (composite visual coding) layer for extracting local details and capturing global semantics. The ConvToMe layer extracts local details through the ConvNeXt (convolutional neural network) layer and captures global semantics in conjunction with the ToMe ViT (Token Merging Vision Transformer) layer. The composite visual coding achieves the following technical effects: Dual advantages in global and local features: The ConvToMe layer improves the feature retention rate of small targets (such as 5-20 pixels) and the global correlation of large targets (such as greater than 50 pixels) through the composite encoding of "ConvNeXt local detail extraction" and "ToMe ViT global semantic capture". On the remote sensing small target dataset, the MAP (Mean Average Precision) is improved compared with SkyEyeGPT.

[0019] Breakthrough in balancing accuracy and speed: The ToMe module reduces Transformer computation by 30%-40% through Token merging, and ConvNeXt's depthwise separable convolution reduces the number of parameters by about 20%. Combined with low-rank decomposition of loop inference, it can reduce model inference latency and GPU memory usage, achieving a dual optimization of accuracy and speed improvement.

[0020] Enhanced robustness of remote sensing scenes: ConvNeXt's 7×7 large convolutional kernel enhances anti-interference capabilities against cloud and fog and noise interference in remote sensing data, and ToMe ViT's location encoding incorporates geographic coordinate information. In optical images containing 20% ​​cloud and fog, the target detection recall rate is still close to 90%.

[0021] Strong module connectivity: Clearly define the feature dimension adaptation logic between the ConvToMe layer and the Q-Former layer (both mapped to 768 dimensions), and adopt a unified normalization strategy for BERT text features and visual features to reduce cross-modal fusion loss and ensure distortion-free feature transfer. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0023] Figure 1 The flowchart illustrates a method for improving the performance of large remote sensing models based on composite visual coding, as provided in an embodiment of this application. Figure 2 This document illustrates a flowchart illustrating the performance enhancements provided by the embodiments of this application in enabling a large remote sensing model to answer user questions. Figure 3 This illustration shows a schematic diagram of the specific preprocessing framework provided in an embodiment of this application; Figure 4 This paper illustrates a schematic diagram of the specific framework of the ConvToMe layer provided in an embodiment of this application. Figure 5 This paper illustrates a schematic diagram of the specific framework of the ConvNeXt layer provided in an embodiment of this application. Figure 6 This paper illustrates a schematic diagram of the specific framework of the ToMe ViT layer provided in an embodiment of this application. Figure 7 A schematic diagram of the specific framework of the BERT layer provided in the embodiments of this application is shown; Figure 8 This paper illustrates a schematic diagram of the specific framework of the Q-Former layer provided in an embodiment of this application. Figure 9 This paper illustrates a schematic diagram of the specific framework of the filtering output layer provided in an embodiment of this application. Figure 10 This paper shows a structural diagram of the remote sensing large model performance improvement device based on composite visual coding provided in an embodiment of this application; Figure 11 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0024] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0026] To address the aforementioned technical problems, embodiments of this application provide a method for improving the performance of large remote sensing models based on composite visual coding, such as... Figure 1 As shown, the method for improving the performance of large remote sensing models based on composite visual coding may include the following steps S101 to S111: Step S101: Based on the open-source visual general language model, adapt to multimodal remote sensing image data and text data, and build a preliminary remote sensing large model. The architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a loop inference calculation layer, and a dynamic threshold filtering system.

[0027] In this step, the visual general language model can be an open-source model such as VisualGLM (Visual General Language Model)-6B, and this embodiment does not limit it.

[0028] BERT (Bidirectional Encoder Representations from Transformers) generates text feature vectors.

[0029] Q-Former (Querying Transformer) connects visual models and language models.

[0030] Multimodal remote sensing image data may include optical data, SAR (Synthetic Aperture Radar) data, hyperspectral data, etc., and this embodiment does not limit this.

[0031] Step S102: Add a composite visual encoder ConvToMe layer to the initial remote sensing large model for extracting local details and capturing global semantics. The ConvToMe layer extracts local details through the convolutional neural network ConvNeXt layer and captures global semantics by combining the token merging visual encoder ToMe ViT layer, resulting in a performance-enhanced remote sensing large model.

[0032] Step S103: In response to a user query that includes multimodal remote sensing image data and text data, the multimodal remote sensing image data in the user query is preprocessed to obtain a multimodal preprocessing matrix.

[0033] Step S104: Divide the multimodal preprocessing matrix according to a preset ratio to obtain a first multimodal preprocessing matrix and a second multimodal preprocessing matrix; input the first multimodal preprocessing matrix into the ConvNeXt layer of the performance-enhanced remote sensing large model to output a local detail feature matrix; input the second multimodal preprocessing matrix into the ToMe ViT layer of the performance-enhanced remote sensing large model to output a global semantic feature matrix; and output a composite visual feature matrix by feature fusion of the local detail feature matrix and the global semantic feature matrix.

[0034] In this step, the preset ratio can be set according to actual needs, such as a preset ratio of 6:4. This embodiment does not limit this.

[0035] Step S105: Input the text data from the user's question into the BERT layer of the performance-enhanced remote sensing model, and output the text feature vector.

[0036] Step S106: Input the composite visual feature matrix and text feature vector into the Q-Former layer of the performance-enhanced remote sensing big model, perform query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, and output the cross-modal feature matrix.

[0037] Step S107: Input the cross-modal feature matrix into the loop inference computation layer of the performance-enhanced remote sensing large model, perform low-rank matrix decomposition and loop inference steps, output the inference feature matrix, and then obtain the preliminary inference result containing confidence and intersection-union ratio (IOU) based on the inference feature matrix.

[0038] In this step, IOU (Intersection Over Union) is a metric used in object detection tasks to measure the degree of overlap between the predicted bounding box and the ground truth bounding box.

[0039] Step S108: Input the preliminary inference results into the dynamic threshold screening system in the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results.

[0040] In this step, the target task corresponding to the preliminary inference result can be determined by the text feature vector.

[0041] Step S109: Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task.

[0042] In step S110, if the preliminary inference result meets the threshold requirements of the target task, then perform post-processing operations and output the final inference result containing the target coordinates, target category, and confidence level.

[0043] In step S111, if the preliminary inference result does not meet the threshold requirements of the target task, the preliminary inference result is sent back to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layer and loop inference calculation layer processes are re-executed.

[0044] This embodiment rewrites the network structure based on the open-source visual general language model, adding a ConvToMe layer for extracting local details and capturing global semantics. The ConvToMe layer extracts local details through the ConvNeXt layer and captures global semantics in combination with the ToMeViT layer, achieving the following technical effects through composite visual encoding: Dual advantages in global and local features: The ConvToMe layer improves the feature retention rate of small targets (such as 5-20 pixels) and the global correlation of large targets (such as greater than 50 pixels) through the composite encoding of "ConvNeXt local detail extraction" and "ToMe ViT global semantic capture". On the remote sensing small target dataset, MAP is improved compared with SkyEyeGPT.

[0045] Breakthrough in balancing accuracy and speed: The ToMe module reduces Transformer computation by 30%-40% through Token merging, and ConvNeXt's depthwise separable convolution reduces the number of parameters by about 20%. Combined with low-rank decomposition of loop inference, it can reduce model inference latency and GPU memory usage, achieving a dual optimization of accuracy and speed improvement.

[0046] Enhanced robustness of remote sensing scenes: ConvNeXt's 7×7 large convolutional kernel enhances anti-interference capabilities against cloud and fog and noise interference in remote sensing data, and ToMe ViT's location encoding incorporates geographic coordinate information. In optical images containing 20% ​​cloud and fog, the target detection recall rate is still close to 90%.

[0047] Strong module connectivity: Clearly define the feature dimension adaptation logic between the ConvToMe layer and the Q-Former layer (both mapped to 768 dimensions), and adopt a unified normalization strategy for BERT text features and visual features to reduce cross-modal fusion loss and ensure distortion-free feature transfer.

[0048] Figure 2 This document illustrates a flowchart of the performance-enhanced remote sensing large model completing user question and answer functions, as provided in an embodiment of this application. Figure 2 The process includes multimodal remote sensing data input, image input preprocessing, a ConvToMe composite visual coding layer (60% of the data is input into the ConvNeXt layer for local detail extraction; 40% of the data is input into the ToMe ViT layer for global semantic capture; attention-weighted fusion yields composite visual features), remote sensing text input, BERT text encoding, merging the output of the ConvToMe composite visual coding layer with the output of the BERT text encoding, inputting Q-Former cross-modal alignment, iterative inference calculation, dynamic threshold filtering, and if the threshold is met, the results are post-processed and output, including JSON (JavaScript Object Notation, a lightweight text data exchange format), PNG (Portable Network Graphics, a bitmap image format using lossless compression algorithms) visualization, and GIS (Geographic Information Systems) coordinates; if the threshold is not met, the results are returned to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layers and iterative inference calculation layers are re-executed.

[0049] This application provides a possible implementation method in which the ConvNeXt layer adopts a four-level architecture consisting of a Stem (backbone) layer, a deep convolutional layer, an inverted bottleneck structure, and a dimension mapping layer. The Stem layer is used to implement input adaptation and preliminary feature extraction, and outputs a preliminary feature matrix. Deep convolutional layers are used to enhance local details and expand the receptive field, outputting an enhanced local feature matrix; The inverted bottleneck structure achieves local feature dimension expansion and compression through channel dimension expansion, feature extraction, and channel dimension compression, outputting a highly expressive local feature matrix. The dimension mapping layer is used to unify feature dimensions and adapt the output, outputting a local detail feature matrix.

[0050] This application provides a possible implementation method. The ToMe ViT layer adopts a four-level architecture of image block segmentation and projection, geo-enhanced location coding, multi-layer Transformer encoder, and global feature reconstruction. The multi-layer Transformer encoder is divided into multiple stages, and a token merging ToMe module is inserted after each stage to form a progressive global modeling process of encoding, merging, and re-encoding.

[0051] This application provides a possible implementation method for geo-enhanced location coding through the following steps A1 to A6: Step A1, geographic coordinate extraction and preprocessing, specifically reads latitude and longitude information from the input remote sensing image metadata, obtains the coordinates of the upper left corner of the image, and calculates the absolute geographic coordinates of the center of each image block; Step A2, coordinate transformation, specifically converting absolute latitude and longitude into relative coordinates, and mapping the relative coordinates to the [0,1] interval to eliminate the influence of differences in geographical range between different images; Step A3, spatial dimension encoding, specifically adopts sinusoidal position encoding, which generates a position vector based on the pixel index and column index of the token in the image, and preserves the pixel-level relative positional relationship between tokens; Step A4, geographic dimension encoding, specifically using linear encoding, maps relative latitude and longitude to a preset dimension vector through a linear layer, and incorporates the actual geographic scale association; Step A5, encoding fusion, specifically, adds the spatial dimension encoding and the geographic dimension encoding element by element to generate the final location encoding vector, ensuring that the token carries both pixel location and geographic scale information simultaneously; Step A6, encoding injection, specifically adds the location encoding vector to the corresponding Token feature vector, without changing the Token dimension, only supplementing spatial geographic association information.

[0052] This embodiment breaks through the limitations of traditional ViT sinusoidal position encoding, incorporates geographic coordinate information from remote sensing images, and ensures that the token sequence retains the "actual geographic scale association" (such as the spatial distance difference of ground objects at different latitudes) during global modeling, thus avoiding global semantic misalignment of large targets due to "pixel-scale position encoding".

[0053] This application provides a possible implementation method, which includes multiple stages: a first stage, a second stage, and a third stage. The first stage focuses on local token association, initially capturing small-scale spatial dependencies. The second stage extends to mid-level association, strengthening the semantics of feature combinations. The third stage achieves global association, extracting global semantics of large targets and scene category information. Here, the first, second, and third stages can be referred to as the shallow stage, the mid-level stage, and the deep stage, respectively.

[0054] This application embodiment provides a possible implementation method in which the ToMe module merges semantically similar tokens through the following steps B1 to B5: Step B1, Token similarity calculation, specifically calculating the cosine similarity between all tokens in the current token sequence; Step B2, similarity threshold, specifically setting the merging threshold; Step B3, similar token clustering, specifically using a preset greedy clustering algorithm, divides tokens with similarity greater than the merging threshold into a cluster, and selects the token with the highest mean similarity within each cluster as the center token. Step B4, merge feature calculation, specifically, the new token feature is the weighted sum of the features of all tokens within the cluster, with the weight being the similarity between the token and the center token; Step B5, residual preservation, specifically involves adding the new token features to the original central token features after merging via residual joins to ensure that the core semantics are not lost.

[0055] This embodiment captures global spatial dependencies between tokens (such as the adjacency relationship between roads and buildings, and the symbiotic relationship between water systems and vegetation) through a multi-layer Transformer encoder. At the same time, it integrates a ToMe Token merging module to reduce the number of tokens while retaining core semantic information, thus solving the problem of "large computational load and semantic redundancy of large remote sensing targets" in traditional ViT.

[0056] The above introduces Figure 1 The embodiments shown have various implementation methods for each step. The following will further explain the performance improvement method for remote sensing large models based on composite visual coding in this application through specific embodiments.

[0057] In a specific embodiment, the Visual General Language Model is the VisualGLM-6B open-source model. Based on the architecture of the VisualGLM-6B open-source model, it is adapted to multimodal remote sensing image data and text data to build a preliminary large-scale remote sensing model. The architecture of the preliminary large-scale remote sensing model includes a text editor BERT layer, a query converter Q-Former layer, a loop inference calculation layer, and a dynamic threshold filtering system. A composite visual encoding ConvToMe layer is added to the preliminary large-scale remote sensing model to extract local details and capture global semantics. The ConvToMe layer extracts local details through the ConvNeXt layer and captures global semantics by combining it with the ToMe ViT layer, resulting in a performance-enhanced large-scale remote sensing model. Through composite visual encoding, it achieves the effect of simultaneously preserving global context and local details, improving robustness and model accuracy.

[0058] The specific implementation plan is as follows: 1. Image input preprocessing (i.e., multimodal differentiation processing) See Figure 3 For the three core remote sensing modes—optical data, SAR data, and hyperspectral data—specific preprocessing and unified format conversion are performed to ensure the consistency and validity of input data.

[0059] 1.1 Modal-specific preprocessing Optical images (such as satellite RGB): Radiometric correction: The FLAASH (Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes) atmospheric correction module of ENVI (Environment for Visualizing Image, a software widely used in remote sensing image processing and analysis) is used to eliminate atmospheric scattering and aerosol effects. Geometric correction: Based on GCP (Ground Control Point), a quadratic polynomial transformation is used to align the image to the WGS84 (World Geodetic Coordinate System 1984) coordinate system, with the error controlled within 1 pixel; Contrast enhancement: CLAHE (Contrast Limit Adaptive Histogram Equalization) is used, with clipLimit (Contrast Limit Threshold) set to 2.0 to enhance the details of ground features at the edges.

[0060] SAR data (such as C-band fully polarimetric SAR data): Speckle noise reduction: Lee filtering (a classic adaptive filtering algorithm with a window size of 5×5) is used to suppress speckle noise; Polarization decomposition: The polarization data is converted into three-channel feature maps, namely "HH+VV, HH-VV, and HV", through Pauli decomposition (a basic polarimetric SAR target decomposition). Here, HH+VV represents the sum of the echoes from horizontal transmission and horizontal reception and vertical transmission and vertical reception; HH-VV represents the difference between the echoes from horizontal transmission and horizontal reception and vertical transmission and vertical reception; and HV represents the echoes from horizontal transmission and vertical reception. Radiation calibration: converting radar echo intensity into backscattering coefficient The formula is: Where K is the calibration coefficient (provided by sensor parameters), DN is a digital value, the original brightness value of the pixel in the SAR image, representing the intensity of the echo.

[0061] Hyperspectral data (such as Hyperion satellite data): Defective pixel repair: The neighborhood mean fill method is used to repair pixels whose gray values ​​deviate from the mean by 3 times the standard deviation within the band. Band selection: Screen 20-30 sensitive bands related to the target using MIC (Maximal Information Coefficient) (e.g., NDVI-related bands for vegetation monitoring); NDVI is the Normalized Difference Vegetation Index, which is calculated from the reflectance of the near-infrared and red light bands and is often used to characterize the vegetation growth status. Dimensionality reduction: PCA (Principal Component Analysis) was used to reduce the screened bands to 10 dimensions, retaining more than 95% of the information.

[0062] 1.2 Unified Format Conversion See also Figure 3 This includes adaptive scaling, pixel value normalization, 3D matrix transformation, and outputting a 3D matrix, as detailed below: Adaptive scaling: A bilinear interpolation algorithm is used to uniformly scale all modal data to 512×512 pixels. This is just an illustration; other pixels may also be used. Pixel value normalization: Min-max normalization is used, with the following formula: This maps all pixel values ​​to the [0,1] range, eliminating brightness and intensity differences between different modalities; among them, It is the normalized value, where x is the pixel value. It is the minimum pixel value. It is the maximum pixel value; 3D matrix transformation: Convert the preprocessed data into a 3D matrix (height H × width W × number of channels C), where: H=512, W=512, number of channels C=3 for optical data, number of channels C=3 for polarization data, and number of channels C=10 for hyperspectral data; Output: A three-dimensional matrix, which serves as the input to the subsequent ConvToMe layer.

[0063] 2. ConvToMe layer: See [link / reference] Figure 4 The preprocessed 3D matrix is ​​divided in a 6:4 ratio. 60% is input into the ConvNeXt layer to extract local details, and 40% is input into the ToMe ViT layer to capture global semantics. Finally, composite visual features are output through feature fusion. This ratio allocation is reasonable because local details are often the focus of remote sensing images, and also because the two structures have different computational costs. Compared to a 5:5 allocation, this is more reasonable and can save computational costs globally.

[0064] 2.1 ConvNeXt layer Input: 60% of the data of the three-dimensional matrix of the preprocessed remote sensing image, with a size of 512×512×C (C is the number of channels), and the pixel values ​​have been normalized to the range of [0,1]. Output: A local detail feature matrix of 512×512×768, which has the same dimension as the global feature matrix output by the ToMe ViT layer, ensuring that there is no dimensional conflict in subsequent attention-weighted fusion.

[0065] Overall, see Figure 5 The ConvNeXt layer adopts a four-level architecture: "Stem layer - deep convolutional layer - inverted bottleneck structure - dimension mapping layer". Each level is optimized for the characteristics of remote sensing data, and the specific design is as follows: 2.1.1 Stem Layer: Input Adaptation and Preliminary Feature Extraction Convolution operation: Use 4×4 non-overlapping convolution, stride=4, padding=0, and the number of convolution kernels is set to 128; Non-overlapping design: ensures that each 4×4 pixel block of the input image corresponds to only one output feature point, avoids pixel information crossover and confusion, and adapts to the "intra-block integrity" requirement of small remote sensing targets (such as the top structure of a 4×4 pixel utility pole). Step size and padding: Step size = 4 compresses the output feature map size from 512×512 to 128×128 (512 / 4=128). No padding ensures that there is no redundant filling in the edge area, which fits the geographic boundary of the remote sensing image. Activation and normalization: LayerNorm (layer normalization) is applied after convolution, with the normalization dimension being the channel dimension (128 dimensions) to avoid the sensitivity of BatchNorm (batch normalization) to the "large brightness difference within batch" of remote sensing data; no activation function is applied to preserve the linear expression of the initial features and provide richer original information for subsequent deep convolution. Output specifications: a preliminary feature matrix of 128×128×128, with the number of channels uniformly converted from input C to 128, completing input adaptation.

[0066] The advantage of this approach is that it replaces the traditional ConvNeXt's "7×7 convolution + pooling" structure, reducing edge feature loss while uniformly converting the number of input channels to the base number of channels adapted to subsequent networks, thus avoiding excessive compression of small target details by the initial pooling.

[0067] 2.1.2 Deep Convolutional Layers: Local Detail Enhancement and Receptive Field Expansion Network layers and structure: A total of 8 depthwise separable convolutional layers are set, divided into 2 stages (4 layers in each stage). The number of channels is adjusted between stages by 1×1 convolution (keeping 128 dimensions unchanged) to avoid feature dimension expansion; Convolution kernel size: 7×7 (replacing the traditional 3×3), by expanding the receptive field (receptive field range = 7 + (7-1) × (4-1) = 19 pixels, corresponding to the original image 19 × 4 = 76 pixels), covering the complete area of ​​small targets (such as a 20-pixel ship), avoiding feature fragmentation; Step size and padding: Step size = 1, padding = 3 ((7-1) / 2 = 3), to ensure that the output feature map size is consistent with the input (128×128) and that edge details are not lost; Number of groups: Set to 128 (consistent with the number of input channels) to achieve "channel-by-channel convolution" and enhance the local feature representation of each channel (e.g., the "HH+VV" channel of SAR data focuses on ship echo features, and the "HV" channel focuses on vegetation scattering features). Activation and normalization: After every 2 depthwise convolutions, insert one LayerNorm (channel dimension normalization) and one GELU (Gaussian Error Linear Unit) activation function; GELU activation: Compared to ReLU (Rectified Linear Unit), it can smoothly activate "weak edge features" (such as low-contrast road boundaries) in remote sensing data, avoiding feature loss; Normalization interval: Inserted once every 2 layers to balance "feature stability" and "computational efficiency" and avoid inference delay caused by frequent normalization; Output specifications: 128×128×128 enhanced local feature matrix, preserving key details such as the edges of ground features (e.g., right angles of buildings, broken lines of coastlines) and textures (e.g., parallel textures of crop rows).

[0068] This allows for the expansion of the receptive field through large-size, depth-separable convolutions, enabling precise capture of local edge and texture features of remotely sensed objects while reducing the number of parameters and computational load, thus balancing the accuracy and efficiency of detail extraction.

[0069] 2.1.3 Inverted Bottleneck Structure: Local Feature Dimension Expansion and Compression Dimension expansion stage: 1×1 convolution is used to expand the number of input channels from 128 dimensions to 512 dimensions (expansion coefficient = 4). The purpose of this expansion is to provide a richer dimensional space for local features and to capture fine-grained differences in remote sensing data (such as subtle differences in the RGB spectra of different crops and differences in SAR echo data of different building materials). Convolution parameters: number of kernels = 512, stride = 1, padding = 0, output feature map size remains 128×128; Feature enhancement stage: The GELU activation function is applied to perform nonlinear transformation on the expanded 512-dimensional features to enhance the distinction between "effective detail features" (such as high-value features of green bands in vegetation) and "ineffective interference features" (such as single-point noise); no normalization layer is applied to avoid blurring of details caused by excessive smoothing. Dimension compression stage: 1×1 convolution is used to compress the number of channels from 512 dimensions back to 128 dimensions (consistent with the output channels of the deep convolutional layer). Compression aims to eliminate redundant dimensions, retain core local features, and avoid dimensional imbalance during subsequent fusion. Convolution parameters: number of kernels = 128, stride = 1, padding = 0, output feature map size remains 128×128; Residual connections: Add residual connections at both ends of the inverted bottleneck structure (input features are directly superimposed on output features) to avoid gradient vanishing in deep networks, ensure that weak detail features of small remote sensing targets (such as the shadow of a 10-pixel telephone pole) can be effectively backpropagated, and improve training stability. Output specifications: A highly expressive local feature matrix of 128×128×128, which integrates multi-dimensional detail association information (such as the combined features of "building edge + material texture").

[0070] This inverted design of "channel dimension expansion - feature extraction - channel dimension compression" can enhance the nonlinear expression capability of local features and adapt to the complex correlation of "multi-dimensional details" in remote sensing data (such as the RGB texture of optical images and the polarization features of SAR data).

[0071] 2.1.4 Dimension Mapping Layer: Feature Dimension Unification and Output Adaptation Convolution operation: 1×1 convolution is used to map the number of channels from 128 dimensions to 768 dimensions; Convolution kernel parameters: number of convolution kernels = 768, stride = 1, padding = 0, output feature map size remains 128×128; Weight initialization: Xavier initialization is used to ensure that the feature mean and variance are stable during the mapping process and to avoid feature distortion caused by dimension transformation; Feature upsampling: The 128×128×768 feature matrix is ​​upsampled to 512×512×768 using bilinear interpolation; The purpose of upsampling is to restore the image to the same size as the input image (512×512) to ensure that local features are aligned with the global features of the ToMeViT layer at the "pixel level" (e.g., in a 512×512 image, the local edge features of the (x,y) coordinates are precisely matched with the global semantic features). Interpolation parameters: interpolation kernel size = 3×3, to ensure that the edge features after upsampling are smooth and jagged (such as maintaining the straightness of building boundaries after upsampling). Final normalization: LayerNorm (channel dimension normalization) is used to perform a final stability optimization on the 768-dimensional features to ensure that the mean of the output features is 0 and the variance is 1, which is suitable for the subsequent weight calculation logic of attention fusion. Output specifications: a 512×512×768 local detail feature matrix, where each pixel corresponds to local detail information of the input image (such as edge intensity, texture direction, and material properties), which can be directly input into the attention fusion module for combination with global features.

[0072] The advantage of this step is that it maps the 128-dimensional local features output by the inverted bottleneck structure to 768 dimensions consistent with the ToMe ViT layer, providing a dimensional basis for the attention-weighted fusion of the ConvToMe layer, while strengthening the semantic relevance of local features.

[0073] 2.2 ToMe ViT Layer Input: 40% of the data of the three-dimensional matrix of the preprocessed remote sensing image, with a size of 512×512×C (C is the number of channels), and the pixel values ​​have been normalized to the [0,1] range; Output: A global semantic feature matrix of 512×512×768, which has the same dimension as the local detail feature matrix output by the ConvNeXt layer, ensuring that there is no dimensional conflict in subsequent attention-weighted fusion.

[0074] See Figure 6Overall, the ToMe ViT layer adopts a four-level architecture: "image patch segmentation and projection - geo-enhanced location coding - Transformer encoder (integrated ToMe module) - global feature reconstruction". Each level is optimized for the global semantic requirements of remote sensing data, and the specific design is as follows: 2.2.1 Image Patch Segmentation and Linear Projection: Construction of Global Feature Units The input 512×512 remote sensing image is segmented into fixed-size image blocks (Tokens), which are then transformed into high-dimensional feature vectors through linear projection. This provides the basic feature units for subsequent global modeling by the Transformer, while ensuring that the Token size adapts to the spatial distribution of large remote sensing targets. Image block segmentation rule: Use 16×16 pixel non-overlapping segmentation to evenly divide the 512×512 image into 32×32=1024 image blocks; The segmentation size adaptability of 16×16 pixel blocks corresponds to the actual geographical range of remote sensing images at different resolutions (e.g., 0.5-meter resolution corresponds to 8×8 meters, and 30-meter resolution corresponds to 480×480 meters). It can cover small targets (e.g., small buildings with 16×16 pixels) and efficiently build global associations of large targets (e.g., at 30-meter resolution, 10 tokens can cover a contiguous farmland of 4.8×4.8 kilometers). Linear projection operation: Each 16×16×C image patch is transformed into a 768-dimensional feature vector (Token) through a linear projection layer; Projection parameters: Linear layer input dimension = 16×16×C (e.g., when C=3 for optical images, the input dimension = 768), output dimension = 768, weights are initialized with He to ensure stable feature variance after projection; Output specifications: 1024×768 Token sequence (1024 Tokens, each Token is a 768-dimensional feature vector), each Token corresponds to a 16×16 pixel region of the input image, carrying the local channel fusion information of that region.

[0075] 2.2.2 Geo-enhanced location coding: Preservation of remote sensing spatial correlation Breaking through the limitations of traditional ViT's sinusoidal location encoding, this approach incorporates geographic coordinate information from remote sensing images, ensuring that the token sequence retains "actual geographic scale correlation" (such as spatial distance differences between ground features at different latitudes) during global modeling. This avoids global semantic misalignment of large targets due to "pixel-scale location encoding." The specific implementation is as follows: Geographic coordinate extraction and preprocessing: Read latitude and longitude information from the input remote sensing image metadata, obtain the coordinates of the upper left corner of the image, and calculate the absolute geographic coordinates of the center of each image block; Coordinate transformation: Convert absolute latitude and longitude to relative coordinates, and map the relative coordinates to the [0,1] interval to eliminate the influence of differences in geographical range between different images; Spatial dimension encoding: Traditional sinusoidal position encoding is adopted, and position vectors are generated based on the pixel index and column index of the token in the image, preserving the pixel-level relative positional relationship between tokens; Geographic dimension encoding: Linear encoding is used to map relative latitude and longitude to a 768-dimensional vector through a linear layer, which is then incorporated into the actual geographic scale correlation; Encoding fusion: The spatial dimension encoding and the geographic dimension encoding are added element by element to generate the final location encoding vector, ensuring that the token carries both "pixel location + geographic scale" information. Encoding Injection: The location encoding vector is added to the corresponding Token feature vector without changing the Token dimension, only supplementing spatial geographic association information; Output specifications: 1024×768 enhanced token sequence, each token contains both feature information of the image patch and carries precise spatial-geographical associations, providing geographic anchors for global semantic modeling.

[0076] 2.2.3 Transformer Encoder (integrated with ToMe module): Global Semantic Modeling and Computational Optimization The overall architecture of the Transformer encoder consists of 12 layers, divided into 3 stages (4 layers per stage). A ToMe module is inserted after each stage, forming a progressive global modeling process of "encoding-merging-re-encoding".

[0077] Functional division of labor in each stage: Shallow stage (layers 1-4): Focus on local associations of tokens (such as the association of features around a 16×16 pixel block) to initially capture small-scale spatial dependencies; Mid-level stage (layers 5-8): Expand to mid-level associations (such as road segments or farmland blocks composed of multiple tokens) to strengthen the semantics of land feature combinations; Deeper stages (layers 9-12): Achieve global association (such as a spatial network of full-image tokens) and extract global semantics and scene category information of large targets.

[0078] The internal structure of a single Transformer layer: Each layer includes three steps: Multi-head Self-Attention (MSA), Layer Normalization (LN), and Feedforward Neural Network (FFN). The parameters are optimized for remote sensing data. Multi-head self-attention (MSA): Head number setting: 8 attention heads, each head dimension = 768 / 8 = 96, balancing global capture capability and computational efficiency; Attention calculation: Scaling dot product attention is used, and the formula is as follows: ,in Q , K , V These are the query, key, and value matrices, respectively, all generated by linear transformations of the token sequence. softmax The () function "compresses" a vector containing arbitrary real numbers and transforms it into a probability distribution such that the value of each element is in the interval (0,1) and the sum of all elements is 1. d k This is the scaling factor; Remote sensing adaptation: When calculating attention weights, tokens with small differences in geographic dimension encoding (i.e., close actual geographic distance) are given a base weight coefficient (1.2 times) to enhance the capture of spatial associations between adjacent features (such as semantic associations between adjacent buildings in a city).

[0079] Layer Normalization (LN): Insert LN (normalization dimension = 768) once before and after MSA to avoid feature distribution shift caused by brightness fluctuations in remote sensing data.

[0080] Feedforward Neural Network (FFN): Employs a "linear layer-GELU activation-linear layer" structure, with an intermediate dimension of 3072 (4 times the token dimension). The formula is as follows: ,in, W 1. W 2 represents the weight. b 1. b 2 is a coefficient. GELU () is the activation function; Activation function selection: Compared with ReLU, GELU can smoothly activate the "weak semantic features" of remote sensing data (such as the semantics of the transition area between grassland and desert), avoiding feature loss; Dimensional Design: The 3072-dimensional intermediate layer provides ample space for semantic expression, adapting to the complex land feature category associations in remote sensing scenes (such as the multi-category semantic combination of "farmland-irrigation canal-road").

[0081] ToMe Token Merging Module: The ToMe module is inserted after each stage of Transformer encoding to achieve "semantically similar token merging". The specific process is as follows: Token similarity calculation: Calculate the cosine similarity between all tokens in the current token sequence; Similarity threshold: Set to 0.85 as the merging threshold to only merge tokens with high semantic similarity (such as adjacent 16×16 pixel blocks of tokens in a contiguous field of farmland) and avoid erroneous merging of tokens with large semantic differences (such as farmland and road tokens). Similar Token Clustering: A greedy clustering algorithm is used to divide tokens with a similarity of ≥0.85 into a cluster, and select one center token (the token with the highest average similarity within the cluster) for each cluster. Merged feature calculation: The new token feature is the weighted sum of the features of all tokens within the cluster, with the weight being the similarity between the token and the center token; Residual Preservation: After merging, the new token features are added to the central token features before merging through residual joins to ensure that the core semantics are not lost; Merging effect: The number of tokens is reduced by 30%-40% after each stage of merging. After 12 layers of encoding and 3 ToMe mergings, the number of tokens is reduced from 1024 to 600-700, and the amount of computation is reduced by more than 40%, while the global semantic association of the large target is preserved. Output specifications: A simplified token sequence of 600-700×768, with each token carrying global semantic association information (e.g., "lake token" contains semantic associations with surrounding wetlands and shorelines), providing high-quality semantic units for subsequent global feature reconstruction.

[0082] The technical effect of this approach is to capture the global spatial dependencies between tokens (such as the adjacency relationship between roads and buildings, and the symbiotic relationship between water systems and vegetation) through a multi-layer Transformer encoder. At the same time, it integrates a ToMe Token merging module to reduce the number of tokens while retaining core semantic information, thus solving the problem of "large computational load and semantic redundancy of large remote sensing targets" in traditional ViT.

[0083] 2.2.4 Global Feature Reconstruction: Aligning Semantic Features with Input Image Size The core function is to reconstruct the simplified token sequence output by the Transformer encoder into a global semantic feature matrix with the same size as the input image (512×512), ensuring "pixel-level" alignment with the local detail features output by the ConvNeXt layer. This provides a spatial matching basis for attention-weighted fusion. The specific implementation is as follows: Token sequence expansion: The simplified token sequence of 600-700×768 is rearranged into a 32×32×768 feature matrix according to the spatial index of the original image patch segmentation (restoring the segmented image patch grid structure). Missing Token Handling: In the extremely rare case where there is no Token at a certain grid position after merging, the mean value of the Token features in the 3×3 neighborhood is used for filling to ensure the integrity of the feature matrix; Global Average Pooling: For the feature matrix of 32×32×768, global average pooling is performed at the grid positions corresponding to each 16×16 pixel block, mapping the 32×32 grid features into a compact feature matrix of 32×32×768; Purpose of Pooling: To further strengthen the global semantic association at each grid position and avoid semantic deviation caused by local Token fluctuations (such as the influence of individual noise Tokens in a contiguous farmland area); Bilinear Interpolation Upsampling: Upsample the compact feature matrix of 32×32×768 to 512×512×768, which is exactly the same size as the input image; Interpolation Parameters: Interpolation kernel size = 3×3, stride = 16 (32×16 = 512), ensuring that the global semantic features after upsampling accurately correspond to the geographical coordinates of the input image at each pixel position. For example, the global semantic features of the (x, y) pixel in a 512×512 image are exactly matched with the Token semantics of the 16×16 block to which the pixel belongs; Final Normalization: Connect LayerNorm (normalization dimension = 768) to standardize the reconstructed global semantic feature matrix, ensuring that the feature mean = 0 and variance = 1, which is consistent with the distribution of the local detail features output by the ConvNeXt layer and adapts to the weight calculation logic of attention weighted fusion; Output Specification: A global semantic feature matrix of 512×512×768, where each pixel corresponds to the global semantic information of the input image (such as the probability of ground object category, attribution of large targets, and spatial association strength), which can be directly input into the attention fusion module to combine with local detail features.

[0084] 3. Remote Sensing Specific BERT Text Encoder (Text Feature Generation) See Figure 7 , for remote sensing texts (such as the query instruction "Identify the damaged farmland area in the SAR data image" and the annotation text "The NDVI value of 0.6 - 0.8 is the vegetation coverage area"), generate high-precision text features. <​​​​​​​

[0086] 3.2 BERT Model Fine-tuning and Encoding Base model: bert-base-chinese-remote-sensing (fine-tuned on the remote sensing text dataset RSICD); Text feature vector generation: After the text is encoded by BERT, the output of [CLS] token is taken as the text feature vector, with a dimension of 768. Semantic enhancement: By embedding remote sensing knowledge graphs (encoding the relationships such as "farmland → vegetation" and "SAR → microwave remote sensing" into vectors) and weighting them with BERT features (with a weight ratio of 0.8:0.2), the semantic relevance of the text is enhanced.

[0087] 4. Q-Former layer (cross-modal feature alignment) See Figure 8 It receives ConvToMe features (512×512×768) and BERT text features (768-dimensional), performs query vector interaction, organizes multiple rounds of alignment, and finally fuses features to achieve visual and textual semantic unity. 4.1 Learnable Query Vector Design Initialize 32 learnable query vectors ( Q The query vector has 768 dimensions, consistent with the text feature dimensions. It is optimized through remote sensing scene pre-training (input is a "remote sensing image-text" pair) to ensure that it has the ability to associate vision with text.

[0088] 4.2 Multi-round cross-attention alignment Perform three rounds of cross-attention interaction, each round consisting of two steps: visual guidance text and text-guided visual interaction. Visual guiding text: Visual features are incorporated into the query vector; among them, It is the text feature of the current round. It is a visual feature. Attention () represents cross attention. It is a ConvToMe feature; Text-guided visuals: This maps textual semantics to visual features. Alignment loss calculation: using contrastive learning loss This yields the minimum visual-text feature distance, where These are the textual features of each round.

[0089] 4.3 Cross-modal feature fusion The aligned visual features (512×512×768) and text features (768 dimensions) are fused through dimensional expansion and element-wise multiplication: the text features are expanded to 512×512×768 and then multiplied with the visual features to generate a 512×512×768 cross-modal feature matrix, which is then input into the recurrent inference layer.

[0090] 5. Low-rank cyclic inference computation (efficiency optimization) See Figure 9 For cross-modal feature matrices, low-rank decomposition and multi-round inference are used to reduce computational complexity while maintaining accuracy. 5.1 Low-rank matrix decomposition For the inference layer weight matrix (e.g., 1024×1024) after the cross-modal feature matrix input, SVD (Singular Value Decomposition) is used to decompose it into two low-rank matrices: W=W_1×W_2, for example W_1∈R^(1024×64), W_2∈R^(64×1024). In this way, the number of parameters is reduced from 1,048,576 to 131,072 (a reduction of 88%).

[0091] 5.2 Circular Reasoning Mechanism Perform 3-5 rounds of inference iterations, with each round's process as follows: Feature update: F_{t+1}=F_t×W_1×W_2+Residual(F_t) (residual connections preserve the original features); Accuracy verification: Calculate the MAE (mean absolute error) of features in two adjacent rounds. If the MAE < 0.01 or the number of iterations reaches 5, terminate the loop. Output optimization: The final round of features is normalized by LayerNorm to generate a 512×512×768 inference feature matrix.

[0092] 6. Dynamic threshold filtering and result output See also Figure 9 The filtering rules are adaptively adjusted according to the type of remote sensing task to ensure the accuracy of the output results. 6.1 Task Scenario Threshold Mapping In one embodiment, a mapping table is established from task type to threshold, wherein: The confidence threshold for the land classification task is ≥0.85, with no IOU threshold, and the core requirement is high boundary accuracy; The confidence threshold for disaster monitoring tasks is ≥0.7, and the IOU threshold is ≥0.5. The core requirement is high recall to avoid missed detections. The confidence threshold for maritime target detection tasks is ≥0.8, and the IOU threshold is ≥0.6. The core requirement is high accuracy in detecting small targets.

[0093] In another embodiment, a general threshold can be preset, such as a confidence threshold of ≥0.8 and an IOU threshold of ≥0.6. This embodiment does not impose any restrictions on this.

[0094] 6.2 Result Screening and Secondary Optimization Valid results: If the inference result meets the threshold requirements of the corresponding task, post-processing is performed (such as non-maximum suppression (NMS) to remove duplicate detection boxes and morphological dilation to optimize segmentation boundaries), and standardized output (such as target coordinates, category, and confidence score) is generated and fed back to the user; Invalid result: If the result is less than the threshold, the result is returned to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layer and loop inference layer processes are re-executed.

[0095] 6.3 Output Format Spatial information: Target coordinates are in WGS84 latitude and longitude (e.g., "116.39°E, 39.91°N"). Attribute information: Encapsulated in JSON format (e.g., {"Target Category":"Farmland","Damage Level":"Moderate","Confidence Level":0.88}); Visualization: Generates a PNG image containing the original image, target annotations, and confidence scores, supporting API (Application Programming Interface) calls and file export.

[0096] This completes the user question input, which includes both image information and text.

[0097] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.

[0098] Based on the methods for improving the performance of large remote sensing models based on composite visual coding provided in the above embodiments, and based on the same inventive concept, this application also provides a device for improving the performance of large remote sensing models based on composite visual coding.

[0099] Figure 10 This is a structural diagram of the remote sensing large model performance improvement device based on composite visual coding provided in the embodiments of this application. Figure 10 As shown, the remote sensing large model performance enhancement device based on composite visual coding may specifically include a construction unit 1010, an enhancement unit 1020, and an inference unit 1030.

[0100] Unit 1010 is used to build a preliminary remote sensing large model based on an open-source visual general language model, adapted to multimodal remote sensing image data and text data. The architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a loop inference calculation layer, and a dynamic threshold filtering system. The enhancement unit 1020 is used to add a composite visual encoder ConvToMe layer to the initial remote sensing large model for extracting local details and capturing global semantics. The ConvToMe layer extracts local details through the convolutional neural network ConvNeXt layer and captures global semantics by combining the token merging visual encoder ToMe ViT layer, resulting in a remote sensing large model with improved performance. Inference unit 1030 is configured to, in response to a user query including multimodal remote sensing image data and text data, preprocess the multimodal remote sensing image data in the user query to obtain a multimodal preprocessing matrix; and The multimodal preprocessing matrix is ​​divided according to a preset ratio to obtain a first multimodal preprocessing matrix and a second multimodal preprocessing matrix. The first multimodal preprocessing matrix is ​​input into the ConvNeXt layer of the performance-enhanced remote sensing model to output a local detail feature matrix. The second multimodal preprocessing matrix is ​​input into the ToMe ViT layer of the performance-enhanced remote sensing model to output a global semantic feature matrix. By fusing the local detail feature matrix and the global semantic feature matrix, a composite visual feature matrix is ​​output. The text data from user queries is input into the BERT layer of a performance-enhancing remote sensing model, outputting text feature vectors; and The composite visual feature matrix and text feature vector are input into the Q-Former layer of the performance-enhanced remote sensing model. This layer performs query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, outputting a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the recurrent inference computation layer of the performance-enhanced remote sensing large model. Low-rank matrix factorization and recurrent inference are performed, outputting the inference feature matrix. Based on this matrix, preliminary inference results including confidence scores and Intersection over Union (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system within the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results; and Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; and If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layer and loop inference calculation layer processes are re-executed.

[0101] This application provides a possible implementation method in which the ConvNeXt layer adopts a four-level architecture of Stem layer, deep convolutional layer, inverted bottleneck structure, and dimension mapping layer. The Stem layer is used to implement input adaptation and preliminary feature extraction, and outputs a preliminary feature matrix. Deep convolutional layers are used to enhance local details and expand the receptive field, outputting an enhanced local feature matrix; The inverted bottleneck structure achieves local feature dimension expansion and compression through channel dimension expansion, feature extraction, and channel dimension compression, outputting a highly expressive local feature matrix. The dimension mapping layer is used to unify feature dimensions and adapt the output, outputting a local detail feature matrix.

[0102] This application provides a possible implementation method. The ToMe ViT layer adopts a four-level architecture of image block segmentation and projection, geo-enhanced location coding, multi-layer Transformer encoder, and global feature reconstruction. The multi-layer Transformer encoder is divided into multiple stages, and a token merging ToMe module is inserted after each stage to form a progressive global modeling process of encoding, merging, and re-encoding.

[0103] This application embodiment provides a possible implementation, wherein the inference unit 1030 is further configured to implement geo-enhanced location coding through the following steps: Geographic coordinate extraction and preprocessing specifically involves reading latitude and longitude information from the input remote sensing image metadata, obtaining the coordinates of the upper left corner of the image, and calculating the absolute geographic coordinates of the center of each image block. Coordinate transformation specifically converts absolute latitude and longitude into relative coordinates, and maps the relative coordinates to the [0,1] interval to eliminate the influence of geographical differences between different images; Spatial dimension encoding, specifically using sinusoidal position encoding, generates position vectors based on the pixel index and column index of the token in the image, preserving the pixel-level relative positional relationship between tokens; Geographic dimension encoding specifically employs linear encoding, mapping relative latitude and longitude to a preset dimension vector through a linear layer, and integrating it with the actual geographic scale correlation. Encoding fusion specifically involves adding the spatial dimension encoding and the geographic dimension encoding element by element to generate the final location encoding vector, ensuring that the token carries both pixel location and geographic scale information simultaneously. Encoding injection involves adding the location encoding vector to the corresponding token feature vector without changing the token dimension, only supplementing spatial geographic association information.

[0104] This application provides a possible implementation method, which includes multiple stages: a first stage, a second stage, and a third stage. The first stage focuses on local token association to initially capture small-scale spatial dependencies. The second stage extends to mid-level association to strengthen the semantics of land feature combinations. The third stage achieves global association to extract global semantics and scene category information of large targets.

[0105] This application embodiment provides a possible implementation, wherein the inference unit 1030 is further used by the ToMe module to merge semantically similar tokens through the following steps: Token similarity calculation, specifically calculating the cosine similarity between all tokens in the current token sequence; Similarity threshold, specifically set the merging threshold; Similar token clustering specifically uses a preset greedy clustering algorithm to divide tokens with similarity greater than the merging threshold into a cluster, and selects the token with the highest mean similarity within each cluster as the center token. The merged feature calculation specifically involves the new token feature being the weighted sum of the features of all tokens within the cluster, with the weight being the similarity between the token and the center token. Residual preservation specifically involves adding the new token features to the original central token features after merging via residual joins, ensuring that core semantics are not lost.

[0106] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the work order quality inspection method based on a large language model of any of the above embodiments.

[0107] In an exemplary embodiment, an electronic device is provided, such as Figure 11 As shown, Figure 11 The illustrated electronic device 1100 includes a processor 1101 and a memory 1103. The processor 1101 and the memory 1103 are connected, for example, via a bus 1102. Optionally, the electronic device 1100 may also include a transceiver 1104. It should be noted that in practical applications, the transceiver 1104 is not limited to one unit, and the structure of this electronic device 1100 does not constitute a limitation on the embodiments of this application.

[0108] Processor 1101 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1101 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0109] Bus 1102 may include a pathway for transmitting information between the aforementioned components. Bus 1102 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1102 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0110] The memory 1103 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0111] The memory 1103 is used to store computer program code that executes the scheme of this application, and its execution is controlled by the processor 1101. The processor 1101 is used to execute the computer program code stored in the memory 1103 to implement the content shown in the foregoing method embodiments.

[0112] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0113] Based on the same inventive concept, this application also provides a storage medium storing a computer program, wherein the computer program is configured to execute the performance improvement method for remote sensing large models based on composite visual coding of any of the above embodiments at runtime.

[0114] Based on the same inventive concept, this application also provides a computer program product, including a computer program configured to execute the performance improvement method for remote sensing large models based on composite visual coding of any of the above embodiments at runtime.

[0115] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0116] Those skilled in the art will understand that the technical solution of this application, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0117] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.

[0118] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.

Claims

1. A method for improving the performance of large remote sensing models based on composite visual coding, characterized in that, The method includes: Based on an open-source visual general language model, and adapted to multimodal remote sensing image data and text data, a preliminary remote sensing large model is built. The architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a loop inference calculation layer, and a dynamic threshold filtering system. In the initial large remote sensing model, a composite visual encoder ConvToMe layer is added to extract local details and capture global semantics. The ConvToMe layer extracts local details through the convolutional neural network ConvNeXt layer and captures global semantics by combining the token merging visual encoder ToMe ViT layer, resulting in a large remote sensing model with improved performance. In response to user queries that include multimodal remote sensing image data and text data, the multimodal remote sensing image data in the user query is preprocessed to obtain a multimodal preprocessing matrix; The multimodal preprocessing matrix is ​​divided according to a preset ratio to obtain a first multimodal preprocessing matrix and a second multimodal preprocessing matrix. The first multimodal preprocessing matrix is ​​input into the ConvNeXt layer in the performance-enhanced remote sensing large model to output a local detail feature matrix. The second multimodal preprocessing matrix is ​​input into the ToMe ViT layer in the performance-enhanced remote sensing large model to output a global semantic feature matrix. By fusing the local detail feature matrix and the global semantic feature matrix, a composite visual feature matrix is ​​output. The text data from user questions is input into the BERT layer of a performance-enhancing remote sensing model, and the output is a text feature vector. The composite visual feature matrix and text feature vector are input into the Q-Former layer of the performance-enhanced remote sensing model. The steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion are performed to output a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the recurrent inference computation layer of the performance-enhanced remote sensing large model, and the steps of low-rank matrix factorization and recurrent inference are performed to output the inference feature matrix. Then, based on the inference feature matrix, the preliminary inference results containing confidence and intersection-union ratio (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system in the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results. Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layer and loop inference calculation layer processes are re-executed.

2. The method according to claim 1, characterized in that, ConvNeXt layers adopt a four-level architecture consisting of Stem layers, deep convolutional layers, inverted bottleneck structures, and dimension mapping layers. The Stem layer is used to implement input adaptation and preliminary feature extraction, and outputs a preliminary feature matrix. Deep convolutional layers are used to enhance local details and expand the receptive field, outputting an enhanced local feature matrix; The inverted bottleneck structure achieves local feature dimension expansion and compression through channel dimension expansion, feature extraction, and channel dimension compression, outputting a highly expressive local feature matrix. The dimension mapping layer is used to unify feature dimensions and adapt the output, outputting a local detail feature matrix.

3. The method according to claim 1 or 2, characterized in that, The ToMe ViT layer adopts a four-level architecture: image patch segmentation and projection, geo-enhanced location coding, multi-layer Transformer encoder, and global feature reconstruction. The multi-layer Transformer encoder is divided into multiple stages, and a token merging ToMe module is inserted after each stage to form a progressive global modeling process of encoding, merging, and re-encoding.

4. The method according to claim 3, characterized in that, Geo-enhanced location coding is achieved through the following steps: Geographic coordinate extraction and preprocessing specifically involves reading latitude and longitude information from the input remote sensing image metadata, obtaining the coordinates of the upper left corner of the image, and calculating the absolute geographic coordinates of the center of each image block. Coordinate transformation specifically converts absolute latitude and longitude into relative coordinates, and maps the relative coordinates to the [0,1] interval to eliminate the influence of geographical differences between different images; Spatial dimension encoding, specifically using sinusoidal position encoding, generates position vectors based on the pixel index and column index of the token in the image, preserving the pixel-level relative positional relationship between tokens; Geographic dimension encoding specifically employs linear encoding, mapping relative latitude and longitude to a preset dimension vector through a linear layer, and integrating it with the actual geographic scale correlation. Encoding fusion specifically involves adding the spatial dimension encoding and the geographic dimension encoding element by element to generate the final location encoding vector, ensuring that the token carries both pixel location and geographic scale information simultaneously. Encoding injection involves adding the location encoding vector to the corresponding token feature vector without changing the token dimension, only supplementing spatial geographic association information.

5. The method according to claim 3, characterized in that, The process involves multiple stages, including the first stage, the second stage, and the third stage. The first stage focuses on local token associations to initially capture small-scale spatial dependencies. The second stage expands to mid-level associations to strengthen the semantics of land feature combinations. The third stage achieves global associations, extracting global semantics and scene category information for large targets.

6. The method according to claim 3, characterized in that, The ToMe module achieves semantically similar token merging through the following steps: Token similarity calculation, specifically calculating the cosine similarity between all tokens in the current token sequence; Similarity threshold, specifically set the merging threshold; Similar token clustering specifically uses a preset greedy clustering algorithm to divide tokens with similarity greater than the merging threshold into a cluster, and selects the token with the highest mean similarity within each cluster as the center token. The merged feature calculation specifically involves the new token feature being the weighted sum of the features of all tokens within the cluster, with the weight being the similarity between the token and the center token. Residual preservation specifically involves adding the new token features to the original central token features after merging via residual joins, ensuring that core semantics are not lost.

7. A device for improving the performance of large remote sensing models based on composite visual coding, characterized in that, The device includes: The building unit is used to build a preliminary remote sensing large model based on the open-source visual general language model, adapted to multimodal remote sensing image data and text data. The architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a loop inference calculation layer, and a dynamic threshold filtering system. The enhancement unit is used to add a composite visual encoder ConvToMe layer to the initial large remote sensing model for extracting local details and capturing global semantics. The ConvToMe layer extracts local details through the convolutional neural network ConvNeXt layer and captures global semantics by combining the token merging visual encoder ToMe ViT layer, resulting in a large remote sensing model with improved performance. The inference unit is used to respond to user queries that include multimodal remote sensing image data and text data, preprocess the multimodal remote sensing image data in the user query to obtain a multimodal preprocessing matrix; and The multimodal preprocessing matrix is ​​divided according to a preset ratio to obtain a first multimodal preprocessing matrix and a second multimodal preprocessing matrix. The first multimodal preprocessing matrix is ​​input into the ConvNeXt layer of the performance-enhanced remote sensing model to output a local detail feature matrix. The second multimodal preprocessing matrix is ​​input into the ToMe ViT layer of the performance-enhanced remote sensing model to output a global semantic feature matrix. By fusing the local detail feature matrix and the global semantic feature matrix, a composite visual feature matrix is ​​output. The text data from user queries is input into the BERT layer of a performance-enhancing remote sensing model, outputting text feature vectors; and The composite visual feature matrix and text feature vector are input into the Q-Former layer of the performance-enhanced remote sensing model. This layer performs query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, outputting a cross-modal feature matrix. The cross-modal feature matrix is ​​input into the recurrent inference computation layer of the performance-enhanced remote sensing large model. Low-rank matrix factorization and recurrent inference are performed, outputting the inference feature matrix. Based on this matrix, preliminary inference results including confidence scores and Intersection over Union (IOU) are obtained. The preliminary inference results are input into the dynamic threshold filtering system within the performance-enhanced remote sensing large model to determine the target task corresponding to the preliminary inference results; and Based on the confidence level and IOU in the preliminary inference results, determine whether the preliminary inference results meet the threshold requirements of the target task; and If the preliminary inference result meets the threshold requirements of the target task, then post-processing is performed to output the final inference result containing the target coordinates, target category, and confidence level. If the initial inference result does not meet the threshold requirements of the target task, the initial inference result is sent back to the ConvToMe layer for re-inference, and the ConvToMe layer and subsequent Q-Former layer and loop inference calculation layer processes are re-executed.

8. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method for improving the performance of large remote sensing models based on composite visual coding as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method for improving the performance of large remote sensing models based on composite visual coding as described in any one of claims 1 to 6 at runtime.

10. A computer program product, comprising a computer program, characterized in that, The computer program is configured to execute the method for improving the performance of large remote sensing models based on composite visual coding as described in any one of claims 1 to 6 at runtime.

Images