A remote sensing large model performance improvement method and device based on composite visual coding

Abstract

The application provides a remote sensing large model performance improvement method and device based on composite visual coding, and relates to the technical field of remote sensing image large language model. The method rewrites the network structure based on an open source visual general language model, adds a composite visual coding ConvToMe layer for extracting local details and capturing global semantics, the ConvToMe layer extracts local details through a ConvNeXt layer, combines a ToMe ViT layer to capture global semantics, obtains a remote sensing large model with improved performance, and through the composite visual coding, the global context and the local details are retained at the same time, the robustness is improved, and the model precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image large language model, and particularly relates to a remote sensing large model performance improvement method and device based on composite visual coding. BACKGROUND

[0002] In recent years, the vigorous development of LLM (Large Language Model) has injected new vitality into the field of artificial intelligence, giving birth to artificial intelligence based on LLM such as Visual ChatGPT (Visual ChatGPT), WorldGPT (WorldGPT), HuggingGPT (HuggingGPT) and TreeGPT (TreeGPT). This kind of artificial intelligence takes LLM as the core, can accurately understand user input, and then formulate reasonable plans and solutions, and can efficiently complete various complex tasks with the help of tool chains. For example, Visual ChatGPT successfully realizes the function of visual question and answer through structured prompt interaction with ChatGPT (ChatGPT) and collaborative cooperation between multiple models.

[0003] However, in the field of remote sensing data interpretation, the existing methods have many shortcomings. Traditional remote sensing algorithms are usually designed for specific application scenarios, and their deployment and operation often depend on the operation of professional technicians, which makes it difficult for ordinary users to independently interpret remote sensing data, greatly limiting the wide application of remote sensing data.

[0004] Therefore, in recent years, many LLMs and VLMs (Visual Language Model) have emerged in the field of remote sensing tasks, and the combination of the two forms a remote sensing large model. The remote sensing large model has the ability to integrate multiple high-performance remote sensing image processing models, which enables it to flexibly select appropriate tools according to different problems and effectively cope with diversified task requirements. Its multi-tool integration feature provides more abundant means for solving complex remote sensing problems, greatly improving the possibility of handling complex tasks. Remote sensing artificial intelligence also supports multi-round dialogue and can understand the dialogue context. This interactive advantage helps better understand user intent, provides more user demand-oriented services, makes the remote sensing interpretation process more intelligent and humanized, improves the interaction efficiency between users and remote sensing interpretation systems, enhances user experience, and lays a foundation for wide application.

[0005] In the field of remote sensing, SkyEyeGPT, EarthGPT and other remote sensing large models have solved the problem of "scene solidification and high operation threshold" of traditional algorithms, but still have two major defects, and the defects directly affect the practicality of remote sensing interpretation:

[0006] Global context and local detail imbalance: Existing models mostly use a single visual encoder, such as pure ViT (Vision Transformer) or pure convolutional network. Pure ViT can capture global semantics (such as large-area vegetation distribution), but has weak ability to extract local details of small targets (such as 5x5 pixel power poles and small ships), and is prone to lose key features due to Token merging. Pure convolutional networks, such as ResNet (Residual Network), can preserve local edges (such as building outlines), but have limited receptive fields and are difficult to capture global spatial relationships of large targets (such as 100x100 pixel lakes and mountains), resulting in "large target semantic ambiguity and small target missed detection".

[0007] Precision and speed are difficult to balance: To improve precision, existing models often increase network layers or expand parameter quantity, such as ViT-L / 16, where L stands for Large, and 16 represents a Patch size of 16x16 pixels, resulting in a high inference delay of 1.5-2 seconds per image (512x512 image), and GPU memory usage of over 16GB, which cannot meet the real-time requirements of disaster emergency monitoring and other real-time requirements. If the model is compressed, such as ViT-B / 16, where B stands for Base, and 16 represents a Patch size of 16x16 pixels, the precision decreases by 8%-12%, which cannot meet the high-precision requirements of land classification and other tasks.

[0008] Therefore, the above technical problems need to be solved. SUMMARY

[0009] In view of the above problems, the present application is proposed to provide a remote sensing large model performance improvement method and device based on composite visual coding, related products, which overcome the above problems or at least partially solve the above problems. The technical solution is as follows:

[0010] In the first aspect, a remote sensing large model performance improvement method based on composite visual coding is provided, which comprises:

[0011] Based on an open-source visual general language model, multi-modal remote sensing image data and text data are adapted to build a preliminary remote sensing large model, wherein the architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a recurrent inference calculation layer, and a dynamic threshold screening system.

[0012] In the preliminary remote sensing large model, a composite visual coding ConvToMe layer for extracting local details and capturing global semantics is added, the ConvToMe layer extracts local details through a convolutional neural network ConvNeXt layer, and combines a token merging visual encoder ToMe ViT layer to capture global semantics, so as to obtain a remote sensing large model with improved performance;

[0013] In response to a user query including multi-modal remote sensing image data and text data, the multi-modal remote sensing image data in the user query is preprocessed to obtain a multi-modal preprocessing matrix;

[0014] The multi-modal preprocessing matrix is divided according to a preset proportion to obtain a first multi-modal preprocessing matrix and a second multi-modal preprocessing matrix; the first multi-modal preprocessing matrix is input into the ConvNeXt layer in the remote sensing large model with improved performance, and a local detail feature matrix is output; the second multi-modal preprocessing matrix is input into the ToMe ViT layer in the remote sensing large model with improved performance, and a global semantic feature matrix is output; the local detail feature matrix and the global semantic feature matrix are fused to output a composite visual feature matrix;

[0015] The text data in the user query is input into the BERT layer in the remote sensing large model with improved performance, and a text feature vector is output;

[0016] The composite visual feature matrix and the text feature vector are input into the Q-Former layer in the remote sensing large model with improved performance, and steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion are performed to output a cross-modal feature matrix;

[0017] The cross-modal feature matrix is input into the recurrent reasoning calculation layer in the remote sensing large model with improved performance, and steps of low-rank matrix decomposition and recurrent reasoning are performed to output a reasoning feature matrix, and then a preliminary reasoning result including confidence and intersection over union IOU is obtained according to the reasoning feature matrix;

[0018] The preliminary reasoning result is input into the dynamic threshold screening system in the remote sensing large model with improved performance, and a target task corresponding to the preliminary reasoning result is determined;

[0019] According to the confidence and IOU in the preliminary reasoning result, it is judged whether the preliminary reasoning result meets the threshold requirements of the target task;

[0020] If the preliminary reasoning result meets the threshold requirements of the target task, a post-processing operation is performed to output a final reasoning result including target coordinates, target categories, and confidence;

[0021] If the preliminary inference result does not meet the threshold requirement of the target task, the preliminary inference result is returned to the ConvToMe layer, re-inferred, and the ConvToMe layer and subsequent Q-Former layer, loop inference calculation layer processes are re-executed.

[0022] In a possible implementation, the ConvNeXt layer adopts a four-level architecture of a Stem layer, a deep convolutional layer, an inverted bottleneck structure, and a dimension mapping layer.

[0023] The Stem layer is configured to implement input adaptation and preliminary feature extraction, and output a preliminary feature matrix.

[0024] The deep convolutional layer is configured to implement local detail enhancement and receptive field expansion, and output an enhanced local feature matrix.

[0025] The inverted bottleneck structure is configured to implement local feature dimension expansion and compression through steps of channel dimension expansion, feature extraction, and channel dimension compression, and output a high-expression local feature matrix.

[0026] The dimension mapping layer is configured to implement feature dimension unification and output adaptation, and output a local detail feature matrix.

[0027] In a possible implementation, the ToMe ViT layer adopts a four-level architecture of image block segmentation and projection, geographical enhanced position encoding, a multi-layer Transformer encoder, and global feature reconstruction.

[0028] In a possible implementation, the geographical enhanced position encoding is implemented through the following steps:

[0029] Geographical coordinate extraction and preprocessing, specifically reading latitude and longitude information from input remote sensing image metadata, obtaining the top-left corner coordinates of the image, and calculating the absolute geographical coordinates of the center of each image block.

[0030] Coordinate conversion, specifically converting the absolute latitude and longitude into relative coordinates, mapping the relative coordinates to the [0, 1] interval, and eliminating the influence of different geographical ranges of images.

[0031] Spatial dimension encoding, specifically adopting sinusoidal position encoding, generating a position vector based on the pixel index and column index of the token Token in the image, and preserving the pixel-level relative position relationship between Tokens.

[0032] Geographical dimension encoding, specifically adopting linear encoding, mapping the relative latitude and longitude through a linear layer into a preset dimension vector, and incorporating the actual geographical scale correlation.

[0033] Encoding fusion, specifically element-wise addition of spatial dimension encoding and geographic dimension encoding, generates the final position encoding vector, ensuring that the Token carries dual information of pixel position and geographic scale;

[0034] Encoding injection, specifically adding the position encoding vector to the corresponding Token feature vector, does not change the Token dimension, but only supplements the spatial and geographic correlation information.

[0035] In one possible implementation, the plurality of stages includes a first stage, a second stage, and a third stage; the first stage focuses on local association of Tokens, and preliminarily captures small-range spatial dependence; the second stage extends to middle-layer association, and strengthens the semantic combination of geographic features; and the third stage realizes global association, and extracts large-target global semantic and scene category information.

[0036] In one possible implementation, the ToMe module realizes merging of semantically similar Tokens through the following steps:

[0037] Token similarity calculation, specifically calculating the cosine similarity between all Tokens in the current Token sequence;

[0038] Similarity threshold, specifically setting a merging threshold;

[0039] Similar Token clustering, specifically using a preset greedy clustering algorithm to divide Tokens with a similarity greater than the merging threshold into a cluster, and selecting the Token with the highest average similarity in each cluster as the center Token;

[0040] Merging feature calculation, specifically the new Token feature is the weighted sum of all Token features in the cluster, and the weight is the similarity between the Token and the center Token;

[0041] Residual reservation, specifically adding the new Token feature to the center Token feature before merging through residual connection, to ensure that the core semantic is not lost.

[0042] In a second aspect, a device for improving performance of a remote sensing large model based on composite visual encoding is provided, and the device includes:

[0043] A building unit is configured to build a preliminary remote sensing large model based on an open-source visual general language model, adapt multi-modal remote sensing image data and text data, and build a preliminary remote sensing large model, wherein the architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a recurrent inference calculation layer, and a dynamic threshold screening system.

[0044] The lifting unit is configured to add a composite visual encoding ConvToMe layer for extracting local details and capturing global semantics in the preliminary remote sensing large model, the ConvToMe layer extracts local details through a convolutional neural network ConvNeXt layer, and captures global semantics in combination with a token merging visual encoder ToMe ViT layer, so as to obtain the remote sensing large model with improved performance.

[0045] The reasoning unit is configured to, in response to a user query including multi-modal remote sensing image data and text data, pre-process the multi-modal remote sensing image data in the user query to obtain a multi-modal pre-processing matrix; and

[0046] The multi-modal pre-processing matrix is divided according to a preset proportion to obtain a first multi-modal pre-processing matrix and a second multi-modal pre-processing matrix; the first multi-modal pre-processing matrix is input into the ConvNeXt layer in the remote sensing large model with improved performance to output a local detail feature matrix; the second multi-modal pre-processing matrix is input into the ToMe ViT layer in the remote sensing large model with improved performance to output a global semantic feature matrix; the local detail feature matrix and the global semantic feature matrix are fused to output a composite visual feature matrix; and

[0047] The text data in the user query is input into the BERT layer in the remote sensing large model with improved performance to output a text feature vector; and

[0048] The composite visual feature matrix and the text feature vector are input into the Q-Former layer in the remote sensing large model with improved performance to perform steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion to output a cross-modal feature matrix; and

[0049] The cross-modal feature matrix is input into the recurrent reasoning calculation layer in the remote sensing large model with improved performance to perform steps of low-rank matrix decomposition and recurrent reasoning to output a reasoning feature matrix, and then a preliminary reasoning result including a confidence and an intersection over union (IOU) is obtained according to the reasoning feature matrix; and

[0050] The preliminary reasoning result is input into the dynamic threshold screening system in the remote sensing large model with improved performance to determine a target task corresponding to the preliminary reasoning result; and

[0051] According to the confidence and the IOU in the preliminary reasoning result, it is determined whether the preliminary reasoning result meets the threshold requirement of the target task; and

[0052] If the preliminary reasoning result meets the threshold requirement of the target task, a post-processing operation is performed to output a final reasoning result including a target coordinate, a target category, and a confidence.

[0053] If the preliminary inference result does not meet the threshold requirement of the target task, the preliminary inference result is returned to the ConvToMe layer, re-inference is performed, and the ConvToMe layer and subsequent Q-Former layer and loop inference calculation layer processes are re-executed.

[0054] In a third aspect, an electronic device is provided, including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the remote sensing large model performance improvement method based on composite visual coding according to any one of the preceding aspects.

[0055] In a fourth aspect, a storage medium is provided, which stores a computer program, wherein the computer program is configured to perform the remote sensing large model performance improvement method based on composite visual coding according to any one of the preceding aspects when running.

[0056] In a fifth aspect, a computer program product is provided, which includes a computer program configured to perform the remote sensing large model performance improvement method based on composite visual coding according to any one of the preceding aspects when running.

[0057] By means of the above technical solutions, the remote sensing large model performance improvement method and device based on composite visual coding, and related products provided by the embodiments of the present application, the method rewrites the network structure based on an open-source visual general language model, adds a ConvToMe (composite visual coding) layer for extracting local details and capturing global semantics, the ConvToMe layer extracts local details through a ConvNeXt (convolutional neural network) layer, combines a ToMe ViT (Token Merging Vision Transformer, token merging vision encoder) layer to capture global semantics, and achieves the following technical effects through composite visual coding:

[0058] Global and local feature dual optimization: the ConvToMe layer combines the composite coding of "ConvNeXt local detail extraction" and "ToMe ViT global semantic capture" to improve the feature retention rate of small targets (such as 5-20 pixels) and the global correlation degree of large targets (such as greater than 50 pixels), and the MAP (Mean Average Precision, average precision mean) on the remote sensing small target dataset is improved compared with SkyEyeGPT.

[0059] Precision and speed balance breakthrough: the ToMe module reduces the Transformer calculation amount by 30%-40% through token merging, the depth separable convolution of ConvNeXt reduces the parameter amount by about 20%, and combined with the low-rank decomposition of loop inference, the model inference delay and GPU memory occupation can be reduced, realizing dual optimization of precision and speed improvement.

[0060] Robustness enhancement of remote sensing scenarios: For cloud and fog, noise interference of remote sensing data, the 7x7 large convolution kernel of ConvNeXt enhances the anti-interference ability, and the position encoding of ToMe ViT integrates geographic coordinate information. In optical images containing 20% cloud and fog, the target detection recall rate is still close to 90%.

[0061] Strong module connection: The feature dimension adaptation logic of ConvToMe layer and Q-Former layer is clear (both are mapped to 768 dimensions), the unified normalization strategy is used for BERT text features and visual features, the cross-modal fusion loss is reduced, and the feature transmission is distortionless. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0063] Figure 1 The flowchart of the remote sensing large model performance improvement method based on composite visual coding provided by the embodiments of the present application is shown;

[0064] Figure 2 The flowchart of the remote sensing large model performance improvement method based on composite visual coding provided by the embodiments of the present application is shown;

[0065] Figure 3 The specific framework schematic diagram of the pre-processing provided by the embodiments of the present application is shown;

[0066] Figure 4 The specific framework schematic diagram of the ConvToMe layer provided by the embodiments of the present application is shown;

[0067] Figure 5 The specific framework schematic diagram of the ConvNeXt layer provided by the embodiments of the present application is shown;

[0068] Figure 6 The specific framework schematic diagram of the ToMe ViT layer provided by the embodiments of the present application is shown;

[0069] Figure 7 The specific framework schematic diagram of the BERT layer provided by the embodiments of the present application is shown;

[0070] Figure 8 The specific framework schematic diagram of the Q-Former layer provided by the embodiments of the present application is shown;

[0071] Figure 9 The specific framework schematic diagram of the screening output layer provided by the embodiments of the present application is shown;

[0072] Figure 10A structural diagram of the remote sensing large model performance improvement device based on composite visual coding provided by the embodiment of the application is shown.

[0073] Figure 11 A structural diagram of an electronic device provided by the embodiment of the application is shown. DETAILED DESCRIPTION

[0074] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and so that the scope of the present application can be completely conveyed to those skilled in the art.

[0075] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that such use can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "comprising" and its variants are to be interpreted as meaning "including but not limited to" an open-ended term.

[0076] To solve the above technical problems, the embodiment of the present application provides a remote sensing large model performance improvement method based on composite visual coding, as shown in the figure, which can include the following steps S101-S111: Figure 1

[0077] Step S101, based on an open-source visual general language model, adapt multi-modal remote sensing image data and text data, and build a preliminary remote sensing large model, wherein the architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a recurrent inference calculation layer, and a dynamic threshold screening system.

[0078] In this step, the visual general language model can be VisualGLM (Visual General Language Model)-6B or other open-source models, and the present embodiment does not limit this.

[0079] BERT (Bidirectional Encoder Representations from Transformers, Bidirectional Encoder Representations from Transformers), generates a text feature vector.

[0080] ​Q-Former (Querying Transformer), connecting visual models and language models.

[0081] The multi-modal remote sensing image data can include optical data, SAR (Synthetic Aperture Radar) data, hyperspectral data, etc., and the present embodiment is not limited thereto.

[0082] In step S102, a composite visual encoding ConvToMe layer for extracting local details and capturing global semantics is added to the preliminary remote sensing large model, the ConvToMe layer extracts local details through a convolutional neural network ConvNeXt layer, and combines a token merging visual encoder ToMe ViT layer to capture global semantics, thereby obtaining a performance-improved remote sensing large model.

[0083] In step S103, in response to a user query including multi-modal remote sensing image data and text data, the multi-modal remote sensing image data in the user query is preprocessed to obtain a multi-modal preprocessed matrix.

[0084] In step S104, the multi-modal preprocessed matrix is divided according to a preset ratio to obtain a first multi-modal preprocessed matrix and a second multi-modal preprocessed matrix; the first multi-modal preprocessed matrix is input into the ConvNeXt layer of the performance-improved remote sensing large model to output a local detail feature matrix; the second multi-modal preprocessed matrix is input into the ToMe ViT layer of the performance-improved remote sensing large model to output a global semantic feature matrix; and the local detail feature matrix and the global semantic feature matrix are fused to output a composite visual feature matrix.

[0085] In this step, the preset ratio can be set according to actual needs, such as a preset ratio of 6:4, and the present embodiment is not limited thereto.

[0086] In step S105, the text data in the user query is input into the BERT layer of the performance-improved remote sensing large model to output a text feature vector.

[0087] In step S106, the composite visual feature matrix and the text feature vector are input into the Q-Former layer of the performance-improved remote sensing large model to perform steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion, and output a cross-modal feature matrix.

[0088] In step S107, the cross-modal feature matrix is input into the recurrent reasoning calculation layer of the performance-improved remote sensing large model to perform steps of low-rank matrix decomposition and recurrent reasoning, and output a reasoning feature matrix, and then obtain a preliminary reasoning result containing confidence and intersection over union (IOU) according to the reasoning feature matrix.

[0089] In this step, IOU (Intersection Over Union) is an index used to measure the degree of overlap between the predicted bounding box and the true bounding box in the target detection task.

[0090] In step S108, the preliminary reasoning result is input into the dynamic threshold screening system in the performance-improved remote sensing large model to determine the target task corresponding to the preliminary reasoning result.

[0091] In this step, the target task corresponding to the preliminary reasoning result can be determined by the text feature vector.

[0092] In step S109, whether the preliminary reasoning result meets the threshold requirement of the target task is determined according to the confidence and IOU in the preliminary reasoning result.

[0093] In step S110, if the preliminary reasoning result meets the threshold requirement of the target task, a post-processing operation is performed to output the final reasoning result containing the target coordinates, target category and confidence.

[0094] In step S111, if the preliminary reasoning result does not meet the threshold requirement of the target task, the preliminary reasoning result is returned to the ConvToMe layer for re-reasoning, and the ConvToMe layer and the subsequent Q-Former layer and the loop reasoning calculation layer process are re-executed.

[0095] In this embodiment, the network structure of the open-source visual general language model is rewritten, and the ConvToMe layer for extracting local details and capturing global semantics is added. The ConvToMe layer extracts local details through the ConvNeXt layer, captures global semantics through the ToMeViT layer, and achieves the following technical effects through the composite visual coding:

[0096] Global and local feature dual optimization: ConvToMe layer through "ConvNeXt local detail extraction", combined with "ToMe ViT global semantic capture" composite coding, improves small target (such as 5-20 pixels) feature retention rate, and improves large target (such as greater than 50 pixels) global correlation degree. The MAP on the remote sensing small target dataset is improved compared with SkyEyeGPT.

[0097] Precision and speed balance breakthrough: ToMe module reduces 30%-40% of the Transformer calculation amount through Token merging, and the depth separable convolution of ConvNeXt reduces about 20% of the parameter amount. Combined with the low-rank decomposition of the loop reasoning, the model reasoning delay and GPU memory occupation can be reduced, realizing the dual optimization of precision and speed improvement.

[0098] Robustness enhancement of remote sensing scene: For the cloud and noise interference of remote sensing data, the 7x7 large convolution kernel of ConvNeXt enhances the anti-interference ability, and the position encoding of ToMe ViT integrates the geographic coordinate information. In the optical image containing 20% cloud and fog, the target detection recall rate is still close to 90%.

[0099] Strong module connection: The feature dimension adaptation logic of ConvToMe layer and Q-Former layer is clearly mapped to 768 dimensions, the unified normalization strategy is used for BERT text features and visual features, and the cross-modal fusion loss is reduced to ensure that the feature transmission is distortionless.

[0100] Figure 2 The flowchart of the performance-improved remote sensing large model provided by the embodiment of the application is shown, which completes the user's question and answer. Figure 2 Among them, it includes multi-modal remote sensing data input, image input preprocessing, ConvToMe composite visual coding layer (60% data input ConvNeXt layer for local detail extraction; 40% data input ToMe ViT layer for global semantic capture; attention weighted fusion to obtain composite visual features), remote sensing text input, BERT text encoding, ConvToMe composite visual coding layer output and BERT text encoding output, Q-Former cross-modal alignment, loop reasoning calculation, dynamic threshold screening, if it meets the standard, the result is post-processed and output, including JSON (JavaScript Object Notation, JavaScript object notation, a lightweight text data exchange format) attribute, PNG (Portable Network Graphics, Portable Network Graphics, a bitmapped image format using lossless compression algorithm) visualization, GIS (Geographic Information Systems, Geographic Information Systems) coordinates; if it does not meet the standard, it is returned to the ConvToMe layer, re-reasoning, and re-executing the ConvToMe layer and the subsequent Q-Former layer, loop reasoning calculation layer process.

[0101] In an embodiment of the application, a possible implementation is provided, the ConvNeXt layer adopts a four-level architecture of Stem (backbone) layer, deep convolution layer, inverted bottleneck structure and dimension mapping layer;

[0102] Among them, the Stem layer is used to realize input adaptation and preliminary feature extraction, and outputs a preliminary feature matrix;

[0103] The deep convolution layer is used to realize local detail enhancement and receptive field expansion, and outputs an enhanced local feature matrix;

[0104] The inverted bottleneck structure realizes local feature dimension expansion and compression through the steps of channel dimension expansion, feature extraction and channel dimension compression, and outputs a high-expression local feature matrix.

[0105] The dimension mapping layer is configured to realize feature dimension unification and output adaptation, and output a local detail feature matrix.

[0106] In an embodiment of the present application, a possible implementation is provided, and the ToMe ViT layer adopts a four-level architecture of image block segmentation and projection, geographical enhanced position encoding, a multi-layer Transformer encoder and global feature reconstruction, wherein the multi-layer Transformer encoder is divided into multiple stages, and a token merging ToMe module is inserted after each stage, forming a progressive global modeling process of encoding, merging and re-encoding.

[0107] In an embodiment of the present application, a possible implementation of geographical enhanced position encoding is realized through the following steps A1 to A6.

[0108] Step A1, geographical coordinate extraction and preprocessing, specifically reading latitude and longitude information from input remote sensing image metadata, obtaining the upper left corner coordinates of the image, and calculating the absolute geographical coordinates of the center of each image block;

[0109] Step A2, coordinate conversion, specifically converting the absolute latitude and longitude into relative coordinates, mapping the relative coordinates to the [0, 1] interval, and eliminating the influence of different geographical ranges of images;

[0110] Step A3, spatial dimension encoding, specifically adopting sinusoidal position encoding, generating a position vector based on the pixel index and column index of the token Token in the image, and preserving the pixel-level relative position relationship between Tokens;

[0111] Step A4, geographical dimension encoding, specifically adopting linear encoding, mapping the relative latitude and longitude to a preset dimension vector through a linear layer, and integrating the actual geographical scale correlation;

[0112] Step A5, encoding fusion, specifically adding the spatial dimension encoding and the geographical dimension encoding element by element to generate the final position encoding vector, ensuring that the Token carries dual information of pixel position and geographical scale;

[0113] Step A6, encoding injection, specifically adding the position encoding vector and the corresponding Token feature vector without changing the Token dimension, and only supplementing the spatial geographical correlation information.

[0114] The embodiment breaks through the limitation of the traditional ViT sine position encoding, integrates the geographic coordinate information of the remote sensing image, ensures that the Token sequence retains the "actual geographic scale correlation" (such as the spatial distance difference of different latitude features) when globally modeling, and avoids the global semantic misplacement of large targets caused by "pixel scale position encoding".

[0115] In the embodiment of the application, a possible implementation manner is provided, and the plurality of stages include a first stage, a second stage and a third stage; the first stage focuses on Token local correlation and preliminarily captures small-range spatial dependence; the second stage extends to middle-layer correlation and strengthens feature combination semantics; and the third stage realizes global correlation and extracts large-target global semantics and scene category information. The first stage, the second stage and the third stage can be respectively referred to as a shallow stage, a middle stage and a deep stage.

[0116] In the embodiment of the application, a possible implementation manner is provided, and the ToMe module realizes the merging of semantically similar Tokens through the following steps B1 to B5.

[0117] Step B1, Token similarity calculation, specifically calculating the cosine similarity between all Tokens in the current Token sequence;

[0118] Step B2, similarity threshold, specifically setting a merging threshold;

[0119] Step B3, similar Token clustering, specifically using a preset greedy clustering algorithm to divide Tokens with a similarity greater than the merging threshold into a cluster, and selecting the Token with the highest average similarity in the cluster as the center Token;

[0120] Step B4, merged feature calculation, specifically the new Token feature is the weighted sum of all Token features in the cluster, and the weight is the similarity between the Token and the center Token;

[0121] Step B5, residual reservation, specifically adding the new Token feature to the center Token feature before merging through residual connection to ensure that the core semantics are not lost.

[0122] The embodiment captures the global spatial dependence between Tokens (such as the adjacent relationship between roads and buildings and the symbiotic relationship between water systems and vegetation) through a multi-layer Transformer encoder, and integrates the ToMe Token merging module to reduce the number of Tokens while retaining core semantic information, thereby solving the problems of large calculation and semantic redundancy of remote sensing large targets in the traditional ViT.

[0123] The above introduces Figure 1The various aspects of the embodiments shown will be further described below through specific embodiments of the composite visual coding-based remote sensing large model performance improvement method of the embodiments of the application.

[0124] In specific embodiments, the visual general language model is a VisualGLM-6B open source model, and based on the architecture of the VisualGLM-6B open source model, a preliminary remote sensing large model is built by adapting multi-modal remote sensing image data and text data, wherein the architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a recurrent inference calculation layer, and a dynamic threshold screening system; a composite visual coding ConvToMe layer for extracting local details and capturing global semantics is added to the preliminary remote sensing large model, the ConvToMe layer extracts local details through a ConvNeXt layer and captures global semantics in combination with a ToMe ViT layer, thereby obtaining a remote sensing large model with improved performance, and through the composite visual coding, the global context and local details are retained at the same time, the robustness is improved, and the model precision is improved.

[0125] The specific implementation scheme is as follows:

[0126] 1. Image input pre-processing (i.e., multi-modal differentiation processing)

[0127] Referring to Figure 3 , special pre-processing and unified format conversion are performed on three core remote sensing modalities of optical data, SAR data, and hyperspectral data to ensure the consistency and effectiveness of the input data.

[0128] 1.1 Special pre-processing of each modality

[0129] Optical image (such as satellite RGB):

[0130] Radiometric correction: using the FLAASH (Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes) atmospheric correction module of ENVI (Environment for Visualizing Image, a software widely used in remote sensing image processing and analysis) to eliminate atmospheric scattering and aerosol effects;

[0131] Geometric correction: based on GCP (Ground Control Point), using a second-order polynomial transformation to align the image to the WGS84 (World Geodetic System 1984) coordinate system, with an error controlled within 1 pixel;

[0132] Contrast enhancement: CLAHE (Contrast Limited Adaptive Histogram Equalization) is used, with clipLimit (contrast limit threshold) set to 2.0, to enhance the edge details of features.

[0133] SAR data (e.g. C-band full-polarization SAR data):

[0134] Speckle noise reduction: Lee filter (a classic adaptive filter algorithm, window size 5x5) is used to suppress speckle noise.

[0135] Polarization decomposition: Pauli decomposition (a basic polarization SAR target decomposition) is used to convert the polarization data into "HH+VV, HH-VV, HV" three-channel feature maps; here, HH+VV represents the sum of horizontally transmitted and vertically received echoes, HH-VV represents the difference between horizontally transmitted and vertically received echoes, and HV represents horizontally transmitted and vertically received echoes.

[0136] Radiometric calibration: radar echo intensity is converted into backscatter coefficient , the formula is: where K is the calibration coefficient (provided by sensor parameters), DN is the digital number, the original brightness value of the pixel in the SAR image, representing the intensity of the echo.

[0137] Hyperspectral data (e.g. Hyperion satellite data):

[0138] Bad pixel repair: the neighborhood mean filling method is used to repair pixels with gray value deviating from the mean value by 3 times the standard deviation in the band;

[0139] Band selection: MIC (Maximal Information Coefficient) is used to select 20-30 sensitive bands related to the target (e.g. NDVI related bands for vegetation monitoring); here NDVI is the normalized difference vegetation index, calculated by the reflectivity of near-infrared and red bands, commonly used to represent the growth state of vegetation;

[0140] Dimensionality reduction: PCA (Principal Component Analysis) is used to reduce the selected bands to 10 dimensions, retaining more than 95% of the information.

[0141] 1.2 Unified format conversion

[0142] Continued from Figure 3 , including adaptive scaling, pixel value normalization, three-dimensional matrix conversion, and output of three-dimensional matrix, as follows:

[0143] Adaptive scaling: all modal data are scaled to 512x512 pixels using bilinear interpolation, which is only illustrative here and can be other pixels;

[0144] Pixel value normalization: min-max normalization is used, and the formula is: All pixel values are mapped to the [0, 1] interval to eliminate the brightness and intensity differences of different modalities; where, is the normalized value, x is the pixel value, is the minimum pixel value, is the maximum pixel value;

[0145] Three-dimensional matrix conversion: the preprocessed data is converted into a three-dimensional matrix (height Hx width Wx channel number C), where: H=512, W=512, the channel number C=3 for optical data, the channel number C=3 for polarization data, and the channel number C=10 for hyperspectral data;

[0146] Output: three-dimensional matrix, as the input of the subsequent ConvToMe layer.

[0147] 2. ConvToMe layer: see Figure 4 The preprocessed three-dimensional matrix is divided in a 6:4 ratio, with 60% input into the ConvNeXt layer to extract local details and 40% input into the ToMe ViT layer to capture global semantics, and finally output the composite visual features through feature fusion. This ratio allocation is on the one hand because local details are often the focus of remote sensing images, and on the other hand because the computational consumption of the two structures is different, a 5:5 computational load allocation is more reasonable, which can save computational overhead from a global perspective.

[0148] 2.1 ConvNeXt layer

[0149] Input: 60% of the preprocessed remote sensing image three-dimensional matrix data, with a specification of 512x512xC (C is the channel number), and the pixel value has been normalized to the [0, 1] interval;

[0150] Output: 512x512x768 local detail feature matrix, which is consistent in dimension with the global feature matrix output by the ToMe ViT layer, ensuring that there is no dimension conflict in subsequent attention weighted fusion.

[0151] Overall, see Figure 5 The ConvNeXt layer adopts a four-level architecture of "Stem layer - deep convolutional layer - inverted bottleneck structure - dimension mapping layer", each level is optimized for the characteristics of remote sensing data, and the specific design is as follows:

[0152] 2.1.1 Stem layer: input adaptation and preliminary feature extraction

[0153] Convolution operation: 4x4 non-overlapping convolution, stride=4, padding=0, number of convolution kernels set to 128;

[0154] Non-overlapping design: ensures that each 4x4 pixel block of the input image corresponds to only one output feature point, avoiding confusion of pixel information and adapting to the "block integrity" requirement of small remote sensing targets (such as the 4x4 pixel pole top structure);

[0155] Stride and padding: stride=4 compresses the output feature map size from 512x512 to 128x128 (512 / 4=128), and no padding ensures that there is no redundant padding in the edge area, which is consistent with the geographical boundary authenticity of remote sensing images;

[0156] Activation and normalization: after convolution, LayerNorm (layer normalization) is connected, with the normalization dimension being the channel dimension (128 dimensions), avoiding the BatchNorm (batch normalization) sensitivity problem of remote sensing data "large brightness difference within the batch"; no activation function is connected, preserving the linear representation of the initial features, providing more rich original information for subsequent deep convolution;

[0157] Output specification: 128x128x128 preliminary feature matrix, the number of channels is uniformly converted from input C to 128, completing input adaptation.

[0158] The advantage of this is to replace the traditional ConvNeXt "7x7 convolution + pooling" structure, reducing edge feature loss while uniformly converting the input channel number to the basic channel number for subsequent network adaptation, avoiding excessive compression of small target details by initial pooling.

[0159] 2.1.2 Deep convolution layer: local detail enhancement and receptive field expansion

[0160] Network layer and structure: a total of 8 layers of depth separable convolution are set, divided into 2 stages (4 layers per stage), and the channel number is adjusted through 1x1 convolution between stages (maintaining 128 dimensions) to avoid feature dimension expansion;

[0161] Convolution kernel size: 7x7 (instead of traditional 3x3), by expanding the receptive field (receptive field range=7+(7-1)x(4-1)=19 pixels, corresponding to 19x4=76 pixels of the original image), covering the complete area of small targets (such as 20-pixel ships), avoiding feature fragmentation;

[0162] Stride and padding: stride=1, padding=3 ((7-1) / 2=3), ensuring that the output feature map size is consistent with the input (128x128), without losing edge details;

[0163] Group number: Set to 128 (consistent with the number of input channels), implement "channel-wise convolution", and strengthen the local feature expression of each channel (such as the "HH+VV" channel of SAR data focusing on ship echo features, and the "HV" channel focusing on vegetation scattering features);

[0164] Activation and normalization: After every 2 layers of deep convolution, insert 1 LayerNorm (channel dimension normalization) and 1 GELU (Gaussian Error Linear Unit) activation function;

[0165] GELU activation: Compared with ReLU (Rectified Linear Unit), it can perform smooth activation on "weak edge features" of remote sensing data (such as low-contrast road boundaries), avoiding feature loss;

[0166] Normalization interval: Inserted once every 2 layers, balance "feature stability" and "computing efficiency", and avoid inference delay caused by frequent normalization;

[0167] Output specification: 128x128x128 enhanced local feature matrix, retaining key detail information such as building right angles, coastline polyline, and crop row parallel texture.

[0168] This can expand the receptive field through large-size depth separable convolution, accurately capture local edge and texture features of remote sensing features, while reducing parameter quantity and computational complexity, balancing detail extraction accuracy and efficiency.

[0169] 2.1.3 Inverted bottleneck structure: local feature dimension expansion and compression

[0170] Dimension expansion phase: Use 1x1 convolution to expand the input channel number from 128 dimensions to 512 dimensions (expansion coefficient = 4);

[0171] Expansion purpose: Provide more abundant dimensional space for local features to capture fine-grained differences in remote sensing data (such as subtle differences in RGB spectrum of different crops and SAR data echo differences of different building materials);

[0172] Convolution parameters: number of convolution kernels = 512, step = 1, padding = 0, output feature map size remains 128x128;

[0173] Feature enhancement phase: access GELU activation function to perform nonlinear transformation on the expanded 512-dimensional features, strengthening the distinction between "effective detail features" (such as high-value features of the green band of vegetation) and "ineffective interference features" (such as single-point noise); do not access the normalization layer to avoid detail blurring caused by excessive smoothing;

[0174] Dimension compression stage: 1x1 convolution is used to compress the channel number from 512 dimensions back to 128 dimensions (consistent with the output channel of the deep convolutional layer);

[0175] Compression purpose: eliminate redundant dimensions, retain core local features, and avoid dimension imbalance during subsequent fusion;

[0176] Convolution parameters: number of convolution kernels = 128, step = 1, padding = 0, output feature map size remains 128x128;

[0177] Residual connection: add residual connection at both ends of the inverted bottleneck structure (input features are directly superimposed on output features), avoid gradient vanishing of deep network, ensure that weak detail features of remote sensing small targets (such as 10-pixel power pole shadow) can be effectively returned, and improve training stability;

[0178] Output specification: 128x128x128 high-expression local feature matrix, which integrates multi-dimensional detail correlation information (such as combined features of "building edge + material texture").

[0179] In this way, through the "channel dimension expansion-feature extraction-channel dimension compression" inverted design, the nonlinear expression ability of local features can be enhanced, and the complex correlation of "multi-dimensional details" (such as RGB texture of optical image and polarization features of SAR data) in remote sensing data can be adapted.

[0180] 2.1.4 Dimension mapping layer: feature dimension unification and output adaptation

[0181] Convolution operation: 1x1 convolution is used to map the channel number from 128 dimensions to 768 dimensions;

[0182] Convolution kernel parameters: number of convolution kernels = 768, step = 1, padding = 0, output feature map size remains 128x128;

[0183] Weight initialization: Xavier initialization is used to ensure that the feature mean and variance are stable during the mapping process, and to avoid feature distortion caused by dimension conversion;

[0184] Feature up-sampling: 128x128x768 feature matrix is up-sampled to 512x512x768 through bilinear interpolation;

[0185] Up-sampling purpose: restore to the same size as the input image (512x512), ensure that local features can be aligned with global features at the "pixel level" (such as 512x512 image, (x, y) coordinates of local edge features and global semantic features are accurately matched);

[0186] Interpolation parameters: interpolation kernel size = 3x3, ensure the edge features after upsampling are smooth and without jaggedness (such as the building boundary after upsampling still maintains linearity);

[0187] Final normalization: access LayerNorm (channel dimension normalization), the last stability optimization for 768-dimensional features, ensures that the mean of the output features is 0 and the variance is 1, and adapts the weight calculation logic of the subsequent attention fusion;

[0188] Output specification: 512x512x768 local detail feature matrix, each pixel point corresponds to the local detail information (such as edge intensity, texture direction, material attribute) of the input image, which can be directly input into the attention fusion module and combined with global features.

[0189] The advantage of this link is to map the 128-dimensional local features output by the inverted bottleneck structure to the 768-dimensional consistent with the ToMe ViT layer, providing a dimensional basis for the attention weighted fusion of the ConvToMe layer, while strengthening the semantic relevance of local features.

[0190] 2.2 ToMe ViT layer

[0191] Input: 40% of the pre-processed remote sensing image three-dimensional matrix, with a specification of 512x512xC (C is the number of channels), and the pixel value has been normalized to the [0,1] interval;

[0192] Output: 512x512x768 global semantic feature matrix, consistent with the dimension of the local detail feature matrix output by the ConvNeXt layer, ensuring that there is no dimensional conflict in the subsequent attention weighted fusion.

[0193] Reference Figure 6 Overall, the ToMe ViT layer adopts a four-level architecture of "image block segmentation and projection-geographical enhanced position encoding-Transformer encoder (integrated ToMe module)-global feature reconstruction", each level is optimized for the global semantic needs of remote sensing data, and the specific design is as follows:

[0194] 2.2.1 Image block segmentation and linear projection: global feature unit construction

[0195] The input 512x512 remote sensing image is divided into fixed-size image blocks (Token), which are converted into high-dimensional feature vectors through linear projection, providing basic feature units for subsequent Transformer global modeling, while ensuring that the Token size adapts to the spatial distribution of remote sensing large targets:

[0196] Image block segmentation rule: use 16x16 non-overlapping segmentation to evenly divide the 512x512 image into 32x32=1024 image blocks;

[0197] The segmentation size adaptation 16x16 pixel block corresponds to the actual geographic range of different resolution remote sensing images (such as 0.5 meter resolution corresponds to 8x8 meters, 30 meter resolution corresponds to 480x480 meters), which can cover small targets (such as 16x16 pixel small buildings) and efficiently construct the global correlation of large targets (such as 10 Tokens can cover 4.8x4.8 kilometers of contiguous farmland under 30 meter resolution);

[0198] Linear projection operation: each 16x16xC image block is converted into a 768-dimensional feature vector (Token) through a linear projection layer;

[0199] Projection parameters: linear layer input dimension = 16x16xC (such as C=3 for optical images, input dimension = 768), output dimension = 768, and weight is initialized with He to ensure stable feature variance after projection;

[0200] Output specification: 1024x768 Token sequence (1024 Tokens, each Token is a 768-dimensional feature vector), each Token corresponds to a 16x16 pixel region of the input image, and carries the local channel fusion information of the region.

[0201] 2.2.2 Geographical enhanced position coding: remote sensing spatial correlation preservation

[0202] Breakthrough the limitations of traditional ViT sinusoidal position coding, integrate the geographical coordinate information of remote sensing images, ensure that the Token sequence retains the "actual geographical scale correlation" (such as the spatial distance difference of ground objects at different latitudes) when globally modeling, and avoid the global semantic misalignment of large targets caused by "pixel scale position coding", the specific implementation is as follows:

[0203] Geographical coordinate extraction and preprocessing: read the latitude and longitude information from the input remote sensing image metadata, obtain the top-left corner coordinates of the image, and calculate the absolute geographical coordinates of the center of each image block;

[0204] Coordinate conversion: convert the absolute latitude and longitude into relative coordinates, map the relative coordinates to the [0,1] interval, and eliminate the influence of different image geographical range differences;

[0205] Spatial dimension coding: use traditional sinusoidal position coding to generate position vectors based on the pixel index and column index of Token in the image, and preserve the pixel-level relative position relationship between Tokens;

[0206] Geographical dimension coding: use linear coding to map the relative latitude and longitude to a 768-dimensional vector through a linear layer, and integrate the actual geographical scale correlation;

[0207] Encoding fusion: element-wise addition of spatial dimension encoding and geographic dimension encoding to generate the final position encoding vector, ensuring that the Token carries both "pixel position + geographic scale" dual information;

[0208] Encoding injection: add the position encoding vector to the corresponding Token feature vector without changing the Token dimension, only supplementing spatial-geographic association information;

[0209] Output specification: 1024x768 enhanced Token sequence, each Token contains both image block feature information and precise spatial-geographic position association, providing geographic anchors for global semantic modeling.

[0210] 2.2.3 Transformer encoder (integrated ToMe module): global semantic modeling and computational optimization

[0211] Transformer encoder overall architecture: a total of 12 layers of Transformer encoder, divided into 3 stages (4 layers per stage), and each stage is inserted with a ToMe module, forming a progressive global modeling process of "encoding-merging-re-encoding".

[0212] Stage function division:

[0213] Shallow stage (1-4 layers): focus on Token local association (such as the surrounding feature association of 16x16 pixel blocks), and preliminarily capture small-range spatial dependencies;

[0214] Middle stage (5-8 layers): expand to middle-level association (such as road segments and farmland blocks composed of multiple Tokens), and strengthen the combination semantics of features;

[0215] Deep stage (9-12 layers): realize global association (such as the spatial network of all image Tokens), and extract large target global semantics and scene category information.

[0216] Single Transformer layer internal structure: each layer contains "multi-head self-attention (MSA, Multi-head Self-Attention) - layer normalization (LN, LayerNorm) - feedforward neural network (FFN, Feedforward Neural Network)" three-step operation, and the parameters are optimized for remote sensing data:

[0217] Multi-head self-attention (MSA):

[0218] Number of heads: 8 attention heads, each head dimension = 768 / 8 = 96, balancing global capture ability and computational efficiency;

[0219] Attention computation: scaled dot-product attention is adopted, with the formula as follows: where Q , K , V are query, key, value matrices respectively, all generated by linear transformation of Token sequence, softmax softmax function "compresses" and converts a vector containing arbitrary real numbers into a probability distribution, so that the value of each element is in the interval (0, 1), and the sum of all elements is 1, d k is a scaling factor;

[0220] Remote sensing adaptation: when calculating attention weights, Tokens with small differences in geographical dimension coding (i.e. close actual geographical distance) are given an increased base weight coefficient (1.2 times), strengthening the capture of spatial correlation of adjacent features (such as the semantic correlation of adjacent buildings in a city).

[0221] Layer normalization (LN): 1 LN is inserted before and after MSA (normalization dimension = 768), to avoid feature distribution deviation caused by remote sensing data brightness fluctuations.

[0222] Feedforward neural network (FFN): "linear layer-GELU activation-linear layer" structure is adopted, with intermediate dimension = 3072 (4 times the Token dimension), with the formula as follows: where W 1, W 2 are weights, b 1, b 2 are coefficients, GELU () is the activation function;

[0223] Activation function selection: GELU, compared to ReLU, can smoothly activate "weak semantic features" of remote sensing data (such as the transition area semantics between grassland and desert), avoiding feature loss;

[0224] Dimension design: 3072-dimensional intermediate layer provides sufficient space for semantic expression, adapting to complex feature category association in remote sensing scenarios (such as "farmland-irrigation canal-road" multi-category semantic combination).

[0225] ToMe Token merging module: ToMe module is inserted after each stage of Transformer encoding, realizing "semantic similar Token merging", with the specific process as follows:

[0226] Token similarity calculation: calculate the cosine similarity between all Tokens in the current Token sequence;

[0227] Similarity threshold: Set 0.85 as the merging threshold, only merge Tokens with high semantic similarity (such as adjacent 16x16 pixel block Tokens in contiguous farmland), avoid merging Tokens with large semantic differences (such as farmland and road Tokens);

[0228] Similar Token clustering: Use a greedy clustering algorithm to divide Tokens with similarity ≥ 0.85 into a cluster, and select 1 center Token (the Token with the highest average similarity within the cluster) for each cluster;

[0229] Merging feature calculation: The new Token feature is the weighted sum of all Token features within the cluster, with the weight being the similarity between the Token and the center Token;

[0230] Residual preservation: After merging, the new Token feature is added to the center Token feature before merging to ensure that the core semantics are not lost;

[0231] Merging effect: After each stage of merging, the number of Tokens is reduced by 30-40%, and after 12 layers of encoding + 3 times of ToMe merging, the number of Tokens is reduced from 1024 to 600-700, with a reduction of more than 40% in computational load, while preserving the global semantic association of large targets;

[0232] Output specification: 600-700x768 simplified Token sequence, each Token carries global semantic association information (such as "lake Token" containing the semantic association of surrounding wetlands and shorelines), providing high-quality semantic units for subsequent global feature reconstruction.

[0233] The technical effect of this is to capture the global spatial dependencies between Tokens (such as the adjacent relationship between roads and buildings, and the symbiotic relationship between water systems and vegetation) through multiple Transformer encoders, while integrating the ToMe Token merging module to reduce the number of Tokens while preserving core semantic information, solving the problem of "large computational load and semantic redundancy of remote sensing large targets" in traditional ViT.

[0234] 2.2.4 Global feature reconstruction: aligning semantic features with input image size

[0235] The core function is to reconstruct the simplified Token sequence output by the Transformer encoder into a global semantic feature matrix consistent with the input image size (512x512), ensuring that the local detail features output by the ConvNeXt layer are aligned at the "pixel level", providing a spatial matching basis for attention weighted fusion. The specific implementation is as follows:

[0236] Token sequence expansion: The compact Token sequence of 600-700x768 is rearranged into a feature matrix of 32x32x768 based on the spatial indices of the original image block segmentation (restored to the segmented image block grid structure);

[0237] Missing Token processing: If there is no Token in a certain grid position after merging (rare cases), use the 3x3 neighborhood Token feature mean to fill in, ensuring the integrity of the feature matrix;

[0238] Global average pooling: For the feature matrix of 32x32x768, perform global average pooling at each 16x16 pixel block corresponding grid position, map the 32x32 grid feature to a compact feature matrix of 32x32x768;

[0239] Pooling purpose: Further strengthen the global semantic association of each grid position, avoid semantic bias caused by local Token fluctuations (such as the influence of individual noise Tokens in contiguous farmland);

[0240] Bilinear interpolation up-sampling: up-sample the compact feature matrix of 32x32x768 to 512x512x768, which is exactly the same size as the input image;

[0241] Interpolation parameters: interpolation kernel size = 3x3, step = 16 (32x16 = 512), to ensure that the global semantic features after up-sampling accurately correspond to the geographic coordinates of each pixel position in the input image, such as the global semantic features of (x, y) pixel in a 512x512 image, which completely match the Token semantics of the 16x16 block to which the pixel belongs;

[0242] Final normalization: access LayerNorm (normalization dimension = 768) to standardize the reconstructed global semantic feature matrix, ensuring that the feature mean = 0, variance = 1, consistent with the distribution of local detail features output by the ConvNeXt layer, adapting to the weight calculation logic of attention weighted fusion;

[0243] Output specification: global semantic feature matrix of 512x512x768, each pixel point corresponds to the global semantic information of the input image (such as feature class probability, large target attribution, spatial correlation strength), which can be directly input into the attention fusion module and combined with local detail features.

[0244] 3. BERT text encoder for remote sensing (text feature generation)

[0245] Referring to Figure 7 , for remote sensing text (such as query instruction "identify farmland damage area in SAR data image", labeled text "NDVI value 0.6-0.8 is vegetation cover area"), generate high-precision text features.

[0246] 3.1 Text normalization

[0247] Tokenization: Use remote sensing-specific jiaba tokenization (integrate professional term dictionaries such as "SAR, NDVI, polarization decomposition") to avoid term splitting;

[0248] Remove stop words: Filter meaningless words (such as "of, in") and keep remote sensing keywords (such as "damage, vegetation, polarization");

[0249] Text truncation or completion: Uniform text length to 512Tokens (truncate if exceeded).

[0250] 3.2 BERT model fine-tuning and encoding

[0251] Base model: Use bert-base-chinese-remote-sensing (fine-tuned on remote sensing text dataset RSICD);

[0252] Text feature vector generation: After BERT encoding, take the output of the [CLS] token as the text feature vector, with a dimension of 768;

[0253] Semantic reinforcement: Through remote sensing knowledge graph embedding (encode "farmland -> vegetation", "SAR -> microwave remote sensing" and other related relationships into vectors), and weighted fusion with BERT features (weight ratio of former and latter is 0.8:0.2), to enhance text semantic relevance.

[0254] 4. Q-Former layer (cross-modal feature alignment)

[0255] Referring to Figure 8 , receive ConvToMe features (512x512x768) and BERT text features (768 dimensions), query vector interaction, organize multiple rounds of alignment, and finally feature fusion to achieve visual and text semantic unity:

[0256] 4.1 Design of learnable query vector

[0257] Initialize 32 learnable query vectors ( Q ) with a dimension of 768, consistent with the text feature dimension. Query vectors are optimized through remote sensing scene pre-training (input is "remote sensing image-text" pairs) to ensure their ability to associate visual and text.

[0258] 4.2 Multi-round cross-attention alignment

[0259] Perform 3 rounds of cross-attention interaction, each round containing visual-guided text and text-guided visual:

[0260] Visual guided text: , where, is the text feature of the current round, is the visual feature, Attention () is the cross-attention, is the ConvToMe feature;

[0261] Text guided visual: , where,

[0262] Alignment loss calculation: adopt contrastive learning loss , get the minimum distance of visual text feature, where is the text feature of each round.

[0263] 4.3 Cross-modal feature fusion

[0264] After alignment, the visual feature (512x512x768) and the text feature (768 dimensions) are fused by dimension expansion and element-wise multiplication: the text feature is expanded to 512x512x768 and multiplied with the visual feature to generate a 512x512x768 cross-modal feature matrix, which is input into the recurrent inference layer.

[0265] 5. Low-rank recurrent inference calculation (efficiency optimization)

[0266] Referring to Figure 9 , for the cross-modal feature matrix, through low-rank decomposition and multi-round inference, the calculation complexity is reduced while the accuracy is guaranteed:

[0267] 5.1 Low-rank matrix decomposition

[0268] For the inference layer weight matrix (such as 1024x1024) after the input of the cross-modal feature matrix, SVD (singular value decomposition) is adopted to decompose it into two low-rank matrices, which is the model weight matrix: W=W_1xW_2, for example, W_1∈R^(1024x64), W_2∈R^(64x1024), so that the number of parameters is reduced from 1,048,576 to 131,072 (88% reduction).

[0269] 5.2 Recurrent inference mechanism

[0270] Perform 3-5 rounds of inference iterations, and the process of each round is:

[0271] Feature update: F_{t+1}=F_txW_1xW_2+Residual(F_t) (residual connection retains the original feature);

[0272] Precision verification: Calculate the MAE (Mean Absolute Error) of the features of the two adjacent rounds, if MAE < 0.01 or the number of iterations reaches 5, terminate the loop;

[0273] Output optimization: The final round of features is standardized by LayerNorm to generate a 512x512x768 inference feature matrix.

[0274] 6. Dynamic threshold screening and result output

[0275] Continuing to refer to Figure 9 , according to the remote sensing task type, the screening rules are adaptively adjusted to ensure the accuracy of the output results:

[0276] 6.1 Task scenario threshold mapping

[0277] In one embodiment, a mapping table of task type to threshold is established, wherein:

[0278] The confidence threshold for land classification tasks is ≥ 0.85, there is no IOU threshold, and the core requirement is high boundary accuracy;

[0279] The confidence threshold for disaster monitoring tasks is ≥ 0.7, the IOU threshold is ≥ 0.5, and the core requirement is high recall rate to avoid missing detection;

[0280] The confidence threshold for marine target detection tasks is ≥ 0.8, the IOU threshold is ≥ 0.6, and the core requirement is high small target detection accuracy.

[0281] In another embodiment, a general threshold can be set in advance, such as a confidence threshold of ≥ 0.8 and an IOU threshold of ≥ 0.6, which is not limited in this embodiment.

[0282] 6.2 Result screening and secondary optimization

[0283] Valid results: If the inference results meet the threshold requirements of the corresponding task, post-processing is performed (such as non-maximum suppression NMS to remove duplicate detection boxes, morphological dilation to optimize segmentation boundaries), standardized output (such as target coordinates, class, confidence) is generated, and feedback is output to the user;

[0284] Invalid results: If less than the threshold, the results are returned to the ConvToMe layer, re-inferred, and the ConvToMe layer and subsequent Q-Former layer, loop inference layer processes are re-executed.

[0285] 6.3 Output format

[0286] Spatial information: Target coordinates use WGS84 latitude and longitude (such as "116.39°E, 39.91°N");

[0287] Attribute information: encapsulated in JSON format (such as {"target category": "farmland", "damage level": "moderate", "confidence": 0.88});

[0288] Visualization: generate a PNG image of "original image + target label + confidence", support API (Application Programming Interface, Application Programming Interface) calling and file exporting.

[0289] At this point, the user's input including image information and text is completed.

[0290] It should be noted that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In actual application, all possible implementation manners described above can be combined in any combination to form possible embodiments of the present application, which will not be repeated here.

[0291] Based on the composite visual coding-based remote sensing large model performance improvement method provided in each of the above embodiments, based on the same inventive concept, the embodiments of the present application also provide a composite visual coding-based remote sensing large model performance improvement device.

[0292] Figure 10 is the structural diagram of the composite visual coding-based remote sensing large model performance improvement device provided by the embodiments of the present application. As Figure 10 shown, the composite visual coding-based remote sensing large model performance improvement device can specifically include a building unit 1010, an improvement unit 1020, and an inference unit 1030.

[0293] The building unit 1010 is configured to adapt multi-modal remote sensing image data and text data based on an open-source visual general language model, and build a preliminary remote sensing large model, wherein the architecture of the preliminary remote sensing large model includes a text editor BERT layer, a query converter Q-Former layer, a recurrent inference calculation layer, and a dynamic threshold screening system.

[0294] The improvement unit 1020 is configured to add a composite visual coding ConvToMe layer for extracting local details and capturing global semantics in the preliminary remote sensing large model, wherein the ConvToMe layer extracts local details through a convolutional neural network ConvNeXt layer, and captures global semantics by combining a token merging visual encoder ToMe ViT layer, to obtain a performance-improved remote sensing large model.

[0295] The reasoning unit 1030 is configured to, in response to a user question including multi-modal remote sensing image data and text data, pre-process the multi-modal remote sensing image data in the user question to obtain a multi-modal pre-processing matrix; and

[0296] The multi-modal pre-processing matrix is divided according to a preset ratio to obtain a first multi-modal pre-processing matrix and a second multi-modal pre-processing matrix; the first multi-modal pre-processing matrix is input into a ConvNeXt layer in the performance-improved remote sensing large model to output a local detail feature matrix; the second multi-modal pre-processing matrix is input into a ToMe ViT layer in the performance-improved remote sensing large model to output a global semantic feature matrix; the local detail feature matrix and the global semantic feature matrix are fused to output a composite visual feature matrix; and

[0297] The text data in the user question is input into a BERT layer in the performance-improved remote sensing large model to output a text feature vector; and

[0298] The composite visual feature matrix and the text feature vector are input into a Q-Former layer in the performance-improved remote sensing large model to perform steps of query vector interaction, multi-round cross-attention alignment, and cross-modal feature fusion to output a cross-modal feature matrix; and

[0299] The cross-modal feature matrix is input into a recurrent reasoning calculation layer in the performance-improved remote sensing large model to perform steps of low-rank matrix decomposition and recurrent reasoning to output a reasoning feature matrix, and then a preliminary reasoning result including a confidence and an intersection over union (IOU) is obtained according to the reasoning feature matrix; and

[0300] The preliminary reasoning result is input into a dynamic threshold screening system in the performance-improved remote sensing large model to determine a target task corresponding to the preliminary reasoning result; and

[0301] Whether the preliminary reasoning result meets a threshold requirement of the target task is determined according to the confidence and the IOU in the preliminary reasoning result; and

[0302] If the preliminary reasoning result meets the threshold requirement of the target task, a post-processing operation is performed to output a final reasoning result including a target coordinate, a target category, and a confidence.

[0303] If the preliminary reasoning result does not meet the threshold requirement of the target task, the preliminary reasoning result is returned to the ConvToMe layer for re-reasoning, and the ConvToMe layer and subsequent Q-Former layer and recurrent reasoning calculation layer processes are re-executed.

[0304] In the embodiments of the present application, a possible implementation manner is provided, the ConvNeXt layer adopts a four-level architecture of a Stem layer, a deep convolutional layer, an inverted bottleneck structure, and a dimension mapping layer.

[0305] wherein, the Stem layer is configured to implement input adaptation and preliminary feature extraction, and output a preliminary feature matrix;

[0306] the deep convolution layer is configured to implement local detail enhancement and receptive field expansion, and output an enhanced local feature matrix;

[0307] the inverted bottleneck structure is configured to implement local feature dimension expansion and compression through the steps of channel dimension expansion, feature extraction and channel dimension compression, and output a high-expression local feature matrix;

[0308] the dimension mapping layer is configured to implement feature dimension unification and output adaptation, and output a local detail feature matrix.

[0309] In an embodiment of the present application, a possible implementation is provided, the ToMe ViT layer adopts a four-level architecture of image block segmentation and projection, geographical enhanced position encoding, multi-layer Transformer encoder and global feature reconstruction, wherein the multi-layer Transformer encoder is divided into multiple stages, and a token merging ToMe module is inserted after each stage, forming a progressive global modeling process of encoding, merging and re-encoding.

[0310] In an embodiment of the present application, a possible implementation is provided, the inference unit 1030 is further configured to implement geographical enhanced position encoding through the following steps:

[0311] geographical coordinate extraction and preprocessing, specifically reading latitude and longitude information from input remote sensing image metadata, obtaining the upper left corner coordinates of the image, and calculating the absolute geographical coordinates of the center of each image block;

[0312] coordinate conversion, specifically converting absolute latitude and longitude into relative coordinates, mapping the relative coordinates to the [0, 1] interval, and eliminating the influence of different image geographical range differences;

[0313] spatial dimension encoding, specifically adopting sinusoidal position encoding, generating a position vector based on the pixel index and column index of the token Token in the image, and preserving the pixel-level relative position relationship between Tokens;

[0314] geographical dimension encoding, specifically adopting linear encoding, mapping the relative latitude and longitude through a linear layer into a preset dimension vector, and integrating the actual geographical scale correlation;

[0315] encoding fusion, specifically adding the spatial dimension encoding and the geographical dimension encoding element by element to generate the final position encoding vector, and ensuring that the Token carries dual information of pixel position and geographical scale;

[0316] encoding injection, specifically adding the position encoding vector and the corresponding Token feature vector without changing the Token dimension, and only supplementing the spatial geographical correlation information.

[0317] In an embodiment of the present application, a possible implementation is provided, and the plurality of stages include a first stage, a second stage, and a third stage. The first stage focuses on local association of tokens, and preliminarily captures spatial dependence in a small range. The second stage extends to middle-layer association, and strengthens semantic combination of ground objects. The third stage realizes global association, and extracts global semantic information of large targets and scene category information.

[0318] In an embodiment of the present application, a possible implementation is provided, and the inference unit 1030 is further configured to implement merging of semantically similar tokens by the ToMe module through the following steps.

[0319] Token similarity calculation, specifically, cosine similarity between all tokens in the current token sequence is calculated.

[0320] Similarity threshold, specifically, a merging threshold is set.

[0321] Similar token clustering, specifically, a preset greedy clustering algorithm is used to divide tokens with a similarity greater than the merging threshold into a cluster, and the token with the highest average similarity in the cluster is selected as the center token.

[0322] Merging feature calculation, specifically, the feature of the new token is the weighted sum of the features of all tokens in the cluster, and the weight is the similarity between the token and the center token.

[0323] Residual reservation, specifically, the new token feature is added to the center token feature before merging through residual connection, to ensure that the core semantic information is not lost.

[0324] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, including a processor and a memory. The memory stores a computer program, and the processor is configured to run the computer program to perform the ticket quality inspection method based on the large language model of any one of the above embodiments.

[0325] In an exemplary embodiment, an electronic device is provided, as shown in Figure 11 , as shown in Figure 11 The electronic device 1100 shown in the figure includes a processor 1101 and a memory 1103. The processor 1101 and the memory 1103 are connected, such as through a bus 1102. Optionally, the electronic device 1100 can also include a transceiver 1104. It should be noted that in actual applications, the transceiver 1104 is not limited to one, and the structure of the electronic device 1100 does not constitute a limitation on the embodiments of the present application.

[0326] The processor 1101 can be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in connection with the disclosure. The processor 1101 can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0327] The bus 1102 can include a path for transmitting information between the above-mentioned components. The bus 1102 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 1102 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 11 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0328] The memory 1103 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but not limited to this.

[0329] The memory 1103 is configured to store computer program codes for implementing the scheme of the present application, and the processor 1101 is configured to control the execution. The processor 1101 is configured to execute the computer program codes stored in the memory 1103 to implement the content shown in the foregoing method embodiments.

[0330] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (for example, a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 11 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0331] Based on the same inventive concept, the embodiments of the present application also provide a storage medium having a computer program stored therein, wherein the computer program is configured to execute the remote sensing large model performance improvement method based on composite visual coding of any one of the foregoing embodiments when running.

[0332] Based on the same inventive concept, the embodiments of the present application also provide a computer program product comprising a computer program, the computer program being configured to execute the remote sensing large model performance improvement method based on composite visual coding of any one of the foregoing embodiments when running.

[0333] Those skilled in the art can clearly understand the specific working process of the system, device, and module described above, which can refer to the corresponding process in the foregoing method embodiments, and for the sake of brevity, no further description is given here.

[0334] Those skilled in the art can understand that the technical solutions of the present application can be embodied in the form of a software product in essence or in whole or part of the technical solutions, and the computer software product is stored in a storage medium, which includes a plurality of program instructions for causing an electronic device (such as a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application when running the program instructions. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0335] Alternatively, all or part of the steps of the foregoing method embodiments can be completed by program instruction related hardware (such as an electronic device of a personal computer, a server, or a network device, etc.), and the program instructions can be stored in a computer readable storage medium, and when the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the method described in the embodiments of the present application.

[0336] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; even though the present application has been described in detail with reference to the above embodiments, those ordinarily skilled in the art should understand that, within the spirit and principle of the present application, the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features thereof can be replaced by equivalent replacements; and these modifications or replacements do not cause the corresponding technical solutions to depart from the scope of the present application.

Claims

1. A remote sensing large model performance improvement method based on composite visual coding, characterized by, The method comprises: Based on the open source visual universal language model, the multi-modal remote sensing image data and the text data are adapted, and a preliminary remote sensing large model is built, wherein the architecture of the preliminary remote sensing large model comprises a text editor BERT layer, a query converter Q-Former layer, a cycle reasoning calculation layer and a dynamic threshold screening system; A composite visual coding ConvToMe layer for extracting local details and capturing global semantics is added to the preliminary remote sensing large model, the ConvToMe layer extracts local details through a convolutional neural network ConvNeXt layer, and combines a token merging visual encoder ToMe ViT layer to capture global semantics, thereby obtaining a performance-improved remote sensing large model; In response to a user query comprising multi-modal remote sensing image data and text data, the multi-modal remote sensing image data in the user query is preprocessed to obtain a multi-modal preprocessing matrix; The multi-modal preprocessing matrix is divided according to a preset proportion to obtain a first multi-modal preprocessing matrix and a second multi-modal preprocessing matrix; the first multi-modal preprocessing matrix is input into the ConvNeXt layer of the performance-improved remote sensing large model to output a local detail feature matrix; the second multi-modal preprocessing matrix is input into the ToMe ViT layer of the performance-improved remote sensing large model to output a global semantic feature matrix; and the local detail feature matrix and the global semantic feature matrix are fused to output a composite visual feature matrix; The text data in the user query is input into the BERT layer of the performance-improved remote sensing large model to output a text feature vector; The composite visual feature matrix and the text feature vector are input into the Q-Former layer of the performance-improved remote sensing large model to perform the steps of query vector interaction, multi-round cross-attention alignment and cross-modal feature fusion, and output a cross-modal feature matrix; The cross-modal feature matrix is input into the cycle reasoning calculation layer of the performance-improved remote sensing large model to perform the steps of low-rank matrix decomposition and cycle reasoning, and output a reasoning feature matrix, and then a preliminary reasoning result containing confidence and intersection over union (IOU) is obtained according to the reasoning feature matrix; The preliminary reasoning result is input into the dynamic threshold screening system of the performance-improved remote sensing large model to determine a target task corresponding to the preliminary reasoning result; According to the confidence and the IOU in the preliminary reasoning result, it is judged whether the preliminary reasoning result meets the threshold requirement of the target task; If the preliminary reasoning result meets the threshold requirement of the target task, a post-processing operation is performed to output a final reasoning result containing target coordinates, a target category and confidence; If the preliminary reasoning result does not meet the threshold requirement of the target task, the preliminary reasoning result is returned to the ConvToMe layer for re-reasoning, and the ConvToMe layer and the subsequent Q-Former layer and cycle reasoning calculation layer processes are re-executed.

2. The method of claim 1, wherein, The ConvNeXt layer adopts a four-level architecture of a Stem layer, a deep convolutional layer, an inverted bottleneck structure and a dimension mapping layer; The Stem layer is used for input adaptation and preliminary feature extraction, and outputs a preliminary feature matrix; The deep convolutional layer is used for local detail enhancement and receptive field expansion, and outputs an enhanced local feature matrix; The inverted bottleneck structure realizes local feature dimension expansion and compression through the steps of channel dimension expansion, feature extraction and channel dimension compression, and outputs a high-expression local feature matrix. The dimension mapping layer is used to realize feature dimension unification and output adaptation, and outputs a local detail feature matrix.

3. The method according to claim 1 or 2, characterized in that, The ToMe ViT layer adopts a four-level architecture of image block segmentation and projection, geographical enhanced position encoding, multi-layer Transformer encoder and global feature reconstruction.

4. The method of claim 3, wherein, The geographical enhanced position encoding is realized through the following steps: Geographical coordinate extraction and preprocessing, specifically reading the latitude and longitude information from the input remote sensing image metadata, obtaining the top-left corner coordinates of the image, and calculating the absolute geographical coordinates of the center of each image block; Coordinate conversion, specifically converting the absolute latitude and longitude into relative coordinates, mapping the relative coordinates to the [0, 1] interval, and eliminating the influence of different image geographical range differences; Spatial dimension encoding, specifically using sinusoidal position encoding to generate a position vector based on the pixel index and column index of the token Token in the image, preserving the pixel-level relative position relationship between Tokens; Geographical dimension encoding, specifically using linear encoding to map the relative latitude and longitude to a preset dimension vector through a linear layer, incorporating the actual geographical scale correlation; Encoding fusion, specifically adding the spatial dimension encoding and the geographical dimension encoding element by element to generate the final position encoding vector, ensuring that the Token carries both pixel position and geographical scale information; Encoding injection, specifically adding the position encoding vector to the corresponding Token feature vector without changing the Token dimension, only supplementing the spatial and geographical correlation information.

5. The method of claim 3, wherein, The multiple stages include a first stage, a second stage and a third stage; the first stage focuses on Token local correlation and preliminarily captures small-range spatial dependence; the second stage extends to middle-layer correlation and strengthens the semantic combination of geographical objects; and the third stage realizes global correlation and extracts large target global semantic and scene category information.

6. The method of claim 3, wherein, The ToMe module realizes semantic similar Token merging through the following steps: Token similarity calculation, specifically calculating the cosine similarity between all Tokens in the current Token sequence; Similarity threshold, specifically setting a merging threshold; Similar Token clustering, specifically using a preset greedy clustering algorithm to divide Tokens with a similarity greater than the merging threshold into a cluster, and selecting the Token with the highest average similarity in each cluster as the center Token; Merging feature calculation, specifically the new Token feature is the weighted sum of all Token features in the cluster, and the weight is the similarity between the Token and the center Token; Residual reservation, specifically adding the new Token feature to the center Token feature before merging through residual connection to ensure that the core semantics are not lost.

7. A remote sensing large model performance improvement device based on composite visual coding, characterized by, The device comprises: The building unit is used for adapting multi-modal remote sensing image data and text data based on an open-source visual general language model to build a preliminary remote sensing large model, wherein the architecture of the preliminary remote sensing large model comprises a text editor BERT layer, a query converter Q-Former layer, a recurrent reasoning calculation layer and a dynamic threshold screening system. The performance improvement unit is used for adding a composite visual coding ConvToMe layer for extracting local details and capturing global semantics to the preliminary remote sensing large model, the ConvToMe layer extracts local details through a convolutional neural network ConvNeXt layer, and combines a token merging visual encoder ToMe ViT layer to capture global semantics, so as to obtain a performance-improved remote sensing large model. The reasoning unit is used for responding to a user question comprising multi-modal remote sensing image data and text data, pre-processing the multi-modal remote sensing image data in the user question to obtain a multi-modal pre-processing matrix; and The multi-modal pre-processing matrix is divided according to a preset proportion to obtain a first multi-modal pre-processing matrix and a second multi-modal pre-processing matrix; the first multi-modal pre-processing matrix is input into the ConvNeXt layer in the performance-improved remote sensing large model to output a local detail feature matrix; the second multi-modal pre-processing matrix is input into the ToMe ViT layer in the performance-improved remote sensing large model to output a global semantic feature matrix; the local detail feature matrix and the global semantic feature matrix are fused to output a composite visual feature matrix; and The text data in the user question is input into the BERT layer in the performance-improved remote sensing large model to output a text feature vector; and The composite visual feature matrix and the text feature vector are input into the Q-Former layer in the performance-improved remote sensing large model to perform the steps of query vector interaction, multi-round cross-attention alignment and cross-modal feature fusion to output a cross-modal feature matrix; and The cross-modal feature matrix is input into the recurrent reasoning calculation layer in the performance-improved remote sensing large model to perform the steps of low-rank matrix decomposition and recurrent reasoning to output a reasoning feature matrix, and then a preliminary reasoning result comprising a confidence and an intersection over union (IOU) is obtained according to the reasoning feature matrix; and The preliminary reasoning result is input into the dynamic threshold screening system in the performance-improved remote sensing large model to determine a target task corresponding to the preliminary reasoning result; and Whether the preliminary reasoning result meets the threshold requirement of the target task is determined according to the confidence and the IOU in the preliminary reasoning result; and If the preliminary reasoning result meets the threshold requirement of the target task, a post-processing operation is performed to output a final reasoning result comprising a target coordinate, a target category and a confidence; If the preliminary reasoning result does not meet the threshold requirement of the target task, the preliminary reasoning result is returned to the ConvToMe layer for re-reasoning, and the ConvToMe layer and subsequent Q-Former layer and recurrent reasoning calculation layer processes are re-executed.

8. An electronic device, comprising: A device comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the composite visual coding-based remote sensing large model performance improvement method of any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program is configured to execute the remote sensing large model performance improvement method based on composite visual coding in any one of claims 1 to 6 when running.

10. A computer program product comprising a computer program, characterized in that, The computer program is configured to execute the remote sensing large model performance improvement method based on composite visual coding in any one of claims 1 to 6 when running.

Images