A multi-modal remote sensing pre-training collaborative acceleration method and system based on semantic perception

Abstract

The application discloses a kind of multi-modal remote sensing pre-training collaborative acceleration method and system based on semantic perception, belong to remote sensing image processing technical field, first acquisition geospatially aligned multi-source remote sensing image sample, according to unified geospatial grid division each modality image, obtain multi-modal image block;Based on physical pixel information is carried out semantic importance evaluation, generates single modality specific score and cross-modal comprehensive score;Image block is carried out scale self-adapting embedding and is input into intra-modal space encoder;According to cross-modal comprehensive score, space-modal asymmetric mask is executed, context area feature is retained, target area feature is separated and redundant area feature is discarded;After cross-modal fusion encoder obtains fusion context representation, with the target area hidden space representation extracted by teacher network to complete input as supervision, hidden space feature prediction training is executed.The application can reduce the multi-modal remote sensing pre-training calculation overhead, improve training efficiency and keep representation accuracy.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a multimodal remote sensing pre-training collaborative acceleration method and system based on semantic awareness. Background Technology

[0002] In recent years, with the rapid development of Earth observation technology, the ability to acquire multi-source remote sensing data, such as optical remote sensing imagery, synthetic aperture radar (SAR) imagery, and multispectral imagery, has been significantly enhanced. Different sensors, based on different physical imaging mechanisms, can observe the same surface area and provide complementary information such as texture, structure, backscattering, and spectral reflectance. Therefore, multimodal remote sensing data has been widely used in tasks such as natural resource monitoring, disaster emergency response, land cover classification, urban environmental analysis, and ecological environment assessment.

[0003] However, remote sensing imagery typically features wide coverage, high spatial resolution, large data volume, and high costs associated with manual annotation. A large amount of remote sensing data lacks accurate annotation, while traditional remote sensing interpretation models largely rely on manually labeled samples for supervised training, making it difficult for models to fully utilize the massive amounts of unlabeled remote sensing data. Furthermore, different sensor data exhibit significant differences in imaging mechanisms, spatial resolution, spectral channels, noise distribution, and data structure, making it difficult for models trained on a single modality or dataset to achieve stable generalization across sensors, regions, and mission scenarios.

[0004] To reduce reliance on manually labeled samples, existing technologies are increasingly incorporating self-supervised learning into remote sensing image representation learning. Self-supervised learning utilizes the data itself to construct supervisory signals, enabling the model to learn general feature representations from unlabeled remote sensing images. In multimodal remote sensing pre-training tasks, to mitigate the impact of differences in underlying imaging from different sensors on the pre-training objective, existing technologies further employ joint embedding prediction architectures. These architectures typically do not directly reconstruct the original pixels but instead predict the semantic representation of the target region in the feature space or latent space, thus providing a foundation for unified representation learning of multi-source heterogeneous remote sensing data.

[0005] Multimodal remote sensing pre-training methods based on joint embedding prediction architectures typically encode optical, SAR, and multispectral modal inputs into a shared feature space and predict the latent space features of the target region through contextual regions to achieve semantic alignment and complementary information fusion between different modalities. These methods can, to some extent, improve the model's transfer and generalization capabilities in downstream tasks such as classification, segmentation, and disaster identification.

[0006] However, extending multimodal remote sensing pre-training methods based on joint embedding prediction architectures to large-scale, high-resolution, multi-source heterogeneous remote sensing data still faces significant computational efficiency bottlenecks. First, high-resolution remote sensing images have wide coverage and high pixel density, generating a large number of tokens after image patching. When multiple modalities such as optical, SAR, and multispectral are input into the model, the number of input tokens further increases with spatial resolution, coverage, and the number of modalities. Especially when using Transformer-type encoders for spatial modeling and cross-modal fusion, the computational complexity of self-attention typically increases quadratically with the number of tokens, leading to high memory consumption during model pre-training, long training cycles, and overall low training efficiency.

[0007] Secondly, existing multimodal remote sensing pre-training methods based on joint embedding prediction architectures typically treat the input spatial location and different modalities as approximately indiscriminate, failing to fully utilize the spatial redundancy and cross-modal spatial consistency prevalent in remote sensing images. Specifically, remote sensing images often contain large areas of water, bare land, farmland, woodland, road backgrounds, or homogeneous building areas. These areas have high local spatial similarity and relatively low information density; while urban scene boundaries, disaster areas, areas with vegetation type changes, and complex terrain boundary areas often contain higher semantic information. If existing methods allocate the same computational resources to all areas, a large amount of computational overhead is easily consumed in low-information-content redundant areas.

[0008] Furthermore, although multimodal remote sensing data originate from different sensors, after geospatial alignment, different modalities within the same geographical location often exhibit spatial correspondences. This cross-modal spatial consistency could be used to guide models in recognizing the information importance of different regions and selectively compressing input data or model-internal features. However, existing multimodal remote sensing pre-training methods based on joint embedding prediction architectures typically focus on modality fusion or latent space prediction, lacking spatial semantic importance assessment mechanisms geared towards computational acceleration, and making it difficult to dynamically adjust computational resource allocation based on differences in information density across different geographical regions and modalities.

[0009] Furthermore, while existing general vision domains offer model acceleration methods such as token pruning, token merging, random masking, or fixed-ratio sampling, these methods largely rely on static rules, random strategies, or single-modal visual features, lacking a targeted understanding of the spatial semantic distribution differences in remote sensing images and the characteristics of multimodal sensors. Directly applying these methods to multimodal remote sensing pre-training processes can easily lead to the following problems: First, regions with important semantic features may be misclassified as redundant and excessively discarded; second, the spatial correspondence between different modalities at the same geographical location may be disrupted; third, it is difficult to dynamically determine the compression intensity based on the differences in the physical imaging characteristics of optical, SAR, and multispectral modalities; and fourth, it may weaken the model's ability to represent key ground feature structures, modal complementarity relationships, and spatial contextual information.

[0010] Meanwhile, existing pre-training acceleration solutions mostly optimize from a single level. For example, performing random masking or token dropping only at the input data level can reduce the length of the input sequence, but may result in the loss of key spatial details and modal complementarity information; performing token merging or feature compression only within the model can reduce some encoding computation, but has limited improvement on the overall training throughput. For high-resolution, multimodal remote sensing pre-training tasks, single-level acceleration methods cannot simultaneously achieve computational efficiency, representation accuracy, and cross-domain generalization ability.

[0011] Furthermore, multimodal remote sensing foundational models not only need to maintain high accuracy in downstream tasks with data from the same source as the pre-training data, but also need to adapt to external geographical regions, novel sensors, or different data distributions not seen during the pre-training phase. If acceleration strategies lack semantic constraints, cross-modal consistency constraints, and latent space prediction constraints, they may impair the model's multimodal representation capabilities while increasing training speed, leading to a decline in the model's generalization performance in cross-regional, cross-sensor, or small-sample downstream tasks.

[0012] Therefore, how to fully utilize the cross-modal spatial consistency, local redundancy, and spatial semantic importance differences of remote sensing data in a multimodal remote sensing pre-training framework based on joint embedding prediction architecture, and perform semantically aware collaborative compression of input data and model internal features, while reducing the computational cost of pre-training high-resolution multi-source remote sensing data and maintaining the model's multimodal representation capability, downstream task accuracy, and cross-domain generalization capability, has become an urgent technical problem to be solved in this field. Summary of the Invention

[0013] The technical problem this invention aims to solve is to address the shortcomings of the prior art by providing a semantically aware multimodal remote sensing pre-training collaborative acceleration method and system. This method addresses the shortcomings of existing multimodal remote sensing pre-training methods based on joint embedding prediction architectures. When dealing with high-resolution, multi-source heterogeneous remote sensing data, the computational cost increases quadratically with the number of input tokens, resulting in long training cycles and low overall training efficiency. Furthermore, existing methods lack awareness of differences in spatial semantic distribution and cross-modal spatial consistency in remote sensing images, making it difficult to differentiate between low-information-content redundant regions and key semantic regions. Using general token pruning, random masking, or token merging strategies can easily disrupt the spatial correspondence between different modalities, thus affecting the model's multimodal representation accuracy and cross-domain generalization ability.

[0014] The present invention adopts the following technical solution: A semantically aware multimodal remote sensing pre-training collaborative acceleration method includes the following steps: S1. Obtain geospatial aligned multi-source remote sensing image samples, divide each modal image according to a unified geospatial grid, and obtain multimodal image blocks corresponding to the same geographic region under the same grid index; S2. Based on the physical pixel information of multimodal image blocks, perform semantic importance evaluation to generate single-modal specific scores and cross-modal comprehensive scores; S3. Perform scale-adaptive embedding on multimodal image blocks and map them to a shared latent space to obtain the feature sequences of sub-blocks for each modality; S4. Input the sub-block feature sequence into the intramodal spatial encoder and perform adaptive token merging based on the single-modal specific score to obtain the image block-level features of each modality; S5. Construct positive sample pairs based on image block-level features of different modalities under the same grid index, and construct negative sample pairs based on image block-level features of different grid indices, and calculate cross-modal contrast loss; S6. Based on the cross-modal integrated score, perform spatial-modal asymmetric masking on the image block-level features, retain context region features, separate target region features and discard redundant region features; S7. Input the context region features into the cross-modal fusion encoder to obtain the fused context representation; S8. Based on the fusion context representation, predict the latent space features of the target region. The teacher network generates the real latent space target features of the target region based on the complete unmasked image block-level features of all modalities. Based on the cross-modal contrast loss and the latent space prediction loss between the predicted latent space features of the target region and the latent space target features of the target region, train the multimodal remote sensing pre-trained model.

[0015] Furthermore, the multi-source remote sensing image samples include at least one of optical modal images, synthetic aperture radar modal images, and multispectral modal images; When dividing the image into blocks, the same geospatial grid unit is mapped to the pixel region in the corresponding modal image according to the spatial resolution of each modal image, so that different modal image blocks under the same grid index represent the same geographic region.

[0016] Further, step S2 includes: Pixel-level statistics are extracted based on the physical imaging characteristics of different modal images, and the pixel-level statistics are aggregated into single-modality specific scores at the image block level. After standardizing the single-modal specific score and mapping it to a preset numerical range, a cross-modal comprehensive score is generated through a cross-modal fusion operator.

[0017] Furthermore, when generating single-mode specific scores, for optical modal images, pixel-level statistics are extracted based on texture complexity; for synthetic aperture radar modal images, pixel-level statistics are extracted based on backscattering intensity; and for multispectral modal images, pixel-level statistics are extracted based on spectral reflectance changes.

[0018] Further, step S3 includes: The multimodal image block is divided into a sequence of sub-blocks of fixed pixel size. The sub-block sequence is mapped to a shared latent space through a linear projection layer that matches the corresponding modality. A positional code is added to the feature sequence of the mapped sub-block. The positional code contains the spatial resolution information and geographical location information of the corresponding modality.

[0019] Further, step S4 includes: The token merging rate of an image block is determined based on the specific score of the single modality corresponding to that image block. When an image patch belongs to a high-resolution region, a lower token merging rate is used to preserve the feature details of the sub-blocks; When an image patch belongs to a low-resolution region, a higher token merging rate is used to compress redundant sub-block features.

[0020] Furthermore, when performing adaptive token merging, the sub-block feature sequences corresponding to image blocks with the same token merging rate are divided into the same computation group, and the sub-block feature sequences within the same computation group are recombined into standard tensors for parallel computation. Different computational groups sequentially pass through a shared intramodal spatial encoder, and the gradients of each computational group are accumulated during the backpropagation phase.

[0021] Further, step S6 includes: The context region and target region are determined based on the cross-modal synthesis score, wherein the context region is preferentially assigned to the high-scoring grid index, and the target region is assigned to the medium-scoring grid index. Perform intramodal masking within the context region, replacing the masked modal features with learnable mask tokens, and ensuring that at least one modality's valid input is preserved at each spatial location.

[0022] Further, step S7 includes: attaching block-level absolute location encoding based on a unified geospatial grid index to the context region features, and performing global context modeling and heterogeneous feature fusion on the context region features of different modalities through the self-attention mechanism and cross-attention mechanism in the cross-modal fusion encoder.

[0023] Furthermore, the multimodal remote sensing pre-trained model includes a student network and a teacher network; The student network includes an intramodal spatial encoder, a crossmodal fusion encoder, and a predictor; the teacher network includes an intramodal spatial encoder and a crossmodal fusion encoder. The teacher network receives complete unmasked image block-level features from all modalities and generates true latent space features for the target region. The student network generates predicted latent space features for the target region based on contextual region features. The parameters of the teacher network are updated using an exponential moving average based on the parameters of the student network.

[0024] Secondly, embodiments of the present invention provide a semantically aware multimodal remote sensing pre-training collaborative acceleration system, comprising: The image patch segmentation module is used to acquire geospatial aligned multi-source remote sensing image samples and divide each modal image according to a unified geospatial grid to obtain multimodal image patches corresponding to the same geographic region under the same grid index. The semantic importance assessment module is used to generate single-modality specific scores and cross-modality comprehensive scores based on the physical pixel information of multimodal image patches; The scale-adaptive embedding module is used to perform scale-adaptive embedding on multimodal image patches and map them to a shared latent space to obtain the feature sequences of sub-patterns for each modality. The intramodal spatial coding module is used to perform adaptive token merging on the sub-block feature sequence based on a single modality specific score to obtain image block-level features for each modality. The asymmetric masking module is used to perform spatial-modal asymmetric masking on image block-level features based on cross-modal synthesis scores to obtain context region features, target region features, and redundant region features. The cross-modal fusion module is used to perform cross-modal fusion of context region features to obtain a fused context representation; The latent space prediction training module is used to predict the latent space features of the target region based on the fused context representation, and to train a multimodal remote sensing pre-trained model based on the latent space prediction loss between the predicted latent space features and the real latent space features.

[0025] Furthermore, the semantic importance assessment module includes a single-modal scoring unit and a cross-modal comprehensive scoring unit; The single-modal scoring unit is used to extract pixel-level statistics for the physical imaging characteristics of different modal images and generate single-modal specific scores. The cross-modal comprehensive scoring unit is used to standardize and interval map specific scores of a single modality, and generate a cross-modal comprehensive score.

[0026] Furthermore, the intramodal spatial coding module includes a token merging unit and a grouped parallel computing unit; The token merging unit is used to determine the token merging rate of the sub-block feature sequence based on a specific score of a single modality; The grouped parallel computing unit is used to divide sub-block feature sequences with the same token merging rate into the same computing group, and to perform parallel encoding on the sub-block feature sequences within the same computing group.

[0027] Furthermore, the multimodal remote sensing pre-trained model includes a student network and a teacher network; The student network includes an intramodal spatial encoder, a crossmodal fusion encoder, and a predictor; the teacher network includes an intramodal spatial encoder and a crossmodal fusion encoder. The teacher network is used to generate true latent space features of the target region based on the complete unmasked image block-level features of all modalities, and the student network is used to generate predicted latent space features of the target region based on the context region features.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects: A semantically aware multimodal remote sensing pre-training collaborative acceleration method is proposed, forming a complete multimodal remote sensing pre-training collaborative acceleration process. Model-level token merging is driven by single-modal specific scores, and data-level asymmetric masking is driven by cross-modal integrated scores. This prioritizes the allocation of computational resources to high-information regions, reducing the participation of low-information redundant tokens in computation, thereby reducing pre-training computational overhead while maintaining the quality of latent space representation.

[0029] Furthermore, it adapts to remote sensing data with different spatial resolutions and from different sensor sources, ensuring that multimodal image patches under the same grid index correspond to the same geographic region. This setting maintains cross-modal spatial correspondence and avoids spatial mismatches in subsequent masking, fusion, and prediction processes.

[0030] Furthermore, image patch information density is evaluated directly before model encoding, avoiding reliance on deep features to determine importance. This approach enables earlier identification of high-information-content and low-redundancy regions, providing a unified scoring basis for subsequent token merging and asymmetric masking.

[0031] Furthermore, semantic importance assessment is made consistent with the physical imaging mechanisms of different modalities. Optical images emphasize texture, SAR images emphasize scattering structure, and multispectral images emphasize spectral variations, thereby avoiding the misclassification of important information from certain modalities as redundant due to the use of a single visual indicator.

[0032] Furthermore, the token merging rate is determined based on the specific score of a single modality, with a low merging rate for high-scoring regions and a high merging rate for low-scoring regions, achieving model-level adaptive compression. High-scoring key regions retain more fine-grained sub-block features, while low-scoring homogeneous regions compress more redundant sub-block features, thereby reducing the computational load within the encoder while minimizing the loss of key semantic details.

[0033] Furthermore, image blocks with the same token merging rate are grouped into the same computation group, with parallel processing within the group and serial processing between groups, and gradient accumulation is performed to address the issue of inconsistent sequence lengths when different image blocks are processed with different merging rates. By performing parallel computation in groups, sequences within the same group can be recombined into standard tensors, improving hardware parallel efficiency, while maintaining consistent model parameter updates through a shared encoder and gradient accumulation.

[0034] Furthermore, the context region and target region are determined based on the cross-modal synthesis score, and intra-modal masking is performed within the context region. At least one modal input is retained at each spatial location, achieving data-level spatial-modal asymmetric compression. High-scoring regions are prioritized as context input, moderately important regions are used as prediction targets, and low-information regions are discarded. Retaining at least one modal input avoids completely destroying multimodal semantic cues at the same spatial location. The technical disclosure document details the target region separation, redundant feature discarding, and mask token replacement mechanisms.

[0035] Furthermore, block-level absolute position encoding is added to the contextual region features, and global context modeling and heterogeneous feature fusion are performed through self-attention and cross-attention mechanisms. This restores the ability to model cross-regional and cross-modal associations on the compressed contextual features. Block-level absolute position encoding provides a unified spatial coordinate reference, self-attention models spatial context, and cross-attention enhances the fusion of complementary modal information.

[0036] Furthermore, an asymmetric teacher-student pre-training architecture is established. The student network performs accelerated context prediction, while the teacher network generates a stable target representation based on the complete input. The two networks work together to constrain the student network to learn a high-quality latent space representation through latent space prediction loss. The teacher network parameters are updated through exponential moving average, which helps improve the stability of the training target.

[0037] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0038] In summary, this invention achieves multimodal spatial alignment through a unified geographic grid, utilizes semantic importance scores to drive model-level token merging and data-level asymmetric masking respectively, reducing low-information-content redundant token computation; and combines cross-modal fusion and teacher-student latent space prediction training to improve pre-training efficiency while maintaining multimodal representation accuracy and cross-domain generalization ability.

[0039] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall process and architecture of the method of the present invention. Detailed Implementation

[0041] This invention provides a semantically aware collaborative acceleration method for multimodal remote sensing pre-training. First, an asymmetric teacher-student network paradigm is constructed based on a joint embedding prediction architecture. Second, a multi-scale semantic importance assessment mechanism is used to quantify the physical pixel-level information density, generating single-modality specific scores and cross-modality comprehensive scores. Finally, guided by these scores, a two-layer collaborative acceleration mechanism between data and model is executed. Specifically, at the data level, asymmetric masking operations are performed on the spatial and modal aspects based on the cross-modal comprehensive scores to simplify the input sequence; and at the model level, adaptive token merging is performed in the intra-modal spatial encoder based on the single-modality specific scores to compress redundant features. This semantically aware collaborative acceleration method for multimodal remote sensing pre-training effectively overcomes the computational bottleneck caused by the high redundancy of high-resolution remote sensing images by dynamically mining the inherent information distribution of multi-source data, achieving approximately 1.9 times speedup in large-scale multimodal pre-training while effectively balancing high-precision model representation and cross-domain generalization capabilities in downstream tasks.

[0042] The method of this invention is based on a joint embedding prediction architecture, which constructs an asymmetric teacher-student network paradigm. The overall architecture and modules are as follows: In the asymmetric network paradigm, the student network consists of a two-stage decoupled encoder and a predictor; the teacher network contains only a two-stage decoupled encoder and no predictor. The two-stage decoupled encoder is the core feature extraction module, and it is divided into two levels: The first stage is a weight-shared intramodal spatial encoder, used for local feature aggregation of single-modal sub-block sequences; The second level is a cross-modal fusion encoder, used for global context interaction of multimodal features.

[0043] The teacher network and the student network have the same structure in the feature extraction module, but they differ in the data flow and acceleration strategy in the forward propagation.

[0044] The teacher network receives a complete sequence of image patches as input and does not perform feature merging or masking operations during the feature extraction process. Its weights are updated by the student network's weights using an exponential moving average. After encoding the complete input, the teacher network outputs the features of the corresponding target region, which serve as the true latent space representation for subsequent predictions. Student Network: It proactively introduces a collaborative acceleration mechanism during forward propagation. Specifically, for the input image patch sequence, it performs data-level spatial-modal asymmetric masking and model-level adaptive merging. Finally, it utilizes its unique predictor to predict the features of the target region based on the extracted global context features.

[0045] Please see Figure 1 This invention discloses a multimodal remote sensing pre-training collaborative acceleration method based on semantic awareness, with forward propagation and feature prediction of the student network as the main thread. The specific steps are as follows: S1. Sampling and image patching of multi-source remote sensing datasets A training corpus containing multiple heterogeneous remote sensing datasets is acquired. In each training iteration, a target dataset is randomly selected from this corpus, and remote sensing image samples aligned geographically according to their corresponding batch size are obtained. To adapt to single-temporal model settings, static images at a single time point are extracted from the acquired samples as model input. Simultaneously, multi-source data from multiple sensors are categorized into three data types: optical modality, synthetic aperture radar modality, and multispectral modality. This classification is based on the physical imaging characteristics of the sensors, rather than referring to a specific single sensor. This invention is compatible with single-modal input or multi-modal input with missing modalities; image samples only need to contain at least one of the above data categories.

[0046] For multimodal image samples with different spatial resolutions, a unified geospatial grid is constructed in the geospatial domain to divide the data according to the physical block size corresponding to the target dataset.

[0047] Specifically, instead of physically cropping the original pixel image, the same geographic grid cell is mapped to the corresponding pixel region of each modality based on its own spatial resolution, thereby extracting the image patch corresponding to that grid cell. This partitioning method ensures that for each sample in the current batch, the image patches corresponding to different modalities under the same grid index can represent the same geographic region, thus achieving cross-modal grid-level spatial alignment at the model input.

[0048] In one specific embodiment, the pre-training corpus is built based on the GeoPlex multi-source remote sensing dataset. To accommodate heterogeneous multi-source inputs, the data in the corpus is divided into three basic modalities according to physical characteristics: (1) Visible light mode (opt), covering high-resolution aerial imagery, NAIP and SPOT6 / 7 data; (2) Synthetic Aperture Radar Modes (SAR), covering Sentinel-1 and ALOS-2 data; (3) Multispectral modes (spec), covering Sentinel-2 and Landsat 7 / 8 / 9 data.

[0049] In each training iteration, a single dataset is randomly selected from the corpus, and the corresponding single temporal phase is extracted. Static image samples. In actual training, the input samples do not need to necessarily include all three modalities; the model can be jointly pre-trained on multiple datasets that are compatible with missing modalities.

[0050] For those with different ground resolutions With the number of channels The multimodal samples are segmented according to the physical block size originally set in the target dataset. The image is divided into non-overlapping image patches to construct a unified geospatial grid. This partitioning method ensures that any grid index... Heterogeneous image blocks All correspond strictly to the same surface region, achieving cross-modal grid-level spatial alignment at the model input end.

[0051] S2, Multi-scale semantic importance assessment Before feature extraction in the network, the information density of each image patch is directly quantified at the physical pixel level, which serves as an evaluation criterion for measuring the semantic importance of different geographical regions.

[0052] First, pixel-level statistics are extracted based on the physical imaging characteristics of different modalities of the sample. Then, these pixel-level statistics are aggregated into modality-specific scores at the image block level. The modality-specific scores are standardized and mapped to the (0,1) interval. Finally, a cross-modal fusion operator is used to generate a cross-modal comprehensive score.

[0053] Based on the cross-modal integrated score, a spatial sampling distribution is constructed, and the retention and discard of image blocks are determined in advance at the grid index level: the top-ranked high-scoring grid indices are selected first, and their corresponding image blocks are used as the context regions provided to the student network; the medium-scoring grid indices are selected, and their corresponding image blocks are used as the target regions to be predicted; the image blocks corresponding to the remaining grid indices are redundant regions to be discarded.

[0054] The specific calculation process is as follows: S201, Modal-Specific Feature Extraction and Block-Level Score Aggregation Visible light mode (opt): Quantizes texture complexity using the gradient magnitude of the directional gradient histogram (HOG).

[0055] Its block-level fraction is:

[0056] in, This represents a truncated mean operation, used to filter local noise and anomalous interference. In a specific embodiment, the truncation ratio constant is set to... .

[0057] Synthetic Aperture Radar Mode (SAR): Converts backscattering intensity to the decibel domain To alleviate speckle noise.

[0058] truncated mean within weighted fusion blocks With truncation standard deviation The formula is:

[0059] In one specific embodiment, the weight parameter is set to , .

[0060] Multispectral mode (spec): This refers to the pixel reflectance along the spectral band. Smooth interpolation is a pseudo-continuous spectral curve Calculate the maximum absolute value of its discrete second-order difference. Approximate spectral curvature. Block-level fraction. From within the block The truncated mean of the response is obtained.

[0061] S202, Normalization and Cross-Modal Synthesis Using the statistical median of modality scores in a single sample Interquartile range Perform robust standardization

[0062] The unimodal normalized score is then obtained by truncating the Sigmoid function and mapping it to the (0,1) interval.

[0063] in, A cutoff constant to avoid the influence of extreme values.

[0064] Finally, based on the principle that "a region that is significant in any modality is an important region", a smooth LogSumExp approximate maximum function is used for cross-modal synthesis to calculate the cross-modal synthesis score. :

[0065] In one specific embodiment, a smaller temperature coefficient is introduced. This is to ensure that salient features of a single modality dominate the overall score.

[0066] At this point, the two types of scores output in this step will drive the subsequent two-layer collaborative acceleration mechanism: among them, the single-modal normalized score Adaptive sub-block compression for controlling the intra-modal feature extraction stage (i.e., model-level acceleration in step S4); cross-modal synthesis score Used to determine the asymmetric space and modal mask distribution of image patches at the grid index level (i.e., data-level acceleration of step S6).

[0067] S3, Scale-Adaptive Embedding To address the differences in spatial resolution and channel count among different modalities, the image patches extracted in step S1 are segmented into a sequence of sub-patches of fixed pixel size. Then, a modality-specific linear projection layer is used to uniformly map the sub-patch sequences of different dimensions to a shared latent feature space. After mapping, an adaptive positional encoding is applied to each sub-patch. This positional encoding incorporates the spatial resolution of the corresponding modality during generation, thereby endowing the sub-patch features with realistic physical scale and absolute geographical location information.

[0068] In one specific embodiment, the grid index extracted in step S1 Heterogeneous image patches with different resolutions and number of channels The scale adaptation module first divides it into a sequence of sub-patch blocks of fixed pixel size; Subsequently, the flattened sub-block sequence is input into the corresponding mode. Specific linear projection layer In this context, a unified mapping is used to define a shared dimension. The latent space vector; After mapping, adaptive location codes are added to each sub-block to give it true physical scale and absolute geographical location information. Following the above embedding process, the grid index... A single modal image patch at a given location is transformed into a dimension-aligned feature sequence. This serves as a standardized input for subsequent feature extraction.

[0069] S4, Intramodal Spatial Encoding and Model-Level Acceleration Based on Adaptive Merging The sub-block sequences of each modality are sequentially input into the weight-shared intra-modal spatial encoder for feature aggregation. Inside the encoder, for each image block in the current modality, dynamic token merging is performed on its internal sub-block sequences based on the modality-specific score corresponding to the image block calculated in step S2.

[0070] Specifically, if an image patch belongs to a high-resolution critical region, a low merging rate is used to preserve fine-grained semantic details; if an image patch belongs to a low-resolution homogeneous region, a high merging rate is used for local depth compression. To address the issue of irregular tensor lengths caused by differences in merging rates among different image patches, which affects hardware parallelism, this step employs a grouped parallel computation strategy. Sub-block sequences corresponding to image patches with the same merging rate are recombined into a standard tensor, and intra-group parallel and inter-group serial computation and gradient accumulation are performed. Finally, the aggregation of sub-block sequences is completed through this intra-modal encoding process, outputting the image patch-level features of each modality.

[0071] In one specific embodiment, the above feature sequence The inputs are sequentially fed into an intramodal spatial encoder composed of Transformer blocks with shared weights. In order to improve feature extraction efficiency, in Internally, adaptive token merging (AdaptiveToMe) is strictly limited to a single image patch within a single modality. Specifically, a semantic importance threshold is introduced. The single-mode normalized score output in step 2 is used directly. Dynamically modulate the current image patch in the first... The merging behavior of a layer assigns two levels of discrete merging rates, high and low, to its internal layers. :

[0072] in, and Corresponding to the first Both the conservative and aggressive merging rates of the layers follow a decay strategy of "more merging in shallow layers and fewer merging in deep layers".

[0073] In one specific embodiment, the model-level importance grouping threshold is set to... The initial merger rates for conservative and aggressive approaches were set as follows: and The merging scale of each network layer strictly follows the merging quantity starting from the first layer. The number of merges decreases linearly until the number of merges in the last layer is zero.

[0074] During network forward propagation, a grouping parallel calculation mechanism based on the aforementioned merging rate is executed. This is done according to the scores corresponding to the image patches. , will be greater than or equal to the threshold The sub-block sequence corresponding to the image block is classified into Group (executing low merge rate); will be less than the threshold The sub-block sequence corresponding to the image block is classified into Groups (perform high merging rates). Since sub-block sequences within the same group have the same number of tokens merged at any level, and their sequence lengths are always aligned, the model reassembles them into standard tensors to achieve fully parallel computation within the group. For sequences with varying lengths... Groups and The model is passed through a shared Transformer layer sequentially between groups; during the backpropagation phase, the gradients generated by each group are automatically accumulated and updated uniformly. The model parameters are then determined. Finally, the highly compressed and aggregated sub-block sequence is output as a single block-level feature vector. .

[0075] S5. Preliminary cross-modal feature alignment based on contrastive learning Before performing cross-modal feature fusion, cross-modal contrast loss is calculated. Image patch features from different modalities at the same geographical location are used as positive sample pairs, and image patch features from different geographical locations are used as negative sample pairs, thereby driving the initial semantic alignment between different modal features.

[0076] In one specific embodiment, before performing cross-modal feature fusion, the cross-modal consistency loss is calculated. This drives the initial representation alignment of multi-source heterogeneous data. The loss function directly applies to the single-modal block-level features output by the intra-modal spatial encoder in step 4. Spatial locations within the same geographic grid Different modal features are used as positive sample pairs, and the calculation formula is as follows:

[0077] in, Indicates a temperature coefficient The exponential cosine similarity.

[0078] In one specific embodiment, the temperature coefficient of contrast loss is set to... .

[0079] S6. Data-level acceleration based on spatial-modal asymmetric masks Block-level absolute position encoding is applied to the features of each modal image patch after initial alignment. This encoding relies only on the absolute index of the image patch in the shared geographic grid, thereby establishing a unified spatial coordinate reference for subsequent cross-modal interactions. Subsequently, the features of each modality are concatenated along the sequence dimension, and spatial masking and intra-modal masking are sequentially performed on the concatenated features according to the spatial sampling strategy generated in advance based on the preset sampling ratio in step S2.

[0080] Specifically, firstly, image patch features corresponding to the grid index of the context region are extracted. The features corresponding to the grid index of the target region are then separated to serve as the prediction benchmark for the subsequent predictor and are not input into the cross-modal encoder. At the same time, redundant features corresponding to unselected indices are discarded. Subsequently, random intramodal masking is performed on the previously extracted context region features. Learnable mask tokens are used to replace the masked modal features, ensuring that at least one modality's valid input is retained at each spatial location.

[0081] In one specific embodiment, block-level location codes dependent on the shared geographic grid absolute index are added to the initially aligned modal block-level features to establish a unified spatial coordinate reference. This is based on the cross-modal synthesis score output in step 2. A non-uniform spatial sampling strategy is constructed to simplify the input sequence. The specific process includes spatial masking and intramodal masking: Spatial masking: Two probability distributions are constructed based on the comprehensive score set, sampling the context region and the prediction target region respectively. For the context region, which serves as the visible input to the student network, the following probability distribution is constructed:

[0082] By prioritizing the sampling of high-resolution seed blocks using this distribution, and generating a continuous rectangular mask centered on these seed blocks, a context space index set is obtained. .

[0083] In one specific embodiment, the corresponding spatial sampling temperature coefficient is set to The retention ratio of the context region is set to 25% of the total number of grids.

[0084] For the prediction target region that serves as the prediction target for the teacher network, guide it to cover the moderately important region to force the model to perform deep semantic reasoning.

[0085] First, calculate the median of the scores at each location and the overall score. Distance offset Transform it into a distribution By sampling non-overlapping rectangular blocks using this distribution, a set of target spatial indexes is obtained. .

[0086] In one specific embodiment, the corresponding spatial sampling temperature coefficient is set to The retention rate of the target area is set to 15% of the total number of grids, and redundant features corresponding to unselected background blocks are completely discarded.

[0087] Modal intrinsic mask: Determines the context space Then, the corresponding image patch features are extracted, and a random modal masking operator is performed in this region. .

[0088] For each position This operator randomly discards tokens of a specific modality at a certain ratio. The student network ultimately receives the asymmetric context feature set. Represented as:

[0089] In one specific embodiment, the cross-modal masking probability is set to 40%. This mechanism requires that at least one modality's valid input be retained at each spatial location, using learnable mask tokens to replace the masked modal features.

[0090] S7, cross-modal fusion coding The set of asymmetric context features of the student network retained after the above asymmetric masking process. Input to cross-modal fusion encoder middle. By combining absolute position encoding, and through self-attention and cross-attention mechanisms, global context modeling and deep fusion of heterogeneous features are performed, ultimately outputting a grid index. Corresponding fusion context representation ,in .

[0091] S8. Target Feature Prediction and Joint Optimization Using the student network's unique predictor based on the multimodal block-level features, latent space prediction is performed on the target region features separated in step S6. Subsequently, the joint embedding prediction latent space prediction loss is calculated between the predicted features output by the predictor and the true latent space features of the target region extracted by the teacher network after cross-modal fusion encoding. The overall joint optimization loss of the network model is composed of the cross-modal contrastive loss from step S5 and the latent space prediction loss from this step. Finally, end-to-end training of the entire architecture is completed through backpropagation.

[0092] After initial alignment and cross-modal global context fusion, the model performs cross-modal feature prediction following a joint embedding prediction architecture. Specifically, the predictor of the student network... Characterized solely by the fusion context output of the cross-modal fusion encoder For visible information, combined with the target space index set The spatial location information is used to output the predicted features of the region in the latent space. Simultaneously, the teacher network receives a complete set of multimodal image patches. Without activating token merging, these patches are sequentially processed through a front-end embedding module and a two-level feature extraction module to generate a high-quality, realistic latent space representation of the corresponding target region. Calculate predictive features Representation of the real latent space The mean squared error over the target region yields the cross-modal feature prediction loss. :

[0093] After obtaining the latent space prediction loss, the overall pre-training joint optimization objective of the network model is... It is composed of the weighted sum of the cross-modal consistency loss from step 5 and the prediction loss from this step:

[0094] The model ultimately achieves the overall optimization objective through backpropagation. It calculates gradients and completes end-to-end pre-training updates for the entire asymmetric architecture.

[0095] In another embodiment of the present invention, a semantically aware multimodal remote sensing pre-training collaborative acceleration system is provided. This system can be used to implement the above-mentioned semantically aware multimodal remote sensing pre-training collaborative acceleration method. Specifically, the semantically aware multimodal remote sensing pre-training collaborative acceleration system includes an image patch segmentation module, a semantic importance evaluation module, a scale-adaptive embedding module, an intramodal spatial coding module, an asymmetric masking module, a cross-modal fusion module, and a latent space prediction training module.

[0096] The image patch division module is used to acquire geospatial aligned multi-source remote sensing image samples and divide each modal image according to a unified geospatial grid to obtain multimodal image patches corresponding to the same geographic region under the same grid index. The semantic importance assessment module is used to generate single-modality specific scores and cross-modality comprehensive scores based on the physical pixel information of multimodal image patches; The scale-adaptive embedding module is used to perform scale-adaptive embedding on multimodal image patches and map them to a shared latent space to obtain the feature sequences of sub-patterns for each modality. The intramodal spatial coding module is used to perform adaptive token merging on the sub-block feature sequence based on a single modality specific score to obtain image block-level features for each modality. The asymmetric masking module is used to perform spatial-modal asymmetric masking on image block-level features based on cross-modal synthesis scores to obtain context region features, target region features, and redundant region features. The cross-modal fusion module is used to perform cross-modal fusion of context region features to obtain a fused context representation; The latent space prediction training module is used to predict the latent space features of the target region based on the fused context representation, and to train a multimodal remote sensing pre-trained model based on the latent space prediction loss between the predicted latent space features and the real latent space features.

[0097] The semantic importance assessment module includes a single-modal scoring unit and a cross-modal comprehensive scoring unit; The single-modal scoring unit is used to extract pixel-level statistics for the physical imaging characteristics of different modal images and generate single-modal specific scores. The cross-modal comprehensive scoring unit is used to standardize and interval map specific scores of a single modality, and generate a cross-modal comprehensive score.

[0098] The intramodal spatial coding module includes a token merging unit and a grouped parallel computing unit; The token merging unit is used to determine the token merging rate of the sub-block feature sequence based on a specific score of a single modality; The grouped parallel computing unit is used to divide sub-block feature sequences with the same token merging rate into the same computing group, and to perform parallel encoding on the sub-block feature sequences within the same computing group.

[0099] The multimodal remote sensing pre-trained model includes a student network and a teacher network; The student network includes an intramodal spatial encoder, a crossmodal fusion encoder, and a predictor; the teacher network includes an intramodal spatial encoder and a crossmodal fusion encoder. The teacher network is used to generate true latent space features of the target region based on the complete unmasked image block-level features of all modalities, and the student network is used to generate predicted latent space features of the target region based on the context region features.

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0101] In a specific model training and validation embodiment, both the model pre-training and downstream fine-tuning are performed in a graphics processing unit (GPU) environment, specifically using an NVIDIA H100 GPU. In the network optimization step, the AdamW optimizer is used for parameter iteration, with a base learning rate set to [value missing]. The system is configured with a cosine annealing strategy featuring linear preheating. The parameters of the teacher network are updated using an exponential moving average mechanism with an exponential moving average decay rate set to 0.996. The overall loss weights are set to 0.996 for the overall optimization objective of the pre-training phase. Set to 1.0.

[0102] The pre-trained model can be adapted to various multimodal remote sensing downstream evaluation tasks. These tasks cover in-distribution tasks with data from the same source as the pre-training corpus, as well as out-of-distribution tasks with external data not seen during pre-training. In-distribution tasks include TreeSatAI-TS multi-label tree species classification and FLAIR high-resolution land cover semantic segmentation. Out-of-distribution tasks include Sen1Floods11 flood segmentation, So2Sat local climate zone classification, and HLS BurnScar fire scar segmentation. Depending on the task characteristics, two standard evaluation settings are adopted during the evaluation phase: end-to-end fine-tuning, which updates all parameters of the network; and linear probing, which completely freezes the pre-trained backbone network and trains only a lightweight evaluation head specific to the task. The specific implementation methods for feature extraction and mapping for different task types are detailed below: 1) Specific implementation for graph-level classification tasks: For TreeSatAI-TS and So2Sat classification tasks, a learnable global token is added inside the cross-modal fusion encoder. After the pre-trained model receives multi-source image patch input and performs forward propagation, it extracts the multimodal representation corresponding to the global token, inputs it into the linear classifier for decoding, and outputs the graph-level classification prediction result. This type of task implementation uses the macro-F1 score or overall accuracy as the evaluation metric.

[0103] 2) Specific implementation for pixel-level segmentation tasks: For FLAIR, Sen1Floods11, and HLS BurnScar segmentation tasks, the single modality with the spatial resolution closest to the target annotation is selected as the benchmark. Local shallow sub-block features are extracted from this benchmark modality through a scale-adaptive embedding layer encoding, and these local shallow sub-block features are concatenated with the deep multimodal features output by the cross-modal fusion encoder along the channel dimension. The concatenated fused features are input into the segmentation decoding head for the specific task, mapping to pixel-by-pixel segmentation prediction results. This type of task implementation uses the average intersection-union ratio (IUU) as the evaluation metric.

[0104] Verification of the technical effects of the embodiments and comparative examples of the present invention: To verify the technical advancement of this invention, this embodiment not only compares the baseline model (Ours-Base) without the acceleration module with the accelerated model (Ours-Acc) of this invention, but also introduces the mainstream advanced architectures in the existing multimodal remote sensing field as a comparison baseline.

[0105] Pre-training efficiency verification: As shown in Table 1, the acceleration mechanism of this invention filters out low-information-content redundant regions, ultimately retaining approximately 15% of the core tokens for cross-modal fusion computation. This mechanism enables the accelerated model to achieve a pre-training speedup of approximately 1.9 times under different data volumes. Furthermore, as the training sample size increases, the accelerated model maintains a stable speedup ratio, and the absolute time savings increase proportionally, indicating that the proposed solution has good scalability when processing large-scale remote sensing data.

[0106] Table 1: Comparison of Pre-training Efficiency between Baseline and Accelerated Models

[0107] Validation of downstream task accuracy and feature representation capabilities: The pre-trained model was then adapted to the downstream evaluation tasks. These tasks were divided into in-distribution tasks (results shown in Table 2) and out-of-distribution tasks (results shown in Table 3). The data in the tables show that the baseline model without acceleration achieved high accuracy in all tasks, and its feature representation capability outperformed some existing mainstream baseline models. Based on this baseline, the accelerated model of this invention increased the data throughput from 46k images per minute to 87k images per minute, achieving an acceleration of approximately 1.9 times. Its specific accuracy performance is as follows: Table 2: Comparison of Performance and Modalities of Downstream Tasks within Distribution

[0108] In-distribution task evaluation analysis: In the intra-distribution evaluation, for the TreeSatAI-TS multi-label tree species classification task, the accelerated model filters out low-information background noise through an asymmetric masking mechanism, prompting the model to focus on core features, and its macro F1 score improves from 75.6 to 76.2 compared to the baseline model. For the boundary-sensitive FLAIR land cover semantic segmentation task, the accelerated model adopts a strategy of densely concatenating shallow local features and deep cross-modal features to compensate for the information loss caused by the discarding of deep spatial labels, keeping the decrease in the average intersection-union ratio within 0.3 (from 56.3 to 56.0). These results show that the acceleration mechanism significantly improves efficiency while enhancing the discriminative power of specific classification tasks and maintaining the accuracy of high-resolution segmentation tasks.

[0109] Table 3: Comparison of Performance and Generalization Ability of Off-Distribution Downstream Tasks

[0110] In out-of-distribution evaluation, the model's generalization ability on external datasets not seen in the pre-training corpus was tested. For the Sen1Floods11 flood segmentation and So2Sat climate zone classification tasks across geographic regions, the accelerated model maintained stable feature representation capabilities under the linear probe setting with a frozen backbone network. The overall So2Sat accuracy improved by 0.3 (reaching 59.7), and the mean intersection-over-union (MIU) fluctuation of Sen1Floods11 was controlled within 0.4 (reaching 91.1). For the cross-sensor transfer task, when facing novel sensor modalities (HLS) not included in the pre-training corpus, the accelerated model achieved an end-to-end fine-tuned MUI of 90.5 and a linear probe MUI of 87.5, with fluctuations within 0.3 compared to the baseline model.

[0111] Table 4: Comparison of ablation experiments using the dual-layer synergistic acceleration strategy

[0112] Ablation validation of the synergistic effect of the two-layer acceleration mechanism: As shown in Table 4, the ablation experiment results demonstrate that neither data-level acceleration nor model-level acceleration alone can simultaneously achieve both processing efficiency and accuracy across different task types. This invention combines data-level spatial-modal asymmetric masking with model-level adaptive token merging, utilizing model-level feature smoothing to offset the sampling bias of the data-level mask. The final data shows that the two-layer collaborative mechanism achieves the highest throughput and high overall classification accuracy, while maintaining stable performance in complex segmentation tasks, proving the necessity of this two-layer collaborative mechanism.

[0113] In summary, this invention presents a semantically aware multimodal remote sensing pre-training collaborative acceleration method and system. It divides multi-source remote sensing images into image patches using a unified geospatial grid, ensuring that different modalities correspond to the same geographic region under the same grid index, providing a spatially consistent foundation for subsequent cross-modal alignment, masking, and fusion. Through semantic importance assessment based on physical pixel information, it generates single-modal specific scores and cross-modal comprehensive scores, enabling the model to identify key semantic regions and low-information redundant regions before encoding. Furthermore, this invention utilizes single-modal specific scores to control adaptive token merging during the intra-modal spatial encoding stage, reducing the feature computation cost of homogeneous regions; and uses cross-modal comprehensive scores to perform spatial-modal asymmetric masking, preserving high-information context regions, separating target regions, and discarding redundant regions, thereby reducing token computation overhead at both the data and model levels. By using a cross-modal fusion encoder and a teacher-student latent space prediction training mechanism, this invention improves the throughput and training efficiency of multimodal remote sensing pre-training while maintaining the semantics of key ground objects, modal complementarity, and spatial context representation capabilities, thereby enhancing the model's generalization performance in classification, segmentation, and cross-regional and cross-sensor tasks.

Claims

1. A multimodal remote sensing pre-training collaborative acceleration method based on semantic awareness, characterized in that, Includes the following steps: S1. Obtain geospatial aligned multi-source remote sensing image samples, divide each modal image according to a unified geospatial grid, and obtain multimodal image blocks corresponding to the same geographic region under the same grid index; S2. Based on the physical pixel information of multimodal image blocks, perform semantic importance evaluation to generate single-modal specific scores and cross-modal comprehensive scores; S3. Perform scale-adaptive embedding on multimodal image blocks and map them to a shared latent space to obtain the feature sequences of sub-blocks for each modality; S4. Input the sub-block feature sequence into the intramodal spatial encoder and perform adaptive token merging based on the single-modal specific score to obtain the image block-level features of each modality; S5. Construct positive sample pairs based on image block-level features of different modalities under the same grid index, and construct negative sample pairs based on image block-level features of different grid indices, and calculate cross-modal contrast loss; S6. Based on the cross-modal integrated score, perform spatial-modal asymmetric masking on the image block-level features, retain context region features, separate target region features and discard redundant region features; S7. Input the context region features into the cross-modal fusion encoder to obtain the fused context representation; S8. Based on the fusion context representation, predict the latent space features of the target region. The teacher network generates the latent space target features of the target region based on the block-level features of the complete unmasked image of all modalities. Based on the cross-modal contrast loss and the latent space prediction loss between the predicted latent space features of the target region and the latent space target features of the target region, train the multimodal remote sensing pre-trained model.

2. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 1, characterized in that, The multi-source remote sensing image samples include at least one of optical modal images, synthetic aperture radar modal images, and multispectral modal images; When dividing the image into blocks, the same geospatial grid unit is mapped to the pixel region in the corresponding modal image according to the spatial resolution of each modal image, so that different modal image blocks under the same grid index represent the same geographic region.

3. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 1, characterized in that, Step S2 includes: Pixel-level statistics are extracted based on the physical imaging characteristics of different modal images, and the pixel-level statistics are aggregated into single-modality specific scores at the image block level. After standardizing the single-modal specific score and mapping it to a preset numerical range, a cross-modal comprehensive score is generated through a cross-modal fusion operator.

4. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 3, characterized in that, When generating single-mode specific scores, for optical modal images, pixel-level statistics are extracted based on texture complexity; for synthetic aperture radar modal images, pixel-level statistics are extracted based on backscattering intensity; and for multispectral modal images, pixel-level statistics are extracted based on spectral reflectance changes.

5. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 1, characterized in that, Step S3 includes: The multimodal image block is divided into a sequence of sub-blocks of fixed pixel size. The sub-block sequence is mapped to a shared latent space through a linear projection layer that matches the corresponding modality. A positional code is added to the feature sequence of the mapped sub-block. The positional code contains the spatial resolution information and geographical location information of the corresponding modality.

6. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 1, characterized in that, Step S4 includes: The token merging rate of an image block is determined based on the specific score of the single modality corresponding to that image block. When an image patch belongs to a high-resolution region, a lower token merging rate is used to preserve the feature details of the sub-blocks; When an image patch belongs to a low-resolution region, a higher token merging rate is used to compress redundant sub-block features.

7. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 6, characterized in that, When performing adaptive token merging, the sub-block feature sequences corresponding to image blocks with the same token merging rate are divided into the same computation group, and the sub-block feature sequences within the same computation group are recombined into standard tensors for parallel computation. Different computational groups sequentially pass through a shared intramodal spatial encoder, and the gradients of each computational group are accumulated during the backpropagation phase.

8. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 1, characterized in that, Step S6 includes: The context region and target region are determined based on the cross-modal synthesis score, wherein the context region is preferentially assigned to the high-scoring grid index, and the target region is assigned to the medium-scoring grid index. Perform intramodal masking within the context region, replacing the masked modal features with learnable mask tokens, and ensuring that at least one modality's valid input is preserved at each spatial location.

9. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 1, characterized in that, Step S7 includes: attaching block-level absolute location encoding based on a unified geospatial grid index to the context region features, and performing global context modeling and heterogeneous feature fusion on the context region features of different modalities through the self-attention mechanism and cross-attention mechanism in the cross-modal fusion encoder.

10. The semantically aware multimodal remote sensing pre-training collaborative acceleration method according to claim 1, characterized in that, The multimodal remote sensing pre-trained model includes a student network and a teacher network; The student network includes an intramodal spatial encoder, a crossmodal fusion encoder, and a predictor; the teacher network includes an intramodal spatial encoder and a crossmodal fusion encoder. The teacher network receives complete unmasked image block-level features from all modalities and generates latent space target features for the target region. The student network generates predicted latent space features for the target region based on context region features. The parameters of the teacher network are updated using an exponential moving average based on the parameters of the student network.

11. A semantically aware multimodal remote sensing pre-training collaborative acceleration system, characterized in that, include: The image patch segmentation module is used to acquire geospatial aligned multi-source remote sensing image samples and divide each modal image according to a unified geospatial grid to obtain multimodal image patches corresponding to the same geographic region under the same grid index. The semantic importance assessment module is used to generate single-modality specific scores and cross-modality comprehensive scores based on the physical pixel information of multimodal image patches; The scale-adaptive embedding module is used to perform scale-adaptive embedding on multimodal image patches and map them to a shared latent space to obtain the feature sequences of sub-patterns for each modality. The intramodal spatial coding module is used to perform adaptive token merging on the sub-block feature sequence based on a single modality specific score to obtain image block-level features for each modality. The asymmetric masking module is used to perform spatial-modal asymmetric masking on image block-level features based on cross-modal synthesis scores to obtain context region features, target region features, and redundant region features. The cross-modal fusion module is used to perform cross-modal fusion of context region features to obtain a fused context representation; The latent space prediction training module is used to predict the latent space features of the target region based on the fused context representation, and to train a multimodal remote sensing pre-trained model based on the latent space prediction loss between the predicted latent space features and the real latent space features.

12. The semantically aware multimodal remote sensing pre-training collaborative acceleration system according to claim 11, characterized in that, The semantic importance assessment module includes a single-modal scoring unit and a cross-modal comprehensive scoring unit; The single-modal scoring unit is used to extract pixel-level statistics for the physical imaging characteristics of different modal images and generate single-modal specific scores. The cross-modal comprehensive scoring unit is used to standardize and interval map specific scores of a single modality, and generate a cross-modal comprehensive score.

13. The semantically aware multimodal remote sensing pre-training collaborative acceleration system according to claim 11, characterized in that, The intramodal spatial coding module includes a token merging unit and a grouped parallel computing unit; The token merging unit is used to determine the token merging rate of the sub-block feature sequence based on a specific score of a single modality; The grouped parallel computing unit is used to divide sub-block feature sequences with the same token merging rate into the same computing group, and to perform parallel encoding on the sub-block feature sequences within the same computing group.

14. The semantically aware multimodal remote sensing pre-training collaborative acceleration system according to claim 11, characterized in that, The multimodal remote sensing pre-trained model includes a student network and a teacher network; The student network includes an intramodal spatial encoder, a crossmodal fusion encoder, and a predictor; the teacher network includes an intramodal spatial encoder and a crossmodal fusion encoder. The teacher network is used to generate true latent space features of the target region based on the complete unmasked image block-level features of all modalities, and the student network is used to generate predicted latent space features of the target region based on the context region features.

Images