Remote sensing visual basic model pre-training method for cross-imaging-mode unified modeling

Abstract

The invention discloses a remote sensing visual basic model pre-training method for cross-imaging-modal unified modeling, and relates to the technical field of remote sensing image processing, and the method comprises the steps: constructing a pre-training data set, and randomly selecting remote sensing image sequences of two modalities for mask processing; inputting a remote sensing visual basic model, extracting embedded features through a spatio-temporal feature adaptive embedded layer, processing through a hybrid expert encoder to obtain decoupling features, processing through a cross attention encoder to obtain cross features, and reconstructing to obtain a reconstructed image of a corresponding mode; constructing a loss function according to the difference between the reconstructed image and the remote sensing image, and optimizing model parameters according to the loss function; and under the condition of model convergence, obtaining a pre-trained remote sensing visual basic model. According to the method, a dynamic parameter adjustment mechanism is introduced, deployment requirements of different places can be considered, good expansibility and universality are achieved, and a universal and telescopic technical scheme can be provided for efficient processing and intelligent analysis of earth observation big data.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a pre-training method for a unified modeling method for remote sensing visual fundamental models across imaging modalities. Background Technology

[0002] As a core support for the development of artificial intelligence-driven Earth observation, remote sensing fundamental models provide a new paradigm for the intelligent processing of complex remote sensing tasks. Compared with traditional mission-specific models, remote sensing fundamental models not only significantly reduce reliance on high-cost labeled data, but also possess the ability to migrate across missions and perform rapid fine-tuning. They can widely support key applications such as ecological environment monitoring, climate change analysis, disaster assessment, land surveys, and smart city construction, demonstrating extremely high scientific research value and application potential.

[0003] Currently, most mainstream remote sensing fundamental models are limited to single-modal or single-temporal image processing. Some related technologies attempt to fuse multispectral and synthetic aperture radar data to enhance representation capabilities, but they struggle to effectively capture the spatiotemporal evolution patterns of surface processes. To address modal heterogeneity, some related technologies employ multi-scale feature fusion or spatiotemporal decoupling strategies for feature alignment and joint learning. However, these methods typically rely on complex modality-specific preprocessing procedures or fixed fusion structures, lacking adaptability to different task scenarios. This limits model generalization and makes it difficult to flexibly extend to new modalities.

[0004] Furthermore, as model sizes continue to increase, many related technologies are deployed by designing and pre-training multiple dedicated models for different hardware platforms. This not only leads to redundant training costs but also violates the requirement of "modeling once, adapting to multiple devices" for efficient and intelligent analysis. Especially in resource-constrained environments such as edge devices, large models often face problems such as high inference latency and large memory consumption, severely restricting their practical applications.

[0005] In summary, although remote sensing fundamental models have made significant progress in promoting intelligent Earth observation, they still have many shortcomings in areas such as cross-modal unified modeling, spatiotemporal dynamic information fusion, and flexible model deployment. Therefore, there is an urgent need to propose a new technical solution that can achieve multimodal collaborative sensing, scalable modeling, and efficient application. Summary of the Invention

[0006] This application provides a pre-training method for a unified modeling approach across imaging modalities for remote sensing visual fundamental models, in order to address the shortcomings of the aforementioned related technologies. The technical solution is as follows: In a first aspect, this application provides a pre-training method for a unified modeling approach for remote sensing vision across imaging modalities, including: Construct a pre-training dataset of remote sensing image sequences that include at least two modalities; Two modalities of remote sensing image sequences are randomly selected from the pre-trained dataset, and each remote sensing image is masked separately. The masked remote sensing image sequence is input into the remote sensing vision basic model, and the corresponding modality embedding features are extracted through the spatiotemporal feature adaptive embedding layer of the remote sensing vision basic model. The hybrid expert encoder of the remote sensing vision basic model processes the embedding features of the two modalities respectively to obtain the decoupled features of the corresponding modalities. The decoupled features of the two modalities are processed by the cross-attention encoder of the remote sensing vision basic model to obtain the cross features of the corresponding modalities. The reconstructed image of the corresponding modalities is then obtained by using a projection network based on the cross features. The loss function of the remote sensing visual basic model is constructed based on the difference between the reconstructed image and the corresponding remote sensing image, and the model parameters of the remote sensing visual basic model are optimized based on the loss function. If the model converges, a pre-trained remote sensing vision basic model is obtained.

[0007] In one alternative embodiment of the first aspect, the masking process for each remote sensing image includes: Each remote sensing image is processed by image block-level sampling, and a spatial mask region or a temporal mask region is generated in each remote sensing image to obtain a masked remote sensing image.

[0008] In one alternative embodiment of the first aspect, the step of extracting features through a spatiotemporal feature adaptive embedding layer of a remote sensing visual basic model to obtain the embedded features of the corresponding modality includes: The remote sensing images of the two modalities after masking are processed by a spatiotemporal feature adaptive embedding layer, and the spectral features and temporal features are mapped to a high-dimensional space. The interaction relationship between the spectral features and temporal features is mined based on a dynamic attention mechanism to generate a fused feature representation. High-dimensional feature representations of remote sensing images are generated based on fused feature representations, and convolutional layers are used to map these high-dimensional feature representations into image block-level embedded features.

[0009] In one alternative of the first aspect, the hybrid expert encoder includes multiple different expert subnetworks; The hybrid expert encoder, based on the remote sensing vision model, processes the embedding features of the two modalities respectively to obtain the decoupled features of the corresponding modalities, including: The hybrid expert encoder processes the embedded features of the input through a learnable gating network and activates a preset number of expert sub-networks based on the embedded features. The activation weights of each activated expert subnetwork are calculated using the gated network. Obtain the encoding result of each activated expert subnetwork based on the embedded features; The encoding results are weighted and summed based on the activation weights of each activated expert subnetwork to obtain the decoupling features of the corresponding modality.

[0010] In one alternative embodiment of the first aspect, the step of processing the decoupled features of the two modalities respectively through the cross-attention encoder of the remote sensing vision basic model to obtain the cross features of the corresponding modalities, and using a projection network to reconstruct the reconstructed image of the corresponding modality based on the cross features, includes: The query vector corresponding to mode one is generated based on the decoupling features of mode one, and the key vector and value vector corresponding to mode one are generated based on the decoupling features of mode two. Cross attention is calculated based on the query vector, key vector and value vector corresponding to mode one to obtain the cross features of mode one. The query vector corresponding to mode two is generated based on the decoupling features of mode two, and the key vector and value vector corresponding to mode two are generated based on the decoupling features of mode one. Cross attention is calculated based on the query vector, key vector and value vector corresponding to mode two to obtain the cross features of mode two. The pixel values ​​of the mask region are reconstructed using a projection network based on the cross features of mode 1 and mode 2, respectively, to obtain the reconstructed image of the corresponding mode.

[0011] In one alternative embodiment of the first aspect, the loss function for constructing the remote sensing visual basis model based on the difference between the reconstructed image and the corresponding remote sensing image includes: The reconstruction loss term is constructed based on the difference between each remote sensing image and its corresponding reconstructed image in the remote sensing image sequence before masking. An auxiliary loss term is constructed based on the activation weight distribution of each expert sub-network in the hybrid expert encoder; The loss function of the remote sensing vision basic model is obtained by constructing the reconstruction loss term, auxiliary loss term, and hyperparameters of the auxiliary loss term. The process of optimizing the model parameters of the remote sensing vision basic model based on the loss function, and obtaining the pre-trained remote sensing vision basic model upon determining model convergence, includes: Backpropagation is performed based on the value of the loss function to optimize the model parameters of the remote sensing visual basic model; If the value of the loss function is less than a preset threshold, or reaches the iteration number threshold, then the model is determined to have converged, and a pre-trained remote sensing vision basic model is obtained. Otherwise, proceed to the masking process and subsequent steps for the next iteration.

[0012] In one alternative of the first aspect, after obtaining the pre-trained remote sensing visual basic model, the target task is processed through the obtained pre-trained remote sensing visual basic model, wherein the input features of the target task include at least one modality of remote sensing image sequence. The process of processing the target task includes: The remote sensing image sequence of at least one modality is input into the remote sensing visual basic model, and the steps of extracting the embedding features of the corresponding modality through the spatiotemporal feature adaptive embedding layer of the remote sensing visual basic model and subsequent steps are performed until the decoupled features of the corresponding modality are obtained. The output of the hybrid expert encoder of the remote sensing vision basic model is connected to the prediction head of the corresponding target task. The decoupled features are processed by the prediction head of the corresponding target task, and the prediction result of the target task is output.

[0013] Secondly, this application also provides a pre-training device for a unified modeling of remote sensing visual fundamental models across imaging modalities, comprising: Dataset building unit, used to build a pre-training dataset of remote sensing image sequences including at least two modalities; The masking unit is used to randomly select two modalities of remote sensing image sequences from the pre-training dataset and perform masking processing on each remote sensing image separately. The model pre-training unit is used to input the masked remote sensing image sequence into the remote sensing vision basic model, and extract the embedding features of the corresponding modality through the spatiotemporal feature adaptive embedding layer of the remote sensing vision basic model. The model pre-training unit is also used to process the embedding features of the two modalities respectively through the hybrid expert encoder of the remote sensing vision basic model to obtain the decoupled features of the corresponding modalities. The model pre-training unit is also used to process the decoupled features of the two modalities respectively through the cross-attention encoder of the remote sensing vision basic model to obtain the cross features of the corresponding modalities, and to reconstruct the corresponding modalities based on the cross features using the projection network. The model pre-training unit is also used to construct a loss function for the remote sensing visual basic model based on the difference between the reconstructed image and the corresponding remote sensing image, and to optimize the model parameters of the remote sensing visual basic model based on the loss function. The model pre-training unit is also used to obtain a pre-trained remote sensing visual basic model when the model is determined to have converged.

[0014] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method provided by the first aspect of this application or any implementation thereof.

[0015] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of this application or any implementation thereof.

[0016] The beneficial effects of the technical solution provided in this application include at least the following: This application leverages the inherent characteristics of spectral and temporal attributes to construct a unified representation space, thereby eliminating the rigid dependence of traditional models on sensor modality and spatiotemporal resolution, and achieving consistent interpretation of heterogeneous Earth observation data sources. The remote sensing vision foundation model in this application employs a hybrid expert encoder, adaptively activating the optimal sub-network based on input data features and task requirements, achieving dynamic adaptation and efficient inference in cross-modal and multi-task scenarios. Combined with a cross-modal attention-guided mask reconstruction pre-training method, the quality of multi-source information fusion is effectively improved, allowing for parameter adjustment of the model based on multimodal data. This application outperforms traditional methods in various tasks with single-modal / multimodal inputs and sparse / missing annotations, demonstrating high accuracy, high efficiency, and strong generalization ability, providing an efficient and robust solution to the challenges of existing remote sensing foundation models.

[0017] Furthermore, this application achieves unified representation learning of multi-source remote sensing image data through adaptive spatiotemporal feature modeling and a lightweight multimodal coding structure. By introducing a dynamic parameter adjustment mechanism, it can accommodate the deployment requirements of hardware equipment in different scenarios. Through a pre-trained remote sensing vision base model, it can flexibly scale model parameters while maintaining cross-modal robustness, and also achieve end-to-end adaptation for various remote sensing interpretation tasks such as land use classification, disaster monitoring, and ecological environment assessment. This application has good scalability and versatility, providing a general and scalable technical solution for the efficient processing and intelligent analysis of Earth observation big data. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating a method for pre-training a remote sensing visual basic model for unified modeling across imaging modalities, provided in an embodiment of this application. Figure 2 This is the second flowchart illustrating a method for pre-training a remote sensing visual basic model for unified modeling across imaging modalities, as provided in an embodiment of this application. Figure 3This is a schematic diagram illustrating the effects of different modal inputs provided in the embodiments of this application; Figure 4 This is one of the visual comparison diagrams of different modal task results provided in the embodiments of this application; Figure 5 This is the second of the visualization comparison diagrams of different modal task results provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a remote sensing visual basic model pre-training device for unified modeling across imaging modalities provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0022] It should be noted that the terms "first" and "second" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than those described or illustrated herein.

[0023] The present application will now be described in detail with reference to specific embodiments.

[0024] Next, combine Figure 1 This paper introduces a pre-training method for a unified modeling approach across imaging modalities in remote sensing vision, provided by embodiments of this application. For details, please refer to... Figure 1 , Figure 1 This illustration shows a flowchart of a pre-training method for a unified modeling approach across imaging modalities in remote sensing vision, provided in an embodiment of this application. Figure 1As shown, the method includes the following steps: S101, Construct a pre-training dataset of remote sensing image sequences including at least two modalities; S102, randomly select two modalities of remote sensing image sequences from the pre-training dataset, and perform masking processing on each remote sensing image; S103, input the masked remote sensing image sequence into the remote sensing visual basic model, and extract the embedding features of the corresponding modality through the spatiotemporal feature adaptive embedding layer of the remote sensing visual basic model. S104, the hybrid expert encoder of the remote sensing vision basic model processes the embedding features of the two modalities respectively to obtain the decoupled features of the corresponding modalities; S105, the decoupled features of the two modes are processed by the cross-attention encoder of the remote sensing vision basic model to obtain the cross features of the corresponding modes, and the reconstructed image of the corresponding mode is obtained by using the projection network based on the cross features. S106, construct the loss function of the remote sensing visual basic model based on the difference between the reconstructed image and the corresponding remote sensing image, and optimize the model parameters of the remote sensing visual basic model based on the loss function. S107, if the model converges, the pre-trained remote sensing vision basic model is obtained.

[0025] In some embodiments, the pre-trained dataset constructed in S101 is represented as ,in, This represents a remote sensing image sequence, where the subscript i indicates the modality number. For timing length, For the number of channels, Image height, The image width is specified. The pre-trained dataset includes at least two different modalities of remote sensing images. The modalities include, but are not limited to, visible light RGB images, pseudo-color images, multispectral images, hyperspectral images, and synthetic aperture radar images. This application does not limit the specific modalities.

[0026] In some embodiments, S102 may extract data from the pre-trained dataset in each iteration round. Two spatially aligned remote sensing image sequences are randomly selected from the data and used for pre-training of the remote sensing vision basic model; these sequences are denoted as remote sensing image sequences. and remote sensing image sequences For example, a visible light image (size 1m) from WorldView-3 with a ground sampling distance (GSD) of 1m can be selected. ) and GSD 10m multispectral multi-temporal images from Sentinel-2 (resampled size is ) ).

[0027] Furthermore, each remote sensing image in the two modal remote sensing image sequences can be masked to generate masked regions. Masking can not only be used to simulate scenarios with missing information, but also enhance the model's self-supervised learning ability and provide constraints for cross-modal reconstruction and robust feature extraction. Specifically, in the masking process, each remote sensing image is processed by image block-level sampling to generate a spatial mask region or a temporal mask region in each remote sensing image, thus obtaining the masked remote sensing image.

[0028] In some embodiments, S103, after inputting the masked remote sensing image sequence into the remote sensing visual base model, the masked remote sensing image sequence is processed through the spatiotemporal feature adaptive embedding layer of the remote sensing visual base model, specifically including: The remote sensing images of the two modalities after masking are processed by a spatiotemporal feature adaptive embedding layer, and the spectral features and temporal features are mapped to a high-dimensional space. The interaction relationship between the spectral features and temporal features is mined based on a dynamic attention mechanism to generate a fused feature representation. High-dimensional feature representation of remote sensing images generated based on fusion feature representation The high-dimensional feature representation is mapped to image patch-level embedding features using convolutional layers. ; in, Indicates the number of image patches. Where is the image patch size, and D is the feature dimension.

[0029] Further, S104 is executed to process the embedded features through a hybrid expert encoder; It should be noted that a hybrid expert encoder can be understood as a Transformer encoder that incorporates a Mixture of Experts (MoE) module. In other words, the feedforward network layer in a standard Transformer encoder is replaced by a hybrid expert module. The hybrid expert module includes multiple different expert sub-networks, each with the same structure and different network parameters. Different network parameters can be pre-configured for each expert sub-network to handle different types and modalities of data.

[0030] Specifically, S104 includes: S1041, the hybrid expert encoder processes the embedded features of the input through a learnable gating network and activates a preset number of expert sub-networks according to the embedded features; for example, the expert sub-network with the strongest correlation to the corresponding embedded features can be determined according to the modality type, task type, etc. of the embedded features, for example, two expert sub-networks can be selected to process the embedded features. S1042, The activation weight of each activated expert subnetwork is calculated through the gating network; S1043, Obtain the encoding result of each activated expert subnetwork based on the embedded features; S1044, The encoding results are weighted and summed based on the activation weights of each activated expert sub-network to fuse the encoding results of each activated expert sub-network and obtain the decoupling features of the corresponding modality; For example, processing yields remote sensing image sequences. Corresponding decoupling features and remote sensing image sequences Corresponding decoupling features .

[0031] In some embodiments, in S105, the decoupled features are further input into the cross-attention decoder to realize the interactive fusion of information from different sensors using the cross-modal cross-attention mechanism, generating cross features with cross-modal interaction capabilities, thereby reconstructing and predicting the mask region based on the cross features.

[0032] In S105, the cross-features of the two modes can be processed separately. One mode can be denoted as Mode 1, and the other mode as Mode 2. Specifically, this includes: The query vector corresponding to mode one is generated based on the decoupling features of mode one, and the key vector and value vector corresponding to mode one are generated based on the decoupling features of mode two. Cross attention is calculated based on the query vector, key vector and value vector corresponding to mode one to obtain the cross features of mode one. The query vector corresponding to mode two is generated based on the decoupling features of mode two, and the key vector and value vector corresponding to mode two are generated based on the decoupling features of mode one. Cross attention is calculated based on the query vector, key vector and value vector corresponding to mode two to obtain the cross features of mode two. This allows us to obtain the cross-features of the two modes separately.

[0033] Furthermore, the pixel values ​​of the mask region are reconstructed using a projection network based on the cross features of mode 1 and mode 2, respectively, to obtain the reconstructed image of the corresponding mode, including the reconstructed image. and reconstructed images .

[0034] For example, the process in S102-S105 is as follows: Figure 2 As shown, any two modes of imagery can be selected from high-resolution visible light imagery, multi-temporal synthetic aperture radar imagery, multi-temporal spectral imagery, and hyperspectral imagery. Figure 2The selected modalities in the example are high-resolution visible light image and multi-temporal spectral image. Further masking preprocessing is performed, and the masked modal 1 and masked modal 2 are input into the spatiotemporal feature adaptive embedding layer, respectively. The spatiotemporal feature adaptive embedding layer processes the image based on masked modal 1 to obtain embedding feature 1, and processes it based on masked modal 2 to obtain embedding feature 2. Embedding feature 1 and embedding feature 2 are then input into a hybrid expert encoder, which processes them to obtain decoupling feature 1 and decoupling feature 2. Decoupling feature 1 and decoupling feature 2 are then input into a cross-attention encoder. The cross-attention encoder combines decoupling feature 1 and decoupling feature 2 to obtain reconstructed image 1 and reconstructed image 2 corresponding to modal 1.

[0035] Furthermore, in S106, a loss function for the remote sensing visual baseline model can be constructed based on the differences between the reconstructed image and the corresponding remote sensing image, specifically including: S1061, Construct a reconstruction loss term based on the difference between each remote sensing image in the remote sensing image sequence before masking and the corresponding reconstructed image. For example, the mean absolute error L1 loss is used to calculate the input image. With reconstructed images The difference between them yields the reconstruction loss term. and input images With reconstructed images The difference between them yields the reconstruction loss term. , ; S1062, construct the auxiliary loss term based on the activation weight distribution of each expert sub-network in the hybrid expert encoder. The auxiliary loss term can be evaluated based on the squared coefficient of variation of the activation weights of each expert sub-network to prevent features from becoming overly dependent on a few expert sub-models. S1063, Based on Reconstruction Loss Item Auxiliary loss items and the hyperparameters of the auxiliary loss term. The loss function for constructing the basic remote sensing visual model is given by the following formula: ; S1064, perform backpropagation based on the value of the loss function to optimize the model parameters of the remote sensing visual basic model; for example, AdamW can be selected as the optimizer during the backpropagation process.

[0036] Next, execute S107. The model convergence can be determined based on the value of the loss function and the number of current iterations. If the value of the loss function is less than a preset threshold, or the number of current iterations reaches the iteration threshold, the model convergence is determined, and a pre-trained remote sensing vision basic model is obtained.

[0037] Otherwise, proceed to step S102 and subsequent steps for the next iteration until the model converges.

[0038] It should be noted that a preset threshold for the number of iterations can be selected according to the specific situation, and the optimized model parameters can be saved after each preset number of iterations of training. This application embodiment does not limit this.

[0039] In some embodiments, after obtaining a pre-trained remote sensing visual basic model, the target task can be processed through the obtained pre-trained remote sensing visual basic model. The input features of the target task may include at least one modality of remote sensing image sequence. The process of processing a given target task using a pre-trained remote sensing vision base model includes: Input the remote sensing image sequence of at least one modality into the remote sensing visual basic model, and execute steps S103-S104 through the remote sensing visual basic model until the decoupling features of the corresponding modality are obtained. The output of the hybrid expert encoder of the remote sensing vision basic model is connected to the prediction head of the corresponding target task. The decoupled features are processed by the prediction head of the corresponding target task, and the prediction result of the target task is output.

[0040] For example, when the target task is image classification, the connected prediction head is a classification prediction head; when the target task is image segmentation, the connected prediction head is a segmentation prediction head; when the target task is image change detection, the connected prediction head is a change detection head. This is only an example of the type of prediction head, and the embodiments of this application do not limit the type of prediction head.

[0041] In some embodiments, the pre-trained remote sensing visual basic model can be trained using visible light imagery and a digital elevation model (DEM) as training modalities during the training process, and the pre-trained remote sensing visual basic model can be used to process glacier mapping tasks.

[0042] For example, such as Figure 3 As shown, when processing glacier mapping tasks, in three cases—modal loss, cross-modal migration, and simultaneous presence of modal loss and cross-modal migration—the performance of the remote sensing visual basic model in this application is superior to that of related technologies. The IoU (Intersection over Union) parameter of the remote sensing visual basic model in this application is also higher than that of the models in related technologies.

[0043] It should be noted that modality loss can be understood as the situation where the modal types of the images input to the remote sensing visual basic model do not cover all the training modalities of the remote sensing visual basic model; cross-modal transfer can be understood as the modal types of the images input to the remote sensing visual basic model including additional modalities other than all the training modalities of the remote sensing visual basic model; when both modality loss and cross-modal transfer exist, the modal types of the images input to the remote sensing visual basic model do not cover all the training modalities of the remote sensing visual basic model, and also include additional modalities other than the training modalities.

[0044] In one specific embodiment, such as Figures 4-5 As shown, in various cases where the input images are visible light images, multi-temporal visible light images, combinations of visible light images and hyperspectral images, multispectral images, multispectral multi-temporal images, combinations of multispectral images and DEMs, hyperspectral images, or synthetic aperture radar images, the remote sensing visual basic model pre-trained in the embodiments of this application can achieve processing results superior to related technologies, and the processing results are closer to the true labels.

[0045] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0046] Please see below. Figure 6 The diagram below illustrates the structure of a pre-training device for a unified modeling approach for remote sensing vision across imaging modalities, as provided in an exemplary embodiment of this application. The device includes: Dataset building unit, used to build a pre-training dataset of remote sensing image sequences including at least two modalities; The masking unit is used to randomly select two modalities of remote sensing image sequences from the pre-training dataset and perform masking processing on each remote sensing image separately. The model pre-training unit is used to input the masked remote sensing image sequence into the remote sensing vision basic model, and extract the embedding features of the corresponding modality through the spatiotemporal feature adaptive embedding layer of the remote sensing vision basic model. The model pre-training unit is also used to process the embedding features of the two modalities respectively through the hybrid expert encoder of the remote sensing vision basic model to obtain the decoupled features of the corresponding modalities. The model pre-training unit is also used to process the decoupled features of the two modalities respectively through the cross-attention encoder of the remote sensing vision basic model to obtain the cross features of the corresponding modalities, and to reconstruct the corresponding modalities based on the cross features using the projection network. The model pre-training unit is also used to construct a loss function for the remote sensing visual basic model based on the difference between the reconstructed image and the corresponding remote sensing image, and to optimize the model parameters of the remote sensing visual basic model based on the loss function. The model pre-training unit is also used to obtain a pre-trained remote sensing visual basic model when the model is determined to have converged.

[0047] It should be noted that the apparatus provided in the above embodiments, when executing a method for pre-training a remote sensing visual basic model with unified modeling across imaging modalities, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the apparatus provided in the above embodiments and the embodiment of a method for pre-training a remote sensing visual basic model with unified modeling across imaging modalities belong to the same concept, and the implementation process is detailed in the method embodiment, which will not be repeated here.

[0048] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0049] Please see Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0050] like Figure 7 As shown, the electronic device includes a processor and a memory.

[0051] In this embodiment, the processor is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array).

[0052] A processor can also include a main processor and a coprocessor. The main processor is used to process data in the wake-up state and is also called the CPU (Central Processing Unit). The coprocessor is a low-power processor used to process data in the standby state.

[0053] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the non-transitory computer-readable storage media in the memory are used to store at least one instruction, which is executed by a processor to implement the methods in the embodiments of this application.

[0054] In some embodiments, the electronic device further includes a peripheral device interface and at least one peripheral device. The processor, memory, and peripheral device interface are connected via a bus or signal line. Each peripheral device is connected to the peripheral device interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes: a display screen, a camera, and audio circuitry. The peripheral device interface can be used to connect at least one I / O (Input / Output) related peripheral device to the processor and memory.

[0055] In some embodiments of this application, the processor, memory, and peripheral device interfaces are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor, memory, and peripheral device interfaces can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.

[0056] The electronic device structural block diagrams shown in the embodiments of this application do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0057] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0058] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A pre-training method for a remote sensing visual fundamental model that unifies modeling across imaging modalities, characterized in that, include: Construct a pre-training dataset of remote sensing image sequences that include at least two modalities; Two modalities of remote sensing image sequences are randomly selected from the pre-trained dataset, and each remote sensing image is masked separately. The masked remote sensing image sequence is input into the remote sensing vision basic model, and the corresponding modality embedding features are extracted through the spatiotemporal feature adaptive embedding layer of the remote sensing vision basic model. The hybrid expert encoder of the remote sensing vision basic model processes the embedding features of the two modalities respectively to obtain the decoupled features of the corresponding modalities. The decoupled features of the two modalities are processed by the cross-attention encoder of the remote sensing vision basic model to obtain the cross features of the corresponding modalities. The reconstructed image of the corresponding modalities is then obtained by using a projection network based on the cross features. The loss function of the remote sensing visual basic model is constructed based on the difference between the reconstructed image and the corresponding remote sensing image, and the model parameters of the remote sensing visual basic model are optimized based on the loss function. If the model converges, a pre-trained remote sensing vision basic model is obtained.

2. The method for pre-training a remote sensing visual fundamental model for unified modeling across imaging modalities according to claim 1, characterized in that, The masking process for each remote sensing image includes: Each remote sensing image is processed by image block-level sampling, and a spatial mask region or a temporal mask region is generated in each remote sensing image to obtain a masked remote sensing image.

3. The method for pre-training a remote sensing visual fundamental model for unified modeling across imaging modalities according to claim 1, characterized in that, The feature extraction through the spatiotemporal feature adaptive embedding layer of the remote sensing visual basic model to obtain the embedded features of the corresponding modality includes: The remote sensing images of the two modalities after masking are processed by a spatiotemporal feature adaptive embedding layer, and the spectral features and temporal features are mapped to a high-dimensional space. The interaction relationship between the spectral features and temporal features is mined based on a dynamic attention mechanism to generate a fused feature representation. High-dimensional feature representations of remote sensing images are generated based on fused feature representations, and convolutional layers are used to map these high-dimensional feature representations into image block-level embedded features.

4. The method for pre-training a remote sensing visual fundamental model for unified modeling across imaging modalities according to claim 2, characterized in that, The hybrid expert encoder includes multiple different expert subnetworks; The hybrid expert encoder, based on the remote sensing vision model, processes the embedding features of the two modalities respectively to obtain the decoupled features of the corresponding modalities, including: The hybrid expert encoder processes the embedded features of the input through a learnable gating network and activates a preset number of expert sub-networks based on the embedded features. The activation weights of each activated expert subnetwork are calculated using the gated network. Obtain the encoding result of each activated expert subnetwork based on the embedded features; The encoding results are weighted and summed based on the activation weights of each activated expert subnetwork to obtain the decoupling features of the corresponding modality.

5. The method for pre-training a remote sensing visual fundamental model for unified modeling across imaging modalities according to claim 4, characterized in that, The method involves processing the decoupled features of the two modalities using a cross-attention encoder based on a remote sensing vision model to obtain the cross features of the corresponding modalities, and then using a projection network to reconstruct the reconstructed image of the corresponding modality based on the cross features. This includes: The query vector corresponding to mode one is generated based on the decoupling features of mode one, and the key vector and value vector corresponding to mode one are generated based on the decoupling features of mode two. Cross attention is calculated based on the query vector, key vector and value vector corresponding to mode one to obtain the cross features of mode one. The query vector corresponding to mode two is generated based on the decoupling features of mode two, and the key vector and value vector corresponding to mode two are generated based on the decoupling features of mode one. Cross attention is calculated based on the query vector, key vector and value vector corresponding to mode two to obtain the cross features of mode two. The pixel values ​​of the mask region are reconstructed using a projection network based on the cross features of mode 1 and mode 2, respectively, to obtain the reconstructed image of the corresponding mode.

6. The method for pre-training a remote sensing visual fundamental model for unified modeling across imaging modalities according to claim 4, characterized in that, The loss function for constructing the basic remote sensing visual model based on the differences between the reconstructed image and the corresponding remote sensing image includes: The reconstruction loss term is constructed based on the difference between each remote sensing image and its corresponding reconstructed image in the remote sensing image sequence before masking. An auxiliary loss term is constructed based on the activation weight distribution of each expert sub-network in the hybrid expert encoder; The loss function of the remote sensing vision basic model is obtained by constructing the reconstruction loss term, auxiliary loss term, and hyperparameters of the auxiliary loss term. The process of optimizing the model parameters of the remote sensing vision basic model based on the loss function, and obtaining the pre-trained remote sensing vision basic model upon determining model convergence, includes: Backpropagation is performed based on the value of the loss function to optimize the model parameters of the remote sensing visual basic model; If the value of the loss function is less than a preset threshold, or reaches the iteration number threshold, then the model is determined to have converged, and a pre-trained remote sensing vision basic model is obtained. Otherwise, proceed to the masking process and subsequent steps for the next iteration.

7. A method for pre-training a remote sensing visual fundamental model for unified modeling across imaging modalities according to any one of claims 1-6, characterized in that, After obtaining the pre-trained remote sensing visual basic model, the target task is processed through the obtained pre-trained remote sensing visual basic model. The input features of the target task include remote sensing image sequences of at least one modality. The process of processing the target task includes: The remote sensing image sequence of at least one modality is input into the remote sensing visual basic model, and the steps of extracting the embedding features of the corresponding modality through the spatiotemporal feature adaptive embedding layer of the remote sensing visual basic model and subsequent steps are performed until the decoupled features of the corresponding modality are obtained. The output of the hybrid expert encoder of the remote sensing vision basic model is connected to the prediction head of the corresponding target task. The decoupled features are processed by the prediction head of the corresponding target task, and the prediction result of the target task is output.

8. A pre-training device for a remote sensing visual fundamental model that unifies modeling across imaging modalities, characterized in that, include: Dataset building unit, used to build a pre-training dataset of remote sensing image sequences including at least two modalities; The masking unit is used to randomly select two modalities of remote sensing image sequences from the pre-training dataset and perform masking processing on each remote sensing image separately. The model pre-training unit is used to input the masked remote sensing image sequence into the remote sensing vision basic model, and extract the embedding features of the corresponding modality through the spatiotemporal feature adaptive embedding layer of the remote sensing vision basic model. The model pre-training unit is also used to process the embedding features of the two modalities respectively through the hybrid expert encoder of the remote sensing vision basic model to obtain the decoupled features of the corresponding modalities. The model pre-training unit is also used to process the decoupled features of the two modalities respectively through the cross-attention encoder of the remote sensing vision basic model to obtain the cross features of the corresponding modalities, and to reconstruct the corresponding modalities based on the cross features using the projection network. The model pre-training unit is also used to construct a loss function for the remote sensing visual basic model based on the difference between the reconstructed image and the corresponding remote sensing image, and to optimize the model parameters of the remote sensing visual basic model based on the loss function. The model pre-training unit is also used to obtain a pre-trained remote sensing visual basic model when the model is determined to have converged.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images