Remote sensing time sequence frame completion method, device and equipment based on joint embedded prediction architecture and diffusion model

Abstract

The invention provides a remote sensing time sequence frame completion method based on a joint embedded prediction architecture and a diffusion model, and the method comprises the steps: collecting a first time sequence remote sensing data set, and carrying out the data preprocessing, the first time sequence remote sensing data set being a training data set; carrying out representation learning on the preprocessed time sequence remote sensing data set based on the joint embedded prediction architecture model so as to extract target features; inputting the target features into the constructed diffusion model to train the diffusion model, and obtaining a double-component model comprising a joint embedded prediction architecture and the diffusion model; and collecting a second time sequence remote sensing data set, and inputting the second time sequence remote sensing data set into the double-component model to obtain a target intermediate frame so as to complete the remote sensing time sequence frame of the second time sequence remote sensing data set. The invention further provides a remote sensing time sequence frame complementing device and equipment based on the joint embedded prediction architecture and the diffusion model.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and computer vision technology, and specifically to a remote sensing temporal frame completion method, apparatus, and device based on a joint embedding prediction architecture and a diffusion model. Background Technology

[0002] In today's rapidly developing digital and information-based world, remote sensing technology, as a long-distance, non-contact method of Earth observation, provides crucial data support for key areas such as environmental monitoring and resource exploration. However, due to the influence of sensor performance, adverse weather conditions, and unforeseen external interference, remote sensing time-series data often suffers from missing frames, which severely restricts the effectiveness of subsequent data applications. Therefore, in-depth research on time-series frame completion based on remote sensing time-series data is of great significance.

[0003] Currently, remote sensing video frame interpolation technology has become a research hotspot, with various algorithms and methods emerging. Traditional frame interpolation algorithms are mostly based on signal processing and image processing techniques, such as inter-frame difference methods, which infer the pixel values ​​of intermediate frames by calculating the differences between adjacent frames. This method is relatively simple in principle and has low computational cost, but it is only suitable for situations where scene changes are relatively gentle and the motion patterns are relatively simple. Once encountering complex dynamic scenes, the generated intermediate frames often suffer from problems such as blurring and ghosting, making it difficult to meet the needs of practical applications. Optical flow methods predict the position and grayscale value of intermediate frame pixels by estimating the motion vectors of pixels in video frames. This method can handle the motion information of objects to a certain extent, but the accuracy of optical flow estimation drops significantly when facing occlusion, fast movement, and complex backgrounds, resulting in poor frame interpolation effects.

[0004] Despite continuous development and progress in existing remote sensing video frame interpolation techniques, numerous challenges remain. In complex scenes, remote sensing videos contain a large number of dynamic targets and intricate terrain variations, making it extremely difficult to accurately capture the motion information and appearance features of objects. Factors such as occlusion between different ground features, rapid changes in lighting conditions, and sensor noise can also severely interfere with the performance of frame interpolation algorithms, leading to errors or distortions in the generated intermediate frames. Therefore, how to improve the computational efficiency of algorithms while ensuring frame interpolation quality, and achieve fast and accurate remote sensing video intermediate frames for completion, is a key problem that urgently needs to be solved in current research. Summary of the Invention

[0005] The first aspect of the present invention provides a remote sensing temporal frame completion method based on a joint embedding prediction architecture and a diffusion model, comprising: acquiring a first temporal remote sensing dataset and performing data preprocessing, wherein the first temporal remote sensing dataset is a training dataset; performing representation learning on the preprocessed first temporal remote sensing dataset based on the joint embedding prediction architecture model to extract target features; inputting the target features into a constructed diffusion model to train the diffusion model and obtain a dual-component model containing the joint embedding prediction architecture and the diffusion model; acquiring a second temporal remote sensing dataset and inputting the second temporal remote sensing dataset into the dual-component model to obtain target intermediate frames, thereby achieving remote sensing temporal frame completion for the second temporal remote sensing dataset.

[0006] The above scheme also includes: using a second time-series remote sensing dataset to fine-tune the two-component model through low-rank adaptive tuning.

[0007] In the above scheme, a first time-series remote sensing dataset is collected and data preprocessing is performed, including: spatiotemporal standardization and scene segmentation processing of the collected first time-series remote sensing dataset.

[0008] In the above scheme, based on the joint embedding prediction architecture model, representation learning is performed on the preprocessed time-series remote sensing dataset, including: designing a spatiotemporally decoupled self-attention learning mechanism based on the joint embedding prediction architecture, and realizing semantic representation learning of the first time-series remote sensing dataset through a multi-stage pre-training strategy.

[0009] In the above scheme, the spatiotemporal decoupling self-attention learning mechanism includes: separating spatial and temporal feature calculations to first capture spatial information and then analyze temporal dynamics.

[0010] In the above scheme, multi-stage pre-training includes a first-stage pre-training and a second-stage pre-training. The multi-stage pre-training process includes: training using progressive tubular masks based on multiple frames of images in the first time-series remote sensing dataset to complete the first-stage pre-training; and fine-tuning through multimodal masks and freezing the spatial self-attention layer and the multilayer perceptron layer to complete the second-stage pre-training.

[0011] In the above scheme, the target features are input into the constructed diffusion model to train the diffusion model. The training process includes: combining the self-attention neural network architecture with the diffusion model and extracting long-term remote sensing data through a multi-head attention mechanism to generate video frames; extracting text features and video features from the video frames respectively; and completing the causal rationality of the video frames based on the text features and video features through a preset 3D causal variational autoencoder.

[0012] In the above scheme, the target features are input into the constructed diffusion model to train the diffusion model. The training process also includes: projecting the target features onto the target feature representation space through a multilayer perceptron to complete feature alignment; after the target features are aligned, they are input into the diffusion model to complete the training of the diffusion model.

[0013] A second aspect of the present invention provides a remote sensing time-series frame completion device based on a joint embedding prediction architecture and a diffusion model, comprising: a training data acquisition module for acquiring a first time-series remote sensing dataset and performing data preprocessing, wherein the first time-series remote sensing dataset is a training dataset; a feature extraction module for performing representation learning on the preprocessed first time-series remote sensing dataset based on the joint embedding prediction architecture model to extract target features; a diffusion model training module for inputting the target features into a constructed diffusion model to train the diffusion model and obtain a dual-component model containing the joint embedding prediction architecture and the diffusion model; and a target intermediate frame acquisition module for acquiring a second time-series remote sensing dataset and inputting the second time-series remote sensing dataset into the dual-component model to obtain target intermediate frames, thereby achieving remote sensing time-series frame completion for the second time-series remote sensing dataset.

[0014] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] The technical solutions of the embodiments of the present invention have at least the following beneficial effects:

[0016] (1) This method is designed for representation learning of remote sensing spatiotemporal data, and a self-supervised learning framework based on joint embedding prediction architecture is designed. This framework replaces the traditional pixel-level image reconstruction task with non-pixel-level representation space prediction, which effectively avoids the interference of noise and redundant details in remote sensing images on representation learning.

[0017] (2) This method injects the features extracted by the joint embedding prediction architecture into the diffusion model, which greatly improves the model's ability to understand and generate remote sensing data. In order to adapt to the image spatial resolution and complex land cover distribution of remote sensing images, the diffusion model is fine-tuned. Based on a large amount of remote sensing image data, the key parameters of the pre-trained model are adjusted by the transfer learning strategy to improve the land cover feature capture ability, so as to realize the accurate output of intermediate frames that conform to the changes of land cover in the remote sensing image data in the urban spatiotemporal dynamic interpolation task. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a remote sensing temporal frame completion method based on a joint embedding prediction architecture and a diffusion model according to an embodiment of the present invention is shown.

[0019] Figure 2 This illustration schematically shows a flowchart of multi-stage pre-training during semantic representation learning according to an embodiment of the present invention;

[0020] Figure 3 The schematic diagram illustrates a structural block diagram of a remote sensing temporal frame completion device based on a joint embedding prediction architecture and a diffusion model according to an embodiment of the present invention; and

[0021] Figure 4 A block diagram illustrating an electronic device suitable for implementing a remote sensing temporal frame completion method based on a joint embedding prediction architecture and a diffusion model, according to an embodiment of the present invention, is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0023] Figure 1 The flowchart illustrates a remote sensing temporal frame completion method based on a joint embedding prediction architecture and a diffusion model according to an embodiment of the present invention.

[0024] Please refer to the following for details. Figure 1 In an embodiment of the present invention, the specific process of the remote sensing temporal frame completion method based on the joint embedding prediction architecture and diffusion model includes operations S110 to S140.

[0025] During operation S110, the first time-series remote sensing dataset is collected and preprocessed. The first time-series remote sensing dataset is used as the training dataset.

[0026] In an embodiment of the present invention, a first time-series remote sensing dataset is collected and data preprocessing is performed, including: spatiotemporal standardization and scene segmentation processing of the collected first time-series remote sensing dataset.

[0027] It is understood that this embodiment uses multi-source satellite data to collect the first time-series remote sensing dataset. After the dataset is collected, spatiotemporal standardization preprocessing and scene segmentation are performed to construct a high-quality training dataset.

[0028] For example, after selecting several time-series remote sensing datasets under strict screening, the image formats are unified and stitched together into a video format, for example, cropped to 448×448 pixels. Then, a command-line tool is used to detect scene changes based on a perceptual hashing algorithm (grayscale conversion, DCT transformation, hash distance calculation). Finally, the video is segmented according to a 3-10 second threshold, low-quality data is removed, and the data is decoupled into background and target layers, providing a structured, high-quality dataset for subsequent model training input.

[0029] In operation S120, based on the joint embedding prediction architecture model, representation learning is performed on the preprocessed time-series remote sensing dataset to extract target features.

[0030] In embodiments of the present invention, representation learning is performed on a preprocessed time-series remote sensing dataset based on a joint embedding prediction architecture model. This includes: designing a spatiotemporally decoupled self-attention learning mechanism based on the joint embedding prediction architecture, and implementing semantic representation learning of the first time-series remote sensing dataset through a multi-stage pre-training strategy. The self-attention learning mechanism based on spatiotemporally decoupled design of the joint embedding prediction architecture will be described in detail below.

[0031] Specifically, the spatiotemporal decoupling self-attention learning mechanism includes: separating spatial and temporal feature calculations to first capture spatial information and then analyze temporal dynamics.

[0032] For example, the joint embedding prediction architecture is designed using a method that combines non-pixel-level semantic representation with spatiotemporal decoupling. A visual self-attention neural network architecture maps the image to a low-dimensional representation space, predicting the high-level semantic representation of the target block from the context block. The loss is optimized using Euclidean distance, avoiding pixel-level noise interference. Through embodiments of this invention, a self-supervised learning framework based on the joint embedding prediction architecture is designed for representation learning of remote sensing spatiotemporal data. This framework replaces the traditional pixel-level image reconstruction task with non-pixel-level representation space prediction, effectively avoiding the interference of noise and redundant details in remote sensing images on representation learning.

[0033] Furthermore, a spatiotemporal decoupling self-attention mechanism is used to separate the calculation of spatial and temporal features. Spatial information such as ground texture is captured first, and then temporal dynamics are analyzed, reducing computational complexity and improving the efficiency of processing spatiotemporal data. This allows the model to better learn the spatiotemporal dependencies in remote sensing data.

[0034] Figure 2 The flowchart illustrating the multi-stage pre-training process in semantic representation learning according to an embodiment of the present invention is shown.

[0035] Please refer to the following for details. Figure 2 In an embodiment of the present invention, multi-stage pre-training includes a first-stage pre-training and a second-stage pre-training, and the multi-stage pre-training process includes operations S210 to S220.

[0036] In operation S210, based on multiple frames of images in the first time-series remote sensing dataset, progressive tubular masks are used for training to complete the first stage of pre-training.

[0037] For example, the encoder (excluding temporal self-attention) of the jointly embedded predictive architecture model can, for instance, employ a large visual self-attention neural network architecture with 24 layers and a dimension of 1024, while the predictor consists of 6 layers with a dimension of 384.

[0038] Furthermore, in the first stage of pre-training, for example, spatial awareness can be trained using over 8 million static images from the first time-series remote sensing dataset, using progressive tubular masks (e.g., mask ratio of 0.7~0.95), and the model can be trained for 66 epochs using a pre-defined optimizer with a learning rate of 5e. -4 Linear warm-up is performed for the first 3 epochs. Then the maximum and minimum mask rates are set to 0.95 and 0.7 respectively, the batch size is 2240, and gradient accumulation is used to adapt to the large batch size.

[0039] In operating the S220, fine-tuning is performed using multimodal masks, and the spatial self-attention layer and multilayer perceptron layer are frozen to complete the second stage of pre-training.

[0040] Furthermore, in the second pre-training stage, the same optimizer and learning rate configuration are used to fine-tune the model. The entire fine-tuning process lasts for 16 epochs. Temporal inference capabilities are fine-tuned using multimodal masks (tubular, causal, intermediate frames). The maximum and minimum mask rates used in this stage are consistent with those in the first stage. In this stage, the spatial components are frozen, and training is only performed on the temporal self-attention and other model components. It should be noted that the spatial self-attention and multilayer perceptron layers remain frozen.

[0041] Based on this, a multi-stage pre-training strategy is used to learn the semantic representation of the first time-series remote sensing dataset in order to extract target features, which can be spatiotemporal features.

[0042] Through the embodiments of the present invention, the joint embedding prediction architecture relies on a two-stage pre-training strategy to decouple spatial perception and causal reasoning. The self-attention mechanism can capture spatial texture and temporal dynamics respectively, effectively improving the hierarchical ability of feature representation. Furthermore, the model introduces a progressive multimodal masking strategy to gradually increase the prediction difficulty, prompting the model to learn more robust spatiotemporal dependencies.

[0043] In operation S130, the target features are input into the constructed diffusion model to train the diffusion model and obtain a two-component model that includes the joint embedding prediction architecture and the diffusion model.

[0044] Understandably, in this embodiment, a diffusion model is used as the foundation, integrating a visual self-attention neural network and a 3D causal variational autoencoder. The extracted target features are then used to train the diffusion model, generating a two-component model that includes a joint embedding prediction architecture and the diffusion model. This achieves accurate modeling of the spatiotemporal dynamics of remote sensing data, addressing the shortcomings of traditional generative models in terms of geographic feature consistency and long-term temporal dependencies. The spatiotemporal features of the joint embedding prediction architecture are used as conditional inputs to the diffusion model, guiding it to focus on geographic features.

[0045] In an embodiment of the present invention, in order to train the diffusion model to generate a two-component model comprising a joint embedding prediction architecture and a diffusion model, the training process includes: combining a visual self-attention neural network architecture with the diffusion model, and extracting long-term time-series remote sensing data through a multi-head attention mechanism to generate video frames; extracting text features and video features from the video frames respectively; and, based on the text features and video features, completing the causal rationality assessment of the video frames through a preset 3D causal variational autoencoder.

[0046] Specifically, the visual self-attention neural network architecture is combined with a diffusion model to capture long temporal dependencies through a multi-head attention mechanism. In text-to-video generation, text features are first encoded, and then the video sequence is transformed into a form suitable for processing by the visual self-attention neural network. Semantic video frames are generated through progressive denoising.

[0047] Furthermore, by using pre-defined text and visual experts, text and visual features are processed separately. With the diffusion time step as input, scaling, offsetting, and gating mechanisms are used to promote the alignment of text and video modalities in the feature space and reduce modal differences.

[0048] Furthermore, the 3D causal variational autoencoder, composed of an encoder, decoder, and regularizer, achieves 2× downsampling and upsampling through symmetrically arranged residual blocks, with some blocks performing 3D downsampling. This compresses the spatiotemporal dimension of the video while retaining key information, ensuring the logical coherence of the generated frames on the time axis. This solves the problem of causal rationality in the evolution of dynamic targets in remote sensing videos, thereby achieving causal rationality for video frames.

[0049] Furthermore, the training process also includes: projecting the target features onto the target feature representation space through a multilayer perceptron to complete feature alignment; and inputting the aligned target features into the diffusion model to complete the training of the diffusion model.

[0050] Specifically, the spatiotemporal features extracted from the pre-trained joint embedding prediction architecture are used as conditional inputs and projected onto the target feature representation space through a multilayer perceptron. Here, the target feature representation space is an aligned and optimized unified text feature representation space. Therefore, after aligning the feature shapes, the spatiotemporal features are input into the constructed diffusion model, guiding the model to focus on geographic features.

[0051] In embodiments of this invention, spatiotemporal features obtained using a joint embedding prediction architecture are input as conditional information into a diffusion model. After being mapped to the embedding space by a multilayer perceptron, the features deeply participate in the denoising process of the diffusion model, thereby ensuring that the generated remote sensing video remains coherent in time series, has a visual effect close to the real scene, and strictly follows the physical laws of remote sensing images, effectively improving the quality of video generation.

[0052] In operation S140, a second time-series remote sensing dataset is acquired and input into the dual-component model to obtain the target intermediate frame, thereby completing the remote sensing time-series frame of the second time-series remote sensing dataset.

[0053] In an embodiment of the present invention, a second time-series remote sensing dataset is used to fine-tune the dual-component model through low-rank adaptive tuning.

[0054] Understandably, a second temporal remote sensing dataset is collected, which is the dataset to be used for temporal frame interpolation. Based on these datasets, a two-component model containing a joint embedding prediction architecture and a diffusion model is fine-tuned using low-rank adaptive techniques, enabling efficient parameter fine-tuning and accurate spatiotemporal feature modeling in remote sensing video frame interpolation tasks.

[0055] For example, satellite imagery of a specific region from 1984 to 2020 was collected. 300 cities were cropped at a 512×512 pixel standard as the training set, and 75 cities as the validation set, totaling 14,250 frames, encompassing various terrains including mountains and plains. Furthermore, using this training set containing 300 cities, training was performed using eight A40 GPUs with a low-rank adaptive technique, a rank of 256, and a learning rate of 1e. -4 The dual-component model was fine-tuned over 3000 steps. Furthermore, for the fine-tuned dual-component model, a sampler from a classifier-less denoising diffusion probability model was used during inference. This improved generation efficiency and reduced computational resource consumption without class constraints, while maintaining sample diversity and spatial feature authenticity, thus adapting to unconditional prediction scenarios of urban expansion.

[0056] Furthermore, the validation set is input into the fine-tuned dual-component model to output the target intermediate frame, which is to realize the accurate output of intermediate frames of remote sensing image data that conform to the changes of ground features in the urban spatiotemporal dynamic interpolation task.

[0057] Through embodiments of this invention, features extracted by a joint embedding prediction architecture are injected into a diffusion model, significantly enhancing the model's ability to understand and generate remote sensing data. To adapt to the spatial resolution and complex distribution of ground features in remote sensing images, the diffusion model is fine-tuned. Based on a large amount of remote sensing image data, a transfer learning strategy is employed to adjust key parameters of the pre-trained model, improving its ability to capture ground feature characteristics. This enables accurate output of intermediate frames that reflect changes in ground features from remote sensing image data in urban spatiotemporal dynamic interpolation tasks. This method addresses the shortcomings of existing remote sensing temporal frame completion techniques, which suffer from poor completion results due to insufficient temporal datasets, extreme motion, or extremely complex scenes, thereby improving the overall quality of temporal frame completion.

[0058] Based on the aforementioned remote sensing temporal frame completion method based on a joint embedding prediction architecture and a diffusion model, this invention also provides a remote sensing temporal frame completion device based on a joint embedding prediction architecture and a diffusion model. The following will combine... Figure 3 The device is described in detail.

[0059] Figure 3 The diagram illustrates the structure of a remote sensing temporal frame completion device based on a joint embedding prediction architecture and a diffusion model according to an embodiment of the present invention.

[0060] like Figure 3 As shown, the remote sensing temporal frame completion device 300 based on the joint embedding prediction architecture and diffusion model in this embodiment includes a training data acquisition module 310, a feature extraction module 320, a diffusion model training module 330, and a target intermediate frame acquisition module 340.

[0061] The training data acquisition module 310 is used to acquire a first time-series remote sensing dataset and perform data preprocessing. The first time-series remote sensing dataset is the training dataset. In one embodiment, the training data acquisition module 310 can be used to perform the operation S110 described above, which will not be repeated here.

[0062] The feature extraction module 320 is used to perform representation learning on a preprocessed time-series remote sensing dataset based on a joint embedding prediction architecture model to extract target features. In one embodiment, the feature extraction module 320 can be used to perform the operation S120 described above, which will not be repeated here.

[0063] The diffusion model training module 330 is used to input the target features into the constructed diffusion model to train the diffusion model and obtain a two-component model that includes the joint embedding prediction architecture and the diffusion model. In one embodiment, the diffusion model training module 330 can be used to perform the operation S130 described above, which will not be repeated here.

[0064] The target intermediate frame acquisition module 340 is used to acquire the second time-series remote sensing dataset and input the second time-series remote sensing dataset into the dual-component model to obtain the target intermediate frame, thereby completing the remote sensing time-series frames of the second time-series remote sensing dataset. In one embodiment, the target intermediate frame acquisition module 340 can be used to perform the operation S140 described above, which will not be repeated here.

[0065] According to embodiments of the present invention, any multiple modules among the training data acquisition module 310, feature extraction module 320, diffusion model training module 330, and target intermediate frame acquisition module 340 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of the present invention, at least one of the training data acquisition module 310, feature extraction module 320, diffusion model training module 330, and target intermediate frame acquisition module 340 can be at least partially implemented as hardware circuitry, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the training data acquisition module 310, feature extraction module 320, diffusion model training module 330, and target intermediate frame acquisition module 340 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0066] Figure 4 A block diagram illustrating an electronic device suitable for implementing a remote sensing temporal frame completion method based on a joint embedding prediction architecture and a diffusion model, according to an embodiment of the present invention, is shown.

[0067] like Figure 4 As shown, an electronic device 400 according to an embodiment of the present invention includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0068] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 402 and / or RAM 403. It should be noted that programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0069] According to an embodiment of the present invention, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0071] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0072] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing time-series frame completion method based on a joint embedding prediction architecture and a diffusion model, characterized in that, The method comprises: collecting a first time-series remote sensing data set and performing data preprocessing, wherein the first time-series remote sensing data set is a training data set; based on a joint embedding prediction architecture model, performing feature learning on the preprocessed time-series remote sensing data set to extract target features; inputting the target features into a constructed diffusion model to train the diffusion model and obtain a dual-component model comprising the joint embedding prediction architecture and the diffusion model; collecting a second time-series remote sensing data set and inputting the second time-series remote sensing data set into the dual-component model to obtain a target intermediate frame, thereby realizing remote sensing time-series frame completion of the second time-series remote sensing data set.

2. The remote sensing time-series frame completion method based on a joint embedding prediction architecture and a diffusion model according to claim 1, characterized in that, The method further comprises: using the second time-series remote sensing data set to fine-tune the dual-component model through low-rank self-adaptation.

3. The remote sensing time-series frame completion method based on a joint embedding prediction architecture and a diffusion model according to claim 1, characterized in that, The collection of the first time-series remote sensing data set and the data preprocessing comprise: performing spatio-temporal standardization and scene segmentation processing on the collected first time-series remote sensing data set.

4. The remote sensing time-series frame completion method based on a joint embedding prediction architecture and a diffusion model according to claim 1, characterized in that, The feature learning based on the joint embedding prediction architecture model comprises: designing a spatio-temporal decoupling self-attention learning mechanism based on the joint embedding prediction architecture, and realizing semantic feature learning of the first time-series remote sensing data set through a multi-stage pre-training strategy.

5. The method of claim 4, wherein, The spatio-temporal decoupling self-attention learning mechanism comprises: separating spatial and temporal feature calculation to capture spatial information first and then analyze temporal dynamics.

6. The remote sensing time-series frame completion method based on a joint embedding prediction architecture and diffusion model according to claim 4, characterized in that, The multi-stage pre-training comprises a first-stage pre-training and a second-stage pre-training, and the process of the multi-stage pre-training comprises: using a progressive tubular mask to train based on multiple frames of images in the first time-series remote sensing data set to complete the first-stage pre-training; completing the second-stage pre-training by multi-modal mask fine-tuning and freezing the spatial self-attention layer and the multi-layer perception layer.

7. The method of claim 1, wherein, The training process of inputting the target features into the constructed diffusion model comprises: combining a self-attention neural network architecture with the diffusion model and extracting long time-series remote sensing data through a multi-head attention mechanism to generate video frames; extracting text features and video features from the video frames, respectively; based on the text features and the video features, completing the causal rationality of the video frames through a preset 3D causal variational autoencoder.

8. The remote sensing time-series frame completion method based on a joint embedding prediction architecture and a diffusion model according to claim 7, characterized in that, The training process of inputting the target features into the constructed diffusion model further comprises: projecting the target features to a target feature representation space through a multi-layer perception to complete feature alignment; after the target features are aligned, inputting them into the diffusion model to complete the training of the diffusion model.

9. A remote sensing time-series frame completion device based on a joint embedding prediction architecture and a diffusion model, characterized in that, The device comprises: a training data collection module for collecting a first time-series remote sensing data set and performing data preprocessing, wherein the first time-series remote sensing data set is a training data set; a feature extraction module for performing feature learning on a preprocessed time-series remote sensing data set based on a joint embedding prediction architecture model to extract target features; and a diffusion model training module for inputting the target features into a constructed diffusion model to train the diffusion model and obtain a dual-component model comprising the joint embedding prediction architecture and the diffusion model. The diffusion model training module is configured to input the target feature into a constructed diffusion model to train the diffusion model and obtain a double-component model including a joint embedding prediction architecture and the diffusion model. The target intermediate frame acquisition module is configured to collect a second time-series remote sensing data set, input the second time-series remote sensing data set into the double-component model, and obtain a target intermediate frame to implement remote sensing time-series frame completion on the second time-series remote sensing data set. 10.An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-8.

Images