A semantic adaptive multi-source satellite image spatio-temporal fusion method

Abstract

The application discloses a kind of semantic adaptive multi-source satellite image space-time fusion methods: first, obtain satellite remote sensing image at different time, determine diffusion variable at initial time step T;Then, build semantic adaptive model S including feature extraction branch, prior image fusion branch and target image reconstruction branch;Deep features of diffusion variable are obtained by feature extraction branch;Shallow fusion features and deep fusion features of remote sensing image at different time are obtained by prior image fusion branch;Intermediate variable corresponding to all features is obtained by target image reconstruction branch, and the diffusion model is reversed to reconstruct and obtain the diffusion variable of next time step;Repeat input semantic adaptive model S, reconstruct after each branch and obtain new intermediate variable, and reverse reconstruction until reaching preset iteration number, finally obtain the intermediate variable corresponding to time step 0, it is the first satellite image of target time.This application can improve the accuracy of multi-source satellite image space-time fusion.

Description

Technical Field

[0001] This invention belongs to the field of computer science, and specifically relates to a semantically adaptive spatiotemporal fusion method for multi-source satellite images. Background Technology

[0002] Satellites enable large-scale, periodic surface observations, supporting various applications such as disaster emergency response, agricultural monitoring, and smart city planning. However, acquiring remote sensing imagery with high spatiotemporal resolution remains a significant challenge due to physical limitations imposed by sensor field of view and satellite orbit design. For example, the MODIS satellite, with a revisit period of 1 to 2 days, provides high-frequency agricultural monitoring data, but its low spatial resolution of 250 to 1000 meters cannot meet the detailed analysis required for precision agriculture. Conversely, while the Landsat satellite boasts a high spatial resolution of 30 meters, its low revisit frequency of approximately 16 days makes it difficult to provide timely observational information. This spatiotemporal trade-off restricts applications such as precision agriculture that rely on high-frequency and high-precision observations.

[0003] To address this challenge, the Spatiotemporal Fusion (STF) technology was developed. Its core idea is to integrate multi-source satellite data and generate continuous observation data with high spatiotemporal resolution through algorithms.

[0004] Existing spatiotemporal fusion (STF) methods are mainly divided into traditional methods and deep learning-based methods. Traditional methods encompass various technical approaches such as weight functions, unmixing, Bayesian methods, fusion strategies, and dictionary learning. These methods perform well when land cover changes are small, but due to their reliance on linear models and manually designed hyperparameters, they often struggle to handle complex dynamic changes spanning long periods. To overcome these limitations, deep learning methods have been introduced. They can automatically extract multi-scale features and model the complex nonlinear relationship between coarse-resolution and fine-resolution images. Here, coarse resolution refers to high temporal resolution but low spatial resolution, while fine resolution refers to low temporal resolution but high spatial resolution.

[0005] Despite the significant advantages of deep learning methods, current research still faces three core challenges:

[0006] 1) Artifact problem: Most existing methods rely on pure data-driven approaches and lack explicit constraints. When fusing images with huge scale differences, especially in complex or dynamic regions, they are prone to producing obvious artifacts.

[0007] 2) Spectral distortion: Due to differences in acquisition time, image sequences often exhibit temporal inconsistencies; most existing methods fail to explicitly model this temporal dynamic, resulting in spectral distortion in the reconstructed images.

[0008] 3) Insensitive to semantic changes: Land cover evolves over time, driven by natural dynamics or human activities. Existing deep learning methods often struggle to fully extract high-frequency information from reference images, thus limiting their ability to capture semantic changes resulting from land cover transformation. Summary of the Invention

[0009] To address the issues of artifacts, spectral distortion, and insensitivity to semantic changes in traditional fusion methods, this invention proposes a semantically adaptive spatiotemporal fusion method for multi-source satellite images, which can improve the accuracy of spatiotemporal fusion of multi-source satellite images.

[0010] The specific steps include:

[0011] Step 1: Acquire remote sensing images of the same size from two satellites at different times, and determine the diffusion variables at the initial time step T. ;

[0012] Time steps include The initial time step is T;

[0013] Specifically, this involves acquiring the first satellite image at time t0. Second satellite image Second satellite image at time t1 ; and the first satellite image at time t2 Second satellite image ;

[0014] The determination method is as follows:

[0015]

[0016] Gaussian noise that follows a standard normal distribution;

[0017] Step 2: Construct a semantic adaptive model S, which includes a feature extraction branch, a prior image fusion branch, and a target image reconstruction branch.

[0018] Step 3: Diffusion variables The input is fed into the feature extraction branch, where multi-scale convolution and downsampling are used to obtain deep features of the diffusion variables. ;

[0019] Step 4: Input the remote sensing images from the two satellites at different times into the prior image fusion branch to obtain a set of shallow fusion features. and deep fusion features ;

[0020] The prior image fusion branch includes a shallow fusion branch and a deep fusion branch.

[0021] The shallow feature fusion branch consists of one embedding layer, five feature extraction units, and four fusion units; specifically:

[0022] First, the embedding layer receives satellite images from two satellites at different times. , , , , Each image is projected into a 64-channel feature space using 3x3 convolution.

[0023] Then, the first four feature extraction units obtain the feature map of the prior image through multi-scale convolution and downsampling. , , , , };

[0024] in Indicates the number of feature layers; values ​​are... =1,2,3,4.

[0025] Next, the fusion unit consists of three convolutional modules: [the modules then combine] the features of the second satellite image. , and Stacking and feeding the data into the first convolutional module yields the second satellite image fusion features. The first satellite image features and Stack the convolutional blocks and input them into the second convolutional block to obtain the first satellite image fusion features. ; Fuse features from second satellite images Features fused with the first satellite image The layers are concatenated, and a shallow fusion feature is generated through a third convolutional block. .

[0026] At the same time, the feature map of the prior image { , , , , The input is fed into the fifth feature extraction unit, where convolution and downsampling are used to obtain the respective deep features. , , , , }

[0027] The deep fusion branch consists of two temporal feature alignment modules and a semantic adaptive fusion module.

[0028] Specifically:

[0029] The first-time feature alignment module receives deep features. , , After feature alignment, the first-time aligned features are obtained. ;

[0030] The second time feature alignment module receives deep features. , After feature alignment, the second time-aligned feature is obtained. ;

[0031] The calculation of the first-time feature alignment module is as follows:

[0032]

[0033]

[0034]

[0035]

[0036] In the formula, This is the first coarse-time dynamic coefficient. This is the first-time feature extraction unit. For feature splicing, The difference coefficient for the first mode. This is the first modal difference extraction unit. This is the first fine-time dynamic coefficient. This is the first cross-attention module. These are the position coordinates of pixels in the feature map;

[0037] The second time feature alignment module is calculated as follows:

[0038]

[0039]

[0040]

[0041]

[0042] This is the second coarse-time dynamic coefficient. This is the second time feature extraction unit. The second modal difference coefficient. This is the second modality difference extraction unit. This is the second fine time dynamic coefficient. This is the second cross-attention module.

[0043] Align two time features , Deep features , and The input is fed into the semantic adaptive fusion module to obtain deep fusion features. :

[0044]

[0045] In the formula, express and Euclidean distance, express and Euclidean distance.

[0046] The change mask is obtained by calculating the Euclidean distance between feature maps; the calculation is as follows:

[0047]

[0048] In the formula, express and Euclidean distance, express and Euclidean distance, express and Euclidean distance;

[0049] Step 5: Deeply fuse features Diffusion variables deep features Deep features and shallow fusion features Input to the target image reconstruction branch to obtain intermediate variables .

[0050] The target image reconstruction branch includes a deep residual removal module and a multi-scale noise removal module;

[0051] The deep residual removal module consists of two temporal convolutional units: each temporal convolutional unit contains two residual convolutional blocks and a multilayer perceptron.

[0052] First, deep features With deep fusion features The difference is calculated and compared with the preset residual strength at the initial time step T. Input the first temporal convolutional unit to obtain the residual components. .

[0053] Then, deep features With residual components The difference is calculated, and the noise intensity is preset at the initial time step T. Input a second temporal convolutional unit to remove deep features. The residual components in.

[0054] The specific formula is as follows:

[0055] First, at the initial time step T, set And residual image For the second satellite image With the first satellite image The difference; that is, the diffusion variable. ;

[0056] Therefore, for deep features The formula for deep residual removal is as follows:

[0057]

[0058]

[0059] In the formula, For learnable feature extractors, For Jacobian matrices, That is, the residual components ; This is a higher-order term.

[0060] Finally, after removing the residual components and shallow fusion features The input is fed into the multi-scale noise removal module, where it undergoes multi-scale convolution and upsampling to obtain intermediate variables. .

[0061] Step Six: Transfer intermediate variables By inputting the inverse reconstruction process equations of the diffusion model, the diffusion variables at the next time step T-1 are obtained. ;

[0062] The equations for the diffusion model's reverse reconstruction process are as follows:

[0063] Step 7: Return to Step 3 and set the diffusion variable for the next time step. The feature extraction branch of the semantic adaptive model S is input, and deep and shallow features are fused together with satellite images at each time step. After reconstructing the target image, new intermediate variables are obtained, and the reverse reconstruction process is repeated until the preset number of iterations is reached, finally obtaining the intermediate variable corresponding to time step 0. That is, the first satellite image at time t1. .

[0064] The advantages of this invention are:

[0065] 1. This invention provides a semantically adaptive spatiotemporal fusion method for multi-source satellite images. It solves the artifact problem in traditional fusion methods by using a step-by-step iterative reverse reconstruction approach. It also uses low-resolution observations and intermediate fusion features to explicitly constrain the solution space and designs a Taylor heuristic decoding process to improve the stability of the reconstruction process.

[0066] 2. The present invention provides a semantically adaptive spatiotemporal fusion method for multi-source satellite images. It uses a time feature alignment module to solve the spectral distortion problem inherent in traditional fusion amplification. By projecting a reference image with temporal dynamics onto a unified latent space, the spectral consistency of the reconstructed image is effectively ensured.

[0067] 3. The present invention provides a semantically adaptive spatiotemporal fusion method for multi-source satellite images. It designs a semantically adaptive fusion (SAF) module, which enhances the spatial detail reconstruction capability of dynamic regions by adaptively migrating high-frequency information from semantic matching regions. Moreover, the semantically adaptive fusion module improves the model's ability to perceive semantic changes. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating a semantically adaptive spatiotemporal fusion method for multi-source satellite images according to the present invention.

[0069] Figure 2 This is a schematic diagram of the semantic adaptive model for dynamic spatiotemporal fusion of the present invention;

[0070] Figure 3 This is a schematic diagram of the fusion results on the CIA dataset in an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of the fusion results on the LGC dataset in an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of the fusion results on the AHB dataset in an embodiment of the present invention;

[0073] Figure 6 Box plots of RMSE, SAM, and SSIM calculated in the variation regions of three test sets according to an embodiment of the present invention;

[0074] Figure 7 This is a schematic diagram of the classification results on the CIA dataset in an embodiment of the present invention;

[0075] Figure 8This is a schematic diagram of the classification results on the LGC dataset in an embodiment of the present invention;

[0076] Figure 9 This is a schematic diagram of the classification results on the AHB dataset in an embodiment of the present invention. Detailed Implementation

[0077] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0078] This invention proposes a semantically adaptive spatiotemporal fusion method for multi-source satellite images. It utilizes Taylor-inspired decoupling to extract low-resolution and high-frequency features to constrain the solution space. By introducing temporal feature alignment and semantically adaptive fusion modules, it projects multimodal images with temporal dynamics into a unified latent space, adaptively enhancing spatial details while maintaining the spectral consistency of the reconstructed images. Experiments on benchmark datasets demonstrate that this invention outperforms existing methods in both quantitative and qualitative evaluation, particularly excelling in complex and dynamic scenes.

[0079] This invention bridges the scale gap between multimodal regions through low-resolution observation and a constraint-guided residual diffusion mechanism. To eliminate artifacts, intermediate fusion features are introduced to explicitly model residuals and noise, and Taylor series-inspired feature decomposition is used to suppress noise and residuals. To reduce spectral distortion caused by differences in acquisition time and phenology, Temporal Feature Alignment (TFA) is proposed, projecting reference images into a unified latent space through linear regression. Furthermore, the Semantic Adaptive Fusion (SAF) module can identify the semantic similarity between changed and unchanged regions and adaptively transfer high-frequency details from semantically matched blocks, thereby reconstructing a fine spatial structure within dynamic regions.

[0080] like Figure 1 and Figure 2 As shown, the specific steps include:

[0081] Step 1: Acquire remote sensing images of the same size from two satellites at different times, and determine the diffusion variables at the initial time step T. ;

[0082] The time step is the number of iterations, specifically including The initial time step is T, which is set to 100 in this embodiment.

[0083] Specifically, this involves acquiring the first satellite image at reference time t0. Second satellite image ; and the first satellite image at reference time t2 Second satellite image The second satellite image at target time t1 ;

[0084] The first satellite image has high spatial resolution and low temporal resolution; the second satellite image has low spatial resolution and high temporal resolution. Temporal resolution refers to the time interval between remote sensing sampling of the same location, i.e., the sampling frequency. Spatial resolution refers to the size of the smallest unit that can be distinguished in detail on the image, i.e., the ground area corresponding to a single pixel. This setting of resolution difference can both preserve the detailed texture information of the high spatial resolution image and effectively alleviate the shortcomings of a single satellite image in terms of spatial detail reconstruction and spectral consistency maintenance by utilizing the spectral characteristics of the high temporal resolution image.

[0085] Given reference time and Two sets of coarse-fine image pairs and and target time A coarse-resolution image The goal of spatiotemporal fusion (STF) is to reconstruct a corresponding fine-resolution image under a conditional distribution. The condition set Due to degraded images It is known that, in addition to the noise diffusion process, a residual diffusion process is also introduced to prioritize determinism and guide inverse image recovery. During the training phase, the model utilizes the predicted residuals. and noise A high-resolution preliminary image was gradually reconstructed from the degraded version. During the inference phase, the image is reconstructed iteratively.

[0086] The diffusion variable is used to characterize the diffusion characteristics of image pixels. It is a state variable that integrates the target image, residual changes and noise perturbations. It is used to limit the reconstruction solution space of the target image and form an effective constraint on the reconstruction process.

[0087] Time step The corresponding diffusion variables are: ;

[0088] In the formula, The noise is Gaussian noise that follows a standard normal distribution, with a mean of 0 and a variance of 1. The first satellite image at time t1, which is to be reconstructed; Preset time steps The residual strength, Preset time steps The noise intensity, residuals, and noise are progressively added to the target high-resolution image. In the middle, when t becomes large enough, It can be approximated as noise and low-resolution observations. A linear combination. When t=T, , ,at this time The expression is as follows:

[0089]

[0090] This diffusion variable It is a fusion of second satellite images The state variables, along with those affected by noise perturbation, are used to limit the reconstruction solution space of the target image, thus providing effective constraints on the reconstruction process.

[0091] Step 2: Construct a semantic adaptive model S, which includes a feature extraction branch, a prior image fusion branch, and a target image reconstruction branch.

[0092] The feature extraction branch extracts features from the input diffusion variables through multi-scale convolution and downsampling to obtain their deep features, which are then input into the target image reconstruction branch.

[0093] The prior image fusion branch performs shallow and deep fusion on remote sensing images observed by two satellites at different times, respectively, to obtain the corresponding shallow fusion features and deep fusion features, and transmits them to the target image reconstruction branch.

[0094] Finally, the target image reconstruction branch reconstructs each feature to obtain intermediate variables. .

[0095] Step 3: Diffusion variables The input is fed into the feature extraction branch, where multi-scale convolution and downsampling are used to obtain deep features of the diffusion variables. ;

[0096] Step 4: Input the remote sensing images from the two satellites at different times into the prior image fusion branch to obtain a set of shallow fusion features. and deep fusion features ;

[0097] The prior image fusion branch includes a shallow fusion branch and a deep fusion branch;

[0098] Satellite images of two satellites at different times , , , , The inputs are fed into the shallow fusion branch respectively to obtain the shallow fusion features. and the deep features corresponding to each image { , , , , };

[0099] The shallow feature fusion branch consists of one embedding layer, five feature extraction units, and four fusion units; specifically:

[0100] First, the embedding layer receives satellite images from two satellites at different times. , , , , Each image is projected into a 64-channel feature space using 3x3 convolution.

[0101] Then, the first four feature extraction units obtain the feature map of the prior image through multi-scale convolution and downsampling. , , , , };in Indicates the number of feature layers; values ​​are... =1,2,3,4.

[0102] Next, the fusion unit consists of three convolutional modules: [the modules then combine] the features of the second satellite image. , and Stacking and feeding the data into the first convolutional module yields the second satellite image fusion features. The first satellite image features and Stack the convolutional blocks and input them into the second convolutional block to obtain the first satellite image fusion features. ; Fuse features from second satellite images Features fused with the first satellite image The layers are concatenated, and a shallow fusion feature is generated through a third convolutional block. .

[0103] At the same time, the feature map of the prior image { , , , , The input is fed into the fifth feature extraction unit, where convolution and downsampling are used to obtain the respective deep features: the first satellite image at time t0. deep features Second satellite image deep features Second satellite image at time t1 deep features First satellite image at time t2 deep features Second satellite image deep features .

[0104] The deep fusion branch consists of two temporal feature alignment modules and a semantic adaptive fusion module, which integrates deep features { , , , , The input is fed into the deep fusion branch to obtain the deep fusion features. .

[0105] In this embodiment, due to differences in acquisition time and phenological variations, multi-temporal images often exhibit inconsistent spectral characteristics. To address this issue, this invention introduces a first temporal feature alignment module. This module adaptively estimates fine temporal dynamic coefficients and difference coefficients to construct a linear regression equation, projecting the features of the first and second resolution images into a unified latent space for alignment. By constructing a linear equation, the first temporal feature alignment module projects reference images with temporal dynamic differences into a unified latent space, eliminating spectral distortion in the reconstructed images caused by temporal differences and phenological variations, effectively ensuring the spectral consistency of the reconstructed images. The second temporal feature alignment module operates on the same logic as the first temporal feature alignment module and will not be described further here.

[0106] Specifically:

[0107] The first-time feature alignment module receives deep features. , , After feature alignment, the first-time aligned features are obtained. ;

[0108] The second time feature alignment module receives deep features. , After feature alignment, the second time-aligned feature is obtained. ;

[0109] The calculation of the first-time feature alignment module is as follows:

[0110]

[0111]

[0112]

[0113]

[0114] In the formula, This is the first coarse-time dynamic coefficient. This is the first-time feature extraction unit. For feature splicing, The difference coefficient for the first mode. This is the first modal difference extraction unit. This is the first fine-time dynamic coefficient. This is the first cross-attention module. These are the position coordinates of pixels in the feature map;

[0115] The second time feature alignment module is calculated as follows:

[0116]

[0117]

[0118]

[0119]

[0120] This is the second coarse-time dynamic coefficient. This is the second time feature extraction unit. The second modal difference coefficient. This is the second modality difference extraction unit. This is the second fine time dynamic coefficient. This is the second cross-attention module.

[0121] Align two time features , Deep features , and The input is fed into the semantic adaptive fusion module to obtain deep fusion features. :

[0122]

[0123] In the formula, express and Euclidean distance, express and Euclidean distance.

[0124] The change mask is obtained by calculating the Euclidean distance between feature maps; the calculation is as follows:

[0125]

[0126] In the formula, express and Euclidean distance, express and Euclidean distance, express and Euclidean distance;

[0127] In this embodiment, the semantic adaptive fusion module adaptively transfers high-frequency details from the semantic matching region, improving the model's ability to perceive semantic changes in land features and achieving accurate reconstruction of spatial details in dynamic regions. In time-series data, if a land cover area remains stable over a long period, it can be inferred that no significant changes occurred at intermediate time points. Based on this assumption, the present invention will... Less than Minimum value ( The region is defined as the "invariant region," while the remaining regions are considered "potentially variable regions." For the invariant region, this invention directly... (First deep feature) serves as the feature representation of the invariant region. For the variable region, this invention first compares... and From adaptive selection, if < Then (First-time alignment features) are used as the feature representation of the changed region; otherwise, they are used... (Second time alignment feature) serves as the feature representation of this region.

[0128] Step 5: Deeply fuse features Diffusion variables deep features Deep features and shallow fusion features Input to the target image reconstruction branch to obtain intermediate variables .

[0129] The target image reconstruction branch includes a deep residual removal module and a multi-scale noise removal module;

[0130] The deep residual removal module consists of two temporal convolutional units: each temporal convolutional unit contains two residual convolutional blocks and a multilayer perceptron.

[0131] First, deep features With deep fusion features The difference is calculated and compared with the preset residual strength at the initial time step T. Input the first temporal convolutional unit to obtain the residual components. .

[0132] Then, deep features With residual components By subtracting the two, we can obtain the combination of the pure feature and the noise feature, and the noise intensity preset at the initial time step T. Input a second temporal convolutional unit to remove deep features. residual components .

[0133] Since the multimodal inputs differ in scale and are collected at different time points, this invention performs multimodal fusion in the latent feature space, thereby achieving better temporal alignment and maintaining the spectral and semantic consistency of the reconstructed image.

[0134] In this embodiment, although the noise and residual components in the diffused variables are a linear combination, they become nonlinearly entangled in the feature space after propagation through the network. Therefore, this invention uses Taylor series expansion at the feature level to approximate this. This representation achieves effective decoupling of noise and residuals, making the noise and residual components in the deep features of the diffused variable linearly separable. Then, a residual removal module and a denoising decoder are used to remove the residual and noise components, improving the reconstruction accuracy of intermediate variables. The specific formula is as follows:

[0135]

[0136] Through Performing a first-order Taylor expansion at this point, we obtain:

[0137]

[0138]

[0139]

[0140] in (⋅) is a learnable feature extractor. Represents higher-order terms. The Jacobian matrix is ​​used as a local linear approximation of nonlinear functions. In network design, The residual component is extracted using a noise encoder. Estimation and removal are performed using a deep residual removal module. Residual image. For the second satellite image With the first satellite image difference;

[0141] This is a deep feature of the diffusion variable completed by removing residuals.

[0142] Finally, after removing the residual components and shallow fusion features The multi-scale noise removal module, input into the target image reconstruction branch, obtains intermediate variables through multi-scale convolution and upsampling. .

[0143] To further narrow down the predicted output Ground Truth The difference between them leads to the introduction of perceptual loss, which is defined as follows:

[0144]

[0145] in This indicates that from the pre-trained VGG-16 network, the first... Feature maps extracted from layers; , and These represent the number of channels, height, and width of the corresponding feature map, respectively.

[0146] Reconstruction loss Alignment loss with time features They are defined as follows:

[0147]

[0148]

[0149] Where (C,H,W) represents The dimension, N represents The number of pixels in, and and Let L1 and L2 represent the norms, respectively. This refers to the depth features extracted from the coarse-resolution image C1 at the target time, while These are the corresponding predicted values ​​generated by the time feature alignment module.

[0150] The total loss function integrates , and Its definition is as follows:

[0151]

[0152] Where parameters , and Used to balance the weights (trade-offs) among the various loss components.

[0153] To effectively utilize the auxiliary functions of the temporal feature alignment module while minimizing its interference with optimization in the early stages of training, this invention employs a warm-up strategy. This strategy allows the network to focus on the primary objective initially and gradually activate its temporal modeling capabilities. The loss function is defined as follows:

[0154]

[0155] Where E represents the total number of training epochs, and e refers to the current training epoch.

[0156] The diffusion step count T was set to 100, and the DDIM sampler was used during the inference phase with 50 sampling steps. Model training consisted of 200 epochs with a batch size of 4. The Adam optimizer was used to minimize the loss function, and the hyperparameters were set as follows: =1、 =0.01 and =0.0001. The initial learning rate is set to 0.0001 and is dynamically halved every 40 rounds.

[0157] Step Six: Transfer intermediate variables By inputting the inverse reconstruction process equations of the diffusion model, the diffusion variables at the next time step T-1 are obtained. ;

[0158] The reverse process of RDDM is defined as follows:

[0159]

[0160] in and Let residuals and noise be represented, respectively, and these are typically predicted by two separate networks. While noise prediction can enhance the diversity of generated results, it introduces instability during training. By analyzing the forward diffusion process, it was observed that both residuals and noise can be... To represent this. Based on this observation, a neural network was constructed to analyze the degraded image at each diffusion step t. Reconstructing intermediate variables (When t=0, it corresponds to) Next, we will explain the introduction from the perspective of forward diffusion. To construct the principle of the reverse reconstruction process.

[0161] The forward diffusion process at time step t is defined as follows:

[0162]

[0163] in A low-resolution image representing the predicted time. For residuals, and Represents noise. According to the definition of residual and formula (16), residual and noise can be expressed as:

[0164]

[0165]

[0166] The equations for the diffusion model's reverse reconstruction process are as follows:

[0167]

[0168]

[0169]

[0170] As the first coefficient, The second coefficient is t, which represents the time step of the diffusion process and is used to indicate the noise level of the current sample. This refers to, at time step T, by... The input network produces an initial output, which is then iteratively used to generate a high-resolution target image.

[0171] The above derivation clearly shows that, during back diffusion, residual and noise components can be detected through intermediate images. Joint modeling is performed, thus providing a theoretical basis for performing reverse reconstruction using a single network.

[0172] Step 7: Return to Step 3 and set the diffusion variable for the next time step. The feature extraction branch of the semantic adaptive model S is input, and deep and shallow features are fused together with satellite images at each time step. After reconstructing the target image, new intermediate variables are obtained, and the reverse reconstruction process is repeated until the preset number of iterations is reached, finally obtaining the intermediate variable corresponding to time step 0. That is, the first satellite image at time t1. .

[0173] Through multiple iterations and optimizations, errors in the fusion process are gradually corrected, noise in the diffusion variables is removed, and the accuracy of feature fusion and the quality of image reconstruction are continuously improved, ultimately obtaining the first satellite image at the target time.

[0174] This invention effectively integrates the advantages of multi-time-series and multi-resolution satellite images, solves the problems of artifacts, spectral distortion and insensitivity to semantic changes in traditional fusion methods, and significantly improves the accuracy of spatiotemporal fusion of multi-source satellite images.

[0175] Existing methods are prone to producing noticeable artifacts when fusing images with significant scale differences, especially in dynamic regions. To address this issue, this invention introduces a diffusion model for the reverse reconstruction process equations. Traditional residual diffusion models require two neural networks to estimate the residuals and noise needed for the reverse reconstruction process. On the one hand, two neural networks significantly increase the model's memory consumption; on the other hand, noise-based reconstruction is unsuitable for high-dimensional image reconstruction tasks such as spatiotemporal fusion of remote sensing images. Therefore, this invention constructs intermediate variables to represent noise and residuals, enabling the construction of a reverse reconstruction process using only a single neural network.

[0176] Dataset Construction and Evaluation

[0177] To verify the performance of the proposed method, experiments were conducted on three benchmark datasets collected from heterogeneous regions:

[0178] The Coleambally Irrigation District (CIA) is located in southern New South Wales and is primarily used for irrigating rice.

[0179] The Lower Gwydir Water Conservancy District (LGC) in northern New South Wales is affected by flooding events.

[0180] And Arukorqin Banner (AHB) in Inner Mongolia Autonomous Region, which represents an agro-pastoral mixed system.

[0181] These datasets encompass diverse land cover types, with land features exhibiting significant phenological and semantic variations. In all three datasets, each image is divided into 256 sub-databases. The slices are 256 in size. Specifically, the CIA dataset contains 1,287 training slices and 117 test slices; the LGC dataset contains 2,527 training slices and 361 test slices; and the AHB dataset contains 2,907 training slices and 327 test slices.

[0182] The table lists the modules, including the Noise Encoder, Shallow Fusion, Deep Fusion, De-Res Block, and Denoising Decoder, and provides the block name, kernel size, number of input / output channels, and feature map size for each stage, as shown in Table 1.

[0183] Table 1

[0184]

[0185] This invention compares the proposed method with five cutting-edge methods belonging to different categories: ESTARFM, MLFF-GAN, SwinSTFM, STFDiff, and STFMamba. Quantitative evaluation employed five metrics: correlation coefficient (CC), structural similarity index (SSIM), dimensionless global relative error of synthesis (ERGAS), root mean square error (RMSE), and spectral angle mapping (SAM). Higher values ​​for CC and SSIM indicate better structural fidelity, while lower values ​​for ERGAS, RMSE, and SAM reflect reduced spatial discrepancies and spectral distortion.

[0186] Results Evaluation and Analysis

[0187] CIA Dataset: The fusion results on the CIA dataset are as follows Figure 3 As shown, a. the true reference (GT) Landsat image (LS) at time t1; b. the MODIS image at time t1; c. the reference LS at time t0; d. the reference LS at time t2; e. ESTARFM; f. MLFF-GAN; g. SwinSTFM; h. STFDiff; i. STFMamba; j. OURS. The first row of (e)–(j) shows the spectral differences between the reconstruction results and the GT. The higher the brightness at t1 (target time), the greater the spectral error. The quantitative evaluation results are listed in Table 2.

[0188] The region is largely covered by farmland and vegetation, characterized by variations in crop phenology, while the surrounding dryland and woodland remain relatively stable. Notably, on the CIA dataset, the traditional method ESTARFM outperforms several deep learning-based methods. This is primarily because the short time span between the reference and predicted images results in limited land cover variation, allowing linear models to produce effective fusion results. Deep learning-based methods (such as STFMamba and SwinSTFM) tend to produce blurred results due to the resolution difference between coarse and fine-resolution images. In contrast, while generative models like STFDiff and MLFF-GAN can generate detailed images, they fail to reconstruct semantic variations due to a lack of constraints. However, our proposed method achieves the best performance thanks to its semantically adaptive diffusion framework, which possesses strong generative capabilities and can adaptively transfer high-frequency details. Therefore, it effectively reconstructs semantic variations while maintaining spectral fidelity.

[0189] Table 2

[0190]

[0191] LGC dataset: Fusion results on the LGC dataset are as follows Figure 4 As shown. a. a. True reference (GT) Landsat image (LS) at time t_1; b. MODIS image at time t_1; c. Reference time LS; d. The references for each time step are: LS; e. ESTARFM; f. MLFF-GAN; g. SwinSTFM; h. STFDiff; i. STFMamba; j. OURS. The first row of (e)–(j) shows the SAM difference between the reconstruction results and the GT. Higher brightness indicates a greater spectral error.

[0192] The quantitative evaluation results are listed in Table 3. In this flood scenario, large-scale vegetation loss led to significant land cover changes between December 28, 2004, and January 13, 2005, while the changes between January 13, 2005, and January 29, 2005, were relatively smaller. ESTARFM performs poorly in such dynamic scenarios. Compared to other deep learning-based methods, STFMamba performs better; however, its data-driven strategy limits its ability to capture drastic changes. In contrast, this invention proposes a temporal feature alignment module to align temporal dynamics and uses a semantically adaptive fusion module to reconstruct the evolution of land cover, thereby improving fusion accuracy.

[0193] Table 3

[0194]

[0195] AHB dataset: Figure 5 Tables 1 and 4 show the visual and quantitative results on the AHB dataset, respectively.

[0196] a. Time-based true reference (GT) Landsat imagery (LS); b. MODIS image at any given time; c. Reference time: LS; d. The references for each time step are: LS; e. ESTARFM; f. MLFF-GAN; g. SwinSTFM; h. STFDiff; i. STFMamba; j. OURS. The first row of (e)–(j) shows the SAM difference between the reconstruction results and the GT. Higher brightness indicates a greater spectral error.

[0197] The region exhibits diverse land cover types and pronounced seasonal phenological dynamics, which increases the challenge of spatiotemporal fusion. Traditional methods such as ESTARFM fail to capture dynamic semantic changes. Although generative models such as MLFF-GAN and STFDiff can reconstruct image details, they cannot preserve spectral information in changing regions due to a lack of necessary constraints. In contrast, SA-STF (the method of this invention) effectively handles long-interval variations by aligning temporal dynamics and adaptively fusing features based on detected changes.

[0198] Table 4

[0199]

[0200] To verify the contributions of the proposed components, ablation experiments were conducted on the LGC dataset, and the results are shown in Table 5. When only the Taylor-inspired decoder is retained, the results are comparable to RDDM except for ERGAS; this is expected because the accuracy of residual estimation decreases in the absence of SAF and TFA modules, thus reducing the advantage of the Taylor structure. Adding Temporal Feature Alignment (TFA) or Semantic Adaptive Fusion (SAF) modules further improves performance. When all components (OURS) are integrated, the model achieves optimality across all five metrics, demonstrating its effectiveness in recovering dynamic changes. Notably, even without the diffusion process (Only TFA+SAF), the model remains highly competitive, indicating that these modules can be used as plug-and-play components for spatiotemporal fusion. However, when the resolution difference between coarse and fine-resolution images is large, removing the diffusion mode leads to a significant decrease in the ERGAS metric and a deterioration in the recovery of fine-grained spatial details.

[0201] Table 5

[0202]

[0203] To evaluate the performance of each method under dynamic changes, Figure 6 Box plots of RMSE, SAM, and SSIM calculated in varying regions across three test sets are presented; a. RMSE box plot; b. SAM box plot; c. SSIM box plot. SA-STF exhibits the narrowest box width and the best median performance, indicating its superior overall performance. The RMSE and SSIM box plots demonstrate that SA-STF can accurately reconstruct high-frequency textures in regions where semantics or phenology undergoes significant changes. Furthermore, SA-STF's excellent SAM score further confirms its ability to maintain spectral consistency and prevent spectral distortion within varying regions.

[0204] The goal of Spatiotemporal Fusion (STF) is to generate high-quality temporal imagery to support downstream tasks such as land cover classification and agricultural assessment. K-means classification of ground truth and fused imagery was performed on three datasets, and fusion quality was evaluated using metrics such as overall accuracy (OA), precision, recall, and F1 score. Classification results and quantitative evaluations on the CIA, LGC, and AHB datasets are detailed in [link to documentation]. Figure 7 Tables 8, 9 and 6. Figure 7 a. Real Reference; b. ESTARFM; c. MLFF-GAN; d. SwinSTFM; e. STFDifff; f. STFMamba; g. OURS; Figure 8 a. Real Reference; b. ESTARFM; c. MLFF-GAN; d. SwinSTFM; e. STFDifff; f. STFMamba; g. OURS; Figure 9 a. Real Reference; b. ESTARFM; c. MLFF-GAN; d. SwinSTFM; e. STFDifff; f. STFMamba; g. OURS.

[0205] Table 6

[0206]

[0207] Traditional methods and deep learning-based approaches often exhibit limited generalization ability across different datasets. In contrast, SA-STF consistently maintains top performance across all evaluation metrics, highlighting its strong generalization ability in downstream tasks. This performance improvement is attributed to the Temporal Feature Alignment (TFA) and Semantic Adaptive Fusion (SAF) modules, which explicitly model dynamic changes in land cover and adaptively transfer high-frequency information from reference imagery. Therefore, even under challenging conditions with complex land cover variations, SA-STF maintains high spectral fidelity and spatiotemporal consistency.

Claims

1. A semantically adaptive spatiotemporal fusion method for multi-source satellite images, characterized in that, The specific steps include: Step 1: Acquire remote sensing images of the same size from two satellites at different times, and determine the diffusion variables at the initial time step T. ; Time step is ; Specifically, this involves acquiring the first satellite image at time t0. Second satellite image Second satellite image at time t1 ; and the first satellite image at time t2 Second satellite image ; Step 2: Construct a semantic adaptive model S, which includes a feature extraction branch, a prior image fusion branch, and a target image reconstruction branch; Step 3: Diffusion variables The input is fed into the feature extraction branch, where multi-scale convolution and downsampling are used to obtain deep features of the diffusion variables. ; Diffusion variables : ; Gaussian noise that follows a standard normal distribution; Step 4: Input the remote sensing images from the two satellites at different times into the prior image fusion branch to obtain a set of shallow fusion features. and deep fusion features ; The prior image fusion branch includes a shallow fusion branch and a deep fusion branch; The shallow feature fusion branch consists of an embedding layer, five feature extraction units, and four fusion units; Embedded layer receives satellite images , , , , And projected onto the 64-channel feature space; The first four feature extraction units obtain feature maps for each satellite image through multi-scale convolution and downsampling. , , , , };in Indicates the number of feature layers; values ​​are... =1,2,3,4; The fusion unit consists of three convolutional modules. The first two convolutional modules stack the image features to obtain their respective fusion features, which are then concatenated and passed through the third convolutional block to generate shallow fusion features. ; The deep fusion branch consists of two temporal feature alignment modules and one semantic adaptive fusion module; First, each image obtains deep features in the fifth feature extraction unit of the shallow feature fusion branch. , , , , Time alignment features are obtained through two time feature alignment modules respectively. , Combining deep features , and The input is fed into the semantic adaptive fusion module to obtain deep fusion features. ; In the formula, express and Euclidean distance, express and Euclidean distance; This is a change mask obtained by calculating the Euclidean distance between feature maps; The change mask is calculated as follows: In the formula, express and Euclidean distance, express and Euclidean distance, express and Euclidean distance; These are the position coordinates of pixels in the feature map; Step 5: Deeply fuse features Diffusion variables deep features Deep features and shallow fusion features Input to the target image reconstruction branch to obtain intermediate variables ; The target image reconstruction branch includes a deep residual removal module and a multi-scale noise removal module; deep features are removed sequentially. The residual components and noise in the data; Step Six: Transfer intermediate variables By inputting the inverse reconstruction process equations of the diffusion model, the diffusion variables at the next time step T-1 are obtained. ; Step 7: Return to Step 3 and set the diffusion variable for the next time step. The feature extraction branch of the semantic adaptive model S is input, and deep and shallow features are fused together with satellite images at each time step. After reconstructing the target image, new intermediate variables are obtained, and the reverse reconstruction process is repeated until the preset number of iterations is reached, finally obtaining the intermediate variable corresponding to time step 0. That is, the first satellite image at time t1. .

2. The method as described in claim 1, characterized in that, In step four, the embedding layer uses 3*3 convolution to project each image into a 64-channel feature space.

3. The method as described in claim 1, characterized in that, In the fusion unit of step four, the features of the second satellite image are... , and Stacking and feeding the data into the first convolutional module yields the second satellite image fusion features. The first satellite image features and Stack the convolutional blocks and input them into the second convolutional block to obtain the first satellite image fusion features. ; Fusing features from second satellite images Features fused with the first satellite image The layers are concatenated, and a shallow fusion feature is generated through a third convolutional block. .

4. The method as described in claim 3, characterized in that, In step four, the feature map of the prior image { , , , , The input is fed into the fifth feature extraction unit, where convolution and downsampling are used to obtain the respective deep features. , , , , } 5. The method as described in claim 4, characterized in that, The deep fusion branch is specifically as follows: The first-time feature alignment module receives deep features. , , After feature alignment, the first-time aligned features are obtained. ; The second time feature alignment module receives deep features. , After feature alignment, the second time-aligned feature is obtained. ; The calculation of the first-time feature alignment module is as follows: In the formula, This is the first coarse-time dynamic coefficient. This is the first-time feature extraction unit. For feature splicing, The difference coefficient for the first mode. This is the first modal difference extraction unit. This is the first fine-time dynamic coefficient. This is the first cross-attention module; The second time feature alignment module is calculated as follows: This is the second coarse-time dynamic coefficient. This is the second time feature extraction unit. The second modal difference coefficient. This is the second modality difference extraction unit. This is the second fine time dynamic coefficient. This is the second cross-attention module.

6. The method as described in claim 2, characterized in that, The deep residual removal module consists of two temporal convolutional units: each temporal convolutional unit contains two residual convolutional blocks and a multilayer perceptron; First, deep features With deep fusion features The difference is calculated and compared with the preset residual strength at the initial time step T. Input the first temporal convolutional unit to obtain the residual components. ; Then, deep features With residual components The difference is calculated, and the noise intensity is preset at the initial time step T. Input a second temporal convolutional unit to remove deep features. residual components ; Finally, after removing the residual components and shallow fusion features The input is fed into the multi-scale noise removal module, where it undergoes multi-scale convolution and upsampling to obtain intermediate variables. .

7. The method as described in claim 6, characterized in that, For deep features The formula for deep residual removal is as follows: In the formula, For learnable feature extractors, For Jacobian matrices, That is, the residual components ; It is a higher-order term; At the initial time step T, set = =1, and the residual image For the second satellite image With the first satellite image The difference; that is, the diffusion variable. .

8. The method as described in claim 7, characterized in that, The formula for the reverse reconstruction process of the diffusion model is as follows: ； For time steps Preset noise level; For time steps Preset residual strength.

Images