A light flow guide-based spatiotemporal fusion method for diffusion model

Abstract

The application discloses a diffusion model space-time fusion method based on optical flow guidance, comprising the following steps: acquiring fine spatial resolution images, coarse spatial resolution images at a reference time and coarse spatial resolution images at a target time of the same observation area; up-sampling the coarse spatial resolution images to obtain up-sampled images of the same size as the fine spatial resolution images, and constructing pseudo-fine resolution images; estimating pixel-level displacement optical flow of the fine spatial resolution images and the pseudo-fine resolution images based on an optical flow estimation network to obtain target scale optical flow; calculating the gradient consistency of the fine spatial resolution images and the pseudo-fine resolution images to construct an optical flow confidence map; establishing a conditional diffusion model to obtain an initial fusion image at the target time; and splicing the initial fusion image and the up-sampled coarse spatial resolution images along the channel dimension to output a high-resolution fusion image at the target time. The application can generate a high-resolution fusion image at the target time which is good in structure consistency, detail clarity and radiation consistency.

Description

Technical Field

[0001] This invention relates to a spatiotemporal fusion method for diffusion models, and more particularly to a spatiotemporal fusion method for diffusion models based on optical flow guidance. Background Technology

[0002] Remote sensing applications (such as land surface change monitoring, ecological environment assessment, and resource management) typically require both high spatial and temporal resolution observation data to characterize ground features and dynamic processes. However, due to limitations in sensor design and orbital revisit cycles, actual data often exhibits a "high spatial-low temporal" or "high temporal-low spatial" pattern. This means that high-resolution sensors have strong detail acquisition capabilities but long revisit cycles, while high-frequency observation sensors can provide dense temporal sequences but lack spatial detail, leading to a long-standing contradiction in spatiotemporal resolution. To alleviate this contradiction, spatiotemporal fusion methods are a suitable choice, capable of integrating complementary information from multi-source, multi-temporal images to generate a fused image with high spatiotemporal resolution at the target time.

[0003] Traditional spatiotemporal fusion methods have been developed for a long time, and common technical approaches include: weighted reflectance fusion, unmixing models based on linear spectral mixing, posterior estimation based on a Bayesian framework, coupled representation methods based on dictionary learning, and hybrid models that integrate multiple mechanisms. These methods usually have explicit physical or statistical assumptions and perform well in some relatively stable or gently changing scenarios. However, they often rely on simplified assumptions such as pixel homogeneity, linear mixing, pre-set priors, or local similarity. In cases of highly heterogeneous land cover, strong changes (such as abrupt changes, displacement, or morphological changes), or complex nonlinear radiative variations, they are prone to problems such as loss of detail, edge misalignment, and artifact accumulation, and have limited ability to express complex spatiotemporal relationships.

[0004] With the development of deep learning, spatiotemporal fusion methods based on convolutional networks, generative adversarial networks, and Transformer structures have emerged, enabling the learning of nonlinear mapping relationships from data and improving upon traditional methods in terms of detail recovery and structural representation. However, most existing deep learning spatiotemporal fusion methods are still often modeled as deterministic regression: that is, directly mapping from input to output. This is often difficult to effectively characterize the uncertainty and multimodal changes in prediction when facing noise, uncertainty, and complex temporal dynamics in real remote sensing scenes. At the same time, when there is significant motion or structural displacement of the target at any given time, relying solely on static pairing learning is prone to boundary drift, texture mismatch, or unstable reconstruction of changed regions.

[0005] As a probabilistic generative model, the diffusion model has advantages such as stable training, strong distribution coverage, and suitability for reconstruction tasks. In recent years, it has been gradually introduced into remote sensing image generation, enhancement, restoration, and multi-source fusion. Applying the diffusion model to the field of spatiotemporal fusion can theoretically better model complex distributions and uncertainties through progressive denoising generation, thereby improving reconstruction quality and robustness. However, many existing diffusion-based spatiotemporal fusion methods still tend to treat fusion as a general generative problem in their design, failing to fully utilize the temporal variation mechanism of the target time and the constraints of available observations. They lack explicit dynamic priors to constrain motion and structural consistency, which may lead to structural instability and unreliable details in changed areas under complex surface changes, temporal mismatches, or occlusion interference. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to propose a spatiotemporal fusion method based on optical flow-guided diffusion model. It achieves robust motion alignment by using high-confidence optical flow features as prior information, and injects alignment priors at multiple scales during the diffusion denoising generation process, thereby generating high-resolution fused images of the target time with consistent structure, clear details, and good radiometric consistency.

[0007] Technical solution: This invention includes the following steps:

[0008] Step 1: Acquire the fine spatial resolution image F0 of the same observation area at reference time t0, the coarse spatial resolution image C0 at reference time t0, and the image at the target time t0. p Coarse spatial resolution image C P ;

[0009] Step 2: Upsample the coarse spatial resolution image to obtain an upsampled image at the same scale as F0, and construct a pseudo fine resolution image F based on edge-preserving smoothness constraints. P1 ;

[0010] Step 3, based on the optical flow estimation network, analyze F0 and F P1 Estimate the pixel-level displacement optical flow to obtain the target-scale optical flow U;

[0011] Step 4, calculate F0 and F P1 The gradient consistency is determined, and a confidence map conf of the optical flow is constructed, which is calculated based on the magnitude of the optical flow gradient. Where σ is a hyperparameter that adjusts gradient sensitivity. It is an exponential function. is the total gradient magnitude, i and j represent the row and column in the matrix, respectively. The confidence map conf ensures that the optical flow with high confidence is injected into the encoder. The optical flow U is filtered according to conf. For low confidence regions, a gating mechanism is used to reduce the contribution of optical flow or neighborhood propagation is used for smoothing to obtain robust optical flow U*.

[0012] Step 5, using F0, C0, C P Using U* as the conditional input, a conditional diffusion model is established. Convolutionally encoded optical flow features are injected into the multi-scale feature layers of the diffusion model to apply motion alignment constraints to the denoising generation process, resulting in the initial fused image F at the target time. P2 ;

[0013] Step 6, F P2 With upsampling C P The image is stitched along the channel dimension, the input residual refinement network predicts and refines the residuals, and then overlays them to output a high-resolution fused image F at the target time. P .

[0014] The specific process of constructing the optical flow confidence map conf in step 4 is as follows: Calculate F0 and F within a local window. P1 The gradient magnitude difference and gradient direction consistency are obtained and mapped to the [0,1] interval to obtain the initial confidence level. Then, the confidence level is multiplied by the confidence factor based on the optical flow gradient magnitude to obtain the final confidence map conf.

[0015] The diffusion model in step 5 recovers the fine-scale image at the target time using a progressively denoised generation method.

[0016] During the training of the diffusion model, noise is added to the target high-resolution sample to form a multi-stage state, and the denoising network is trained under the condition {F0, C0, C...} P} Predict noise or residuals under robust optical flow U*; during inference, start from random noise or prior initialization, and generate the initial fused image F through several denoising steps. P2 Meanwhile, convolutionally encoded optical flow features are injected into the multi-scale feature layers of the diffusion model. At each encoder scale, the mechanism injects the flow estimate corresponding to that scale into the feature channel.

[0017] For the input features of each encoder layer, optical flow pooling is first performed to the corresponding scale, and 3×3 convolution is applied to adjust the features before they are injected into the current feature map.

[0018] The optical flow estimation process is as follows: C is estimated using bilinear interpolation. P Upsampling is performed, and thinning is done using an edge-preserving smoothing filter. This pseudo-image shares the same spatial resolution as F0 and is able to achieve the same result in t. p Estimating optical flow in situations where high-resolution observation data is lacking.

[0019] The pseudo-high resolution image F in step 2 P1 The construction process is as follows: using C PThe upsampling result is used as the initial value, and iterative optimization is performed to maintain C at low frequencies. P The radiation trend is observed, while maintaining a gradient structure similar to F0 in the edge region, resulting in a pseudo-fine resolution image F. P1 .

[0020] This invention also provides a spatiotemporal fusion system for optical flow-guided diffusion models, used to execute the aforementioned spatiotemporal fusion method for optical flow-guided diffusion models, comprising:

[0021] Optical flow estimation module: for F0 and F P1 Estimate the target-scale optical flow U;

[0022] Confidence assessment module: Calculates the optical flow confidence map conf based on gradient consistency, and filters the target-scale optical flow U based on conf to obtain robust optical flow U*;

[0023] Multi-scale feature injection module: downsamples U* to the spatial scale corresponding to each encoder layer of the diffusion network, and after convolutional encoding, performs residual fusion with the feature map of the current layer;

[0024] Conditional diffusion fusion module: based on F0, C0, C P U* is the conditional input, and the initial fused image F is generated through progressive denoising. P2 ;

[0025] Residual refinement module: F P2 With upsampling C P After stitching, predict the high-frequency residual map and output the final fused image F. P .

[0026] The confidence assessment module is designed with a gradient-based confidence estimation module (GFC). GFC measures the intensity of local changes by calculating the spatial gradients of two-dimensional optical flow, horizontal optical flow, and vertical optical flow.

[0027] The multi-scale feature injection module injects the flow estimate corresponding to each encoder scale into the feature channel, maintaining spatial correspondence and displacement trend throughout the encoding process, from fine-grained to coarse-grained scales.

[0028] Beneficial effects: This invention achieves robust motion alignment by using high-confidence optical flow features as prior information, and injects alignment priors at multiple scales during the diffusion denoising generation process, thereby generating high-resolution fused images of the target time with consistent structure, clear details, and good radiometric consistency. The joint mechanism of optical flow and confidence can suppress erroneous displacement under conditions of occlusion, cloud shadows, and noise, improving the alignment reliability of changed areas. The multi-scale injection diffusion model of optical flow enables the generation process to simultaneously satisfy motion alignment and detail generation, reducing edge drift and texture mismatch. Residual refinement compensates for high-frequency information, which can further improve edge sharpness and visual consistency. It has good robustness to different scenes, different land cover types, and certain domain differences, and has a wide range of applications. Attached Figure Description

[0029] Figure 1 This is a flowchart of the present invention;

[0030] Figure 2 The fusion results obtained by the method of the present invention and the comparison method are as follows: (a) is the reference image, (b) is the flexible spatiotemporal fusion method of satellite images of different resolutions (FSDAF), (c) is the spatiotemporal data fusion method based on variation (VSDF), (d) is the spatiotemporal fusion method of remote sensing based on single prior image (OPGAN) based on generative adversarial network (OPGAN), (e) is the spatiotemporal fusion method of remote sensing image based on conditional generative adversarial network (GAN-STFM), (f) is the spatiotemporal fusion method of remote sensing based on dual-stream Transformer (SwinSTFM), (g) is the spatiotemporal fusion method of remote sensing based on diffusion model (STFDiff), and (h) is the STF-FlowDiff of the present invention.

[0031] Figure 3 The classification results of the images fused by the method of the present invention and the comparison method are as follows: (a) is the reference image, (b) is the method of flexible spatiotemporal fusion of satellite images of different resolutions (FSDAF), (c) is the method of spatiotemporal fusion based on change (VSDF), (d) is the method of spatiotemporal fusion of remote sensing images based on generative adversarial network (OPGAN), (e) is the method of spatiotemporal fusion of remote sensing images based on conditional generative adversarial network (GAN-STFM), (f) is the method of spatiotemporal fusion of remote sensing images based on dual-stream Transformer (SwinSTFM), (g) is the method of spatiotemporal fusion of remote sensing images based on diffusion model (STFDiff), and (h) is the STF-FlowDiff of the present invention. Detailed Implementation

[0032] The invention will now be further described with reference to the accompanying drawings.

[0033] Example 1

[0034] likeFigure 1 As shown, the optical flow-guided diffusion model-based spatiotemporal fusion method in this embodiment utilizes optical flow estimation to obtain pixel-level motion information between the reference and target times. Reliable motion estimates are selected through confidence assessment, and motion features are injected into the diffusion generation model in a multi-scale manner. This explicitly constrains the consistency of spatial structure and temporal changes during the generation of high-resolution images at the target time. The overall principle can be summarized as a three-stage collaborative mechanism of "motion estimation—reliability assessment—generation constraints." Specifically, it includes the following steps:

[0035] Step 1: Acquire the fine spatial resolution image F0 of the same observation area at reference time t0, the coarse spatial resolution image C0 at reference time t0, and the image at the target time t0. p Coarse spatial resolution image C P .

[0036] Step 2: To make optical flow estimation more stable under resolution differences, a pseudo-fine resolution image is constructed, using C... P The upsampling result is used as the initial value, and iterative optimization is performed to maintain C at low frequencies. P The radiation trend is observed, while maintaining a gradient structure similar to F0 in the edge region, thereby enhancing texture matching and obtaining a pseudo-fine resolution image F. P1 .

[0037] Step 3, based on the optical flow estimation network, analyze F0 and F P1 The pixel-level displacement optical flow is estimated and scaled to obtain the target-scale optical flow U.

[0038] The optical flow estimation process is specifically implemented as follows: since only the high-resolution image at time t0 is available as input data, while t p Only a coarse-resolution image is provided. To resolve the resolution mismatch issue, C++ is used. P Construct a pseudo-high-resolution image. Specifically, use bilinear interpolation on C. P Upsampling is performed, followed by thinning using an edge-preserving smoothing filter to approximate a fine-scale spatial structure. This pseudo-image shares the same spatial resolution as F0, thus enabling [further processing] at t [scales]. p To reliably estimate optical flow in the absence of truly high-resolution observation data.

[0039] Step 4, calculate F0 and F P1 Gradient consistency is determined, and an optical flow confidence map conf is constructed. Based on conf, the optical flow U is filtered to obtain the robust optical flow U*. F0 and F are calculated within a local window. PThe gradient magnitude difference, gradient direction consistency or structural similarity are mapped to [0,1] confidence level; for low confidence regions, gating is used to reduce optical flow contribution or neighborhood propagation smoothing is used to obtain robust optical flow U*.

[0040] Optical flow inevitably contains unreliable or noisy regions. To quantify local reliability, the spatial gradient magnitude of the flow field is calculated:

[0041]

[0042] Then the total gradient magnitude The square of is defined as:

[0043]

[0044] Substituting this into the Gaussian exponential decay function, the confidence plot of optical flow is defined as follows:

[0045]

[0046] In the above formula, It is an exponential function. It is a hyperparameter that adjusts gradient sensitivity. This represents the total gradient magnitude, where i and j represent the row and column of the matrix, respectively. The optical flow is calculated using optical flow calculations. A confidence map is used to ensure that high-confidence optical flows are injected into the encoder. Therefore, the final optical flow input to the network is U*.

[0047] Step 5, using F0, C0, C P Using U* as the conditional input, a conditional diffusion model is established. Convolutionally encoded optical flow features are injected into the multi-scale feature layers of the diffusion model to apply motion alignment constraints to the denoising generation process, resulting in the initial fused image F at the target time. P2 .

[0048] The diffusion model reconstructs the fine-scale image of the target at a specific time point using a progressively denoised generation method. During training, high-resolution samples of the target are added to introduce noise to form multi-stage states, and the denoising network is trained under the condition {F0, C0, C...} P} Predict noise or residuals based on optical flow prior U*; during inference, start from random noise or prior initialization, and generate the initial fused image F through several denoising steps. P2 Meanwhile, convolutionally encoded optical flow features are injected into the multi-scale feature layers of the diffusion model. At each encoder scale, the mechanism injects the flow estimate corresponding to that scale into the feature channel.

[0049] For the input features of each encoder layer, optical flow pooling is first performed to the corresponding scale, and 3×3 convolution is applied to adjust the features before injecting them into the current feature map (residual injection).

[0050]

[0051] In the above formula, This represents the original input features of the encoder's l-th layer. These are the features obtained after convolution. This indicates a new feature. express By downsampling to match the resolution of each layer, the model achieves dynamic modeling capability of temporal context through multi-scale additive injection.

[0052] This design maintains spatial correspondence and displacement trends throughout the encoding process, from fine-grained to coarse-grained scales. Compared to cascaded or global flow fusion strategies, the multi-scale feature injection module more effectively captures object motion and structural dynamics, applies optical flow constraints to the denoising generation process, and obtains the initial fused image F at the target time. P2 .

[0053] Step 6, F P2 With upsampling C P Stitching along the channel dimension, the input residual refinement network predicts and refines the residuals, which are then superimposed to compensate for high-frequency details and suppress local texture deviations in the diffused output. The output is a high-resolution fused image F at the target time. P .

[0054] Example 2

[0055] This embodiment provides a spatiotemporal fusion system based on an optical flow-guided diffusion model, comprising:

[0056] Optical flow estimation module: for F0 and F P1 Estimate the target-scale optical flow U;

[0057] The confidence assessment module calculates the optical flow confidence map `conf` based on gradient consistency and then filters the target-scale optical flow `U` according to `conf` to obtain the robust optical flow `U*`. Specifically, a gradient-based confidence estimation module (GFC) is designed. GFC measures the intensity of local variations by calculating the spatial gradients of the two-dimensional optical flow, horizontal optical flow, and vertical optical flow. Smaller variations indicate a smoother local region, thus assigning a higher confidence score. This function can be applied to tasks such as edge detection, region-weighted fusion, and occlusion inference.

[0058] The multi-scale feature injection module downsamples U* to the spatial scale corresponding to each encoder layer of the diffusion network, performs residual fusion with the feature map of the current layer after convolutional encoding, and injects the flow estimate corresponding to that scale into the feature channel at each encoder scale. This design can maintain spatial correspondence and displacement trends throughout the encoding process, from fine-grained to coarse-grained scales. Compared with cascaded or global flow fusion strategies, the multi-scale feature injection module can more effectively capture object motion and structural dynamics.

[0059] Conditional diffusion fusion module: based on F0, C0, C P U* is the conditional input, and the initial fused image F is generated through progressive denoising. P2 ;

[0060] Residual refinement module: F P2 With upsampling C P After stitching, predict the high-frequency residual map and output the final fused image F. P .

[0061] Example 3

[0062] This example is an experimental case. The experimental data was acquired by the CIA in the Colombali Irrigation District (34.0034°E, 145.0675°S) in southern New South Wales, Australia. This well-managed irrigation district is a commonly used area for time-series remote sensing studies. The area covers approximately 2193 square kilometers. During the 2001-2002 crop growing season, 17 pairs of cloud-free Landsat-Medium Resolution Imaging Spectroradiometer (Landsat-MODIS) images were acquired, and atmospheric corrections were performed on the Landsat-7 ETM+ images using MODTRAN4. Because the CIA is located in the overlapping area of ​​two adjacent Landsat paths, it has a high temporal revisit frequency (approximately 8 days), making it an ideal benchmark for evaluating the performance of spatiotemporal fusion methods under conventional irrigation patterns and relatively homogeneous agricultural landscapes.

[0063] The experimental data was cropped to 1200 × 1200 pixels, containing six spectral bands, including blue, green, red, near-infrared, and two short-wave infrared bands. Each experiment used a pair of Landsat-MODIS image pairs and one MODIS image as input, and the corresponding t... pThe Landsat image at time step is used as a reference image for accuracy evaluation. Figure 2 shows an example input from the CIA dataset, and the fusion and classification results are analyzed using this example. All images are displayed in near-infrared-red-green (NIR-Red-Green) RGB mode. All methods require only three input images and use their default parameters. All deep learning methods use the same training image pairs. For the proposed method, each image is segmented into 256 × 256 image patches with a stride of 128. The batch size is set to 24, the training period is 300 epochs, and the initial learning rate is 1e-3. All experiments were conducted on a Windows workstation equipped with an i9-13900K processor, 128 GB of RAM, and an NVIDIA A40 GPU (48 GB of VRAM).

[0064] like Figure 1 As shown, the specific implementation steps are as follows:

[0065] Step 1: Acquire the fine spatial resolution image F0 of the same observation area at reference time t0, the coarse spatial resolution image C0 at reference time t0, and the image at the target time t0. p Coarse spatial resolution image C P ;

[0066] Step 2: Upsample the coarse spatial resolution image to obtain an upsampled image at the same scale as F0, and construct a pseudo fine resolution image F based on edge-preserving smoothness constraints. P1 ;

[0067] Step 3, based on the optical flow estimation network, analyze F0 and F P1 The pixel-level displacement optical flow is estimated and scaled to obtain the target-scale optical flow U;

[0068] Step 4, calculate F0 and F P1 The gradient consistency is determined and an optical flow confidence map conf is constructed. Based on conf, the optical flow U is filtered to obtain the robust optical flow U*.

[0069] Step 5, using F0, C0, C P Using robust optical flow U* as conditional input, a conditional diffusion model is established. Convolutionally encoded optical flow features are injected into the multi-scale feature layers of the diffusion model to apply motion alignment constraints to the denoising generation process, resulting in the initial fused image F at the target time. P2 ;

[0070] Step 6, transfer the initial fused image F P2 With upsampling C PThe input residuals are stitched together, the network predicts and refines the residuals, and then the images are stacked to output a high-resolution fused image of the target time. P .

[0071] The proposed method (STF-FlowDiff) was experimentally analyzed on the public dataset CIA and compared with six existing representative methods, including the flexible spatiotemporal fusion method of satellite images of different resolutions (FSDAF), the variation-based spatiotemporal data fusion method (VSDF), the single-prior image remote sensing spatiotemporal fusion method based on generative adversarial networks (OPGAN), the remote sensing image spatiotemporal fusion method based on conditional generative adversarial networks (GAN-STFM), the remote sensing spatiotemporal fusion method based on two-stream Transformer (SwinSTFM), and the remote sensing spatiotemporal fusion method based on diffusion models (STFDiff).

[0072] To evaluate the effectiveness of different spatiotemporal fusion methods, this invention employs six quantitative metrics to assess prediction accuracy. Root mean square error (RMSE) is used to measure pixel-level deviation between the reference image and the fused image; structural similarity index (SSIM) and universal image quality index (UIQI) are used to evaluate the structural fidelity between the ground image and the fused result; normalized global error (ERGAS) is used to characterize the overall spectral distortion at the global scale; and peak signal-to-noise ratio (PSNR) is used to quantify reconstruction quality. For ease of comparison, the best value for each metric in this embodiment is indicated in bold, and the second-best value is indicated by underline. Figure 2 The fusion results of different spatiotemporal methods are presented: (a) reference image; (b) flexible spatiotemporal fusion of satellite images of different resolutions (FSDAF); (c) variation-based spatiotemporal data fusion method (VSDF); (d) single-prior image remote sensing spatiotemporal fusion method based on generative adversarial networks (OPGAN); (e) remote sensing image spatiotemporal fusion method based on conditional generative adversarial networks (GAN-STFM); (f) remote sensing spatiotemporal fusion method based on dual-stream Transformer (SwinSTFM); (g) remote sensing spatiotemporal fusion method based on diffusion model (STFDiff); and (h) the STF-FlowDiff method of this invention. Table 1 lists the quantitative evaluation of the fusion results of different spatiotemporal fusion methods. Combined with the tables, it can be seen that the STF-FlowDiff method of this invention performs best in terms of RMSE, UIQI, and ERGAS, demonstrating its advantages in reducing numerical errors and improving overall image quality. Band-specific analysis further confirms its consistent advantage across all spectral bands, especially the significant improvements in the SWIR-1 and SWIR-2 bands. This demonstrates that STF-FlowDiff can effectively simulate complex spectral characteristics.

[0073] Table 1. Quantitative Evaluation of Fusion Results from Different Spatiotemporal Fusion Methods

[0074]

[0075] To evaluate the performance of different spatiotemporal fusion methods in downstream tasks, unsupervised classification using Gaussian Mixture Model (GMM) was performed on the fusion results. Figure 3 The table shows the classification diagrams obtained by fusing images using different spatiotemporal methods: (a) is the reference image; (b) is the flexible spatiotemporal fusion method of satellite images of different resolutions (FSDAF); (c) is the variation-based spatiotemporal data fusion method (VSDF); (d) is the single-prior image remote sensing spatiotemporal fusion method based on generative adversarial networks (OPGAN); (e) is the remote sensing image spatiotemporal fusion method based on conditional generative adversarial networks (GAN-STFM); (f) is the remote sensing spatiotemporal fusion method based on dual-stream Transformer (SwinSTFM); (g) is the remote sensing spatiotemporal fusion method based on diffusion model (STFDiff); and (h) is the STF-FlowDiff of this invention. Table 2 also lists the corresponding classification accuracies. As can be seen from the figure, among all the comparison methods, STF-FlowDiff has the best overall classification performance. Its Kappa value and mean intersection-over-union ratio (mIoU) value are the highest, and its overall accuracy (OA) and mean accuracy (AA) value are also the second highest. This indicates that STF-FlowDiff provides more reliable class-level discrimination and better overall classification consistency.

[0076] Table 2. Classification accuracy of images fused using different spatiotemporal methods

[0077]

[0078] Table 3 Ablation Experiment Results

[0079]

[0080] In addition, to demonstrate the effectiveness of the optical flow estimation module, confidence assessment module, and multi-scale feature injection module required by this invention, ablation experiments were conducted. The specific experimental results are shown in Table 3, proving that the performance of the method of this invention is better than other similar methods.

Claims

1. A spatiotemporal fusion method based on optical flow-guided diffusion models, characterized in that, Includes the following steps: Step 1: Acquire the fine spatial resolution image F0 of the same observation area at reference time t0, the coarse spatial resolution image C0 at reference time t0, and the image at the target time t0. p Coarse spatial resolution image C P ; Step 2: Upsample the coarse spatial resolution image to obtain an upsampled image at the same scale as F0, and construct a pseudo fine resolution image F based on edge-preserving smoothness constraints. P1 ; Step 3, based on the optical flow estimation network, analyze F0 and F P1 Estimate the pixel-level displacement optical flow to obtain the target-scale optical flow U; Step 4, calculate F0 and F P1 The gradient consistency is determined, and a confidence map conf of the optical flow is constructed, which is calculated based on the magnitude of the optical flow gradient. ,in, It is an exponential function. σ is the total gradient magnitude, i and j are hyperparameters for adjusting gradient sensitivity, i and j represent the rows and columns in the matrix, respectively; the optical flow U is filtered according to conf, and for low confidence regions, a gating mechanism is used to reduce the optical flow contribution or neighborhood propagation is used for smoothing to obtain robust optical flow U*. Step 5, using F0, C0, C P Using U* as the conditional input, a conditional diffusion model is established. Convolutionally encoded optical flow features are injected into the multi-scale feature layers of the diffusion model to apply motion alignment constraints to the denoising generation process, resulting in the initial fused image F at the target time. P2 ; Step 6, F P2 With upsampling C P The image is stitched along the channel dimension, the input residual refinement network predicts and refines the residuals, and then overlays them to output a high-resolution fused image F at the target time. P .

2. The spatiotemporal fusion method based on optical flow-guided diffusion model according to claim 1, characterized in that, The specific process of constructing the optical flow confidence map in step 4 is as follows: Calculate F0 and F within a local window. P1 The gradient magnitude difference and gradient direction consistency are obtained and mapped to the [0,1] interval to obtain the initial confidence level. Then, the confidence level is multiplied by the confidence level factor based on the optical flow gradient magnitude to obtain the final confidence level map.

3. The spatiotemporal fusion method based on optical flow-guided diffusion model according to claim 1, characterized in that, The diffusion model in step 5 recovers the fine-scale image at the target time using a progressively denoised generation method.

4. The spatiotemporal fusion method based on optical flow-guided diffusion model according to claim 1 or 3, characterized in that, During the training of the diffusion model, noise is added to the target high-resolution sample to form a multi-stage state, and the denoising network is trained under the condition {F0, C0, C...} P } Predict noise or residuals under robust optical flow U*; during inference, start from random noise or prior initialization, and generate the initial fused image F through several denoising steps. P2 Meanwhile, convolutionally encoded optical flow features are injected into the multi-scale feature layers of the diffusion model. At each encoder scale, the mechanism injects the flow estimate corresponding to that scale into the feature channel.

5. The spatiotemporal fusion method based on optical flow-guided diffusion model according to claim 4, characterized in that, For the input features of each encoder layer, optical flow pooling is first performed to the corresponding scale, and 3×3 convolution is applied to adjust the features before they are injected into the current feature map.

6. The spatiotemporal fusion method based on optical flow-guided diffusion model according to claim 1, characterized in that, The optical flow estimation process is as follows: C is estimated using bilinear interpolation. P Upsampling is performed, and thinning is done using an edge-preserving smoothing filter. This pseudo-image shares the same spatial resolution as F0 and is able to achieve the desired result in a t-value. p Estimating optical flow in situations where high-resolution observation data is lacking.

7. The spatiotemporal fusion method for diffusion models based on optical flow guidance according to claim 1, characterized in that, The pseudo-high resolution image F in step 2 P1 The construction process is as follows: using C P The upsampling result is used as the initial value, and iterative optimization is performed to maintain C at low frequencies. P The radiation trend is observed, while maintaining a gradient structure similar to F0 in the edge region, resulting in a pseudo-fine resolution image F. P1 .

8. A spatiotemporal fusion system based on optical flow-guided diffusion models, used to execute the spatiotemporal fusion method based on optical flow-guided diffusion models according to any one of claims 1-6, characterized in that, include: Optical flow estimation module: for F0 and F P1 Estimate the target-scale optical flow U; Confidence assessment module: Calculates the optical flow confidence map conf based on gradient consistency, and filters the target-scale optical flow U based on conf to obtain robust optical flow U*; Multi-scale feature injection module: downsamples U* to the spatial scale corresponding to each encoder layer of the diffusion network, and after convolutional encoding, performs residual fusion with the feature map of the current layer; Conditional diffusion fusion module: based on F0, C0, C P U* is the conditional input, and the initial fused image F is generated through progressive denoising. P2 ; Residual refinement module: F P2 With upsampling C P After stitching, predict the high-frequency residual map and output the final fused image F. P .

9. The spatiotemporal fusion system based on optical flow-guided diffusion model according to claim 8, characterized in that, The confidence assessment module is designed with a gradient-based confidence estimation module (GFC). GFC measures the intensity of local changes by calculating the spatial gradients of two-dimensional optical flow, horizontal optical flow, and vertical optical flow.

10. The spatiotemporal fusion system based on optical flow-guided diffusion model according to claim 8, characterized in that, The multi-scale feature injection module injects the flow estimate corresponding to each encoder scale into the feature channel, maintaining spatial correspondence and displacement trend throughout the encoding process, from fine-grained to coarse-grained scales.

Images