A method and system for generating image sequences based on spatiotemporal diffusion multi-source data fusion

Abstract

This application discloses a method and system for generating image sequences based on spatiotemporal diffusion multi-source data fusion, belonging to the field of image sequence generation technology. The method includes training a synchronous mapping network using existing satellite remote sensing images and existing radar echo reflectance images as input and existing lightning strike point spatial distribution maps as output; training a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images; inputting the future satellite remote sensing images and future radar echo reflectance images into the trained synchronous mapping network to obtain a first predicted image of the future lightning strike point spatial distribution map; inputting the existing lightning strike point spatial distribution map into the spatiotemporal diffusion model to obtain a second predicted image of the future lightning strike point spatial distribution map; and stitching the first and second predicted images together to obtain the final predicted image sequence of the future lightning strike point spatial distribution map, significantly improving prediction accuracy and robustness.

Description

Technical Field

[0001] This application belongs to the field of image sequence generation technology, specifically relating to a method and system for generating image sequences based on spatiotemporal diffusion multi-source data fusion. Background Technology

[0002] Existing spatiotemporal point process models typically compromise with the assumption of conditional independence between time and space, or only allow space to depend on time unilaterally (e.g., predicting spatial location given a time). This dependency constraint severely weakens the model's ability to capture the complexity of spatiotemporal interactions under historical event conditions, thus affecting predictive performance. Secondly, the joint spatiotemporal distribution of events has a vast sample space, making direct fitting extremely difficult. Existing methods often decompose the target distribution into conditional dependency distributions (e.g., fitting temporal density and conditional spatial density separately), but the representation of conditional spatial density is often limited by specific model structures with weak expressive power, such as kernel density estimation (KDE) and continuous normalized flow (CNF). This structural constraint makes it difficult for models to effectively capture the complex coupling correlations between time and space during event occurrence. Furthermore, most existing methods require complex integration operations when calculating likelihood, or must restrict the intensity function to an integrable form, sacrificing computational efficiency for accuracy.

[0003] At the level of deeply integrated artificial intelligence algorithms, a thorough analysis of their mechanisms reveals that state-of-the-art spatiotemporal point prediction models largely aim to generate deterministic predictions, thereby minimizing mean squared error. However, in systems involving multiple physical processes and a large number of dynamic variables, although prediction metrics may improve, they lack the quantification of uncertainty. This deficiency is gradually amplified over a longer lead time and affects the model's ability to describe the joint distribution, thus mechanistically limiting further improvements in prediction accuracy. Summary of the Invention

[0004] Purpose of the invention: This application develops a method and system for generating image sequences based on spatiotemporal diffusion multi-source data fusion, aiming to solve the technical problem of low prediction accuracy in predicting future events in systems involving multiple physical processes and a large number of dynamic variables in the prior art.

[0005] Technical Solution: In a first aspect, this application provides a method for generating image sequences based on spatiotemporal diffusion multi-source data fusion, applied to lightning strike point prediction, including:

[0006] Using existing satellite remote sensing images and existing radar echo reflectance images as inputs, and existing lightning strike spatial distribution maps as outputs, a synchronous mapping network is trained.

[0007] A prediction network is trained using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images;

[0008] The future satellite remote sensing image and the future radar echo reflectance image are input into the trained synchronous mapping network to obtain the first prediction map of the future spatial distribution map of lightning strike points.

[0009] Input the existing spatial distribution map of lightning strike points into the spatiotemporal diffusion model to obtain a second predicted map of the future spatial distribution map of lightning strike points;

[0010] By stitching together the first prediction image and the second prediction image, a final predicted image sequence of the spatial distribution map of the future lightning strike points is obtained.

[0011] In some embodiments, the step of inputting the existing spatial distribution map of lightning strike points into a spatiotemporal diffusion model to obtain a second predicted map of the future spatial distribution map of lightning strike points includes:

[0012] Determine the time interval;

[0013] Based on the time interval, the existing spatial distribution map of the lightning strike points is divided to obtain an existing spatiotemporal event image;

[0014] Based on existing spatiotemporal event images, obtain existing spatiotemporal event image sequences; the representation formula of the spatiotemporal event image sequences includes:

[0015] ;

[0016] in, The spatiotemporal event image sequence; For images containing spatial information A matrix diagram is used to represent the corresponding time step. spatiotemporal event images, , The number of time steps; Used for characterization Corresponding time step The value at the specified spatial location, express Corresponding time step A lightning event occurs at a specified spatial location. express Corresponding time step No lightning event occurred at the specified spatial location;

[0017] The existing spatiotemporal event image sequence is input into the spatiotemporal diffusion model to predict future spatiotemporal event images, which are the second prediction images.

[0018] In some embodiments, the spatiotemporal diffusion model includes:

[0019] A spatiotemporal self-attention encoder, wherein the spatiotemporal self-attention encoder is used to acquire historical state representations;

[0020] A spatiotemporal diffusion process module, comprising a forward noise addition unit and a reverse noise reduction unit;

[0021] The existing spatial distribution map of lightning strike points includes the current spatial distribution map of lightning strike points and the historical spatial distribution map of lightning strike points. The step of predicting future spatiotemporal event images includes:

[0022] Obtain the current spatiotemporal event image based on the spatial distribution map of lightning strike points;

[0023] Forward diffusion modeling of the current spatiotemporal event image is performed using Markov processes in the time and spatial domains, and Gaussian noise is added in each diffusion step to convert the current spatiotemporal event image into pure Gaussian noise.

[0024] Based on the historical spatial distribution map of lightning strike points, historical spatiotemporal event images are obtained, and an effective representation of the historical spatiotemporal event images is obtained, which is a historical state representation;

[0025] Using the historical state representation as a condition, the pure Gaussian noise is inversely denoised to obtain the original spatiotemporal information of the future spatial distribution of lightning strike points;

[0026] The image of the future spatiotemporal event is predicted based on the original spatiotemporal information.

[0027] In some embodiments, the characterization formula for inverse denoising includes:

[0028] ;

[0029] ;

[0030] ;

[0031] in, To reverse denoising to diffusion steps Subsequent spatiotemporal information images; To reverse denoising to diffusion steps The spatial information image afterward; To reverse denoising to diffusion steps The image contains the time information afterward; For the number of diffusion steps The diffusion parameters are used to control the noise attenuation intensity and the signal retention ratio; For the number of diffusion steps The diffusion parameters are used to control the weights of the inverse denoising output and balance the correction magnitude; Modeling diffusion up to the number of diffusion steps The spatial information image afterward; Modeling diffusion up to the number of diffusion steps The image contains the time information afterward; For the front Diffusion parameters during the reverse diffusion process The product; This refers to the inverse denoising unit; Modeling diffusion up to the number of diffusion steps Subsequent spatiotemporal information images; Representing historical states; The time step is used to characterize the number of diffusion steps.

[0032] In some embodiments, prior to inverse denoising, the method further includes:

[0033] Based on the inverse denoising result of the previous diffusion step and the historical state representation, prediction noise is obtained, which includes spatial prediction noise and temporal prediction noise; the characterization formula of the prediction noise includes:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] in, To reverse denoising to diffusion steps Subsequent spatial prediction noise; To reverse denoising to diffusion steps Subsequent time prediction noise; Weights generated for the spatial attention mechanism; Weights generated for the time attention mechanism; This is the spatiotemporal information image of the current inverse denoising step; This is the spatial information image of the current inverse denoising step; This is the temporal information image of the current inverse denoising step; For activation functions; This is the weight matrix for linear transformation of spatial information images; To reverse denoising to diffusion steps The spatial information image afterward; This is the weight matrix for linear transformation of the historical hidden space representation; Representing the hidden space of history; These are the learnable parameters for linear projection; This is the weight matrix for performing a linear transformation on the time-informed image; To reverse denoising to diffusion steps The image contains the time information afterward; This is the weight matrix for linearly transforming the historical hidden time representation; To hide the timeline of history; For the reverse denoising step Location encoding; For sinusoidal position embedding operation; For activation functions; Representing historical states; and This is a weight matrix used to represent historical states. and position encoding Converted into attention weights; For splicing operations;

[0043] The next cycle of inverse denoising is performed based on the predicted noise.

[0044] In some embodiments, the step of obtaining the historical state representation includes:

[0045] Convert the temporal information in historical spatiotemporal event images into temporal embeddings, and convert the spatial information in historical spatiotemporal event images into spatial embeddings.

[0046] Based on the temporal embedding and the spatial embedding, a spatiotemporal embedding is obtained;

[0047] Based on multiple historical spatiotemporal event images, multiple temporal embedding sequences, spatial embedding sequences, and spatiotemporal embedding sequences are obtained;

[0048] The time embedding sequence is input into the time self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden time representation;

[0049] The spatial embedding sequence is input into the spatial self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden spatial representation;

[0050] The spatiotemporal embedding sequence is input into the spatiotemporal self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden spatiotemporal representation;

[0051] The historical state representation is obtained based on the hidden time representation, the hidden space representation, and the hidden spatiotemporal representation.

[0052] In some embodiments, the inverse denoising unit includes a collaborative attention denoising network, and the step of training the collaborative attention denoising network includes:

[0053] Historical spatiotemporal event images are obtained based on the spatial distribution map of lightning strike points described in history;

[0054] The historical spatiotemporal event images are modeled by diffusion using Markov processes in the time and spatial domains to add Gaussian noise and obtain random noise.

[0055] The random noise is used as the input data of the collaborative attention denoising network, and the historical spatiotemporal event image is used as the target data of the collaborative attention denoising network to train the network. The training of the collaborative attention denoising network employs a weighted mean square error loss function, the representation formula of which includes:

[0056] ;

[0057] in, Let the weighted mean square error loss function be used. For the expectation, for , used to characterize the The original spatiotemporal representation of a lightning strike event, The step size of the time step. This is real noise. The number of times noise has been added so far. Used to characterize the original spatiotemporal representation under the original data distribution. and real noise Calculate the average loss to ensure the model's generalization ability; The inverse denoising unit is used to characterize the predicted noise; Noise figure; This represents the historical state.

[0058] In some embodiments, the step of training a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images includes:

[0059] Determine the prediction network that includes a convolutional gating mechanism;

[0060] The existing satellite remote sensing images and the existing radar echo reflectance images are input into the prediction network;

[0061] Based on the convolutional gating mechanism, the prediction network is trained using the existing satellite remote sensing images and the existing radar echo reflectance images to obtain spatiotemporal features for characterizing the spatiotemporal evolution of weather systems.

[0062] Based on the aforementioned spatiotemporal characteristics, future satellite remote sensing images and future radar echo reflectance images are obtained, and their characterization formulas include:

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] in, For input gates; Forgotten Gate; For output gate; Candidate cell state; This is the Sigmoid function, used to compress any numerical value to the range (0,1). Indicates the current time step The input feature map is used to characterize satellite remote sensing images or radar echo reflectance images; For the current time step The hidden state; For the previous time step The hidden state; For the current time step The unit state; For the previous time step The unit state; Input feature map The convolution kernel weights corresponding to the output gate; Input feature map The convolution kernel weights corresponding to the candidate unit states; Input feature map The convolution kernel weights corresponding to the input gates; Input feature map The convolution kernel weights corresponding to the forget gate; The convolution kernel weights are the values ​​corresponding to the input gates in the hidden states; Hidden state The convolution kernel weights corresponding to the forget gate; Hidden state The convolution kernel weights corresponding to the output gate; Hidden state The convolution kernel weights corresponding to the candidate unit states; These are the bias parameters for the input gate; These are the bias parameters for the forget gate; These are the bias parameters for the output gate; The bias parameter for the candidate cell state; This is a convolution operation; For Hadamah accumulation; It is the hyperbolic tangent function.

[0070] In some embodiments, the synchronous mapping network includes a Unet encoder-decoder convolutional network, and the step of obtaining a first predicted map of the future spatial distribution map of lightning strike points includes:

[0071] By stitching together the satellite remote sensing images and the radar echo reflectance images described in the future, multi-channel input data can be obtained;

[0072] The multi-channel input data is downsampled and convolved by the Unet encoder-decoder convolutional network to extract high-level semantic features and obtain a first feature map.

[0073] The first feature map is upsampled and transposed convolutional to restore spatial resolution and obtain the second feature map;

[0074] The second feature map is processed using skip connections and activation functions to obtain the first prediction map.

[0075] Secondly, embodiments of this application also provide a multi-source data fusion image sequence generation system based on spatiotemporal diffusion, applied to lightning strike point prediction, including:

[0076] A mapping training module is used to train a synchronous mapping network by taking existing satellite remote sensing images and existing radar echo reflectivity images as inputs and existing lightning strike point spatial distribution maps as outputs.

[0077] A prediction training module is used to train a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images.

[0078] The first prediction module is used to input the future satellite remote sensing image and the future radar echo reflectance image into the trained synchronous mapping network to obtain a first prediction map of the future spatial distribution map of lightning strike points.

[0079] The second prediction module is used to input the existing spatial distribution map of lightning strike points into the spatiotemporal diffusion model to obtain a second prediction map of the future spatial distribution map of lightning strike points.

[0080] A stitching module is used to stitch together the first prediction image and the second prediction image to obtain a final predicted image sequence of the spatial distribution map of the future lightning strike points.

[0081] Beneficial Effects: Compared with existing technologies, the present application provides a method for generating multi-source data fusion image sequences based on spatiotemporal diffusion. This method includes training a synchronous mapping network using existing satellite remote sensing images and existing radar echo reflectance images as input and existing lightning strike point spatial distribution maps as output; training a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images; inputting the future satellite remote sensing images and future radar echo reflectance images into the trained synchronous mapping network to obtain a first predicted image of the future lightning strike point spatial distribution map; inputting the existing lightning strike point spatial distribution map into a spatiotemporal diffusion model to obtain a second predicted image of the future lightning strike point spatial distribution map; and stitching the first and second predicted images together to obtain the final predicted image sequence of the future lightning strike point spatial distribution map. This application treats lightning strike point prediction as a spatiotemporal point process. By introducing a diffusion model, it directly learns the spatiotemporal joint distribution of lightning events, accurately quantifies the uncertainty in lightning forecasting, and obtains preliminary future lightning strike point spatial distribution images at shorter time intervals. By training a synchronous mapping network using satellite remote sensing images and radar echo reflectance images, preliminary spatial distribution images of future lightning strike points are obtained. The location data from both sources are then stitched together to obtain the final location result. This demonstrates that the method provided in this application can significantly improve the accuracy and robustness of future event predictions in systems with multiple physical processes and a large number of dynamic variables. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 A flowchart illustrating the steps of the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment;

[0084] Figure 2 A flowchart illustrating the steps of obtaining the second prediction image in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment;

[0085] Figure 3 A flowchart illustrating the steps of predicting future spatiotemporal event images in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment;

[0086] Figure 4 A diagram illustrating the steps for obtaining historical state representation in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment;

[0087] Figure 5 A flowchart illustrating the steps of training a collaborative attention denoising network in a multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment;

[0088] Figure 6 A flowchart illustrating the steps of obtaining a first prediction map of the spatial distribution map of future lightning strike points in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment;

[0089] Figure 7 A module connection diagram of a multi-source data fusion image sequence generation system based on spatiotemporal diffusion provided in this application embodiment;

[0090] Figure 8 A flowchart illustrating the method for generating image sequences based on spatiotemporal diffusion using multi-source data fusion, as provided in this application embodiment.

[0091] Figure 9 A flowchart illustrating the reverse denoising process in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment;

[0092] Figure labels: 10, mapping training module; 20, prediction training module; 30, first prediction module; 40, second prediction module; 50, splicing module. Detailed Implementation

[0093] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0094] Timely and accurate prediction of lightning occurrence and development, and quantifying the uncertainty of prediction (the probability of lightning occurrence and development trends), are crucial for guiding relevant industries to reduce disaster impacts and achieve cost-effectiveness trade-offs. They form the basis for severe convective weather forecasting, local weather process analysis, lightning disaster prevention, and disaster analysis, playing a vital role in strengthening emergency preparedness and protective measures. Due to the small-to-medium scale characteristics of weather systems, lightning forecasting particularly focuses on monitoring and nowcasting, placing high demands on the use of data with high spatiotemporal resolution from weather radar, meteorological satellites, and lightning detectors.

[0095] There are two main methods for generating image sequences for nowcasting lightning, heavy precipitation, and other weather events. One is a lightning diagnostic scheme based on NWP (Numerical Weather Prediction) output. This scheme incorporates gridded electrification parameterization or cloud microphysical parameters as surrogates for lightning, obtaining a gridded image sequence of future lightning locations. The other is an extrapolation-based scheme, using satellite, weather radar, NWP model output, and lightning location system measurement data, converting them into images at a certain resolution, and using them individually or in combination. With the rapid increase in modern observation sources, new methods for improving nowcasting of convective storms and lightning are constantly being explored.

[0096] Recent advances in machine learning and deep learning have significantly enhanced data reasoning capabilities and demonstrated immense potential in weather applications. For example, various machine learning models have been developed to characterize unresolved physical processes in coarse-scale climate models, improve medium-range weather forecasts, and process numerical weather model outputs. Deep learning techniques have been successfully applied to predicting the spatiotemporal evolution of convective precipitation based on radar image extrapolation, such as by introducing convolutional long short-term memory neural network models (ConvLSTM) and U-net models to predict the probability of precipitation or cloud-to-ground (CG) lightning using weather radar, satellite, or joint observation data.

[0097] However, existing methods have significant limitations in handling the spatiotemporal dependence and uncertainty quantification in lightning forecasting. Spatiotemporal point processes, as a framework for describing random events occurring in time and space, are widely used in fields such as earthquakes, disease transmission, and urban transportation. Although existing spatiotemporal point process models, such as Poisson processes, Hawkes processes, self-tuning processes, and neural network-based Recurrent Labeled Time Point Processes (RMTPP), Neural Hawkes Processes (NHP), and Transformer Hawkes Processes (THP), have made progress in modeling event sequences, they face fundamental challenges in handling highly entangled spatiotemporal phenomena like lightning events.

[0098] First, existing spatiotemporal point process models typically compromise with the assumption of conditional independence between time and space, or only allow space to depend on time unilaterally (e.g., predicting spatial location given a time). This dependency constraint severely weakens the model's ability to capture the complexity of spatiotemporal interactions under historical event conditions, thus affecting prediction performance. Second, the spatiotemporal joint distribution of lightning events has a huge sample space, making direct fitting extremely difficult. Existing methods often decompose the target distribution into conditional dependency distributions (e.g., fitting temporal density and conditional spatial density separately), but the representation of conditional spatial density is often limited by specific model structures with weak expressive power, such as kernel density estimation (KDE) and continuous normalized flow (CNF). This structural constraint makes it difficult for models to effectively capture the complex coupling correlation between time and space during lightning occurrence. Furthermore, most existing methods require complex integration operations when calculating likelihood, or must restrict the intensity function to an integrable form, sacrificing computational efficiency while pursuing accuracy.

[0099] At the level of deeply integrated artificial intelligence algorithms, a thorough analysis of their mechanisms reveals that state-of-the-art weather forecasting models largely aim to generate deterministic predictions, thereby minimizing mean squared error. However, weather systems involve multiple physical processes and a large number of dynamic variables. Although forecasting indicators have improved, they lack the quantification of uncertainty. This deficiency is gradually amplified over a longer lead time and affects the model's ability to describe joint distributions, thus mechanistically limiting further improvements in forecast accuracy.

[0100] In view of this, this application provides a method for generating image sequences based on spatiotemporal diffusion multi-source data fusion. The method includes training a synchronous mapping network using existing satellite remote sensing images and existing radar echo reflectance images as input, and an existing spatial distribution map of lightning strike points as output; training a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images; inputting the future satellite remote sensing images and future radar echo reflectance images into the trained synchronous mapping network to obtain a first predicted map of the future spatial distribution map of lightning strike points; inputting the existing spatial distribution map of lightning strike points into a spatiotemporal diffusion model to obtain a second predicted map of the future spatial distribution map of lightning strike points; and stitching the first and second predicted maps together to obtain a final predicted image sequence of the future spatial distribution map of lightning strike points. This application treats lightning strike point prediction as a spatiotemporal point process. By introducing a diffusion model, it directly learns the spatiotemporal joint distribution of lightning events, accurately quantifies the uncertainty in lightning forecasting, and obtains preliminary spatial distribution images of future lightning strike points at shorter time intervals. By training a synchronous mapping network using satellite remote sensing images and radar echo reflectance images, preliminary spatial distribution images of future lightning strike points are obtained. The location data from both sources are then stitched together to obtain the final location result. This demonstrates that the method provided in this application can significantly improve the accuracy and robustness of future event predictions in systems with multiple physical processes and a large number of dynamic variables.

[0101] In some embodiments, please refer to Figure 1 and Figure 8 , Figure 1 This is a flowchart illustrating the steps of the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in an embodiment of this application. Figure 8 This is a flowchart of the method for generating image sequences based on spatiotemporal diffusion multi-source data fusion provided in this application embodiment. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion provided in this application embodiment is specifically implemented through steps 100 to 500:

[0102] Step 100: Train the synchronous mapping network using existing satellite remote sensing images and existing radar echo reflectance images as inputs and existing lightning strike point spatial distribution maps as outputs.

[0103] Step 200: Train the prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images.

[0104] In some embodiments, the method for obtaining future satellite remote sensing images and future radar echo reflectance images in this application includes:

[0105] Identify prediction networks that incorporate convolutional gating mechanisms.

[0106] Input existing satellite remote sensing images and existing radar echo reflectance images into the prediction network.

[0107] Based on the convolutional gating mechanism, a prediction network is trained using existing satellite remote sensing images and existing radar echo reflectance images to obtain spatiotemporal features that characterize the spatiotemporal evolution of weather systems.

[0108] The formulas for representing future satellite remote sensing images and future radar echo reflectance images obtained based on spatiotemporal features include:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] in, For input gates; Forgotten Gate; For output gate; Candidate cell state; This is the Sigmoid function, used to compress any numerical value to the range (0,1). Indicates the current time step The input feature map is used to characterize satellite remote sensing images or radar echo reflectance images; For the current time step The hidden state; For the previous time step The hidden state; For the current time step The unit state; For the previous time step The unit state; Input feature map The convolution kernel weights corresponding to the output gate; Input feature map The convolution kernel weights corresponding to the candidate unit states; Input feature map The convolution kernel weights corresponding to the input gates; Input feature map The convolution kernel weights corresponding to the forget gate; Hidden state The convolution kernel weights corresponding to the input gates; Hidden state The convolution kernel weights corresponding to the forget gate; Hidden state The convolution kernel weights corresponding to the output gate; Hidden state The convolution kernel weights corresponding to the candidate unit states; These are the bias parameters for the input gate; These are the bias parameters for the forget gate; These are the bias parameters for the output gate; The bias parameter for the candidate cell state; This is a convolution operation; For Hadamah accumulation; It is the hyperbolic tangent function.

[0116] Specifically, this application uses a ConvLSTM convolutional long short-term memory network to extract spatiotemporal features through a convolutional gating mechanism.

[0117] Step 300: Input the future satellite remote sensing image and the future radar echo reflectance image into the trained synchronous mapping network to obtain the first prediction map of the spatial distribution of future lightning strike points.

[0118] In some embodiments, the synchronization mapping network includes a Unet encoder-decoder convolutional network; see [link to documentation]. Figure 6 , Figure 6 This is a flowchart illustrating the steps of obtaining a first predicted image of the spatial distribution map of future lightning strike points in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment. The method for obtaining the first predicted image of the spatial distribution map of future lightning strike points in this application is specifically implemented through steps 310 to 340:

[0119] Step 310: Stitch together future satellite remote sensing images and future radar echo reflectance images to obtain multi-channel input data.

[0120] Step 320: Downsample the multi-channel input data using the Unet encoder-decoder convolutional network to extract high-level semantic features and obtain the first feature map.

[0121] Step 330: Perform upsampling and transpose convolution on the first feature map to restore the spatial resolution and obtain the second feature map.

[0122] Step 340: Process the second feature map based on skip connections and activation functions to obtain the first prediction map.

[0123] Specifically, this application concatenates the two features output by ConvLSTM into a multi-channel input, extracts high-level semantic features through a downsampling convolutional layer using a Unet encoder-decoder structure, restores spatial resolution through an upsampling transposed convolution, preserves details by combining skip connections, and finally generates a lightning probability map using a Sigmoid activation function in the output layer.

[0124] Step 400: Input the existing spatial distribution map of lightning strike points into the spatiotemporal diffusion model to obtain the second prediction map of the spatial distribution map of future lightning strike points.

[0125] In some embodiments, the spatiotemporal diffusion model includes:

[0126] Spatiotemporal self-attention encoder, used to obtain historical state representation;

[0127] The spatiotemporal diffusion process module includes a forward noise addition unit and a reverse noise reduction unit;

[0128] In some embodiments, the inverse denoising unit includes a collaborative attention denoising network, and the existing lightning strike spatial distribution map includes the current lightning strike spatial distribution map and the historical lightning strike spatial distribution map. In some embodiments, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating the steps of training a collaborative attention denoising network in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment. The method for training the collaborative attention denoising network in this application is specifically implemented through steps 401 to 403:

[0129] Step 401: Obtain historical spatiotemporal event images based on the spatial distribution map of historical lightning strike points.

[0130] Specifically, firstly, historical raw lightning events are collected, and then a sequence of spatiotemporal lightning event images is obtained based on these historical raw lightning events. sequence The time interval between spatiotemporal event images is 1 minute. , For images containing spatial location information A matrix diagram is used to represent the corresponding time step. spatiotemporal event images, , The number of time steps; Used for characterization Corresponding time step The value at the specified spatial location, express Corresponding time step A lightning event occurs at a specified spatial location. express Corresponding time step No lightning events occurred at the specified spatial location.

[0131] Step 402: Based on Markov processes in the time and space domains, perform diffusion modeling on historical spatiotemporal event images to add Gaussian noise and obtain random noise.

[0132] Specifically, through a forward diffusion process, the diffusion of lightning events is modeled as a Markov process on an image pixel grid. The forward diffusion process models the diffusion of lightning events... Diffusion modeling is a Markov process in both the spatial and temporal domains. ,in, It is the number of diffusion steps. This represents the total number of diffusion steps. During this process, the spatiotemporal diffusion model gradually shifts towards the time interval of lightning events. and spatial location Independently add small amounts of Gaussian noise until... Completely becomes pure Gaussian noise , which is random noise.

[0133] Step 403: Use random noise as input data for the collaborative attention denoising network and historical spatiotemporal event image sequences as target data for the collaborative attention denoising network to train the collaborative attention denoising network.

[0134] Specifically, random noise The original spatiotemporal image sequence of lightning events was used as the input data to train the collaborative attention denoising network. The training employed a weighted mean squared error loss function.

[0135] ;

[0136] in, This is the weighted mean squared error loss function; For the expectation, for , used to characterize the The original spatiotemporal representation of a lightning strike event, The step size of the time step. This is real noise. The number of times noise has been added so far. Used to characterize the original spatiotemporal representation under the original data distribution. and real noise Calculate the average loss to ensure the model's generalization ability; This is an inverse denoising unit used to characterize the noise after prediction; Noise figure; This represents a historical state.

[0137] In some embodiments, please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps for obtaining the second prediction map in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment. The method for obtaining the second prediction map in this application is specifically implemented through steps 410 to 440:

[0138] Step 410: Determine the time interval.

[0139] Specifically, the time interval is set to 1 minute in this application.

[0140] Step 420: Divide the existing lightning strike point spatial distribution map based on time intervals to obtain existing spatiotemporal event images.

[0141] Specifically, the current original lightning event is obtained, and the current original lightning event is divided into segments based on a 1-minute time interval to obtain an existing spatiotemporal event image.

[0142] Step 430: Based on existing spatiotemporal event images, obtain existing spatiotemporal event image sequences.

[0143] In some embodiments, the representation formula for spatiotemporal event image sequences includes:

[0144] ;

[0145] in, A sequence of spatiotemporal event images; For images containing spatial information A matrix diagram is used to represent the corresponding time step. spatiotemporal event images, , The number of time steps; Used for characterization Corresponding time step The value at the specified spatial location, express Corresponding time step A lightning event occurs at a specified spatial location. express Corresponding time step No lightning events occurred at the specified spatial location.

[0146] Step 440: Input the existing spatiotemporal event image sequence into the spatiotemporal diffusion model to predict future spatiotemporal event images, which are the second prediction images.

[0147] Specifically, the existing spatial distribution map of lightning strike points includes the current spatial distribution map of lightning strike points and the historical spatial distribution map of lightning strike points. Please refer to [link / reference]. Figure 3 and Figure 9, Figure 3 This is a flowchart illustrating the steps of predicting future spatiotemporal event images in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment. Figure 9 This is a flowchart of the inverse denoising process in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in the embodiments of this application. The method for predicting future spatiotemporal event images in this application is specifically implemented through steps 441 to 445:

[0148] Step 441: Obtain the current spatiotemporal event image based on the current spatial distribution map of lightning strike points.

[0149] Step 442: Model the current spatiotemporal event image by forward diffusion using Markov processes in the time and spatial domains, and add Gaussian noise in each diffusion step to convert the current spatiotemporal event image into pure Gaussian noise.

[0150] Step 443: Obtain historical spatiotemporal event images based on the historical lightning strike spatial distribution map, and obtain an effective representation of the historical spatiotemporal event images as a historical state representation.

[0151] In some embodiments, please refer to Figure 4 , Figure 4 This is a flowchart illustrating the steps for obtaining historical state representation in the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application embodiment. In this application, the method for obtaining historical state representation is specifically implemented through steps 4431 to 4437:

[0152] Step 4431: Convert the temporal information in the historical spatiotemporal event images into temporal embeddings, and convert the spatial information in the historical spatiotemporal event images into spatial embeddings.

[0153] Specifically, timestamps in images of historical spatiotemporal events and spatial location Convert to time embedding and spatial embedding .

[0154] Step 4432: Obtain the spatiotemporal embedding based on temporal embedding and spatial embedding.

[0155] Specifically, the time of each lightning event is embedded and spatial embedding Addition to obtain spatiotemporal embedding .

[0156] Step 4433: Based on multiple historical spatiotemporal event images, obtain multiple time embedding sequences, spatial embedding sequences, and spatiotemporal embedding sequences.

[0157] Specifically, time-embedded sequences Spatial Embedded Sequence Spatiotemporal embedded sequence .

[0158] Step 4434: Embed the time sequence into the temporal self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden time representation.

[0159] Step 4435: Spatial self-attention module of spatial embedded sequence input spatiotemporal self-attention encoder is used to obtain historical hidden spatial representation.

[0160] Step 4436: Input the spatiotemporal embedding sequence into the spatiotemporal self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden spatiotemporal representation.

[0161] Step 4437: Obtain the historical state representation based on the hidden time representation, hidden spatial representation, and hidden spatiotemporal representation.

[0162] Specifically, historical state representation , Representing the hidden space of history, To hide the timeline of history, It hides the temporal and spatial representation of history.

[0163] Step 444: Using the historical state representation as a condition, perform inverse denoising on the pure Gaussian noise to obtain the original spatiotemporal information of the spatial distribution of future lightning strike points.

[0164] In some embodiments, prior to inverse denoising, the method further includes:

[0165] Based on the inverse denoising result of the previous diffusion step and the historical state representation, the prediction noise is obtained, which includes spatial prediction noise and temporal prediction noise; the characterization formula for the prediction noise includes:

[0166] ;

[0167] ;

[0168] ;

[0169] ;

[0170] ;

[0171] ;

[0172] ;

[0173] ;

[0174] in, To reverse denoising to diffusion steps Subsequent spatial prediction noise; To reverse denoising to diffusion steps Subsequent time prediction noise; Weights generated for the spatial attention mechanism; Weights generated for the time attention mechanism; This is the spatiotemporal information image of the current inverse denoising step; This is the spatial information image of the current inverse denoising step; This is the temporal information image of the current inverse denoising step; For activation functions; This is the weight matrix for linear transformation of spatial information images; To reverse denoising to diffusion steps The spatial information image afterward; This is the weight matrix for linear transformation of the historical hidden space representation; Representing the hidden space of history; These are the learnable parameters for linear projection; This is the weight matrix for performing a linear transformation on the time-informed image; To reverse denoising to diffusion steps The image contains the time information afterward; This is the weight matrix for linearly transforming the historical hidden time representation; To hide the timeline of history; For the reverse denoising step Location encoding; For sinusoidal position embedding operation; For activation functions; Representing historical states; and This is a weight matrix used to represent historical states. and position encoding Converted into attention weights; This is for splicing operations.

[0175] Inverse denoising is performed in the next cycle based on the predicted noise.

[0176] Understandably, the core component for achieving inverse denoising is the collaborative attention denoising network. This network is a denoising neural network whose key objective is to capture the complex interdependencies between the spatial and temporal domains, thereby significantly facilitating the learning of the spatiotemporal joint distribution. The collaborative attention denoising network shares the same network structure in each denoising step, receiving the values ​​generated by the previous denoising step. and Denoising steps with position coding and hidden representations from the spatiotemporal self-attention encoder As input, conditional denoising is achieved. The collaborative attention denoising network quantifies the temporal and spatial dependencies by computing mutual attention weights. Finally, the collaborative attention denoising network includes spatial attention and temporal attention, the outputs of which are the predicted noise values. and Based on these predicted noises and The spatiotemporal diffusion model can obtain the first Predicted value of step and The data is then fed back into a collaborative attention denoising network to iteratively predict and progressively approximate clean values ​​for the spatial and temporal dimensions. Through this iterative refinement process, the interdependence between time and space is adaptively and dynamically captured.

[0177] In some embodiments, the characterization formula for the inverse denoising process includes:

[0178] ;

[0179] ;

[0180] ;

[0181] in, To reverse denoising to diffusion steps Subsequent spatiotemporal information images; To reverse denoising to diffusion steps The spatial information image afterward; To reverse denoising to diffusion steps The image contains the time information afterward; For the number of diffusion steps The diffusion parameters are used to control the noise attenuation intensity and the signal retention ratio; For the number of diffusion steps The diffusion parameters are used to control the weights of the inverse denoising output and balance the correction magnitude; Modeling diffusion up to the number of diffusion steps The spatial information image afterward; Modeling diffusion up to the number of diffusion steps The image contains the time information afterward; For the front Diffusion parameters during the reverse diffusion process The product; This is a reverse noise reduction unit; Modeling diffusion up to the number of diffusion steps Subsequent spatiotemporal information images; Representing historical states; This represents the time step, used to characterize the number of diffusion steps.

[0182] Understandably, a pre-trained spatiotemporal self-attention encoder is used to obtain the hidden representations of past event image sequences. Subsequently, a diffusion model, starting from pure Gaussian noise and conditioned on the hidden representations, iteratively predicts the spatiotemporal information of the next lightning event.

[0183] Collaborative attention denoising network As the core denoiser, it receives the denoising results from the previous step. Hidden representation of historical image sequences and diffusion steps As input, by repeating the denoising process, the diffusion model can generate a set of trajectories for a series of future lightning events using different initial noise samples. This inherent probabilistic generation capability is key to the diffusion model's ability to quantify uncertainty, providing a comprehensive view of the possible distribution of lightning strike points, rather than just a single deterministic prediction. The raw spatiotemporal information of these predicted lightning events is then used to generate weather condition data for future time steps. Weather status data It includes predictions of future lightning strike locations or lightning approach trends. This rich probabilistic output is of significant value in guiding critical decisions such as disaster early warning and emergency response.

[0184] Step 445: Predict future spatiotemporal event images based on the original spatiotemporal information.

[0185] Step 500: Stitch together the first and second prediction images to obtain the final predicted image sequence of the spatial distribution map of future lightning strike points.

[0186] Specifically, this application inputs the stitched image into a CNN network, which contains multiple convolutional layers. Each convolutional layer uses a 3x3 convolutional kernel combined with a ReLU activation function for feature extraction. Next, pooling layers are added to reduce spatial dimensionality and retain important features. Then, through the stacking of multiple convolutional and pooling layers, high-level semantic features are extracted progressively. Finally, fully connected layers are used to map the features to the final prediction result, and the output layer uses a softmax function to generate a probability distribution. The entire CNN network is optimized through backpropagation and stochastic gradient descent, with cross-entropy loss as the loss function to ensure that the model can effectively learn the features fused from multi-source data during training. These features are then fused through the CNN network to obtain the final prediction result.

[0187] Understandably, the multi-source data fusion image sequence generation method based on spatiotemporal diffusion provided in this application includes training a synchronous mapping network using existing satellite remote sensing images and existing radar echo reflectance images as input and existing lightning strike point spatial distribution maps as output; training a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images; inputting the future satellite remote sensing images and future radar echo reflectance images into the trained synchronous mapping network to obtain a first predicted map of the future lightning strike point spatial distribution map; inputting the existing lightning strike point spatial distribution map into a spatiotemporal diffusion model to obtain a second predicted map of the future lightning strike point spatial distribution map; and stitching the first and second predicted maps together to obtain the final predicted image sequence of the future lightning strike point spatial distribution map. This application treats lightning strike point prediction as a spatiotemporal point process, and by introducing a diffusion model, directly learns the spatiotemporal joint distribution of lightning events, accurately quantifies the uncertainty in lightning forecasting, and obtains preliminary future lightning strike point spatial distribution images at shorter time intervals. Satellite remote sensing images and radar echo reflectance images are used to train a synchronous mapping network to obtain preliminary spatial distribution images of future lightning strike points. The positioning data from both sources are then stitched together to obtain the final positioning result. It is evident that the method provided in this application can significantly improve the accuracy and robustness of future event predictions in systems with multiple physical processes and a large number of dynamic variables.

[0188] Accordingly, embodiments of this application also provide a multi-source data fusion image sequence generation system based on spatiotemporal diffusion, applied to lightning strike point prediction. Please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a module connection diagram of a multi-source data fusion image sequence generation system based on spatiotemporal diffusion provided in this application. The multi-source data fusion image sequence generation system based on spatiotemporal diffusion provided in this application includes:

[0189] Mapping training module 10 is used to train a synchronous mapping network with existing satellite remote sensing images and existing radar echo reflectivity images as inputs and existing lightning strike point spatial distribution maps as outputs.

[0190] Prediction training module 20 is used to train a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images;

[0191] The first prediction module 30 is used to input future satellite remote sensing images and future radar echo reflectance images into the trained synchronous mapping network to obtain a first prediction map of the spatial distribution map of future lightning strike points.

[0192] The second prediction module 40 is used to input the existing spatial distribution map of lightning strike points into the spatiotemporal diffusion model to obtain a second prediction map of the spatial distribution map of future lightning strike points.

[0193] The stitching module 50 is used to stitch together the first prediction image and the second prediction image to obtain the final predicted image sequence of the spatial distribution map of future lightning strike points.

[0194] In some embodiments, the second prediction module 40 is specifically used for:

[0195] Determine the time interval;

[0196] Based on time intervals, the existing spatial distribution map of lightning strike points is divided to obtain images of existing spatiotemporal events;

[0197] Based on existing spatiotemporal event images, obtain existing spatiotemporal event image sequences; the representation formulas for spatiotemporal event image sequences include:

[0198] ;

[0199] in, A sequence of spatiotemporal event images; For images containing spatial information A matrix diagram is used to represent the corresponding time step. spatiotemporal event images, , The number of time steps; Used for characterization Corresponding time step The value at the specified spatial location, express Corresponding time step A lightning event occurs at a specified spatial location. express Corresponding time step No lightning event occurred at the specified spatial location;

[0200] The existing spatiotemporal event image sequence is input into the spatiotemporal diffusion model to predict future spatiotemporal event images, which are the second prediction images.

[0201] In some embodiments, the second prediction module 40 is specifically used for:

[0202] Obtain the current spatiotemporal event image based on the current spatial distribution map of lightning strike points;

[0203] Forward diffusion modeling of the current spatiotemporal event image is performed using Markov processes in the time and spatial domains, and Gaussian noise is added in each diffusion step to convert the current spatiotemporal event image into pure Gaussian noise.

[0204] Historical spatiotemporal event images are obtained based on the spatial distribution map of historical lightning strike points, and effective representations of these images are obtained to represent historical states.

[0205] Using historical state representation as a condition, inverse denoising is performed on pure Gaussian noise to obtain the original spatiotemporal information of the spatial distribution of future lightning strike points;

[0206] Predicting future spatiotemporal events based on original spatiotemporal information.

[0207] In some embodiments, the spatiotemporally diffused multi-source data fusion image sequence generation system is specifically used for:

[0208] Based on the inverse denoising result of the previous diffusion step and the historical state representation, the prediction noise is obtained, which includes spatial prediction noise and temporal prediction noise; the characterization formula for the prediction noise includes:

[0209] ;

[0210] ;

[0211] ;

[0212] ;

[0213] ;

[0214] ;

[0215] ;

[0216] ;

[0217] in, To reverse denoising to diffusion steps Subsequent spatial prediction noise; To reverse denoising to diffusion steps Subsequent time prediction noise; Weights generated for the spatial attention mechanism; Weights generated for the time attention mechanism; This is the spatiotemporal information image of the current inverse denoising step; This is the spatial information image of the current inverse denoising step; This is the temporal information image of the current inverse denoising step; For activation functions; This is the weight matrix for linear transformation of spatial information images; To reverse denoising to diffusion steps The spatial information image afterward; This is the weight matrix for linear transformation of the historical hidden space representation; Representing the hidden space of history; These are the learnable parameters for linear projection; This is the weight matrix for performing a linear transformation on the time-informed image; To reverse denoising to diffusion steps The image contains the time information afterward; This is the weight matrix for linearly transforming the historical hidden time representation; To hide the timeline of history; For the reverse denoising step Location encoding; For sinusoidal position embedding operation; For activation functions; Representing historical states; and This is a weight matrix used to represent historical states. and position encoding Converted into attention weights; For splicing operations;

[0218] Inverse denoising is performed in the next cycle based on the predicted noise.

[0219] In some embodiments, the second prediction module 40 is specifically used for:

[0220] Convert the temporal information in historical spatiotemporal event images into temporal embeddings, and convert the spatial information in historical spatiotemporal event images into spatial embeddings.

[0221] Based on temporal and spatial embedding, obtain spatiotemporal embedding;

[0222] Based on multiple historical spatiotemporal event images, multiple temporal embedding sequences, spatial embedding sequences, and spatiotemporal embedding sequences are obtained;

[0223] By inputting the time-embedded sequence into the temporal self-attention module of the spatiotemporal self-attention encoder, a historical hidden time representation is obtained.

[0224] The spatial self-attention module of the spatial embedding sequence input spatiotemporal self-attention encoder is used to obtain the historical hidden spatial representation;

[0225] Input the spatiotemporal embedding sequence into the spatiotemporal self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden spatiotemporal representation;

[0226] Historical state representations are obtained based on hidden temporal representation, hidden spatial representation, and hidden spatiotemporal representation.

[0227] This application has provided a detailed description of a method and system for generating image sequences based on spatiotemporal diffusion multi-source data fusion, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating image sequences based on spatiotemporal diffusion multi-source data fusion, characterized in that, Applications include lightning strike point prediction, including: Using existing satellite remote sensing images and existing radar echo reflectance images as inputs, and existing lightning strike spatial distribution maps as outputs, a synchronous mapping network is trained. The prediction network is trained using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images. The future satellite remote sensing image and the future radar echo reflectance image are input into the trained synchronous mapping network to obtain the first prediction map of the future spatial distribution map of lightning strike points. By inputting the existing spatial distribution map of lightning strike points into the spatiotemporal diffusion model, a second predicted map of the future spatial distribution map of lightning strike points is obtained: Determine the time interval; Based on the time interval, the existing spatial distribution map of the lightning strike points is divided to obtain an existing spatiotemporal event image; Based on existing spatiotemporal event images, obtain existing spatiotemporal event image sequences; the representation formula of the spatiotemporal event image sequences includes: S={x1,x2,…x L }; Wherein, S is the spatiotemporal event image sequence; x i For images containing spatial information s i A matrix diagram is used to represent the spatiotemporal event image at time step i, where i = 1, 2...L, and L is the number of time steps; s i Used to characterize x i The value of s at the specified spatial location corresponding to time step i. i =1 means x i A lightning event occurs at a specified spatial location at time step i. i =0 means x i There is no lightning event at the specified spatial location at time step i; The existing spatiotemporal event image sequence is input into the spatiotemporal diffusion model to predict future spatiotemporal event images, which are the second prediction images. By stitching together the first prediction image and the second prediction image, a final predicted image sequence of the spatial distribution map of the future lightning strike points is obtained.

2. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion according to claim 1, characterized in that, The spatiotemporal diffusion model includes: A spatiotemporal self-attention encoder, wherein the spatiotemporal self-attention encoder is used to acquire historical state representations; A spatiotemporal diffusion process module, comprising a forward noise addition unit and a reverse noise reduction unit; The existing spatial distribution map of lightning strike points includes the current spatial distribution map of lightning strike points and the historical spatial distribution map of lightning strike points. The step of predicting future spatiotemporal event images includes: Obtain the current spatiotemporal event image based on the spatial distribution map of lightning strike points; Forward diffusion modeling of the current spatiotemporal event image is performed using Markov processes in the time and spatial domains, and Gaussian noise is added in each diffusion step to convert the current spatiotemporal event image into pure Gaussian noise. Based on the historical spatial distribution map of lightning strike points, historical spatiotemporal event images are obtained, and an effective representation of the historical spatiotemporal event images is obtained, which is a historical state representation; Using the historical state representation as a condition, the pure Gaussian noise is inversely denoised to obtain the original spatiotemporal information of the future spatial distribution of lightning strike points; The image of the future spatiotemporal event is predicted based on the original spatiotemporal information.

3. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion according to claim 2, characterized in that, The characterization formula for the inverse denoising includes: in, The spatiotemporal information image after reverse denoising down to diffusion steps k-1; This is the spatial information image after reverse denoising down to diffusion steps k-1; The image is a temporal information image after reverse denoising down to diffusion steps k-1; α k β is a diffusion parameter related to the number of diffusion steps k, used to control the noise attenuation intensity and the signal retention ratio; k The diffusion parameter is related to the number of diffusion steps k, and is used to control the weights of the inverse denoising output and balance the correction magnitude. Spatial information image after diffusion modeling up to diffusion step k; A time-information image of diffusion modeling up to diffusion step k; It is the product of the diffusion parameters α during the first k steps of reverse diffusion; ∈ θ This refers to the inverse denoising unit; Model the spatiotemporal information image of diffusion up to diffusion step k; h i-1 is the historical state representation; k is the time step, used to characterize the number of diffusion steps.

4. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion according to claim 2, characterized in that, Before performing inverse denoising, the following is also included: Based on the inverse denoising result of the previous diffusion step and the historical state representation, prediction noise is obtained, which includes spatial prediction noise and temporal prediction noise; the characterization formula of the prediction noise includes: e k =SinusoidalPosEmb(k); α s =Softmax(W sa Concat(h i-1 ,And k )); α t =Softmax(W ta Concat(h i-1 ,And k )); in, This is the spatial prediction noise after inverse denoising down to diffusion steps k; For time prediction noise after inverse denoising up to diffusion step k; α s Weights generated for the spatial attention mechanism; α t Weights generated for the time attention mechanism; x i This is the spatiotemporal information image of the current inverse denoising step; x s,i The spatial information image of the current inverse denoising step; x t,i The image represents the temporal information of the current inverse denoising step; σ is the activation function; W s This is the weight matrix for linear transformation of spatial information images; The spatial information image after inverse denoising down to diffusion steps k+1; W sh h is the weight matrix for linearly transforming the historical hidden space representation; s,i-1 Representing the hidden space of history; b s b sh b t b th W is a learnable parameter for linear projection. t This is the weight matrix for performing a linear transformation on the time-informed image; The image is a temporal information image after reverse denoising down to diffusion step k+1; W th h is the weight matrix for linearly transforming the historical hidden time representation; t,i-1 For historical time-hiding representation; e k For the inverse denoising step k, position encoding is used; SinusoidalPosEmb is the sinusoidal position embedding operation; Softmax is the activation function; h i-1 Representing historical states; W sa and W ta This is the weight matrix, used to represent the historical states h. i-1 and position encoding e k Convert to attention weights; Concat is a concatenation operation; The next cycle of inverse denoising is performed based on the predicted noise.

5. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion according to claim 2, characterized in that, The steps for obtaining historical state representations include: Convert the temporal information in historical spatiotemporal event images into temporal embeddings, and convert the spatial information in historical spatiotemporal event images into spatial embeddings. Based on the time embedding and the spatial embedding, obtain the spatiotemporal embedding; Based on multiple historical spatiotemporal event images, multiple temporal embedding sequences, spatial embedding sequences, and spatiotemporal embedding sequences are obtained; The time embedding sequence is input into the time self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden time representation; The spatial embedding sequence is input into the spatial self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden spatial representation; The spatiotemporal embedding sequence is input into the spatiotemporal self-attention module of the spatiotemporal self-attention encoder to obtain the historical hidden spatiotemporal representation; The historical state representation is obtained based on the hidden time representation, the hidden space representation, and the hidden spatiotemporal representation.

6. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion according to claim 2, characterized in that, The inverse denoising unit includes a collaborative attention denoising network, and the steps for training the collaborative attention denoising network include: Historical spatiotemporal event images are obtained based on the spatial distribution map of lightning strike points described in history; The historical spatiotemporal event images are modeled by diffusion using Markov processes in the time and spatial domains to add Gaussian noise and obtain random noise. The random noise is used as the input data of the collaborative attention denoising network, and the historical spatiotemporal event image is used as the target data of the collaborative attention denoising network to train the network. The training of the collaborative attention denoising network employs a weighted mean square error loss function, the representation formula of which includes: in, Let the weighted mean square error loss function be used. For the expectation, For (τ) i ,s i ), used to characterize the original spatiotemporal representation of the i-th lightning strike event, τ is the time step size, ∈ represents the real noise, and k is the current number of times noise is added. Used to characterize the original spatiotemporal representation x under the original data distribution. 0 The average loss is calculated using the real noise ∈ to ensure the model's generalization ability; ∈ θ The inverse denoising unit is used to characterize the predicted noise; α k h is the noise figure. i-1 This represents the historical state.

7. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion according to claim 1, characterized in that, The step of training a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images includes: Determine the prediction network that includes a convolutional gating mechanism; The existing satellite remote sensing images and the existing radar echo reflectance images are input into the prediction network; Based on the convolutional gating mechanism, the prediction network is trained using the existing satellite remote sensing images and the existing radar echo reflectance images to obtain spatiotemporal features for characterizing the spatiotemporal evolution of weather systems. Based on the aforementioned spatiotemporal characteristics, future satellite remote sensing images and future radar echo reflectance images are obtained, and their characterization formulas include: i t =σ(W xi *X t +W hi *H t-1 +b i ); f t =σ(W xf *X t +W hf *H t-1 +b f ); o t =σ(W xo *X t +W ho *H t-1 +b o ); H t =o t ⊙tanh(C t ); Among them, i t For input gate; f t Forgotten Gate; o t For output gate; The candidate cell state is represented by σ, which is the Sigmoid function used to compress any value to the range (0,1); X t This represents the input feature map at the current time step t, used to characterize satellite remote sensing images or radar echo reflectance images; H t The hidden state at the current time step t; H t-1 This represents the hidden state at the previous time step t-1; C t The cell state at the current time step t; C t-1 This represents the cell state at the previous time step t-1; W xo Input feature map X t The corresponding convolutional kernel weights for the output gate; W xc Input feature map X t Convolutional kernel weights corresponding to candidate unit states; W xi Input feature map X t The convolution kernel weights corresponding to the input gate; W xf Input feature map X t The convolution kernel weights corresponding to the forget gate; W hi For the hidden state H t-1 The convolution kernel weights corresponding to the input gate; W hf For the hidden state H t-1 The convolution kernel weights corresponding to the forget gate; W ho For the hidden state H t-1 The corresponding convolutional kernel weights for the output gate; W hc For the hidden state H t-1 Convolutional kernel weights corresponding to candidate unit states; b i b is the bias parameter for the input gate; f b is the bias parameter for the forget gate; o b is the bias parameter for the output gate; c represents the bias parameters for the candidate cell state; * represents the convolution operation; ⊙ represents the Hadamard product; tanh represents the hyperbolic tangent function.

8. The method for generating image sequences based on spatiotemporal diffusion multi-source data fusion according to claim 1, characterized in that, The synchronous mapping network includes a Unet encoder-decoder convolutional network, and the step of obtaining the first predicted map of the future spatial distribution map of lightning strike points includes: By stitching together the satellite remote sensing images and the radar echo reflectance images described in the future, multi-channel input data can be obtained; The multi-channel input data is downsampled and convolved by the Unet encoder-decoder convolutional network to extract high-level semantic features and obtain a first feature map. The first feature map is upsampled and transposed convolutional to restore spatial resolution and obtain the second feature map; The second feature map is processed using skip connections and activation functions to obtain the first prediction map.

9. A multi-source data fusion image sequence generation system based on spatiotemporal diffusion, characterized in that, Applications include lightning strike point prediction, including: The mapping training module (10) is used to train a synchronous mapping network with existing satellite remote sensing images and existing radar echo reflectivity images as inputs and existing lightning strike point spatial distribution maps as outputs. The prediction training module (20) is used to train a prediction network using existing satellite remote sensing images and existing radar echo reflectance images to obtain future satellite remote sensing images and future radar echo reflectance images. The first prediction module (30) is used to input the future satellite remote sensing image and the future radar echo reflectance image into the trained synchronous mapping network to obtain the first prediction map of the future lightning strike point spatial distribution map; The second prediction module (40) is used to input the existing spatial distribution map of lightning strike points into the spatiotemporal diffusion model to obtain a second prediction map of the future spatial distribution map of lightning strike points. Determine the time interval; Based on the time interval, the existing spatial distribution map of the lightning strike points is divided to obtain an existing spatiotemporal event image; Based on existing spatiotemporal event images, obtain existing spatiotemporal event image sequences; the representation formula of the spatiotemporal event image sequences includes: S={x1,x2,…x L }; Wherein, S is the spatiotemporal event image sequence; x i For images containing spatial information s i A matrix diagram is used to represent the spatiotemporal event image at time step i, where i = 1, 2...L, and L is the number of time steps; s i Used to characterize x i The value of s at the specified spatial location corresponding to time step i. i =1 means x i A lightning event occurs at a specified spatial location at time step i. i =0 means x i There is no lightning event at the specified spatial location at time step i; The existing spatiotemporal event image sequence is input into the spatiotemporal diffusion model to predict future spatiotemporal event images, which are the second prediction images. The stitching module (50) is used to stitch together the first prediction image and the second prediction image to obtain the final predicted image sequence of the future lightning strike point spatial distribution map.

Images