Prediction method, system and device for space-time image sequence and storage medium

Abstract

The invention relates to the technical field of space-time image sequence prediction, in particular to a space-time image sequence-oriented prediction method, system and device and a storage medium, and the method comprises the steps: obtaining an image sequence data set in a past time period, and dividing the image sequence data set into a training set, a verification set and a test set; constructing a space-time image sequence prediction model, improving a loss function used by the space-time image sequence prediction model in a training process through a difference dispersion regular term, and performing iterative training on the space-time image sequence prediction model by taking the training set as input and taking minimization of the loss function value as a target to obtain iterative training parameters; and loading the iteration training parameters into the space-time image sequence prediction model, and operating the space-time image sequence prediction model for the to-be-predicted image sequence in the past time period to obtain the image sequence in the future time period. According to the method, more diversified video or image data can be processed, subtle changes between image sequences are effectively captured, and the prediction precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of spatiotemporal image sequence prediction technology, and in particular to a prediction method, system, device and storage medium for spatiotemporal image sequences. Background Art

[0002] Spatiotemporal image sequence prediction is a data analysis method that combines time and space dimensions, aiming to study the changes of data over time and its distribution characteristics in space. This technology has wide application value in many fields such as weather forecasting, traffic management, and environmental monitoring. With the rapid development of big data and machine learning technology, spatiotemporal prediction methods have achieved an organic combination of traditional physical models and data-driven models, thereby significantly improving the accuracy and efficiency of predictions and reducing time costs.

[0003] At present, the main structures used in spatiotemporal image sequence prediction models include recurrent neural networks (RNNs), convolutional neural networks (CNNs), Transformers, and generative models. Among them, RNNs have significant advantages in processing temporal information, while CNNs perform well in processing spatial information. Transformers can capture complex relationships within sequences through the self-attention mechanism, especially when dealing with long-distance dependencies. Generative models, such as generative adversarial networks (GANs) and diffusion models, predict future images from a generative perspective, providing new ideas for spatiotemporal image sequence prediction.

[0004] like Figure 1As shown in the figure, the SimVP model is a CNN-based video prediction model designed to simplify the model structure and improve the prediction performance. The main components of the SimVP model include the encoder (Spatial Encoder), the converter (Inception_UnetTranslator) and the decoder (SpatialDecoder). Among them, the encoder part stacks Ns ConvNormSiLU blocks, each block consists of Conv2d (convolution operation), GroupNorm (group normalization) and SiLU (activation function), which is used to extract the spatial features of the input video; the converter uses Nt Inception modules combined according to the Unet structure. The core feature of Unet is its U-shaped structure, which consists of two parts: the encoder and the decoder. The encoder part consists of the Inception module in the left half, which is used to further extract and encode spatial features. The decoder part consists of the Inception module in the right half, which is used to reconstruct future frames based on the encoded features. The Inception module consists of a bottleneck Conv2d and multiple parallel GroupConv2d operators, which captures the features of the input data at different scales through convolution kernels of different sizes; the decoder part stacks Ns unConvUpSampleNormSiLU blocks, each of which consists of deconvolution operations (unConv2d), pixel reordering upsampling (PixelShuffleUpSample), GroupNorm and SiLU, which is used to reconstruct future frames based on the output of the converter.

[0005] Although the SimVP model has achieved remarkable results in simplifying the structure and improving the prediction performance, it still has some limitations:

[0006] First, the ability to capture long-distance temporal dependencies is limited: the SimVP model is built entirely on CNN, and CNN may have difficulty capturing long-distance temporal dependencies when processing long-sequence video predictions. This results in the accuracy of the model's prediction results may decrease when predicting a large number of frame image sequences at the same time.

[0007] Second, insufficient frequency domain feature extraction: The SimVP model mainly extracts features from video or image sequences in the spatial domain, while ignoring the frequency domain features of single-frame images. This may lead to low prediction accuracy when the model processes image sequences with no fixed shape (such as weather changes) or relatively abstract meanings (such as document clustering).

[0008] Third, the generalization ability is insufficient: Due to the above limitations, the SimVP model may perform well on specific data sets. However, when faced with more complex or diverse video or image data, its generalization ability has not been fully verified, and the model's predictive performance may be affected.

[0009] In summary, the SimVP model, as a CNN-based video prediction model, has achieved remarkable results in simplifying the structure and improving the prediction performance. However, in order to further improve the prediction performance and generalization ability of the model, in-depth research and improvement are needed to address its limitations. Summary of the invention

[0010] In order to solve the above technical problems, the present invention provides a prediction method for spatiotemporal image sequences, which comprises the following steps:

[0011] S1: Obtain an image sequence dataset for a past time period, and divide the image sequence dataset into a training set, a validation set, and a test set;

[0012] S2: constructing a spatiotemporal image sequence prediction model, and improving the loss function used in the training process of the spatiotemporal image sequence prediction model by using a difference dispersion regularization term, taking the training set as input and minimizing the loss function value as the goal, iteratively training the spatiotemporal image sequence prediction model to obtain iterative training parameters;

[0013] S3: Loading the iterative training parameters into the spatiotemporal image sequence prediction model, running the spatiotemporal image sequence prediction model for the image sequence of the past time period to be predicted, and obtaining the image sequence of the future time period.

[0014] In one embodiment of the present invention, in S2, the method for constructing a spatiotemporal image sequence prediction model is: based on the basic architecture of the SimVP model, the spatiotemporal image sequence prediction model includes an input layer, an encoder, a converter, a decoder and an output layer connected in sequence, and a fast Fourier transform module is added to the converter;

[0015] Among them, the input layer normalizes the input data to obtain standardized input data; the encoder is used to extract the spatial features of the standardized input data; the converter processes the input data in parallel through convolution kernels of different sizes based on the spatial features to capture depth features of different scales; the decoder reconstructs the image sequence of the future time period based on the depth features.

[0016] In one embodiment of the present invention, the converter further includes an encoding unit and a decoding unit, the input end of the fast Fourier transform module is connected to the output end of the encoding unit, and the output end thereof is connected to the input end of the decoding unit, and the three constitute a U-net network structure.

[0017] In one embodiment of the present invention, the data processing process of the fast Fourier transform module is as follows:

[0018] Performing shape transformation on the data received at the input end of the fast Fourier transform module to obtain shape-transformed data;

[0019] Performing a two-dimensional discrete Fourier transform on the shape-transformed data to obtain frequency domain data, and decomposing the frequency domain data into real data and imaginary data;

[0020] The real data and the imaginary data are respectively subjected to multiple convolution operations, the real part and the imaginary part after the convolution operation are spliced ​​and reshaped, and the inverse Fourier transform is used to return the frequency domain to the spatial domain to obtain the spatial features after feature enhancement.

[0021] In one embodiment of the present invention, the data processing expression of the fast Fourier transform module is as follows:

[0022] y=F -1 (MLP(F(y)))∈R [B，h，w，T×C]

[0023] Where B represents the number of samples input per batch, T represents the number of time steps of the image sequence, C represents the number of channels per frame of the image sequence, H represents the height of each frame of the image sequence, W represents the width of each frame of the image sequence, F(·) represents the Fourier transform, and F -1 (·) represents the inverse Fourier transform, and MLP(·) represents the convolution operation of the multi-layer perceptron.

[0024] In one embodiment of the present invention, the method for obtaining the differential dispersion regularization term is:

[0025] S21: Calculate the predicted continuous frames The forward difference value of and the forward difference value Δy of the actual continuous frame y i ,as follows:

[0026]

[0027] Δy i =y i+1 -y i

[0028] in, represents the predicted continuous frames obtained by the i-th iteration training of the model, Represents the predicted continuous frames obtained by the i+1th iteration training of the model;

[0029] S22: In the three dimensions of channel, height and width, Δy i Converted to probability distribution value, as follows:

[0030]

[0031]

[0032] in, Indicated by The probability distribution value of the conversion, σ(Δy) represents the value of the conversion from Δy i The probability distribution value of the conversion, C represents the number of channels of each frame image sequence, H represents the height of each frame image sequence, W represents the width of each frame image sequence, i, j, k, l represent the elements in the three dimensions of C, H, and W respectively, j', k', l' represent the serial numbers of the corresponding elements in the three dimensions of C, H, and W respectively, and τ is the temperature coefficient of the softmax function;

[0033] S23: Based on the probability distribution value, calculate the Kulbak-Leibler divergence, that is, the difference dispersion regularization term.

[0034] In one embodiment of the present invention, the differential dispersion regularization term The expression is as follows:

[0035]

[0036] in, represents the predicted continuous frame, y represents the actual continuous frame, D KL (·) represents the function for calculating the Kulbak-Leibler divergence, Represents the prediction of consecutive frames , σ(Δy) represents the probability distribution value converted from the forward difference value of the actual continuous frame y.

[0037] Based on the same inventive concept, the present invention also provides a prediction system for spatiotemporal image sequences, the system comprising the following modules:

[0038] A data set acquisition module is used to acquire an image sequence data set of a past time period, and divide the image sequence data set into a training set, a validation set, and a test set;

[0039] A model training module is used to construct a spatiotemporal image sequence prediction model, and improve the loss function used in the training process of the spatiotemporal image sequence prediction model through a difference dispersion regularization term, and take the training set as input to iteratively train the spatiotemporal image sequence prediction model with the goal of minimizing the loss function value to obtain iterative training parameters;

[0040] The result prediction module is used to load the iterative training parameters into the spatiotemporal image sequence prediction model, run the spatiotemporal image sequence prediction model for the image sequence of the past time period to be predicted, and obtain the image sequence of the future time period.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned prediction method for spatiotemporal image sequences when executing the program.

[0042] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the instructions of the above-mentioned prediction method for spatiotemporal image sequences are implemented.

[0043] The above technical solution of the present invention has the following advantages compared with the prior art:

[0044] 1. Integration of spatiotemporal features and frequency domain features: By constructing a spatiotemporal image sequence prediction model based on the improved SimVP model, the design of its model structure (such as U-net network structure) and data processing flow (such as frequency domain transformation and multi-layer perceptron convolution operation) can process more diverse video or image data, and by introducing the fast Fourier transform module, the input data is analyzed in the frequency domain and the frequency domain features are extracted, further enhancing the model's ability to process image sequences with no fixed shape and weak change rules.

[0045] 2. Improved prediction accuracy and computational efficiency: By optimizing the loss function and introducing the difference dispersion regularization term, this method can constrain the difference between the predicted continuous frames and the actual continuous frames, taking into account the forward difference value, which helps to capture subtle changes between image sequences and further improve the prediction accuracy; although the fast Fourier transform module and the difference dispersion regularization term are introduced, the entire model structure is relatively simple, avoiding the complex recurrent neural network structure, which helps to improve computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 It is the network structure diagram of the SimVP model;

[0048] Figure 2 is a flow chart of a prediction method for a spatiotemporal image sequence provided in the first embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the structure of the converter (Inception_FFTnet) of the spatiotemporal image sequence prediction model (SimVP_FFT_DDR model) provided in the first embodiment of the present invention;

[0050] Figure 4 These are some prediction results of the SimVP_FFT_DDR model on meteorological data, where (a) represents the true value and (b) represents the predicted value;

[0051] Figure 5 These are some prediction results of the SimVP_FFT_DDR model on literature data, where (a) represents the true value and (b) represents the predicted value;

[0052] Figure 6 is a structural diagram of a prediction system for spatiotemporal image sequences provided in the second embodiment of the present invention;

[0053] Description of the accompanying drawings in the specification: 100, data set acquisition module; 200, model training module; 300, result prediction module. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0055] Embodiment 1

[0056] Reference Figure 2 As shown, the present invention provides a prediction method for spatiotemporal image sequences, the method comprising the following steps:

[0057] S1: Obtain an image sequence dataset for a past time period, and divide the image sequence dataset into a training set, a validation set, and a test set;

[0058] S2: construct a spatiotemporal image sequence prediction model (hereinafter named SimVP_FFT_DDR model), and improve the loss function used in the training process of the spatiotemporal image sequence prediction model by using a difference dispersion regularization term, take the training set as input, and iteratively train the spatiotemporal image sequence prediction model with the goal of minimizing the loss function value to obtain iterative training parameters;

[0059] S3: Loading the iterative training parameters into the spatiotemporal image sequence prediction model, running the spatiotemporal image sequence prediction model for the image sequence of the past time period to be predicted, and obtaining the image sequence of the future time period.

[0060] Further, in step S2, the method for constructing a spatiotemporal image sequence prediction model is as follows: based on the basic architecture of the SimVP model, the spatiotemporal image sequence prediction model includes an input layer, an encoder, a converter, a decoder and an output layer connected in sequence, and a fast Fourier transform module is added to the converter;

[0061] Among them, the input layer normalizes the input data to obtain standardized input data; the encoder is used to extract the spatial features of the standardized input data; the converter processes the input data in parallel through convolution kernels of different sizes based on the spatial features to capture depth features of different scales; the decoder reconstructs the image sequence of the future time period based on the depth features.

[0062] Taking the prediction of the next 5 frames from the past 5 frames as an example, each frame of the image sequence is a single channel with a size of 64 × 64. The input and output data shape changes of the SimVP_FFT_DDR model and its each layer structure are shown in Table 1.

[0063] Table 1

[0064]

[0065] The SimVP_FFT_DDR model adds a fast Fourier transform module to the converter (Inception_Unet structure) based on the SimVP model structure. The Inception_Unet structure also includes an encoding unit and a decoding unit. The input end of the fast Fourier transform module is connected to the output end of the encoding unit, and its output end is connected to the input end of the decoding unit. The three constitute a U-net network structure, and the new Inception_FFTnet structure is obtained as follows: Figure 3 Specifically, the data processing process of the fast Fourier transform module is as follows:

[0066] The shape of the data passed into the Inception_FFTnet encoding part is (B, T×C, H, W). The encoder return value is first transformed to obtain data of the shape (B, H×W, T×C) and then passed to the fast Fourier transform module;

[0067] The data received at the input end of the fast Fourier transform module is recombined by performing a shape transformation according to a preset length and width to obtain shape-transformed data, the size of which becomes (B, h, w, T×C);

[0068] Then, the reorganized data is subjected to a two-dimensional discrete Fourier transform according to the two dimensions of h and w to obtain frequency domain data, and decomposed into real data and imaginary data;

[0069] When extracting frequency domain features, the real data and the imaginary data are first convolved using a predetermined weight w1 and a bias b1, and the ReLU activation function is applied. Then, the activated real data and imaginary data are convolved for a second time using a predetermined weight w2 and a bias b2. Finally, the real data and the imaginary data are concatenated and reshaped, and returned to the spatial domain through an inverse transform to obtain the spatial features after feature enhancement.

[0070] In summary, the data processing expression of the fast Fourier transform module is as follows:

[0071] y=F -1 (MLP(F(y)))∈R [B,h,w,T×C]

[0072] Where B represents the number of samples input per batch, T represents the number of time steps of the image sequence, C represents the number of channels per frame of the image sequence, H represents the height of each frame of the image sequence, W represents the width of each frame of the image sequence, F(·) represents the Fourier transform, and F -1 (·) represents the inverse Fourier transform, and MLP(·) represents the convolution operation of the multi-layer perceptron.

[0073] In addition, during the model training process, the training loss function of the SimVP_FFT_DDR model also adds difference dispersion regularization, referring to the method of processing the difference between consecutive frames in the TAU (Temporal Attention Unit) model, to assist the model in learning the difference between consecutive frames and realizing the inherent changes, so that the difference between the predicted frames and the difference between the real frames will not be too far apart, thereby ensuring that the predicted change trend is similar to the real change trend. Therefore, the method for obtaining the difference dispersion regularization term is as follows:

[0074] S21: For a given predicted continuous frame And the actual continuous frame y, calculate the predicted continuous frame The forward difference value of and the forward difference value Δy of the actual continuous frame y i ,as follows:

[0075]

[0076] in, represents the predicted continuous frames obtained by the i-th iteration training of the model, Represents the predicted continuous frames obtained by the i+1th iteration training of the model;

[0077] S22: In the three dimensions of channel, height and width, Δy i Converted to probability distribution value, as follows:

[0078]

[0079]

[0080] in, Indicated by The probability distribution value of the conversion, σ(Δy) represents the value of the conversion from Δy i The probability distribution value of the conversion, C represents the number of channels of each frame of the image sequence, H represents the height of each frame of the image sequence, and W represents the width of each frame of the image sequence; i, j, k, l represent the elements in the three dimensions of C, H, and W respectively, j', k', l' represent the serial numbers of the corresponding elements in the three dimensions of C, H, and W respectively; τ is the temperature coefficient of the softmax function, which is used to adjust the output probability distribution. It is set to 0.1 according to experience to sharpen the probability distribution;

[0081] S23: Based on the probability distribution value, calculate the Kullback-Leibler divergence, that is, the difference dispersion regularization term

[0082]

[0083] in, represents the predicted continuous frame, y represents the actual continuous frame, D KL (·) represents the function for calculating the Kulbak-Leibler divergence, Represents the prediction of consecutive frames , σ(Δy) represents the probability distribution value converted from the forward difference value of the actual continuous frame y.

[0084] In order to verify the effectiveness of the method described in the present invention, Experiments 1 and 2 were designed using different types of data sets. In Experiment 1, geosynchronous satellite data of the East Asia-Southeast Asia Pacific Coast region were intercepted. During data integration, the three water vapor channel data corresponding to high, medium and low altitudes were stored in three NPY (binary file format used by the NumPy library) data files and divided by hour.

[0085] The goal of this experiment is to predict the distribution of water vapor in future time periods. In order to more conveniently observe the actual distribution conditions and prediction results, the NPY files of the three channels are merged for visualization before the experiment. The specific operations include: first, normalizing the data of the three channels; then, converting the high, medium and low two-channel data into RGB three-channel data; finally, saving each corresponding file as a new JPG format file. The dataset is divided into three parts: a training set (506 frames), a validation set (144 frames) and a test set (144 frames). The model uses the method of inputting 5 frames and predicting 5 frames. After 35 rounds of training, the experimental results of the meteorological long-range measurement set are shown in Table 2, and some prediction results of the SimVP_FFT_DDR model are shown in Table 2. Figure 4 shown.

[0086] Table 2

[0087]

[0088] In Experiment 2, the original information data of SCI papers with the theme of "Remote Sensing" were obtained from the Web of Science database, including paper titles, authors, journal names, publication dates, etc. These data can be used to study the research status and trends of this topic within a certain period of time. The data was sliced ​​according to the year of publication, and then the high-dimensional data was reduced to a two-dimensional plane through principal component analysis (PCA). Next, the point data on the two-dimensional plane was reconstructed according to the grid position of the target using the grid statistical method, and finally the two-dimensional image data was obtained. The time range of the data is from January 2001 to January 2016, of which [200101, 200801) is used as the training set, [200801, 201201) is used as the validation set, and [201201, 201601) is used as the test set. The model uses the method of inputting December to predict December. After 35 rounds of training, some prediction results of the SimVP_FFT_DDR model are as follows. Figure 5 The experimental results of the literature test set are shown in Table 3.

[0089] Table 3

[0090]

[0091] From the results of the two experiments, it can be seen that the performance of the SimVP_FFT model with only the Fourier transform module increased on both datasets has decreased, which may be related to the limitations of the SimVP model in processing long-series video prediction. The Simvp_DDR model with only the inter-frame difference module has improved on both datasets, indicating that the difference dispersion regularization term improves the prediction accuracy of the model for multi-frame data. The SimVP_FFT_DDR model combining the two structures performs better than the original SimVP model on both datasets, and performs better than the Simvp_DDR model in some evaluation indicators (such as mae in meteorological data and psnr in literature data).

[0092] Embodiment 2

[0093] Based on the same inventive concept as that of the first embodiment, the present invention also provides a prediction system for spatiotemporal image sequences. Figure 6 As shown, the system includes the following modules:

[0094] The data set acquisition module 100 is used to acquire an image sequence data set of a past time period, and divide the image sequence data set into a training set, a validation set, and a test set;

[0095] A model training module 200 is used to construct a spatiotemporal image sequence prediction model, and improve the loss function used in the training process of the spatiotemporal image sequence prediction model by using a difference dispersion regularization term, and iteratively train the spatiotemporal image sequence prediction model with the training set as input and the goal of minimizing the loss function value to obtain iterative training parameters;

[0096] The result prediction module 300 is used to load the iterative training parameters into the spatiotemporal image sequence prediction model, run the spatiotemporal image sequence prediction model for the image sequence of the past time period to be predicted, and obtain the image sequence of the future time period.

[0097] A prediction system for spatiotemporal image sequences proposed in this embodiment is used to implement the aforementioned prediction method for spatiotemporal image sequences. Therefore, the specific implementation method of the prediction system for spatiotemporal image sequences can be seen in the embodiment part of the aforementioned prediction method for spatiotemporal image sequences. For example, the data set acquisition module 100, the model training module 200 and the result prediction module 300 are respectively used to implement the steps S1, S2 and S3 in the prediction method for spatiotemporal image sequences in the first embodiment. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0098] Embodiment 3

[0099] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the prediction method for spatiotemporal image sequences in the first embodiment when executing the program.

[0100] Embodiment 4

[0101] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the instructions of the prediction method for spatiotemporal image sequences in the above-mentioned embodiment 1 are implemented.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0103] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0106] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A prediction method for spatiotemporal image sequences, characterized in that: The method comprises the following steps: S1: Obtain an image sequence dataset for a past time period, and divide the image sequence dataset into a training set, a validation set, and a test set; S2: constructing a spatiotemporal image sequence prediction model, and improving the loss function used in the training process of the spatiotemporal image sequence prediction model by using a difference dispersion regularization term, taking the training set as input and minimizing the loss function value as the goal, iteratively training the spatiotemporal image sequence prediction model to obtain iterative training parameters; S3: Loading the iterative training parameters into the spatiotemporal image sequence prediction model, running the spatiotemporal image sequence prediction model for the image sequence of the past time period to be predicted, and obtaining the image sequence of the future time period.

2. The prediction method for spatiotemporal image sequences according to claim 1, characterized in that: In S2, the method for constructing a spatiotemporal image sequence prediction model is as follows: based on the basic architecture of the SimVP model, the spatiotemporal image sequence prediction model includes an input layer, an encoder, a converter, a decoder and an output layer connected in sequence, and a fast Fourier transform module is added to the converter; Among them, the input layer normalizes the input data to obtain standardized input data; the encoder is used to extract the spatial features of the standardized input data; the converter processes the input data in parallel through convolution kernels of different sizes based on the spatial features to capture depth features of different scales; the decoder reconstructs the image sequence of the future time period based on the depth features.

3. The prediction method for spatiotemporal image sequences according to claim 2, characterized in that: The converter also includes an encoding unit and a decoding unit. The input end of the fast Fourier transform module is connected to the output end of the encoding unit, and the output end is connected to the input end of the decoding unit. The three constitute a U-net network structure.

4. The prediction method for spatiotemporal image sequences according to claim 2, characterized in that: The data processing process of the fast Fourier transform module is as follows: Performing shape transformation on the data received at the input end of the fast Fourier transform module to obtain shape-transformed data; Performing a two-dimensional discrete Fourier transform on the shape-transformed data to obtain frequency domain data, and decomposing the frequency domain data into real data and imaginary data; Perform multiple convolution operations on the real data and the imaginary data respectively, splice and reshape the real part and the imaginary part after the convolution operation, and return from the frequency domain to the spatial domain through the inverse Fourier transform. Get the spatial features after feature enhancement.

5. The prediction method for spatiotemporal image sequences according to claim 2, characterized in that: The data processing expression of the fast Fourier transform module is as follows: and=F -1 (MLP(F(y)))∈R [B,h,w,T×C] Where B represents the number of samples input per batch, T represents the number of time steps of the image sequence, C represents the number of channels per frame of the image sequence, H represents the height of each frame of the image sequence, W represents the width of each frame of the image sequence, F(·) represents the Fourier transform, and F -1 (·) represents the inverse Fourier transform, and MLP(·) represents the convolution operation of the multi-layer perceptron.

6. The prediction method for spatiotemporal image sequences according to claim 1, characterized in that: The method for obtaining the difference dispersion regularization term is: S21: Calculate the predicted continuous frames The forward difference value of and the forward difference value Δy of the actual continuous frame y i ,as follows: Δy i =y i+1 -y i in, represents the predicted continuous frames obtained by the i-th iteration training of the model, Represents the predicted continuous frames obtained by the i+1th iteration training of the model; S22: In the three dimensions of channel, height and width, Δy i Converted to probability distribution value, as follows: in, Indicated by The probability distribution value of the conversion, σ(Δy) represents the value of the conversion from Δy i The probability distribution value of the conversion, C represents the number of channels of each frame image sequence, H represents the height of each frame image sequence, W represents the width of each frame image sequence, i, j, k, l represent the elements in the three dimensions of C, H, and W respectively, j', k', l' represent the serial numbers of the corresponding elements in the three dimensions of C, H, and W respectively, and τ is the temperature coefficient of the softmax function; S23: Based on the probability distribution value, calculate the Kulbak-Leibler divergence, that is, the difference dispersion regularization term.

7. The prediction method for spatiotemporal image sequences according to claim 1 or 6, characterized in that: The difference dispersion regularization term The expression is as follows: in, represents the predicted continuous frame, y represents the actual continuous frame, D KL (·) represents the function for calculating the Kulbak-Leibler divergence, Represents the prediction of consecutive frames , σ(Δy) represents the probability distribution value converted from the forward difference value of the actual continuous frame y.

8. A prediction system for spatiotemporal image sequences, characterized in that: The system includes the following modules: A data set acquisition module is used to acquire an image sequence data set of a past time period, and divide the image sequence data set into a training set, a validation set, and a test set; A model training module is used to construct a spatiotemporal image sequence prediction model, and improve the loss function used in the training process of the spatiotemporal image sequence prediction model through a difference dispersion regularization term, and take the training set as input to iteratively train the spatiotemporal image sequence prediction model with the goal of minimizing the loss function value to obtain iterative training parameters; The result prediction module is used to load the iterative training parameters into the spatiotemporal image sequence prediction model, run the spatiotemporal image sequence prediction model for the image sequence of the past time period to be predicted, and obtain the image sequence of the future time period.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the prediction method for spatiotemporal image sequences according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements instructions of the prediction method for spatiotemporal image sequences according to any one of claims 1 to 7.

Images