Video prediction model based on operator learning

Abstract

The invention relates to a video prediction model based on operator learning. The video prediction model comprises a coding layer, an operator layer and a decoding layer, the coding layer receives an externally input video, the video is a series of continuous picture sequences, gridding labeling is carried out on the video, and time information and the originally input picture sequences are fused; carrying out downsampling processing on the fused picture sequence, and sending a processing result to an operator layer; the operator layer adopts a self-adaptive Fourier neural operator to carry out Fourier transform on a result processed by the coding layer, and extracts spatial-temporal characteristics; and the decoding layer obtains a predicted picture sequence according to the spatial-temporal characteristics provided by the operator layer. According to the method, the application requirement of space-time flexibility of the prediction task can be met under the conditions of data missing, low input data frame rate and the like, and meanwhile, the precision in the video prediction task is relatively high.

Description

Technical Field

[0001] This invention belongs to the field of spatiotemporal sequence prediction and relates to a video prediction model based on operator learning. Background Technology

[0002] Video prediction of future information has always been a hot topic in various fields. The most common video prediction problem is the prediction of time series data. With the development of big data, fields such as natural sciences, social sciences, industrial engineering, and fintech have accumulated massive amounts of data. Among these massive amounts of data, time series data collected or generated in chronological order is an important component. For this time series data, tasks can be divided into filtering, smoothing, and prediction tasks based on the time of the target task relative to the current time. The prediction task involves extracting data features from observations of certain variables over a past period to predict future events. Using time series data to predict its state over a future period has a wide range of applications. For example, in the financial sector, it is used for cash flow forecasting and stock price forecasting; in the retail industry, it is used for business revenue forecasting and inventory consumption forecasting; in the tourism industry, it is used to predict tourism order volume and customer service volume; and it is also widely used in meteorology and population density forecasting to help decision-makers make important decisions based on data.

[0003] The main objective of video prediction tasks is to optimize the network G so that the predicted output is as close as possible to the actual predicted target. For example... Figure 1 As shown, common video sequence prediction methods are generally divided into two categories according to their principles. One category is the traditional method based on physical laws. For example, the inter-frame difference method predicts future information by using the inter-frame difference of the observed information in the image sequence. The optical flow method is based on the principle of optical continuity, assuming that the pixels between adjacent frames change continuously in time, and uses this continuity to estimate the motion information of the pixels. The other category is the method based on deep learning. The existing video prediction methods based on deep learning generally predict the spatial information of the target in the image and the movement and change of the target. In recent years, the methods based on deep learning have developed rapidly.

[0004] RNNs are a type of neural network capable of processing time-series data. For video prediction problems in video generation, each video frame can be treated as a data point in a time series, and then an RNN can be used to model these data points. This method can consider dependencies between each time step, but its ability to model long-term dependencies is limited. Wang et al. proposed a novel PredRNN structure, which uses recurrent convolutional units (RCUs) to replace standard RNN units to capture longer temporal dependencies. This allows for the generation of future images through learning from historical frames, simultaneously processing both spatial features and temporal changes. Through intra- and inter-layer connections, different RNN layers can pass memory information in both horizontal and vertical directions, thus unifying the model's memory space. Most existing models are modifications of RNNs, LSTMs, and other networks. Figure 2 The diagram illustrates the architecture of a traditional video frame prediction model based on Recurrent Neural Networks (RNNs). The model utilizes a multi-layered stacked structure of spatiotemporal long short-term memory network units, a mainstream approach in early video prediction tasks, aiming to predict future frame sequences from historical frame sequences. Existing video prediction models, especially for video tasks, are constrained by data storage and computational architectures, necessitating the physical discretization of continuous video into a finite sequence of images. The limitation of traditional models lies in their over-reliance on this discretization structure, presupposing fixed time intervals for dynamic modeling. This results in the model only being able to predict discrete sequences at specific frame rates (e.g., only supporting training and prediction at 60fps), unable to flexibly adapt to continuous or varying time spans.

[0005] Modeling with fixed time intervals results in poor flexibility when processing real videos. For example, when a video requires a large number of frames in dynamic regions, existing models struggle to generate more predictions for these regions. While reducing the sampling interval when discretely sampling the data can mitigate this problem to some extent, it increases computational costs and training difficulty exponentially and is fundamentally incapable of handling the flexibility of continuous time.

[0006] Traditional video prediction tasks often assume that both the input and prediction sequences are fixed. The input sequence requires a fixed frame rate with equal intervals, and the prediction sequence is determined by training conditions. However, in real-world applications, limitations such as camera hardware and storage resources make it difficult to obtain the original training frame rate and interval. Sometimes, only low-frame-rate image sequences are available, or storage constraints mean only a fraction of the image sequences (fewer than the training frames) are stored. Faced with these situations, traditional models, due to training limitations, cannot complete the video prediction task with such low-frame-rate image sequences and must be retrained. Furthermore, training at low frame rates results in reduced prediction performance due to the lower information content. Therefore, traditional models cannot adequately meet the flexibility required for real-world applications. Summary of the Invention

[0007] The technical problem solved by this invention is to overcome the shortcomings of the existing technology and propose a video prediction model based on operator learning.

[0008] The solution of the present invention is: A video prediction model based on operator learning includes an encoding layer, an operator layer, and a decoding layer; Encoding layer: Receives externally input video, which is a series of continuous image sequences, performs grid-based annotation on it, fuses the time information with the original input image sequence, performs downsampling on the fused image sequence, and sends the processing result to the operator layer; Operator layer: Adaptive Fourier neural operators are used to perform Fourier transform on the results processed by the coding layer and extract spatiotemporal features; Decoding layer: Based on the spatiotemporal features provided by the operator layer, the predicted image sequence is obtained.

[0009] Preferably, the encoding layer labels the image sequence in chronological order, expands the corresponding time label values ​​to the size of the image sequence, and performs a fusion operation on the expanded time label values ​​and the original input image sequence in the channel dimension.

[0010] Preferably, the encoding layer performs downsampling processing on the fused image sequence, using the following method: S1. Perform a two-dimensional convolution with a stride of 1 on the channel layer of the fused image sequence to obtain the latent space variable sequence; S2. Perform a two-dimensional convolution with a stride of 2 on the latent space variable sequence to obtain an image sequence with reduced spatial resolution. S3. Repeat steps S1-S2 until the preset number of times is reached.

[0011] The preferred method for implementing the operator layer is as follows: An adaptive Fourier neural operator is used to perform a Fourier transform on the latent space variable sequence after the coding layer processing to obtain a tensor with dimensions (h,w,f,c), where h is the height of the latent space variable, w is the width of the latent space variable, f is the frequency dimension after the frequency domain transformation, and c is the number of channels of the latent space variable sequence. The tensor is then divided into blocks in the (h,w) dimension. Multiply the segmented image by the shared matrix to obtain the latent space variables in the frequency domain after segmentation; By concatenating the latent space variables after they are divided in the frequency domain, a complete sequence of latent space variables in the frequency domain is obtained. The complete sequence of latent space variables in the frequency domain is restored to the complete sequence of latent space variables in the time domain by using Fourier transform; The complete sequence of latent space variables in the time domain is input into the spatiotemporal decoupling module to obtain spatiotemporal features.

[0012] The preferred method for implementing the spatiotemporal decoupling module is as follows: 2.1 Perform a two-dimensional convolution with a stride of 1 on the complete sequence of latent space variables in the time domain; 2.2 Perform a 3D convolution with a stride of 1 on the convolution result of step 2.1; 2.3 The input of this 2D convolution and the result of the 3D convolution are concatenated along the channel dimension; 2.4 Perform a 2D convolution with a stride of 1 on the splicing result of 2.3; 2.5 Perform a 3D convolution with a stride of 1 on the convolution result of 2.4; 2.6 Repeat steps 2.3-2.5 until the predetermined number of times is reached. The result of the last three-dimensional convolution is the spatiotemporal feature sequence.

[0013] Preferably, the decoding layer obtains the predicted image sequence based on the spatiotemporal features provided by the operator layer, as follows: 3.1 First, perform a deconvolution with a stride of 2 on the spatiotemporal feature sequence in the channel dimension, and then perform a convolution with a stride of 1 on the deconvolution result; 3.2 Repeat step 3.1 up to the predetermined number of times to obtain the predicted image sequence.

[0014] Preferably, the parameters of each layer of the video prediction model are obtained through training.

[0015] Preferably, during training, several videos are selected as a sample set, and each video is divided into segments along time, with the first segment as input and the second segment as output.

[0016] The advantages of this invention compared to the prior art are: 1. To address the limitation of fixed time intervals in existing video prediction models, this invention treats the video prediction task as a continuous dynamic process. The idea is that a spatiotemporal sequence reflects a continuous changing process, and this dynamic process, due to its physical meaning, is also continuous. After modeling differential equations using the input sequence, flexible frame rate prediction of the target prediction time can be achieved by customizing the solution step size. Simultaneously, considering the increasing computational cost of traditional differential equation solving methods with increasing accuracy, this invention introduces operator learning to solve the spatiotemporal sequence prediction problem. It utilizes neural operators to simulate the solution of continuous functions through discrete sampling. Specifically, the input video sequence is a sample of the equation's observations, and the output is the corresponding sample of the position to be solved.

[0017] This invention introduces an explicit temporal embedding mechanism. Unlike traditional models that only fit discrete sequences, this model essentially aims to learn the continuous evolution equation of dynamic changes in video, and achieves the conversion from the time domain to the frequency domain through Fourier transform and inverse Fourier transform, thereby enabling resampling and prediction based on any given time point, breaking through the limitation of fixed time intervals.

[0018] 2. To address the shortcomings of existing video prediction models, which are prone to high computational complexity and difficulty in balancing model performance requirements when overcoming the limitation of fixed time intervals, this invention proposes a spatiotemporally decoupled feature extraction and fusion scheme. Only spatial features are extracted at the encoding and decoding layer, while spatiotemporal features are extracted and fused at the operator layer, effectively capturing the spatiotemporal correlation of sequences.

[0019] 3. To improve the video prediction performance of the model, this invention applies an adaptive Fourier operator to process feature information in the frequency domain at the Fourier operator layer; a spatiotemporal decoupling module is introduced within the operator layer, which extracts and fuses spatial and temporal features through the alternating use of two-dimensional and three-dimensional convolutions; and interlayer connections are introduced between operator layers to capture spatiotemporal feature correlations, increasing the network exploration depth while maintaining the original feature information, thus avoiding the problem of spatiotemporal correlations being submerged in deep networks. Attached Figure Description

[0020] Figure 1 This is a common video sequence prediction method; Figure 2 This is a traditional video frame prediction model architecture based on recurrent neural networks (RNNs); Figure 3 This is a diagram showing the composition of the model of the present invention; Figure 4 This is a schematic diagram of the coding layer implementation; Figure 5 This is a schematic diagram of the frequency domain transformation part of the operator layer; Figure 6This is a schematic diagram of the spatiotemporal decoupling module. Detailed Implementation

[0021] The invention will now be further described with reference to the accompanying drawings.

[0022] To address the need for flexibility and achieve a certain level of accuracy in video prediction tasks, this invention models spatiotemporal sequences from the perspective of differential equations. The video prediction problem is modeled as fitting a continuous dynamic equation based on observations and obtaining prediction information by solving this equation. Operator learning is introduced to solve the spatiotemporal sequence prediction problem, and neural operators are used to map the input sequence function to the prediction target function. Specifically, video sequences are essentially discrete samples of continuous dynamic processes in the real world. Although the videos we observe are discrete sequences composed of a finite number of frames, the underlying dynamic evolution processes such as object motion and fluid changes are continuous and smooth in the time dimension. Therefore, the video prediction task can be abstracted as the process of predicting the state of a system. Over time Modeling the rate of change. Mathematically, this continuous state evolution process is naturally described by differential equations. We can model the instantaneous state of the video... Over time The rate of change is modeled as a function state Regarding time The derivative is equal to Formally, it can be expressed as:

[0023] Among them, the function This represents the continuous dynamics operator of the system. The goal of video prediction is to solve for the corresponding time-series solutions, thus transforming the prediction problem into a problem of solving differential equations. This invention uses neural operators to learn function fitting, utilizing observed video sequences as observation samples for the differential equations. Specifically, the operator layer uses Fourier transform to fit a complete function in the frequency domain, followed by inverse Fourier transform and sampling in the time domain to obtain the target prediction result. By using the ideas of operator learning and differential equations to model the time dynamics equations, the time dynamics process is modeled as a continuous process through differential equations, and its analytical solution is obtained using operator learning, thereby achieving a certain degree of temporal flexibility in prediction.

[0024] This invention also employs a spatiotemporally decoupled feature extraction method. In the operator structure, Fourier operators are used to process the feature sequences, extracting the spatiotemporal correlation of the sequences in the frequency domain. Following frequency domain processing, a spatiotemporally decoupled module is designed for spatiotemporal feature extraction, extracting and fusing spatial and temporal features. This model can meet the spatiotemporal flexibility requirements of prediction tasks even with missing data or low input frame rates, while maintaining high accuracy in video prediction tasks.

[0025] The model of this invention first inputs an image sequence. After the spatial information of the image sequence is encoded by an encoding layer, it is input into an operator layer. In the operator layer, after Fourier transform, various linear transforms, and spatiotemporal decoupling modules, the processed feature sequence is obtained. Then, the original image information is recovered through a decoding layer, thus completing the extraction of image sequence data information and the prediction of future information. Figure 3 As shown.

[0026] Specifically as follows: First, a continuous sequence of images from the video is input into the encoding layer. Considering the strong spatiotemporal correlation in spatiotemporal prediction, this model performs grid-based annotation on the sequence data, converting time information into coordinate information, as shown in the following formula:

[0027]

[0028] The `get_grid` function discretizes the spatial and temporal dimensions of the input image sequence, using the original input... The dimensions are (C, H, W, T), where C is the number of channels, H is the height of the original image, W is the width of the original image, and T is the number of images contained in the image sequence. For the time information marked, This is the concatenated input. For splicing operations, splicing is performed at the channel dimension.

[0029] The input image is fused with the original image at the channel dimension. Specifically, for the input data, the image sequence is labeled chronologically, for example, [0,1,2,…,n-1] represent the first n frames. The corresponding time-labeled values ​​are then expanded to the size of the image sequence (B,1,T,H,W), where B is the number of image sequences. The expanded time-labeled values ​​and the original input image sequence are then fused at the channel dimension. The advantage of gridded labeling is that it explicitly incorporates temporal information into the image sequence, enhancing the temporal features that are difficult to extract from spatiotemporal features.

[0030] Spatial features of the fused image sequence are extracted. Since the image sequence data is large, increasing the dimensionality to the feature space consumes a lot of computational resources. Considering the training cost of the task, it is necessary to reduce the dimensionality of the feature space.

[0031] The merged image sequence is downsampled using the following method: S1. Perform a two-dimensional convolution with a stride of 1 on the channel layer of the fused image sequence to obtain the latent space variable sequence; S2. Perform a two-dimensional convolution with a stride of 2 on the latent space variables to obtain an image sequence with reduced spatial resolution; S3. Repeat steps S1-S2 until the preset number of times is reached.

[0032] A convolutional layer with a stride of 1 is used to capture the detailed features of the image; a convolutional layer with a stride of 2 achieves spatial downsampling, gradually reducing the size of the feature map to extract more global features.

[0033] The mathematical process of the coding layer is as follows:

[0034]

[0035] The sequence of latent space vectors obtained in step S1. The sequence of latent space vectors obtained in step S2. Given an input image sequence, The step size is 1. This is used to pad values ​​in a convolution operation with a stride of 1, in order to maintain the shape of the image. The step size is 1. This is used to pad values ​​in a convolution operation with a stride of 2. The stride is used alternately in the encoding layer with convolution operations of different stride lengths to downsample the image, reducing the number of model parameters. Since the encoding layer downsamples the image in the spatial dimension, the projection layer needs to restore the image sequence, using invertible convolution to achieve spatial restoration. However, because temporal features are more difficult to extract than spatial features from image sequences, a common approach in spatiotemporal feature extraction can be adopted: extract temporal and spatial features separately, and then fuse them to generate the corresponding spatiotemporal features. The encoding layer implementation is as follows... Figure 4 As shown.

[0036] After spatial encoding by the encoding layer, the sequence input is processed by the operator layer. The main structure of the operator layer is the adaptive Fourier operator. The principle of the adaptive Fourier operator is based on a fundamental principle of operator learning: frequency domain multiplication corresponds to convolution in the complete time domain, independent of input resolution. This principle is used to simplify parameters in the original Fourier neural operator, effectively solving the global convolution problem in the Fourier domain and showing promise in learning challenging partial differential equations. To address the challenges in visual representation learning, such as discontinuities in images and high-resolution inputs, this model introduces an adaptive Fourier neural operator. This operator performs block multiplication operations along the channel dimension and shares weights at different spatial locations in the image. This reduces computation and the number of parameters, and by avoiding high-frequency truncation of the image sequence, it preserves the complete information of the image, instead capturing the frequency domain information of the feature sequence along the channel dimension.

[0037] The operator layer is implemented as follows: (1) The adaptive Fourier neural operator is used to perform Fourier transform on the latent space variable sequence after the coding layer to obtain a tensor with dimension (h,w,f,c), where h is the height of the latent space variable, w is the width of the latent space variable, f is the frequency dimension after frequency domain transformation, and c is the number of channels of the latent space variable sequence. The tensor is divided into blocks in the (h,w) dimension. (2) Multiply the image after segmentation by the shared matrix to obtain the latent space variables after segmentation in the frequency domain; (3) The latent space variables after being divided into blocks in the frequency domain are spliced ​​together to obtain a complete sequence of latent space variables in the frequency domain; (4) The complete hidden space variable sequence in the frequency domain is restored to the complete hidden space variable sequence in the time domain by Fourier transform; (5) Input the complete sequence of latent space variables in the time domain into the spatiotemporal decoupling module to obtain spatiotemporal features.

[0038] Figure 5 This is a schematic diagram of the frequency domain transformation part of the operator layer.

[0039] Following Fourier transform and frequency domain operations, a spatiotemporal decoupling module (implemented via a multilayer perceptron) is introduced to extract spatiotemporal features for predictive learning of image sequences, in order to further extract spatiotemporal correlation information. The spatiotemporal decoupling module used in this model is based on the fundamental idea of ​​decoupling extraction during the spatiotemporal feature extraction process. It recognizes that spatial and temporal features in spatiotemporal data have different characteristics and meanings. Therefore, this model separates and extracts spatial and temporal features separately in the field of spatiotemporal feature extraction, resulting in better feature representation. Specifically, it consists of two convolutional layers. By transforming the dimensionality of the latent variable feature sequence, two-dimensional convolution is first used to extract spatial features, followed by three-dimensional convolution to extract spatiotemporal features. Two-dimensional convolution can effectively capture spatial features in images, while three-dimensional convolution can further capture temporal features in time-series data such as videos. Alternating between two-dimensional and three-dimensional convolution can simultaneously extract spatial features of images and temporal features of videos, thus more comprehensively representing spatiotemporal information and effectively capturing the spatiotemporally related parts of latent variables.

[0040] The spatiotemporal decoupling module is implemented as follows: 2.1 Perform a two-dimensional convolution with a stride of 1 on the complete sequence of latent space variables in the time domain; 2.2 Perform a 3D convolution with a stride of 1 on the convolution result of step 2.1; 2.3 The input of this 2D convolution and the result of the 3D convolution are concatenated along the channel dimension; 2.4 Perform a 2D convolution with a stride of 1 on the splicing result of 2.3; 2.5 Perform a 3D convolution with a stride of 1 on the convolution result of 2.4; 2.6 Repeat steps 2.3-2.5 until the predetermined number of times is reached. The result of the last three-dimensional convolution is the spatiotemporal feature sequence. Figure 6 This is a schematic diagram of the spatiotemporal decoupling module.

[0041] Considering that common feature extraction models perform poorly in extracting temporal features with increasing layer depth, as temporal features are overwhelmed by noise, leading to distorted predicted image sequences, this study, inspired by the success of ResNet and DenseNet in deep networks, introduces a dense connection-like operation between neural operator layers. This fuses the features processed by Fourier transform and frequency domain with the original input features. Inter-layer connections help improve model accuracy by allowing information to flow directly from the input to the layer output, providing the network with the possibility of re-extracting features lost during the deep network extraction process. This makes it easier for the model to capture complex relationships in the data, thereby improving accuracy and ensuring good spatiotemporal feature extraction capabilities within a certain network depth. Specifically, each layer can be fused with the original input, enabling iterative updates through joint learning of the original input and the learned feature maps.

[0042] Furthermore, dense connections, due to their skip connection structure, exhibit better performance in supervised learning. In deep networks, gradient vanishing is a common problem, especially as the number of layers increases. Because each layer introduces parameters and non-linear transformations, the gradient gradually diminishes as it propagates layer by layer during backpropagation. This results in minimal or almost no updates to the weights of shallower layers, making the network difficult to train. Inter-layer connections, however, can directly propagate gradients through skip connections, allowing gradients to spread more rapidly throughout the network. This alleviates the gradient vanishing problem, enabling deeper networks to be trained more effectively.

[0043] This model is trained through supervised learning and achieves modeling of continuous time series through operator learning. By fusing time information with original image information, it achieves video prediction tasks with both high and low frame rates. At the same time, through improvements to the Fourier operator and the addition of multilayer perceptrons and interlayer connections, the model's prediction performance is guaranteed in various application scenarios.

[0044] The decoding layer obtains the predicted image sequence based on the spatiotemporal features provided by the operator layer, as follows: 3.1 First, perform a deconvolution with a stride of 2 on the spatiotemporal feature sequence in the channel dimension, and then perform a convolution with a stride of 1 on the deconvolution result; 3.2 Repeat step 3.1 up to the predetermined number of times to obtain the predicted image sequence.

[0045] This invention introduces skip connections in the decoding layer, linking feature maps from a specific layer in the encoder with those from the corresponding layer in the decoder. This combines low-level detailed features with high-level contextual information, helping the network better recover details and boundaries. However, considering that pooling downsamples feature maps, some details and spatial information may be lost, potentially leading to a decrease in accuracy. Therefore, convolution is used to downsample the image sequence.

[0046] The parameters of each layer in the video prediction model of this invention are obtained through training. Several videos are selected as a sample set, and each video is divided into time segments, with the first segment as input and the second segment as output. Parameters for convolutional layers, deconvolutional layers, and the sharing matrix are then obtained.

[0047] Traditional spatiotemporal prediction tasks often assume that both the input and prediction sequences are fixed. Therefore, the input sequence needs to be at a fixed frame rate with equal intervals, and the prediction sequence is determined by the training conditions. However, in real-world applications, limitations such as camera hardware and storage resources make it difficult to obtain the original training frame rate and interval input sequences. Sometimes, only low-frame-rate image sequences can be obtained, or storage limitations may result in only a subset of image sequences less than the training frame count. Faced with these situations, traditional models, due to training limitations, cannot complete the spatiotemporal prediction task when encountering such low-frame-rate image sequences and must be retrained. Furthermore, training at low frame rates results in reduced prediction performance due to the lower information content. Therefore, traditional models cannot adequately meet the flexibility required for real-world applications.

[0048] The model of this invention achieves prediction tasks for various image sequences by employing the idea of ​​operator learning and explicitly fusing temporal information into the original image sequence. Furthermore, regarding performance improvements for flexible prediction, this model makes the following enhancements: spatial features are extracted in the encoding and decoding stages based on the idea of ​​spatiotemporal feature decoupling; adaptive Fourier operators are applied in the Fourier operator layer to process feature information in the frequency domain; a spatiotemporal decoupling module is introduced within the operator layer to extract spatiotemporal features; and inter-layer connections are introduced between operator layers to capture the correlation of spatiotemporal features. These improvements result in good prediction performance for various tasks such as weather forecasting.

[0049] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A video prediction model based on operator learning, characterized in that: It includes the encoding layer, operator layer, and decoding layer; Encoding layer: Receives externally input video, which is a series of continuous image sequences, performs grid-based annotation on it, fuses the time information with the original input image sequence, performs downsampling on the fused image sequence, and sends the processing result to the operator layer; Operator layer: Adaptive Fourier neural operators are used to perform Fourier transform on the results processed by the coding layer and extract spatiotemporal features; Decoding layer: Based on the spatiotemporal features provided by the operator layer, the predicted image sequence is obtained.

2. The operator-learning-based video prediction model of claim 1, wherein: The encoding layer labels the image sequence in chronological order, expands the corresponding time label values ​​to the size of the image sequence, and performs a fusion operation on the expanded time label values ​​and the original input image sequence in the channel dimension. 3.The operator learning based video prediction model of claim 1, wherein: The encoding layer downsamples the fused image sequence using the following method: S1. Perform a two-dimensional convolution with a stride of 1 on the channel layer of the fused image sequence to obtain the latent space variable sequence; S2. Perform a two-dimensional convolution with a stride of 2 on the latent space variable sequence to obtain an image sequence with reduced spatial resolution. S3. Repeat steps S1-S2 until the preset number of times is reached.

4. The operator-learning-based video prediction model of claim 3, wherein: The operator layer is implemented as follows: An adaptive Fourier neural operator is used to perform a Fourier transform on the latent space variable sequence after the coding layer processing to obtain a tensor with dimensions (h,w,f,c), where h is the height of the latent space variable, w is the width of the latent space variable, f is the frequency dimension after the frequency domain transformation, and c is the number of channels of the latent space variable sequence. The tensor is then divided into blocks in the (h,w) dimension. Multiply the segmented image by the shared matrix to obtain the latent space variables in the frequency domain after segmentation; By concatenating the latent space variables after they are divided in the frequency domain, a complete sequence of latent space variables in the frequency domain is obtained. The complete sequence of latent space variables in the frequency domain is restored to the complete sequence of latent space variables in the time domain by using Fourier transform; The complete sequence of latent space variables in the time domain is input into the spatiotemporal decoupling module to obtain spatiotemporal features.

5. The operator-learning-based video prediction model of claim 4, wherein: The spatiotemporal decoupling module is implemented as follows: 2.1 Perform a two-dimensional convolution with a stride of 1 on the complete sequence of latent space variables in the time domain; 2.2 Perform a 3D convolution with a stride of 1 on the convolution result of step 2.1; 2.3 The input of this 2D convolution and the result of the 3D convolution are concatenated along the channel dimension; 2.4 Perform a 2D convolution with a stride of 1 on the splicing result of 2.3; 2.5 Perform a 3D convolution with a stride of 1 on the convolution result of 2.4; 2.6 Repeat steps 2.3-2.5 until the predetermined number of times is reached. The result of the last three-dimensional convolution is the spatiotemporal feature sequence.

6. The operator-learning-based video prediction model of claim 1, wherein: The decoding layer obtains the predicted image sequence based on the spatiotemporal features provided by the operator layer, as follows: 3.1 First, perform a deconvolution with a stride of 2 on the spatiotemporal feature sequence in the channel dimension, and then perform a convolution with a stride of 1 on the deconvolution result; 3.2 Repeat step 3.1 up to the predetermined number of times to obtain the predicted image sequence.

7. The operator-learning-based video prediction model according to any one of claims 1-6, wherein: The parameters of each layer of the video prediction model are obtained through training.

8. The operator-learning-based video prediction model of claim 7, wherein: During training, several videos are selected as a sample set, and each video is divided into segments along time, with the first segment as input and the second segment as output.

Images