A video prediction method based on motion direction perception

Abstract

This invention discloses a video prediction method based on motion direction perception, belonging to the fields of computer vision, machine learning, and deep learning. Inspired by direction-selective neurons in animal visual systems, this method decomposes the complex motion information of objects in a video into two independent components, vertical and horizontal, to effectively extract the dynamic information of video objects. The direction-selective perception unit is implemented using a multilayer perceptron structure. Based on this, the method establishes a multi-scale feature fusion neural network, including a spatial encoder, a motion direction perception converter, and a spatial decoder, to achieve the extraction and prediction of multi-scale spatiotemporal features. Tests on artificial and real datasets demonstrate that this method can effectively reduce motion blur and improve the accuracy and consistency of video prediction.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, machine learning and deep learning technology, and specifically relates to temporal feature extraction and deep neural network models in video prediction tasks. Background Technology

[0002] Video prediction is an important and challenging task, aiming to generate future video frames based on previously observed video frames. This ability to predict complex spatiotemporal information has attracted widespread research interest in fields such as traffic flow prediction, weather forecasting, human behavior prediction, and autonomous driving. However, building accurate prediction models is a significant challenge, requiring the simultaneous processing of multidimensional data that includes both spatial and temporal information.

[0003] Currently, building deep learning models to solve complex video prediction problems is the mainstream approach. Many deep learning models in recent years have brought significant advancements to video prediction technology. Currently, deep learning-based video image prediction models are generally divided into three types: first, prediction models based on autoencoder networks (AEs); second, prediction models based on generative adversarial networks (GANs); and third, prediction models based on recurrent neural networks (RNNs), with the current mainstream approach being the construction of RNNs for video prediction. Because video sequences simultaneously possess complex information in both temporal and spatial dimensions, models proposed in recent years have adopted an RNN-CNN hybrid architecture to process temporal and spatial information separately. For example, the Convolutional Long Short-Term Memory (ConvLSTM) network, which can simultaneously capture temporal and spatial information, replaces the linear layers in the traditional LSTM network with convolutional layers, enabling it to process spatial information and improving the prediction performance of video images. Based on this, various other RNN+CNN models have been derived. The spatiotemporal memory unit proposed by the PredRNN prediction model can remember not only temporal states but also spatial states, enabling the stacking of deep interactive information between multiple convolutional layers and further improving prediction performance. Despite these models' significant advancements in video prediction performance, a gap remains when visualizing the results compared to real frames, specifically manifested as blurring in the prediction of moving objects. Video prediction tasks place high demands on the model's ability to perceive motion information. Many works have focused on improving the model's ability to extract motion information. For example, optical flow or inter-frame differencing methods are used to enhance dynamics. Optical flow extraction suffers from the problem of consuming substantial computational resources and increasing the input data volume, further burdening computation. While inter-frame differencing can extract motion information simply and effectively, it is not conducive to the model's extraction of motion features and doubles the model's input data volume. Other works have used 3D convolution or visual Transformers to enhance the fusion of spatiotemporal information. However, these methods are either computationally intensive or lack rich representation of motion information, resulting in significant motion blur in the prediction results. Summary of the Invention

[0004] In view of the current problem that deep neural network models are not good enough at extracting motion characteristics in video prediction, this invention provides a video prediction method based on motion direction perception mechanism.

[0005] To achieve the above objectives, the technical solution of the present invention is a video prediction method based on motion direction perception. The method includes: Step 1: Obtaining a dataset for training and validating a video frame prediction model. The dataset for video frame prediction has four dimensions: the number of video frames S, the number of video frame channels C, the height of the video frame image H, and the width of the video frame image W.

[0006] Step 2: Preprocess the acquired dataset to obtain the training dataset;

[0007] Step 3: Construct a video frame prediction model;

[0008] The video frame prediction model consists of three parts: a spatial encoder, a motion orientation-aware converter, and a spatial decoder. The specific steps are as follows:

[0009] (1) Spatial encoder processing method:

[0010] The encoder consists of L convolutional modules connected in series; each convolutional module is composed of a convolutional unit, a group normalization function, and an activation function connected in sequence; in addition, the output of each convolutional module is not only input vertically into the next convolutional module, but also output horizontally to the motion direction sensing converter.

[0011] If the convolutional module is the first convolutional module of the encoder, then the number of input channels of the convolutional units in this module is the number of input video frame channels C; otherwise, the number of input channels is the model hyperparameter C. h h is the number of the convolution module;

[0012] All convolutional modules in the encoder have a kernel size of 3×3 and a grouping normalization group number of 2. If the convolutional module is the odd-numbered convolutional module of the encoder, then the convolutional stride of the convolutional unit of that convolutional module is 1, otherwise the convolutional stride is 2.

[0013]

[0014] in, Represents the output of the i-th convolutional layer, X represents the original video sequence, LeakyReLU(·) represents the activation function, GroupNorm(·) represents group normalization, and Conv i This represents the two-dimensional convolution operation of the i-th layer; performing the operation once on all layers from i=1 to i=L will yield a spatial encoding set. i = 1...L, where L is the total number of elements in the spatial coding set; this set is input into the next layer of motion direction perception converter for temporal feature extraction; this set contains spatial information of different scales extracted by the spatial encoder, and further extraction of these multi-scale spatial information can improve the perception ability of information of different scales;

[0015] (2) The processing method of the motion direction sensing converter is as follows:

[0016] The motion orientation sensing converter consists of L parallel layers, corresponding to the L features output by the spatial encoder; each layer of the motion orientation sensing converter stacks N motion orientation sensing units; specifically, the i-th layer consists of N... i It consists of three MDAUnits for motion direction sensing; each MDAUnit is composed of three parts: a channel reduction part, a motion direction multilayer perceptron, and a channel expansion part; the channel reduction part and the channel expansion part are both composed of 1×1 two-dimensional convolution, batch normalization, and activation function; in this way, the reduction and expansion operations of feature channels are realized, reducing the amount of computation;

[0017] In MDAUnit, two multilayer perceptrons for motion direction exist side-by-side, differing only in the number of groups. The structure of the multilayer perceptron for motion direction includes:

[0018] Inspired by four types of direction-selective neurons in the animal retina, this module decomposes complex motion into two components, horizontal and vertical, reducing the difficulty of representation.

[0019] Horizontal branch: The input features are grouped, and then a fully connected layer is applied along the horizontal direction to perform temporal modeling, outputting horizontal motion features;

[0020] Vertical branch: Same as above, extract temporal features along the vertical direction and output vertical motion features;

[0021] Channel branching: Integrates features from horizontal and vertical branches, applies them along the channel dimension, and outputs the fused features;

[0022] By selectively processing motion information in this direction, the dimensionality and extraction difficulty of spatiotemporal information can be reduced, thereby improving the ability to extract motion information.

[0023]

[0024] in, This is the set after time features have been extracted by the motion direction sensing converter. This is a mapping function for MDAUnit, with the superscript (N) i The ) indicates that the mapping function will be executed N times;

[0025] (3) The processing method of the spatial decoder is as follows:

[0026] The spatial decoder processing flow is the reverse of the spatial encoder processing flow; all parameters are the same as those of the spatial encoder, except that the two-dimensional convolution operation is replaced with deconvolution motion to restore the extracted features to the original video sequence.

[0027]

[0028] Among them, TransConv i (·) represents deconvolution operation. The video sequence represents the prediction;

[0029] Step 3.2: Set the experimental hyperparameters;

[0030] Step 4: Train the video frame prediction model built in Step 3;

[0031] Step 5: Verify and evaluate the accuracy of the trained video frame prediction model, including metrics such as mean squared error, mean absolute error, structural similarity, and peak signal-to-noise ratio.

[0032] Step 6: Perform predictive analysis using the evaluated video prediction model.

[0033] Furthermore, the specific steps of step 2 are as follows:

[0034] Step 2.1: Set a sliding window based on the number of input and output video frames of the video frame prediction model to generate input data for the video frame prediction model and a pair of real data used to calculate the prediction error;

[0035] Step 2.2: Generate an iterable dataset for training and validating the video frame prediction model based on the batch size of the video frame prediction model and the real data pairs.

[0036] Furthermore, the calculation method for step 4 is as follows:

[0037] Step 4.1: Define the error function, using the mean squared error as the training error function, as follows:

[0038]

[0039] in, Y represents the predicted video sequence, T′ represents the actual video sequence, C represents the number of channels in the output video frame, and H and W represent the length and width of the output video frame, respectively.

[0040] Step 4.2: Set the learning rate and iteration rounds;

[0041] Step 4.3: Use gradient descent to update the parameters that the video frame prediction model needs to learn, thereby reducing the model prediction error.

[0042] This invention mainly includes two features. First, by simulating the motion direction selectivity mechanism of the animal visual system, a motion direction multilayer perceptron (MDA-MLP) is proposed to achieve effective representation and extraction of object motion information, reducing motion blur in the prediction results. The animal visual system is highly sensitive to object motion, which can provide inspiration for model design. There are four types of direction-selective neurons in the animal retina, which selectively respond to motion in the up, down, left, and right directions, respectively. Second, a multi-scale feature fusion architecture is adopted to more fully extract object motion information from the video and improve prediction accuracy. Attached Figure Description

[0043] Figure 1 This is a structural diagram of a multilayer perceptron (MDAMLP) for motion direction.

[0044] Figure 2 This is a structural diagram of the Motion Direction Awareness Unit (MDAUnit).

[0045] Figure 3 This is a structural diagram of a video prediction model based on motion direction awareness. Detailed implementation method:

[0046] Step 1: Obtain the dataset for training and validating the video frame prediction model. A suitable dataset for video frame prediction should have four dimensions: number of video frames (S), number of video frame channels (C), video frame image height (H), and video frame image width (W). This method uses three test datasets: Moving MNIST (mobile handwriting dataset), KTH (human motion dataset), and Human 3.6 (human motion dataset). However, this method is not limited to these three datasets. Specific parameters of the datasets are shown in Table 1.

[0047] Table 1 shows the specific parameters of the dataset.

[0048] Moving MNIST 1 64×64 10 10 KTH 1 128×128 10 20 or 40 Human 3.6 3 128×128 4 4

[0049] Step 2: Perform data preprocessing on the acquired dataset. The specific steps are as follows:

[0050] Step 2.1: Set a sliding window based on the number of input and output video frames of the video frame prediction model to generate a pair of model input data and real data used to calculate the prediction error;

[0051] Step 2.2: Generate an iterable dataset object based on the batch size for training and validating the video frame prediction model;

[0052] Step 3: Design the video frame prediction model; the model consists of three parts: a spatial encoder, a motion direction-aware transducer (MD-Translator), and a spatial decoder. The specific steps are as follows:

[0053] (1) Spatial encoder processing method:

[0054] The encoder consists of L convolutional modules connected in series. Each convolutional module is composed of a convolutional unit (Conv2d), a group normalization function (GroupNorm), and an activation function (LeakyReLU) connected sequentially. Furthermore, the output of each convolutional module is not only input vertically to deeper convolutional layers but also output horizontally to the motion orientation sensing converter. If the convolutional module is the first convolutional module of the encoder, then the number of input channels for its convolutional units is the number of input video frame channels C; otherwise, the number of input channels is the model hyperparameter C. h The kernel size of all convolutional modules in the encoder is 3×3, and the number of groups for group normalization is 2. If a convolutional module is the odd-numbered convolutional module of the encoder, then the convolution step of the convolutional unit of that convolutional module is...

[0055] The length is 1, otherwise the convolution stride is 2.

[0056]

[0057] in, Represents the output of the i-th convolutional layer, X represents the original video sequence, LeakyReLU(·) represents the activation function, GroupNorm(·) represents group normalization, and Conv i This represents the two-dimensional convolution operation of the i-th layer. Performing the operation once on all layers from i=1 to i=L will result in a spatial encoding set. i = 1...L. This set is input into the next layer of motion direction perception converter for temporal feature extraction. This set contains spatial information at different scales extracted by the spatial encoder. Further extraction of this multi-scale spatial information can improve the perception capability of information at different scales.

[0058] The main function of the spatial encoder is to encode and upscale the input video sequence for spatial feature extraction. Assume the shape of the video frame sequence input to the model of this invention is (T, C, H, W), which is a set of 4-dimensional tensor data. Here, T represents the number of frames in the input video, C represents the number of channels in the input video, and H and W represent the height and width of the input video, respectively. The convolutional module in the first spatial encoder first maps the channels of the video frames to the hidden layer, changing the number of channels to C. h The data shape at this point becomes (T, C) h(H,W). Subsequently, the convolutional modules in even-numbered layers downsample the data with a stride of 2, reducing the length and width of the data to half of their original values. The convolutional modules in odd-numbered layers, however, have a stride of 1 and do not perform downsampling.

[0059] (2) The processing method of the motion direction sensing converter is as follows:

[0060] The motion orientation sensing converter consists of L layers, corresponding to the L features output by the spatial encoder. Each layer of the motion orientation sensing converter stacks N motion orientation sensing units. Specifically, the i-th layer consists of N... i It consists of 7 motion direction sensing units (MDAUnits). Each MDAUnit is composed of 7 units arranged sequentially:

[0061] 1. Two-dimensional convolution with a kernel size of 1×1;

[0062] 2. Batch normalization function (BatchNorm);

[0063] 3. Activation function (ReLU);

[0064] 4. Two parallel motion direction multilayer perceptrons (MDA-MLP);

[0065] 5. Two-dimensional convolution with a kernel size of 1×1;

[0066] 6. Batch normalization function (BatchNorm);

[0067] 7. Activation function (ReLU);

[0068] For units 1-3, their function is channel reduction, aiming to decrease the number of parameters in the MDA-MLP. Units 5-7 expand the reduced channels to restore the original number of channels. The motion direction multilayer perceptron in unit 4 exists in two parallel configurations within the MDAUnit, differing only in the number of groups. The following describes the processing method for the motion direction multilayer perceptron MDA-MLP in unit 4:

[0069] The motion direction multilayer perceptron (MDA-MLP), inspired by four types of direction-selective neurons in the animal retina, decomposes complex motion into horizontal and vertical components, reducing the difficulty of representation. Its structure includes:

[0070] • Horizontal branch: The input features are grouped and then a fully connected layer is applied along the horizontal direction to perform temporal modeling, outputting horizontal motion features.

[0071] • Vertical branch: Same as above, extract temporal features along the vertical direction and output vertical motion features.

[0072] • Channel Branch: Integrates features from horizontal and vertical branches, applies them to the channel dimension, and outputs the fused features.

[0073] Given input tensor After layer normalization (LayerNorm), the data is input into three branches. Here, we will explain in detail using the horizontal branch as an example. The horizontal branch first divides I into G groups, changing the tensor size to H×W×C / G, and then merges the channels with the H dimension to obtain... Then, the first fully connected layer (FC) in the H branch performs temporal feature extraction along the H dimension. The output of this fully connected layer is then subjected to squeezing and excitation operations along the H dimension to obtain the output of the H branch:

[0074] h = FC(I) m ),

[0075] E h =MLP(AvgPool(h,1)),

[0076] H o =E h ⊙h

[0077] Here, AvgPool(h,1) represents average pooling on the first dimension of h. ⊙ represents element-wise multiplication. Finally, the output data size is transformed back to H×W×C. The W branch performs a similar operation to the above formula, extracting the time features in the W dimension. To account for the relationship between the H and W dimensions, the C branch integrates the excitations of the H and W branches into the C dimension. The output of MDA-MLP is:

[0078] out = H o +W o +FC(I)⊙(E h +E w )

[0079] Among them, W o This indicates the output of branch W. E h and E w These are the excitations on the H-axis and W-axis, respectively.

[0080] Selectively processing motion information in this direction can reduce the dimensionality and extraction difficulty of spatiotemporal information, thereby improving the ability to extract motion information. The processing procedure of the motion direction-aware converter (MD-Translator) based on the motion direction-aware unit (MDAUnit) can be represented as follows:

[0081]

[0082] in, This is the set after time features have been extracted by the motion direction sensing converter. This is a mapping function for MDAUnit, with the superscript (N) i The expression indicates that the mapping function is executed N times. The main function of the Motion Direction Aware Translator (MD-Translator) is to extract the temporal motion information of spatially encoded video frames to predict motion. Assume the feature set input to the MD-Translator has four elements: Each feature is independently and laterally input into the motion orientation-aware converter for motion extraction, resulting in four feature sets with the exact same shape as the input. i = 1, 2, ..., L. This feature set is then input into the spatial decoder for feature fusion and outputs the final predicted video sequence.

[0083] (3) The processing method of the spatial decoder is as follows:

[0084] The spatial decoder process is the reverse of the spatial encoder process. All parameters are the same as the spatial encoder, except that the 2D convolution operation is replaced with deconvolution motion to restore the extracted features to the original video sequence. The spatial decoder needs to process the set of features extracted from the motion orientation-aware converter. The fusion is performed using a pyramid feature fusion method, where the output of the previous layer plus the input of the current layer is used as the final input of the current layer for computation.

[0085]

[0086] Among them, TransConv i (·) represents deconvolution operation. This represents the predicted video sequence.

[0087] The main function of the spatial decoder is to merge the processed multi-scale features, complete the spatial decoding, and output the final predicted video frame. Assume the feature set input to the spatial decoder has four elements: By performing deconvolution operations, starting from the last layer, Upsampling is performed, and the size is restored to... Later and The sums are then fed into the penultimate layer. This process continues until the final output T′×C×H×W is formed.

[0088] Step 3.2: Set the experimental hyperparameters;

[0089] In this method, the model hyperparameters for different datasets are shown in Table 2 below:

[0090] Table 2 Model Hyperparameters

[0091] Moving MNIST 64 {1,8,2,8} 2 {2，4} KTH 64 {1,2,2,4} 4 {4，8} Human 3.6 64 {1,5,2,6} 4 {4，8}

[0092] Step 4: Train the designed video frame prediction model. The specific steps are as follows:

[0093] Step 4.1: Define the error function, using the mean squared error as the training error function, as follows:

[0094]

[0095] in, Y represents the predicted video sequence, and T′ represents the actual video sequence. C represents the number of channels in the output video frame, and H and W represent the length and width of the output video frame, respectively.

[0096] Step 4.2: Set the learning rate and number of epochs; to achieve optimal model performance, different learning rates and number of epochs need to be set for different datasets. Specific experimental parameters are shown in Table 3 below:

[0097] Table 3 Training Parameters

[0098] Moving MNIST 0.01 2000 KTH 0.01 100 Human 3.6 0.01 200

[0099] Step 4.3: Use gradient descent to update the parameters that the video frame prediction model needs to learn, reduce the model prediction error, until the model's loss function converges;

[0100] Step 5: Verify and evaluate the accuracy of the trained video frame prediction model, including mean squared error (MSE), mean absolute error (MAE), structural similarity (SSIM), and peak signal-to-noise ratio (PSNR).

[0101] • MSE stands for Mean Squared Error, which is a widely used metric in statistics and machine learning to measure the difference between model predictions and actual values.

[0102] • MAE stands for Mean Absolute Error, which is one of the commonly used metrics in statistics and machine learning to evaluate the accuracy of model predictions. Mean Absolute Error measures the average difference between the model's predicted values ​​and the actual values.

[0103] SSIM stands for Structural Similarity Index, a metric used to measure the similarity of image quality. SSIM is calculated based on three aspects: contrast, structure, and brightness. It assesses the similarity between two images by comparing their structural information, rather than simply comparing pixel values. A higher SSIM value indicates that the two images are more similar in quality and structure.

[0104] PSNR stands for Peak Signal-to-Noise Ratio, a commonly used metric for measuring image quality. PSNR assesses image quality based on the ratio of signal to noise in an image. It measures the difference between the original and processed images by calculating the mean square error (MSE), and then normalizes the MSE using the dynamic range of pixel values ​​to obtain the PSNR value.

[0105] Tables 4, 5, 6, and 7 below show the experimental results of this method on the Moving MNIST (10→10) dataset, the KTH (10→20&40) dataset, and the Human3.6 (4→4) dataset, respectively.

[0106] Compared to other models, the model of this invention demonstrates superior prediction performance on all three datasets.

[0107] Table 4. Experimental results on the Moving MNIST (10→10) dataset.

[0108] ConvLSTM 103.3 182.9 0.707 PredRNN 56.8 126.1 0.867 SimVP 23.8 68.9 0.948 Ours 16.3 53.4 0.965

[0109] Table 5 Experimental results for the KTH(10→40) dataset.

[0110] ConvLSTM 0.712 24.31 PredRNN 0.839 27.64 SimVP 0.905 33.72 Ours 0.914 34.43

[0111] Table 6 Experimental results for the KTH(10→40) dataset.

[0112] ConvLSTM 0.639 22.93 PredRNN 0.703 24.16 SimVP 0.886 32.93 Ours 0.894 33.07

[0113] Table 7 Experimental results for the Human3.6 (4→4) dataset

[0114] ConvLSTM 50.4 18.9 0.776 PredRNN 47.8 18.7 0.753 SimVP 31.6 15.1 0.904 Ours 31.1 13.4 0.910

Claims

1. A video prediction method based on motion direction perception, the method comprising: Step 1: Obtain the dataset for training and validating the video frame prediction model. The dataset for video frame prediction has four dimensions: number of video frames S, number of video frame channels C, height of video frame image H, and width of video frame image W. Step 2: Preprocess the acquired dataset to obtain the training dataset; Step 3: Construct a video frame prediction model; The video frame prediction model consists of three parts: a spatial encoder, a motion orientation-aware converter, and a spatial decoder. The specific steps are as follows: (1) Spatial encoder processing method: The encoder consists of L convolutional modules connected in series; each convolutional module is composed of a convolutional unit, a group normalization function, and an activation function connected in sequence; in addition, the output of each convolutional module is not only input vertically into the next convolutional module, but also output horizontally to the motion direction sensing converter. If the convolutional module is the first convolutional module of the encoder, then the number of input channels of the convolutional units in this module is the number of input video frame channels C; otherwise, the number of input channels is the model hyperparameter C. h h is the number of the convolution module; All convolutional modules in the encoder have a kernel size of 3×3 and a grouping normalization group number of 2. If the convolutional module is the odd-numbered convolutional module of the encoder, then the convolutional stride of the convolutional unit of that convolutional module is 1, otherwise the convolutional stride is 2. ； in, This represents the output of the i-th convolutional layer. Represents the original video sequence, LeakyReLU(·) represents the activation function, GroupNorm(·) represents group normalization, and Conv i This represents the two-dimensional convolution operation of the i-th layer; performing the operation once on all layers from i=1 to i=L will yield a spatial encoding set. L is the total number of elements in the spatial coding set; this set is input into the next layer of motion direction sensing converter for temporal feature extraction; (2) The processing method of the motion direction sensing converter is as follows: The motion orientation sensing converter consists of L parallel layers, corresponding to the L features output by the spatial encoder; each layer of the motion orientation sensing converter stacks N motion orientation sensing units; specifically, the i-th layer consists of N... i It consists of three MDAUnits for motion direction sensing; each MDAUnit is composed of three parts: a channel reduction part, a motion direction multilayer sensor, and a channel expansion part. Both the channel reduction and channel expansion parts consist of 1×1 two-dimensional convolutions, batch normalization, and activation functions; In MDAUnit, two multilayer perceptrons for motion direction exist side-by-side, differing only in the number of groups. The structure of the multilayer perceptron for motion direction includes: Horizontal branch: The input features are grouped, and then a fully connected layer is applied along the horizontal direction to perform temporal modeling, outputting horizontal motion features; Vertical branch: Same as above, extract temporal features along the vertical direction and output vertical motion features; Channel branching: Integrates features from horizontal and vertical branches, applies them along the channel dimension, and outputs the fused features; ； in, This is the set after time features have been extracted by the motion direction sensing converter. It is a mapping function of MDAUnit, superscript This means that the mapping function will be executed N times; (3) The processing method of the spatial decoder is as follows: The spatial decoder processing flow is the reverse of the spatial encoder processing flow; all parameters are the same as those of the spatial encoder, except that the two-dimensional convolution operation is replaced with deconvolution motion to restore the extracted features to the original video sequence. ； in, Represents deconvolution operation. The video sequence represents the prediction; Step 3.2: Set the experimental hyperparameters; Step 4: Train the video frame prediction model built in Step 3; Step 5: Verify and evaluate the accuracy of the trained video frame prediction model, including metrics such as mean squared error, mean absolute error, structural similarity, and peak signal-to-noise ratio. Step 6: Perform predictive analysis using the evaluated video prediction model.

2. The video prediction method based on motion direction perception as described in claim 1, characterized in that, The specific steps of step 2 are as follows: Step 2.1: Set a sliding window based on the number of input and output video frames of the video frame prediction model to generate input data for the video frame prediction model and a pair of real data used to calculate the prediction error; Step 2.2: Generate an iterable dataset for training and validating the video frame prediction model based on the batch size of the video frame prediction model and the real data pairs.

3. The video prediction method based on motion direction perception as described in claim 1, characterized in that, The calculation method for step 4 is as follows: Step 4.1: Define the error function, using the mean squared error as the training error function, as follows: ； in, Represents the predicted video sequence, It is a real video sequence; C represents the number of video frames, H represents the number of channels in the output video frame, and H and W represent the length and width of the output video frame, respectively. Step 4.2: Set the learning rate and iteration rounds; Step 4.3: Use gradient descent to update the parameters that the video frame prediction model needs to learn, thereby reducing the model prediction error.

Images