Video prediction method based on deep learning non-autoregression model

Abstract

The invention provides a video prediction method based on a deep learning non-autoregression model, and relates to the field of man-machine interaction, climate prediction and automatic driving. According to the method, how to accurately predict a future video sequence according to an existing video sequence is researched, firstly, Encoder is used for decoding video frames, two-dimensional convolution calculation, LayerNorm normalization processing and SILU activation are carried out, and space information between the video frames can be extracted. And secondly, position codes are added in the model, the capability of extracting time information is increased on the basis of extracting the space information, and the time information and the space information are well fused, so that the problem that the space-time information is difficult to obtain by a non-autoregression model is solved. Besides, a previous attention mechanism is improved, a time-space attention mechanism with higher pertinence is designed, the acquisition of time-space information is further improved, and the accuracy of a prediction result is improved.

Description

Technical Field

[0001] The present invention belongs to the fields of human-computer interaction, climate forecasting and autonomous driving. It predicts video sequences in the future based on existing video sequences. The specific design is a non-autoregressive model based on deep learning. Background Art

[0002] In recent years, the field of deep learning has garnered significant attention. However, breakthroughs in some key areas remain elusive. Although numerous deep learning-based video prediction models have been proposed in the field of video prediction, some challenges remain. For example, the ConvLSTM model combines CNN (Convolutional Neural Networks) and LSTM (Long Short-Term Memory Networks) to effectively capture spatiotemporal information. However, due to the model's autoregressive nature, errors accumulate during the computation process, impacting model performance. Other approaches that address similar issues include PredRNN, MotionRNN, and MIM, all of which are autoregressive models.

[0003] Taking this problem of the autoregressive model into consideration, many non-autoregressive models have emerged in this field. These models can avoid the problem of error accumulation and effectively solve the shortcomings of the autoregressive model. However, other problems arise, namely, they cannot well capture the dependencies between time and space, which also leads to reduced performance. Summary of the Invention

[0004] Purpose of the Invention: This invention provides a video prediction method based on a deep learning non-autoregressive model. This method effectively predicts the subsequent development of a video based on existing video information, including human motion, climate change, and object trajectories. Furthermore, it effectively addresses the error accumulation associated with autoregressive models and the limitations of non-autoregressive models in their reliance on spatial and temporal information.

[0005] Technical solution: A video prediction method based on a deep learning non-autoregressive model, including the following steps:

[0006] Step 1: Divide the video sequence into multiple video frames, input the video frames into the encoder according to the time dimension for training, and then input the results into the spatiotemporal predictor, while adding the temporal position information obtained from the position encoding module.

[0007] Step 2: Input the result obtained from the predictor to the decoder for decoding.

[0008] Step 3: The decoded data is used as the input of the final spatiotemporal attention module to capture the spatiotemporal information between the data and convert the data into the final output frame to obtain the predicted video sequence.

[0009] The beneficial technical effects of the present invention are:

[0010] 1. Based on the non-regression model, it effectively avoids the performance degradation problem caused by error accumulation in the autoregression model.

[0011] 2. The existing non-autoregressive model is improved and a position encoding module is added to enhance the ability to capture spatiotemporal information.

[0012] 3. An attention mechanism was added and the existing attention mechanism was improved to better understand the relationship between time and space in video frames and improve the accuracy of prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0014] Figure 1 It is a schematic diagram of the algorithm flow of the method of the present invention. Figure 2 Schematic diagram of the model structure of the method of the present invention DETAILED DESCRIPTION

[0015] The present invention is further illustrated below with reference to specific examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0016] The present invention is a video prediction method based on a deep learning non-autoregressive model. The method is characterized by the use of a novel autoregressive video prediction model to effectively capture the spatiotemporal dependencies between video frames while reducing the accumulation of errors in the prediction process. The model is based on an encoder, predictor, and decoder architecture, and introduces position encoding and spatiotemporal attention modules into the architecture. Position encoding can well capture the temporal position information of frames in a video sequence, while the role of the spatiotemporal attention module is to model the dependencies between time and space, thereby better acquiring spatiotemporal information. The specific implementation steps of the algorithm are as follows:

[0017] Step 1: Split the video sequence into multiple video frames X = {x1, ..., x 1+t}, and feed the video frames into the model as input data.

[0018] Step 2: The processed data passes through the first module of the model, the Encoder module, which extracts the spatial information of the video frame. In this module, the video frame passes through n-1 layers of EncConvBlock, which specifically calculates a 2D convolution operation, a LayerNorm normalization process, and a SILU activation function to obtain the output data z1.

[0019] Step 3: Input the initial video frame into the position encoding module, which calculates as follows:

[0020]

[0021] Where X represents the input sequence, emb represents the calculated frequency information, freq represents all the frequency information, and pos represents the position code of each x. Finally, the sin function and cos function are used to calculate pos and integrate them to obtain the output result of this module.

[0022] Step 4: Input the results obtained from steps 2 and 3 into the spatiotemporal predictor (STP) module together, and obtain the output result after passing through 1 layer of ExtendedConvBlock and n layers of BottleneckConvBlock.

[0023] Step 5: Pass the initial video frame through the last layer EncConvBlock of the encoder, and integrate the result with the result of step 4 to obtain data z2.

[0024] Step 6: Input the result z2 obtained in step 5 into the decoder (Decoder), and pass through n layers of DecConvBlock. The calculation of each layer needs to go through 2D convolution and PixelShuffle upsampling to obtain the output result of the module.

[0025] Step 7: Use the output of step 6 as the input of the next module, the spatiotemporal attention mechanism (STA), and calculate it using the following formula:

[0026]

[0027] Get the final output data Y={y1,........,y 1+t}.

[0028] At this point, the entire process of predicting video data is completed.

Claims

1. A video prediction method based on deep learning non-autoregressive model. The method is characterized by the use of a novel autoregressive video prediction model to effectively capture the spatiotemporal dependencies between video frames while reducing the accumulation of errors in the prediction process. The model is based on an encoder, predictor, and decoder architecture, and introduces position encoding and spatiotemporal attention modules into the architecture. Position encoding can effectively capture the temporal position information of frames in a video sequence, while the spatiotemporal attention module is used to model the dependencies between time and space, thereby better capturing spatiotemporal information. The method mainly consists of the following steps: Step 1: Divide the video sequence into multiple video frames, input the video frames into the encoder according to the time dimension for training, and then input the results into the spatiotemporal predictor, while adding the temporal position information obtained from the position encoding module. Step 2: Input the result obtained from the predictor to the decoder for decoding. Step 3: The decoded data is used as the input of the final spatiotemporal attention module to capture the spatiotemporal information between the data and convert the data into the final output frame to obtain the predicted video sequence.

2. According to claim 1, the encoder is used to extract spatial information from video frames. The encoder consists of n layers of encoding modules, each of which includes a 2D convolutional layer (with a kernel size of 3x3), LayerNorm, and SILU activation function. The spatiotemporal predictor consists of a layer of ExtendedConvBlock and n layers of BottleneckConvBlock, with temporal position information incorporated into the calculations of each layer. The calculation process in the position encoding module is as follows: X=[x1,x2,......,x N ] pos=X T freq Where X represents the input sequence, emb represents the calculated frequency information, freq represents all the frequency information, and pos represents the position code of each x. Finally, the sin function and cos function are used to calculate pos and integrate them to obtain the output result of this module.

3. The decoder of claim 1 is configured to integrate spatiotemporal information to predict future frames. The decoder comprises n layers of decoding modules, each of which includes a 2D convolutional layer (with a kernel size of 3x3) and a PixelShuffle upsampling method.

4. The spatiotemporal attention module is the last module of the model. The calculation formula of this module is: In this formula, Q, K, and V represent the query, key, and value vectors of each attention head, respectively. head_dim=4 indicates the number of heads in the attention mechanism, controlling the granularity of attention.

Images