Video prediction method based on Taylor decoupling and memory unit correction
Through the method of Taylor decoupling and memory unit correction, the video frame is decoupled into Taylor component and residual component, which solves the problem of excessive video prediction parameters and poor long-distance prediction results in the prior art, and achieves more efficient Video prediction effect.
Patent Information
- Application Number
- CN202110431011.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-21
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-04-21
AI Technical Summary
The existing video prediction methods have too many parameters and poor results in long-distance prediction, and the introduction of proxy objects requires additional information, making it difficult to effectively extract the time correlation in the video.
The video prediction method based on Taylor decoupling and memory unit correction is adopted to decouple the video frame into Taylor component and residual component, and prediction is performed separately through the Taylor model and the residual model, and the memory unit correction term is used to avoid error accumulation.
It improves the accuracy and efficiency of video prediction, reduces network parameters, avoids error accumulation in long-distance prediction, and enhances the modeling ability of video sequence variables.
Smart Images

Figure CN113132737B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer vision video technology, in particular to a video prediction method based on Taylor decoupling and memory unit correction. Background Art
[0002] Prediction and reasoning are one of the most prominent features of intelligence and are also the main goals of decision-making systems. Humans’ common sense about the world comes from early learning and knowledge accumulation. By comparing prediction results with actual facts, the brain can extract a model of the world, thereby obtaining more accurate prediction results in subsequent predictions. Video prediction can be seen as an intermediate step between raw video data and decision-making systems. It can extract potential evolution patterns from raw video data and has broad application prospects in meteorology, transportation, health sciences, anomaly detection and other fields.
[0003] Converting observational data into predictive models that can be used by decision-making systems is of great significance in science and engineering. Although video data is being collected at an increasingly faster rate, designing an automatic predictive model from these raw data is still a very challenging problem. Video prediction requires that neural networks can accurately learn the chaotic dynamics present in the raw video data so that they can accurately model future states. This process is completely data-driven.
[0004] Unlike static images, video frames have strong correlations in the continuous time dimension and have certain fixed motion or deformation patterns. In continuous time steps, due to temporal consistency, there are a large number of visually similar local deformations at the same position in the video frame. From a global perspective, consecutive frames have different visual effects, but they are semantically consistent. A flying bird will not turn into an airplane in the next few frames, and a ball rolling off a table will not fly upwards. From these temporal clues, the video prediction model can extract the temporal correlations in the video, and use this temporal correlation to model possible future states.
[0005] A common method for modeling the temporal correlation of videos is to use ConvLSTM or Conv3D. ConvLSTM is a neural network used to process sequence data. Compared with general neural networks, it has memory units and can process data with sequence changes, so that previous data can contribute to the current state. Videos are composed of multiple consecutive video frames, which are sequence data containing sequential changes. They are very suitable for ConvLSTM. In the past few years, many models that have improved ConvLSTM to complete video prediction tasks have achieved good results. Similar to ConvLSTM, Conv3D is also a neural network structure commonly used for three-dimensional modeling. Video frame sequences have a time dimension in addition to static images, so using Conv3D to model dynamic changes is also very effective. In addition to increasing the accuracy of video prediction by improving the memory capacity of the network, another common method is decoupling. By designing targeted network function modules, different parts of the underlying dynamic evolution in the video are modeled separately to further improve the accuracy of video prediction, such as separating foreground and background, content and action, time variables and spatial variables, physical variables and residual variables, etc. In addition, proxy objects can also be introduced, such as segmentation maps and posture maps.
[0006] Simply improving ConvLSTM or Conv3D to perform video prediction often results in a lot of network parameters, and the long-distance prediction effect is not good. The introduction of proxy objects requires additional information. Therefore, the present invention still uses the idea of decoupling, but adopts a new decoupling method to decouple complex video data into Taylor components and residual components, dig out hidden Taylor models from the observed complex dynamic data, and only use the Taylor expansion of the first frame in the derivation process, avoiding error accumulation in long-distance prediction. Summary of the invention
[0007] The purpose of the present invention is to overcome the shortcomings of the prior art, fully consider the underlying physical logic of the video frame sequence, and propose a video prediction method based on Taylor decoupling and memory unit correction.
[0008] The present invention solves the technical problem by adopting the following technical solutions:
[0009] A video prediction method based on Taylor decoupling and memory unit correction comprises the following steps:
[0010] Step 1: Normalize all video frames from the pixel value range [0,255] to [0,1];
[0011] Step 2: Send the first frame x0 of the input video frame to the encoder E to obtain the 16×16×64 deep feature u0, and then send u0 to the Taylor encoder and residual encoder respectively to obtain the Taylor component and the residual component
[0012] Step 3: Taylor component Send it into the Taylor model to derive the predicted Taylor component for the next moment Then Send it to the Taylor decoder to get the predicted Taylor feature
[0013] Step 4: The residual component Send it into the residual model to derive the predicted residual component for the next moment Then Send to the residual decoder to get the predicted residual features
[0014] Step 5: Predict Taylor characteristics and predicted residual features Add them together and send the sum to the decoder D to get the predicted frame for the next moment
[0015] Step 6: Let all the remaining input video frames except the last frame complete steps 2 to 5 in sequence to obtain the predicted input video frame
[0016] Step 7: Let the last frame be input video frame x t-1 Complete steps 2 to 5 in sequence to obtain the first frame prediction target video frame
[0017] Step 8: Let the predicted target video frame (this is the student training mode, the real target video frame is used in the teacher training mode, and the probability of the student mode and the teacher mode is 0.5 each time during training) complete steps 2 to 5 to obtain the predicted target video frame at the next moment;
[0018] Step 9, repeat step 8 until all predicted target video frames are output;
[0019] Step 10: Adjust the pixel value range of all predicted target video frames from [0, 1] to [0, 255] to obtain the final predicted target video sequence.
[0020] Further, the specific method for the encoding module in step 2 to process the input video frame includes the following steps:
[0021] ⑴ Encoder E uses three cascaded convolution layers to extract the deep features of the video frame and expand the low-dimensional video frame information in the high-dimensional feature space. The three convolution kernels are 3×3×32, 3×3×32, and 3×3×64, respectively, and the final deep feature dimension is 16×16×64;
[0022] ⑵The Taylor encoder uses two cascaded convolution layers to extract the Taylor component of the video frame depth feature in the high-dimensional feature space. The two layers of convolution kernels are 3×3×64 and 3×3×64 respectively;
[0023] ⑶ The residual encoder uses two cascaded convolution layers to extract the residual components of the video frame depth features in the high-dimensional feature space. The two convolution kernels are 3×3×64 and 3×3×64 respectively.
[0024] Further, the specific method of deriving and decoding the Taylor component in step 3 includes the following steps:
[0025] ⑴ The Taylor component obtained by inputting the first frame x0 of the video frame Enter the partial differential model to get Will Send it into the partial differential model again to get Reuse memory unit K t right Correction can be performed to obtain the predicted Taylor component at time t
[0026] ⑵Send the predicted Taylor component into two-layer cascaded convolutional layers for decoding, and convert the Taylor feature space into a fusion feature space. The two-layer convolution kernels are 3×3×64 and 3×3×64 respectively.
[0027] Further, the specific method of the memory unit correction in step (1) comprises the following steps:
[0028] ① Send the input video frame at time t-1 to encoder E and Taylor encoder in sequence to obtain the Taylor component of this video frame Then combine it with the hidden state e of the memory cell t-1 Cascade, through a convolution layer to get the update gate z and reset gate r, and then through a convolution layer to get the candidate state Then use the candidate state to update the hidden state to get the hidden state of the memory unit at time t
[0029] ② e t and Cascade, through a convolution layer to get the correction term K at time t t , and then use K t right Correction is performed to obtain the predicted Taylor component at time t
[0030] Further, the specific method of obtaining the partial derivative of the partial differential model in step (1) comprises the following steps:
[0031] ①The Taylor component obtained by inputting the first frame x0 of the video frame is Because many classical physical models (such as heat equation, wave equation, advection-diffusion equation, etc.) satisfy The finite space differential It can be approximated by a convolution kernel;
[0032] ②The size of the convolution kernel w is k×k. We can define a time matrix M(w), which is also k×k in size:
[0033]
[0034] Then when any function h is convolved with w:
[0035]
[0036] Taylor expansion of the function h:
[0037]
[0038] If k = 3, then when h and w convolution can be fitted The above formula can be fitted Similarly, when M(w) approaches Stacking, h passes through the convolution w, and then passes through a 1×1 filter to get The linear combination of
[0039] The partial differential model in ③⑴ consists of two layers of convolution, with convolution kernels of 7×7×49 and 1×1×64 respectively. The time matrix M(w) is constructed for the convolution kernel of the first convolution layer, and the final loss function is added in is a 7×7 matrix with 1 at position (i, j) and 0 at other positions, which makes each filter The effect is similar The second filter gives The linear combination of
[0040] Further, the specific method of deriving and decoding the residual component in step 4 includes the following steps:
[0041] ⑴ Input the input video frame at time t-1 into encoder E and residual encoder in sequence to obtain the residual component of this video frame The residual model is a 3-layer ConvLSTM cascade, with convolution kernels of 3×3×128, 3×3×128, and 3×3×64 respectively. The input of the first layer is The input of the last two layers is the output of the previous layer, and the output of the last layer is the predicted residual component at time t.
[0042] ⑵The predicted residual component is sent to two cascaded convolutional layers for decoding, and the residual feature space is converted into a fusion feature space. The two layers of convolution kernels are 3×3×64 and 3×3×64 respectively.
[0043] Further, the specific method of the decoder D in step 5 includes the following steps:
[0044] ⑴Decoder D uses three layers of cascaded deconvolution layers to integrate the deep features of the video frame and fuse the high-dimensional video frame information in a low-dimensional space. The convolution kernels of the three layers of deconvolution are 3×3×32, 3×3×32, and 3×3×1, respectively, and the predicted video frame size is 64×64×1.
[0045] The advantages and positive effects of the present invention are:
[0046] 1. The present invention uses a new video decoupling method to decouple complex video data into Taylor components and residual components, and uses Taylor components to model the underlying physical logic existing in complex dynamic data.
[0047] 2. When performing temporal derivation of the Taylor components, only the Taylor expansion for the first frame is used to avoid error accumulation in long-term predictions.
[0048] 3. Use memory cells that incorporate all previous frame information to correct the Taylor component, simulating additional variables in the video sequence. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a framework diagram of the video prediction network of Taylor decoupling and memory unit correction of the present invention.
[0050] Figure 2 It is a framework diagram of the Taylor submodule of the present invention. DETAILED DESCRIPTION
[0051] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0052] A video prediction method based on Taylor decoupling and memory unit correction, such as Figure 1 and Figure 2 As shown, the following steps are included:
[0053] Step S1, normalize all video frames from the pixel value range [0,255] to [0,1];
[0054] Step S2: Send the first frame x0 of the input video frame to the encoder E to obtain the 16×16×64 deep feature u0, and then send u0 to the Taylor encoder and the residual encoder respectively to obtain the Taylor component and the residual component
[0055] The specific implementation method of step S2 is as follows:
[0056] Step S2.1, encoder E uses three layers of cascaded convolutional layers to extract the deep features of the video frame and expand the low-dimensional video frame information in the high-dimensional feature space. The three layers of convolution kernels are 3×3×32, 3×3×32, and 3×3×64, respectively, and the final deep feature dimension is 16×16×64;
[0057] Step S2.2, Taylor encoder uses two layers of cascaded convolutional layers to extract Taylor components of video frame depth features in high-dimensional feature space. The two layers of convolutional kernels are 3×3×64 and 3×3×64 respectively;
[0058] Step S2.3: The residual encoder uses two cascaded convolution layers to extract the residual components of the video frame depth features in the high-dimensional feature space. The two convolution kernels are 3×3×64 and 3×3×64 respectively.
[0059] Step S3: Taylor component Send it into the Taylor model to derive the predicted Taylor component for the next moment Then Send to Taylor decoder to get predicted Taylor features
[0060] The specific implementation method of step S3 is as follows:
[0061] Step S3.1: The Taylor component obtained by inputting the first frame x0 of the video frame Enter the partial differential model to get Will Send it into the partial differential model again to get Reuse memory unit K t right Correction can be performed to obtain the predicted Taylor component at time t
[0062] Step S3.2: Send the predicted Taylor component to the two-layer cascaded convolution layer for decoding, and convert the Taylor feature space into a fusion feature space. The two-layer convolution kernels are 3×3×64 and 3×3×64 respectively.
[0063] The specific implementation method of step S3.1 is as follows:
[0064] S3.1.1、The Taylor component of the first frame x0 of the input video frame is Because many classical physical models (such as heat equation, wave equation, advection-diffusion equation, etc.) satisfy The finite space differential It can be approximated by a convolution kernel;
[0065] S3.1.2. The size of the convolution kernel w is k×k. We can define a time matrix M(w) whose size is also k×k:
[0066]
[0067] Then when any function h is convolved with w:
[0068]
[0069] Taylor expansion of the function h:
[0070]
[0071] If k = 3, then when h and w convolution can be fitted The above formula can be fitted Similarly, when M(w) approaches Stacking, h passes through the convolution w, and then passes through a 1×1 filter to get The linear combination of
[0072] The partial differential model in S3.1.3 and (1) consists of two layers of convolution, with convolution kernels of 7×7×49 and 1×1×64 respectively. The time matrix M(w) is constructed for the convolution kernel of the first convolution layer, and the final loss function is added in is a 7×7 matrix with 1 at position (i, j) and 0 at other positions, which makes each filter The effect is similar The second filter gives The linear combination of
[0073] S3.1.4, the input video frame at time t-1 is sent to encoder E and Taylor encoder in sequence to obtain the Taylor component of this video frame Then combine it with the hidden state e of the memory cell t-1 Cascade, through a convolution layer to get the update gate z and reset gate r, and then through a convolution layer to get the candidate state Then use the candidate state to update the hidden state to get the hidden state of the memory unit at time t
[0074] S3.1.5, e t and Cascade, through a convolution layer to get the correction term K at time t t , and then use K t right Correction is performed to obtain the predicted Taylor component at time t
[0075] Step S4: Residual component Send it into the residual model to derive the predicted residual component for the next moment Then Send to the residual decoder to get the predicted residual features
[0076] The specific implementation method of step S4 is as follows:
[0077] S4.1. Input the input video frame at time t-1 into encoder E and residual encoder in sequence to obtain the residual component of this video frame The residual model is a 3-layer ConvLSTM cascade, with convolution kernels of 3×3×128, 3×3×128, and 3×3×64 respectively. The input of the first layer is The input of the last two layers is the output of the previous layer, and the output of the last layer is the predicted residual component at time t.
[0078] S4.2. Send the predicted residual component to two cascaded convolutional layers for decoding, and convert the residual feature space into a fusion feature space. The two layers of convolution kernels are 3×3×64 and 3×3×64 respectively.
[0079] Step S5: Predict Taylor characteristics and predicted residual features Add them together and send the sum to the decoder D to get the predicted frame for the next moment
[0080] The specific implementation method of step S5 is as follows:
[0081] S5.1, decoder D uses three layers of cascaded deconvolution layers to integrate the deep features of the video frame and fuse the high-dimensional video frame information in a low-dimensional space. The convolution kernels of the three layers of deconvolution are 3×3×32, 3×3×32, and 3×3×1, respectively, and the predicted video frame size is 64×64×1.
[0082] Step S6: let all the remaining input video frames except the last frame complete steps 2 to 5 in sequence to obtain the predicted input video frame
[0083] Step S7: Let the last frame input video frame x t-1 Complete steps 2 to 5 in sequence to obtain the first frame prediction target video frame
[0084] Step S8, let the predicted target video frame (this is the student training mode, the real target video frame is used in the teacher training mode, and the probability of the student mode and the teacher mode is 0.5 each time during training) complete steps 2 to 5 to obtain the predicted target video frame at the next moment;
[0085] Step S9, repeating step 8 until all predicted target video frames are output;
[0086] Step S10: adjust the pixel value range of all predicted target video frames from [0, 1] to [0, 255] to obtain a final predicted target video sequence.
[0087] Through the above steps, the predicted target video sequence can be obtained.
[0088] Finally, we train the network with reconstruction loss and temporal matrix loss as the target, and use MSE (Mean Square Error), MAE (Mean Absolute Error) and SSIM (Structural Similarity) to evaluate the network performance. The method is as follows:
[0089] Test environment: Python 3.6; Pytorch deep learning framework; Ubuntu 16.04 system; NVIDIA GTX1080ti GPU.
[0090] Test sequence: The selected dataset is Moving MNIST. Two handwritten digits are randomly selected from the MNIST dataset each time, and then the moving trajectory is randomly generated to generate a 64×64 video sequence. When the digits encounter the edge, they bounce according to a fixed rule. During training, 10 frames are input and 10 frames are predicted. The training set is dynamically generated, and the test set contains 10,000 fixed sequences.
[0091] Testing method: The video prediction model is obtained through training, and the above test set is used for testing to calculate the evaluation index data.
[0092] Test index: The present invention uses MSE, SSIM and other indexes for evaluation. These index data are calculated for different popular algorithms and then the results are compared, proving that the present invention obtains better results in the field of video prediction.
[0093] The test results are as follows:
[0094] Table 1. Performance comparison of the proposed method and other algorithms on the Moving MNIST dataset (10→10)
[0095]
[0096]
[0097] From the above comparative data, it can be seen that the present invention is superior to all other methods. Among them, CrevNet is currently the most advanced model on the MOVING MNIST dataset, but it can only obtain the results in the table by using spatiotemporal LSTM. Spatiotemporal LSTM is an improved spatiotemporal LSTM that can enhance spatiotemporal dependency. Our model only uses ordinary ConvLSTM, but the effect is still better than CrevNet.
[0098] Any matters not described in the present invention are applicable to the prior art.
[0099] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.
Claims
1. A video prediction method based on Taylor decoupling and memory unit correction, characterized in that The following steps are involved: Step 1: Normalize all video frames from the pixel value range [0,255] to [0,1]; Step 2: Send the first frame x0 of the input video frame to the encoder E to obtain the 16×16×64 deep feature u0, and then send u0 to the Taylor encoder and residual encoder respectively to obtain the Taylor component and the residual component Step 3: Taylor component Send it into the Taylor model to derive the predicted Taylor component for the next moment Then Send to Taylor decoder to get predicted Taylor features The specific method of step 3 includes the following steps: (1) the Taylor component obtained by inputting the first frame x0 of the video frame Enter the partial differential model to get Will Send it into the partial differential model again to get Reuse memory unit K t right Correction can be performed to obtain the predicted Taylor component at time t (2) The predicted Taylor component is sent to two cascaded convolutional layers for decoding, and the Taylor feature space is converted into a fusion feature space. The two convolution kernels are 3×3×64 and 3×3×64 respectively; The specific method of memory unit calibration comprises the following steps: (1) The input video frame at time t-1 is sent to encoder E and Taylor encoder in sequence to obtain the Taylor component of this video frame Then combine it with the hidden state e of the memory cell t-1 Cascade, through a convolution layer to get the update gate z and reset gate r, and then through a convolution layer to get the candidate state Then use the candidate state to update the hidden state to get the hidden state of the memory unit at time t (2) e t and Cascade, through a convolution layer to get the correction term K at time t t , and then use K t right Correction is performed to obtain the predicted Taylor component at time t Step 4: The residual component Send it into the residual model to derive the predicted residual component for the next moment Then Send to the residual decoder to get the predicted residual features Step 5: Predict Taylor characteristics and predicted residual features Add them together and send the sum to the decoder D to get the predicted frame for the next moment Step 6: Let all the remaining input video frames except the last frame complete steps 2 to 5 in sequence to obtain the predicted input video frame Step 7: Let the last frame be input video frame x t-1 Complete steps 2 to 5 in sequence to obtain the first frame prediction target video frame Step 8: Let the first frame predict the target video frame Complete steps 2 to 5 to obtain the predicted target video frame at the next moment Step 9, repeat step 8 until all predicted target video frames are output; Step 10: Adjust the pixel value range of all predicted target video frames from [0, 1] to [0, 255] to obtain the final predicted target video sequence.
2. The video prediction method based on Taylor decoupling and memory unit correction according to claim 1, characterized in that: The specific method for the encoding module in step 2 to process the input video frame comprises the following steps: (1) Encoder E uses three cascaded convolution layers to extract the deep features of the video frame, and expands the low-dimensional video frame information in the high-dimensional feature space. The three-layer convolution kernels are 3×3×32, 3×3×32, and 3×3×64, respectively. Finally, the deep feature dimension is 16×16×64. (2) The Taylor encoder uses two cascaded convolution layers to extract the Taylor component of the video frame depth feature in the high-dimensional feature space. The two convolution kernels are 3×3×64 and 3×3×64 respectively. ⑶ The residual encoder uses two cascaded convolutional layers to extract the residual components of the deep features of the video frame in the high-dimensional feature space. The two layers of convolution kernels are 3×3×64 and 3×3×64 respectively.
3. The video prediction method based on Taylor decoupling and memory unit correction according to claim 1, characterized in that: The specific method for obtaining partial derivatives of the partial differential model comprises the following steps: ⑴The Taylor component obtained by inputting the first frame x0 of the video frame is (2) The size of the convolution kernel w is k×k, and a time matrix M(w) is defined, whose size is also k×k: Then when any function h is convolved with w: Taylor expansion of the function h: If k = 3, then when h and w convolution can be fitted The above formula can be fitted Similarly, when M(w) approaches Stacking, h passes through the convolution w, and then passes through a 1×1 filter to get The linear combination of (3) The partial differential model consists of two layers of convolutions, with convolution kernels of 7×7×49 and 1×1×64 respectively. The time matrix M(w) is constructed for the convolution kernel of the first convolution layer, and is added to the final loss function. in is a 7×7 matrix with 1 at position (i, j) and 0 at other positions, which makes each filter The effect is similar The second filter gives The linear combination of 4. The video prediction method based on Taylor decoupling and memory unit correction according to claim 1, characterized in that: The specific method of deriving and decoding the residual component in step 4 comprises the following steps: ⑴ Input the input video frame at time t-1 into encoder E and residual encoder in sequence to obtain the residual component of this video frame The residual model is a 3-layer ConvLSTM cascade, with convolution kernels of 3×3×128, 3×3×128, and 3×3×64 respectively. The input of the first layer is The input of the last two layers is the output of the previous layer, and the output of the last layer is the predicted residual component at time t. ⑵The predicted residual component is sent to two cascaded convolutional layers for decoding, and the residual feature space is converted into a fusion feature space. The two layers of convolution kernels are 3×3×64 and 3×3×64 respectively.
5. The video prediction method based on Taylor decoupling and memory unit correction according to claim 1, characterized in that: The specific method of the decoder D in step 5 comprises the following steps: ⑴Decoder D uses three cascaded deconvolution layers to integrate the deep features of the video frame and fuse the high-dimensional video frame information in the low-dimensional space. The convolution kernels of the three layers of deconvolution are 3×3×32, 3×3×32, and 3×3×1, respectively. Finally, the predicted video frame size is 64×64×1.
