Video prediction method and system combining spatio-temporal memory features and motion perception

Abstract

The application discloses a video prediction method and system combining space-time memory characteristics and motion perception, comprising: adding a modulation gate on the basis of a standard gated recurrent unit, and controlling forgotten information by the modulation gate and a reset gate; introducing a space-time memory and attention mechanism to design a space-time attention fusion unit; introducing motion perception on the basis of a gradient highway unit, proposing a motion gradient highway unit, and obtaining interframe transient changes and motion trends; and cooperating the space-time attention fusion unit with the motion highway modulation unit to construct a new recurrent neural network, and realizing video prediction. The application reduces the loss of information in the feature extraction process and the influence of the gradient vanishing problem, obtains multi-granularity space-time features and motion information, and greatly improves the prediction accuracy and efficiency of the video.

Description

Technical Field

[0001] This invention belongs to the field of electronic communication and information engineering, and specifically relates to a video prediction method and system that combines spatiotemporal memory features and motion perception. Background Technology

[0002] Video prediction is a crucial research area in video processing and analysis. This task aims to estimate potential events in future video frames using continuous video frame input, making predictions before events occur, effectively saving resources and producing desired results. Currently, this technology is widely applied in research fields such as robot control, video interpolation, autonomous driving, and motion planning. Relying on video prediction under motion conditions, robots can successfully manipulate previously unseen objects, and autonomous vehicles can avoid pedestrians in advance. This technology benefits not only the present but also the future, making video prediction a vital research topic in computer vision. With the widespread use of various sensors, devices, and the internet, and the arrival of the 5G era, video data is ubiquitous in people's daily lives and is increasingly becoming a vital carrier of information. To fully mine and utilize the valuable information in this video data, video prediction technology is receiving increasing attention. If video can be predicted quickly and effectively, preventative measures can be taken in advance, reducing significant losses.

[0003] However, for video prediction tasks, models need to abstract various detailed information from images, including not only static information in the spatial dimension but also dynamic information in the temporal dimension. Complex spatiotemporal dynamics have a significant impact on video prediction tasks. Limited dynamic input sequences and the complex variations in motion itself severely limit the model's ability to represent motion information between frames and its ability to transform complex spatiotemporal features.

[0004] Current conventional methods for video prediction are mainly based on dense motion or motion vectors. In particular, optical flow-based methods have proven to be quite effective for video prediction. Optical flow-based video prediction methods calculate the motion velocity of each pixel in the video using optical flow and incorporate this information as input features into the model, enhancing the model's ability to perceive motion changes and improving prediction accuracy. However, optical flow methods often require calculating the motion between pixels between two frames, which is frequently affected by factors such as image texture and lighting. Therefore, motion estimation for different texture regions may be inaccurate. Furthermore, in fast-moving video sequences, the short time intervals between adjacent frames lead to rapid changes in pixel displacement, easily resulting in data loss and impacting prediction performance. Summary of the Invention

[0005] The technical problem this invention aims to solve is to propose a video prediction method and system that combines spatiotemporal memory features and motion perception. Continuous video frames are used as input data to a novel recurrent neural network constructed by combining motion gradient highway units and four stacked spatiotemporal attention fusion units to complete encoding and prediction. This prediction structure reduces the loss of information during feature extraction and memory processes, enabling the model to have stronger spatiotemporal feature learning capabilities and significantly improving prediction accuracy.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] The video prediction method proposed in this invention, which combines spatiotemporal memory features and motion perception, includes the following steps:

[0008] S1. Obtain the video dataset, preprocess it, and convert it into a 5D tensor.

[0009] S2. Construct a spatiotemporal attention fusion unit.

[0010] S3. Construct motion gradient highway units.

[0011] S4. Construct a spatiotemporal prediction network and introduce a joint loss function to train the network to obtain a trained spatiotemporal prediction network containing video sequences of n time steps. Each video sequence includes 4 layers of spatiotemporal attention fusion units and 3 layers of motion gradient highway units.

[0012] S5. The 5D tensor from step S1 is used as the input to the trained spatiotemporal prediction network and input into the first layer spatiotemporal attention fusion unit to obtain the initial spatiotemporal features. The initial spatiotemporal features are adaptively extracted by a layer of motion gradient highway unit to obtain the initial spatiotemporal motion features.

[0013] S6. Input the initial spatiotemporal features and initial spatiotemporal motion features from step S5 into the second-layer spatiotemporal attention fusion unit for feature extraction. To improve the completeness and timeliness of the feature information, the extracted features are input vertically into the motion gradient highway unit between the current time step layers and horizontally into the spatiotemporal attention fusion unit of the next time step. This process continues until the final spatiotemporal features are obtained in the fourth-layer spatiotemporal attention fusion unit. The final predicted sequence is then output, and the corresponding single-frame predicted image is obtained through deconvolution.

[0014] S7. Input the initial spatiotemporal features from step S5 and the final spatiotemporal features from step S6 into the first layer spatiotemporal attention fusion unit of the spatiotemporal prediction network that has been trained at the current time step, and use it together with the 5D tensor from step S1 as input information. Repeat steps S5-S6 again until the predicted video frames for n time steps are obtained.

[0015] Furthermore, in step S2, the specific steps for constructing the spatiotemporal attention fusion unit are as follows:

[0016] S201. The output of the spatiotemporal attention fusion unit at the same layer in the past time state and the input of the current time state are used as time input information, and attention features are obtained by assigning weights to them using an attention mechanism. The specific formula is as follows:

[0017]

[0018] Among them, H att X represents attentional features. t The input represents the current time status. This represents the hidden state of the l-layer spatiotemporal attention fusion unit at time t-1, and ATT represents the attention mechanism.

[0019] S202. The time input information passes through a reset gate, a modulation gate, and an update gate to obtain different gated spatiotemporal features, as shown in the specific formula:

[0020]

[0021]

[0022]

[0023] Where rt, gt, and ut represent the reset gating feature, modulation gating feature, and update gating feature at time step t, respectively; W xr W xg and W xu W represents the weights of the time input information at time step t in the reset gate, modulation gate, and update gate, respectively; hr W hg and W hu These represent the weights of the hidden state in the reset gate, modulation gate, and update gate, respectively. u σ represents the bias of the update gate; σ represents the Sigmoid activation function; tanh represents the tanh activation function.

[0024] S203, the attention features, reset gating features, and modulation gating features are combined and stored in the time storage unit, with the specific formula as follows:

[0025]

[0026] Among them, T M It serves as a time storage unit.

[0027] S204. The top-level output of the past time state and the network input of the current time state are used as spatiotemporal input information. They are processed by attention mechanism, reset gate and update gate respectively to obtain attention features, reset gate features and update gate features. The attention features and reset gate features are combined and stored in the spatiotemporal storage unit. The specific formula is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] Where, r′ t and u′ t M represents the reset gating feature and the update gating feature under the spatiotemporal input information at time step t, respectively. att This represents the attention characteristics under spatiotemporal conditions. S represents the spatial memory information of layer l-1 at time step t. M W′ represents a spatiotemporal storage unit. xr 、W′ xu W′ represents the weights of the spatiotemporal input information in the reset gate and update gate, respectively. mr 、W′ mu b′ represents the weight of spatial memory information in the reset gate and update gate, respectively. u This indicates that the gate bias is being updated.

[0033] S205. The time storage unit and the spatiotemporal storage unit are merged through a gating mechanism to construct a complete spatiotemporal attention fusion unit.

[0034] Furthermore, in step S3, the specific content of constructing the motion gradient highway unit is as follows:

[0035] S301. Introduce an encoding module to encode the input and output features.

[0036] S302, Construct a standard gradient highway unit.

[0037] S303. Build a motion sensing module using differential operations. The specific formula is as follows:

[0038] D t =D t-1 +α(T t-1 -D t-1 )

[0039] Among them, D t It represents the movement trend at the current time step, D.t-1 It is the movement trend learned in the past time step, T t-1 It represents the motion change features captured by the filter in the previous time step.

[0040] S304. Connect the motion sensing module to the gradient highway unit, using the following formula:

[0041]

[0042] Among them, T t It is the motion change feature captured at the current time step, P t It is the converted input, S t It is a control switch.

[0043] S305 introduces decoding and modulation modules to construct a complete motion gradient highway unit.

[0044] Furthermore, in step S4, the specific steps for constructing the trained spatiotemporal prediction network are as follows:

[0045] S401. A basic recurrent neural network is built by stacking ConvLSTM.

[0046] S402. In order to preserve complete temporal and spatial features, the Spatiotemporal Attention Fusion Unit is used to replace the ConvLSTM.

[0047] S403. To avoid gradient explosion caused by the stacking of multiple spatiotemporal attention fusion units and to improve the integrity of spatiotemporal information, motion gradient highway units are inserted between the layers of spatiotemporal attention fusion units and directly connected through high-speed channels to complete the construction of the spatiotemporal prediction network.

[0048] S404. Introduce a joint loss function to train the spatiotemporal prediction network, obtaining the trained spatiotemporal prediction network; the specific formula for the joint loss function is:

[0049]

[0050] Where α is the weight of the L1 loss function, and X is the input at the current time step. This is the output for the current time step. Further, in step S5, obtaining the initial spatiotemporal features and the initial spatiotemporal motion features includes the following sub-steps:

[0051] S501. In order to preserve complete temporal feature information, the 5D tensor and the output of past time states are combined. The amount of 5D tensor retained is controlled by gating and attention mechanisms and stored in the time storage unit as a time memory feature.

[0052] S502. In order to preserve complete spatiotemporal feature information, the 5D tensor and the top-level spatiotemporal information of past time are fused. The retention of information is controlled by attention mechanism and gating mechanism and stored in spatiotemporal storage unit as spatiotemporal memory feature.

[0053] S503, the temporal memory features and spatiotemporal memory features are reduced in dimensionality by convolutional layers and modulated by update gates to obtain the initial spatiotemporal features, as shown in the following formula:

[0054]

[0055] S504 The initial spatiotemporal features output by the spatiotemporal attention fusion unit are encoded and combined with past motion information as input, and then transmitted to the gradient highway unit to obtain gradient feature information.

[0056] S505. Use the motion sensing module to perform differential calculations on the transient change characteristics and motion change trends of the past time to obtain motion change characteristics.

[0057] S506. The gradient feature information and motion change features of the gradient highway unit are weighted and fused to obtain spatiotemporal feature information with motion change features.

[0058] S507. After bilinear interpolation and decoding modulation, the spatiotemporal feature information with motion change characteristics is used to obtain the initial spatiotemporal motion features, as shown in the following formula:

[0059] m t =broadcast(σ(W) m *Enc(H t ))

[0060]

[0061] g t =σ(W g *Concat[Dec(H t '),H t ])

[0062]

[0063] Where, m t This indicates a motion filter, broadcast indicates a broadcast operation, and W represents a motion filter. m Represents the weights of the motion filter, Enc(H) t ) represents the input from the prediction block, concat represents the superposition operation in the channel direction, H t ' represents the motion change characteristics after filtering and interpolation, wrap represents the transformation operation of bilinear interpolation, W g This indicates the weight of the modulation gate.

[0064] Furthermore, in step S7, the spatiotemporal features extracted by the fourth-layer spatiotemporal attention fusion unit are re-decoded into the state of each time step using the deconvolution method to obtain a more accurate predicted video sequence; the predicted video sequence is then deconvolved to map back to the corresponding single-frame predicted image; simultaneously, the spatiotemporal features are transmitted horizontally to the top-level spatiotemporal attention fusion unit of the next time step and zigzagly into the bottom-level spatiotemporal attention fusion unit of the next time step, and used as one of the input information in the next time step to improve the timeliness of the spatiotemporal features.

[0065] Furthermore, this invention also proposes a video prediction system that combines spatiotemporal memory features and motion perception, including...

[0066] The video dataset preprocessing module is used to preprocess the acquired video dataset and convert it into 5D tensors.

[0067] The spatiotemporal attention fusion unit construction module is used to fuse time storage units and spatiotemporal storage units through a gating mechanism to construct a complete spatiotemporal attention fusion unit.

[0068] The motion gradient highway unit building module is used to introduce decoding and modulation modules to build a complete motion gradient highway unit.

[0069] The spatiotemporal prediction network construction module is used to insert motion gradient highway units between layers of the spatiotemporal attention fusion unit and directly connect them through high-speed channels to complete the construction of the spatiotemporal prediction network. The joint loss function is introduced to train the network and obtain the trained spatiotemporal prediction network.

[0070] The initial spatiotemporal motion feature extraction module is used to take the 5D tensor in the video dataset preprocessing module as the input of the trained spatiotemporal prediction network and input it into the first layer spatiotemporal attention fusion unit to obtain the initial spatiotemporal features. The initial spatiotemporal features are then adaptively extracted by a layer of motion gradient highway unit to obtain the initial spatiotemporal motion features.

[0071] The feature extraction module is used to input the initial spatiotemporal motion features extracted from the first motion gradient highway unit and the initial spatiotemporal features from the initial spatiotemporal motion feature extraction module into the second-layer spatiotemporal attention fusion unit for feature extraction.

[0072] The single-frame prediction image acquisition module repeats the initial spatiotemporal motion feature extraction module and feature extraction module twice to obtain the spatiotemporal features of the remaining layers in the trained spatiotemporal prediction network. The final spatiotemporal features are obtained in the fourth layer spatiotemporal attention fusion unit, and the final prediction sequence is output. The corresponding single-frame prediction image is obtained through deconvolution.

[0073] The predicted video frame acquisition module is used to input the initial spatiotemporal features from the initial spatiotemporal motion feature extraction module and the final spatiotemporal features from the single-frame predicted image acquisition module into the first layer spatiotemporal attention fusion unit of the spatiotemporal prediction network that has been trained at the current time step. It is used together with the 5D tensor in the video dataset preprocessing module as input information, and repeats the operation of the initial spatiotemporal motion feature extraction module, feature extraction module and single-frame predicted image acquisition module until the predicted video frames of n time steps are obtained.

[0074] Furthermore, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video prediction method combining spatiotemporal memory features and motion perception described above.

[0075] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the video prediction method described above that combines spatiotemporal memory features and motion perception.

[0076] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0077] This invention designs a spatiotemporal attention fusion unit that combines spatiotemporal memory features and an attention mechanism. Using this unit as the foundational building block of a recurrent neural network (RNN), it fuses past state temporal information and current state spatial information, enhancing motion perception while also considering the spatial static features of the current state. This allows for the full learning and preservation of detailed information during spatiotemporal state transitions. Furthermore, spatiotemporal memory enables communication between visual dynamics learned by different levels of the RNN, ensuring both low-level details and high-level features are preserved. The attention mechanism further facilitates the learning of long-term global motion information from video frames, achieving higher prediction performance with lower computational load, reducing resource waste, and improving the accuracy of video prediction.

[0078] Meanwhile, motion perception is introduced on the basis of the gradient highway unit to obtain an improved motion gradient highway unit. It learns the transient changes and motion trends between frames, and provides a more efficient route for the gradient flow to return from the output to the previous remote input through the inter-layer jump connection, which reduces the impact of gradient explosion or gradient vanishing during backpropagation.

[0079] This invention effectively solves the problems of missing long-term features, poor spatiotemporal dependence, and low prediction accuracy in video. When predicting future video frames under different scenarios, this method demonstrates significantly better overall accuracy than other prediction methods, and also has a shorter prediction cycle. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating the overall implementation of the present invention.

[0081] Figure 2 This is a schematic diagram of the spatiotemporal attention fusion unit of the present invention.

[0082] Figure 3 This is a schematic diagram of the motion gradient highway unit of the present invention.

[0083] Figure 4 This is a schematic diagram of the spatiotemporal prediction network structure of the present invention.

[0084] Figure 5 These are the predicted video frames obtained in the embodiments of the present invention.

[0085] Figure 6 This refers to the optimized predicted video frame in this embodiment of the invention. Detailed Implementation

[0086] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0087] The present invention will now be described in further detail with reference to the accompanying drawings.

[0088] This invention proposes a video prediction method that combines spatiotemporal memory features and motion perception, such as... Figure 1 As shown, it includes the following steps:

[0089] S1. Obtain the video dataset, preprocess it, and convert it into a 5D tensor.

[0090] In this embodiment, a publicly available human motion recognition dataset is used, containing six different human actions: walking, jogging, running, boxing, waving, and clapping. Twenty-five individuals performed these actions in four different environments. Each video lasts an average of 4 seconds, with each second divided into 25 frames. First, the resolution of each frame is adjusted to 128×128. The actions of individuals 1-16 are used for training, and those 17-25 are used for testing, resulting in a training set of 108,717 sequences and a test set of 4,086 sequences. Second, the video dataset is loaded to obtain video frames. These frames are then divided into multiple subframes, and the subframes of each frame are combined into a new time step. The video data is then standardized and normalized to facilitate model learning of the data's features. Finally, the processed video data is combined into a 5D tensor.

[0091] The input sequence can be divided into multiple time steps {X1,X2,…,X...}t ,X t+1}, extract 10 consecutive frames as the input to the network, denoted as X = {X} t-9 ,X t-8 ,…,X t}

[0092] S2, such as Figure 2 As shown, the specific steps for constructing a spatiotemporal attention fusion unit are as follows:

[0093] S201. The output of the spatiotemporal attention fusion unit at the same layer in the past time state and the input of the current time state are used as time input information, and attention features are obtained by assigning weights to them using an attention mechanism. The specific formula is as follows:

[0094]

[0095] Among them, H att X represents attentional features. t The input represents the current time status. This represents the hidden state of the l-layer spatiotemporal attention fusion unit at time t-1, and ATT represents the attention mechanism.

[0096] S202. The time input information passes through a reset gate, a modulation gate, and an update gate to obtain different gated spatiotemporal features, as shown in the specific formula:

[0097]

[0098]

[0099]

[0100] Where rt, gt, and ut represent the reset gate feature, modulation gate feature, and update gate feature at time step t, respectively; W xr W xg and W xu W represents the weights of the time input information at time step t in the reset gate, modulation gate, and update gate, respectively; hr W hg and W hu These represent the weights of the hidden state in the reset gate, modulation gate, and update gate, respectively. u σ represents the bias of the update gate; σ represents the Sigmoid activation function; tanh represents the tanh activation function.

[0101] S203, the attention features, reset gating features, and modulation gating features are combined and stored in the time storage unit, with the specific formula as follows:

[0102]

[0103] Among them, T M It serves as a time storage unit.

[0104] S204. The top-level output of the past time state and the network input of the current time state are used as spatiotemporal input information. They are processed by the attention mechanism, the reset gate, and the update gate to obtain attention features, reset features, and update features, respectively. The attention features and reset features are combined and stored in the spatiotemporal storage unit. The specific formula is as follows:

[0105]

[0106]

[0107]

[0108]

[0109] Where, r′ t and u′ t M represents the reset gate feature and update gate feature under the spatiotemporal input information at time step t, respectively. att This represents the attention characteristics under spatiotemporal conditions. S represents the spatial memory information of layer l-1 at time step t. M W′ represents a spatiotemporal storage unit. xr 、W′ xu W′ represents the weights of the spatiotemporal input information in the reset gate and update gate, respectively. mr 、W′ mu b′ represents the weight of spatial memory information in the reset gate and update gate, respectively. uu This indicates that the gate bias is being updated.

[0110] S205. The time storage unit and the spatiotemporal storage unit are fused through a gating mechanism to construct a complete spatiotemporal attention fusion unit. The specific formula is as follows:

[0111]

[0112] in, W represents the hidden state of the l-layer spatiotemporal attention fusion unit at time t. 1×1 It is a 1×1 convolution.

[0113] S3, such as Figure 3 As shown, the motion gradient highway unit consists of an encoding / decoding module, a gradient highway unit, and a motion sensing module. The specific content of constructing the motion gradient highway unit is as follows:

[0114] S301. Introduce an encoding module to encode the input and output features.

[0115] S302, Construct standard gradient highway units;

[0116] S303. Build a motion sensing module using differential operations. The specific formula is as follows:

[0117] D t =D t-1 +α(T t-1 -D t-1 )

[0118] Among them, D t It represents the movement trend at the current time step, D. t-1 It is the movement trend learned in the past time step, T t-1 It represents the motion change features captured by the filter in the previous time step.

[0119] S304. Connect the motion sensing module to the gradient highway unit, using the following formula:

[0120]

[0121] Among them, T t It is the motion change feature captured at the current time step, P t It is the converted input, S t It is a control switch.

[0122] S305. Introducing decoding and modulation modules to construct a complete motion gradient highway unit, the specific formula is as follows:

[0123]

[0124] also

[0125] Among them, Dec(H′) t Z represents the decoded output. t H represents the initial spatiotemporal motion features captured at the current time step. t-1 This represents the hidden state of the spatiotemporal attention fusion unit at time t-1.

[0126] S4, such as Figure 4 As shown, the spatiotemporal prediction network consists of four stacked spatiotemporal attention fusion units and three embedded motion gradient highway units. The spatiotemporal prediction network is constructed, and a joint loss function is introduced to train it. The specific content of the trained spatiotemporal prediction network is as follows:

[0127] S401. A basic recurrent neural network is built by stacking ConvLSTM.

[0128] S402. In order to preserve complete temporal and spatial features, the Spatiotemporal Attention Fusion Unit is used to replace the ConvLSTM.

[0129] S403. To avoid gradient explosion caused by the stacking of multiple spatiotemporal attention fusion units and to improve the integrity of spatiotemporal information, motion gradient highway units are inserted between the layers of spatiotemporal attention fusion units and directly connected through high-speed channels to complete the construction of the spatiotemporal prediction network.

[0130] S404. Introducing a joint loss function to train the spatiotemporal prediction network can simultaneously ensure both model stability and accuracy. The L1 loss, calculated based on absolute error, penalizes large error samples and offers stronger stability against outliers. In contrast, the L2 loss focuses more on small error samples and is more sensitive to the overall accuracy of the model. The specific formula for the joint loss function is:

[0131]

[0132] Where α is the weight of the L1 loss function, and X is the input at the current time step. This is the output for the current time step. This paper chooses an adaptive training strategy based on the Adam optimizer to dynamically adjust the weights of L1 and L2 during model training.

[0133] S5. The 5D tensor from step S1 is used as the input to the spatiotemporal prediction network and fed into the first layer of spatiotemporal attention fusion unit to obtain the initial spatiotemporal features. The initial spatiotemporal features are adaptively extracted by a layer of motion gradient highway unit to obtain the initial spatiotemporal motion features, including the following sub-steps:

[0134] S501. In this embodiment, the video frame at time step t is first selected as the input X of the trained spatiotemporal prediction network. t spatiotemporal feature information of the previous time step The data is fed into this unit, where an attention mechanism is used to selectively focus on certain features, better handling key objects and improving the accuracy of data feature understanding without increasing computational cost. Secondly, a time modulation gate g is introduced. t By interacting with modulation and reset gates, the input data for human behavior recognition videos is processed, controlling the amount of information to be retained from past time steps. This improves the model's ability to capture complex short-term dynamics between adjacent time steps and allows input data to pass through adaptively at different rates. Then, a set of transformation functions generates a time memory unit T. M This refers to the temporary memory transferred from the previous time node to the current time node, and the amount of time information input in the current state is controlled by an update gate. Furthermore, the spatiotemporal memory state of the current time step is introduced. After passing through the attention mechanism, the spatiotemporal memory state transmits state information across layers and then forwards it to the next time step. This not only extends the state transition path of the trained spatiotemporal prediction network in terms of time, but also adds additional storage units between horizontally adjacent nodes at the same level in terms of space, enabling the network to learn complex nonlinear transition functions under short-term motion. To capture long-term dependencies in the spatiotemporal transition process, a spatiotemporal memory storage unit S is defined. M When information passes through this node, it can read and update the memory state based on its understanding of the spatiotemporal state. The time memory T... M and spacetime memory S M The layers are connected, and 1×1 convolutional layers are applied for dimensionality reduction. An update gate is used to control the amount of input information at the current time, so as to achieve collaborative learning of long-term dynamics and short-term dynamics and obtain spatiotemporal features.

[0135] S502, the spatiotemporal attention fusion unit extracts spatiotemporal features from video sequences for human behavior recognition. The motion gradient highway unit is input, encoded, and then input into the gradient highway module along with motion change information. By combining these approaches, a gating mechanism is used to control the selection and retention of features, adaptively extracting important information from the input features. Meanwhile, the trend of motion changes in the previous time step and information on motion changes The input is fed into the motion sensing module, where it learns the transient changes and motion trends of the input features by calculating the difference. Then, the modulation process begins, in which the adaptively learned long-term spatiotemporal information is modulated. Information on short-term motion changes Fusion can be achieved through a motion filter where m t Broadcast modulation tensor dimension, making it similar to H t The dimensions of '(t)' are kept consistent. Then, the values ​​of edge pixels are calculated through a bilinear interpolation wrap operation to improve the image's expressiveness and visual effect. t It is an output modulation gate that controls the degree to which the input information updates the state by modulating the input and the encoded output, thereby improving the expressiveness and flexibility of the model, and finally obtaining the output spatiotemporal motion feature Z. t .

[0136] The calculation process in the structure is as follows:

[0137] P t =tanh(W p *Concat([Enc(H t ),T t-1 ]))

[0138] St =σ(W S *Concat([Enc(H t ),T t-1 ]))

[0139] D t =D t-1 +α(T t-1 -D t-1 )

[0140]

[0141] m t =broadcast(σ(W) m *Enc(H t ))

[0142]

[0143] g t =σ(W g *Concat[Dec(H t '),H t ])

[0144]

[0145] Among them, P t It is the converted input, T t-1 It represents the motion change features captured by the filter in the previous time step. t It is a control switch that can convert the input P after conversion. t and motion change characteristics T t-1 Adaptive learning is performed between them. P and W S Enc(H) represents the weights assigned to the convolution filter. t ) represents the input from the prediction block. D t-1 It is the motion trend learned from past time steps, D t It represents the movement trend at the current time step, T. t It represents the motion change features captured at the current time step. t For motion filters, g t It is an output modulation gate, Z t The output is the spatiotemporal motion characteristics.

[0146] S6. Input the initial spatiotemporal features and initial spatiotemporal motion features from step S5 into the second-layer spatiotemporal attention fusion unit for feature extraction. To improve the completeness and timeliness of feature information, the extracted features are input vertically into the motion gradient highway unit between the current time step layers and horizontally into the spatiotemporal attention fusion unit of the next time step.

[0147] S7. Input the spatiotemporal features and spatiotemporal motion features extracted in step S6 into the third-layer spatiotemporal attention fusion unit for feature extraction; and input the extracted features vertically into the motion gradient highway unit between the current time step layers and horizontally into the spatiotemporal attention fusion unit of the next time step.

[0148] S8. Input the spatiotemporal features and spatiotemporal motion features extracted in step S7 into the fourth-layer spatiotemporal attention fusion unit for feature extraction, obtain the final spatiotemporal features, output the final prediction sequence, and obtain the corresponding single-frame prediction image of the prediction sequence through the deconvolution method.

[0149] S9. Input the initial spatiotemporal features from step S5 and the final spatiotemporal features from step S8 into the first layer of the spatiotemporal attention fusion unit of the spatiotemporal prediction network that has been trained at the current time step, and use it together with the 5D tensor from step S1 as input information. Repeat steps S5-S8 again until 10 time steps of predicted video frames are obtained. The results are as follows Figure 5 As shown.

[0150] The model's gradients are calculated using backpropagation, and these gradients are used to update the model's parameters. This process is repeated multiple times until convergence, and the output video prediction frames are reconstructed. Utilizing a joint loss function to predict future spatiotemporal sequence data significantly improves the clarity and accuracy of the predicted images. Through iterative optimization, a more stable output can be obtained, as shown in the optimized predicted image. Figure 6 As shown.

[0151] Meanwhile, the structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and perceptual loss (LPIPS) are used to evaluate the performance of the prediction network. The quantitative results of the human action recognition dataset on different models are compared in the table below. Among them, the higher the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR), the better, and the lower the perceptual loss (LPIPS), the better.

[0152] Table 1. Quantitative results of different methods on the KTH dataset (10 frames → 20 frames).

[0153]

[0154] This invention also proposes a video prediction system combining spatiotemporal memory features and motion perception, including a video dataset preprocessing module, a spatiotemporal attention fusion unit construction module, a motion gradient highway unit construction module, a spatiotemporal prediction network construction module, an initial spatiotemporal motion feature extraction module, a feature extraction module, a single-frame prediction image acquisition module, a prediction video frame acquisition module, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention embodiment.

[0155] This invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0156] This invention also proposes a computer-readable storage medium storing a computer program. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0157] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention. Any other corresponding changes and variations made in accordance with the technical concept of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A video prediction method combining spatiotemporal memory features and motion perception, characterized in that, The method comprises the following steps: S1, obtaining a video dataset for preprocessing and converting into a 5D tensor; S2, constructing a space-time attention fusion unit; S3, constructing a motion gradient highway unit; specifically: S301, introducing an encoding module to encode the input and output features; S302, building a standard gradient highway unit; S303, building a motion perception module using a difference operation, specifically: ； wherein, is the motion trend learned at the current time step, is the motion trend learned at the past time step, is the motion change feature captured by the filter at the previous time step; S304, connecting the motion perception module with the gradient highway unit, specifically: ； wherein, is a motion change feature captured at the current time step, is the transformed input, is a control switch; S305, introducing a decoding and modulation module to build a complete motion gradient highway unit, specifically: ； Furthermore ; wherein, denotes the decoded output, denotes the initial spatio-temporal motion feature captured at the current time step, denotes the hidden state of the spatio-temporal attention fusion unit at time t-1; S4, constructing a space-time prediction network and introducing a joint loss function to train the network, obtaining a trained space-time prediction network containing n video sequences at different time steps, each video sequence including 4 space-time attention fusion units and 3 motion gradient highway units; S5, inputting the 5D tensor in step S1 into the first space-time attention fusion unit of the trained space-time prediction network to obtain initial space-time features; the initial space-time features are adaptively extracted by the first motion gradient highway unit to obtain initial space-time motion features; S6, inputting the initial space-time features and the initial space-time motion features in step S5 into the second space-time attention fusion unit for feature extraction; the extracted features are inputted into the motion gradient highway unit at the current time step and the space-time attention fusion unit at the next time step respectively, and this process is repeated until the final space-time features are obtained at the fourth space-time attention fusion unit, and the final prediction sequence is outputted, and a single-frame prediction picture corresponding to the prediction sequence is obtained through the deconvolution method; S7, inputting the initial space-time features in step S5 and the final space-time features in step S6 into the first space-time attention fusion unit of the trained space-time prediction network at the current time step, and inputting the 5D tensor in step S1 as input information, and repeating steps S5-S6 again until n prediction video frames at different time steps are obtained.

2. The method of claim 1, wherein, In step S1, the specific content of converting the video dataset into a 5D tensor is: loading the video dataset to obtain video frames, dividing the video frames into multiple sub-frames, combining the sub-frames of each video frame into new time steps, performing standardization and normalization processing, and combining the processed video data into a 5D tensor.

3. The method of claim 1, wherein, In step S2, the specific steps of constructing the space-time attention fusion unit are as follows: S201, the output of the same layer space-time attention fusion unit at the past time state and the input at the current time state are used as time input information, and the attention mechanism is used to assign weights to obtain attention features, specifically: ； wherein H att represents the attention feature, X t represents the input of the current time state, represents the hidden state of the space-time attention fusion unit of the l layer at the t-1 moment, and ATT represents an attention mechanism. S202, the time input information passes through the reset gate, the modulation gate and the update gate respectively to obtain different gated space-time features, specifically: ； wherein, , and denote the reset-gating feature, the modulation-gating feature and the update-gating feature at the t-th time step, respectively; xr , W xg and W xu denote the weights of the t-th time step time input information in the reset gate, the modulation gate and the update gate, respectively; hr , W hg and W hu denote the weights of the hidden state in the reset gate, the modulation gate and the update gate, respectively, b u denotes the bias of the update gate; denotes the Sigmoid activation function; tanh denotes the tanh activation function; S203, the attention features, the reset gate features and the modulation gate features are combined and stored in the time storage unit, specifically: ； T M is a time storage unit; S204, the top layer output of the past time state and the input of the current time state are taken as the space-time input information, and are processed by using the attention mechanism, the reset gate and the update gate respectively to obtain the attention feature, the reset gate feature and the update gate feature; the attention feature and the reset gate feature are combined and stored in the space-time storage unit, and the specific formula is: ； wherein, and respectively represent the reset gate feature and the update gate feature under the spatiotemporal input information at the tth time step, M att represents the attention feature under the spatiotemporal state, represents the spatial memory information of the l-1th layer at the tth time step, S M represents the spatiotemporal storage unit, respectively represent the weights of the spatiotemporal input information in the reset gate and the update gate, respectively represent the weights of the spatial memory information in the reset gate and the update gate, represents the bias of the update gate; S205, the time storage unit and the space-time storage unit are fused through the gating mechanism to construct a complete space-time attention fusion unit, and the specific formula is: ； wherein, denotes the hidden state of the space-time attention fusion unit at layer l at time t, W 1×1 is a 1 x 1 convolution.

4. The method of claim 1, wherein, In step S4, the specific steps of constructing the trained space-time prediction network are: S401, a basic recurrent neural network is constructed by stacking ConvLSTM; S402, the space-time attention fusion unit is used to replace ConvLSTM; S403, the motion gradient highway unit is inserted between the layers of the space-time attention fusion unit, and is directly connected through the high-speed channel to complete the construction of the space-time prediction network; S404, a joint loss function is introduced to train the space-time prediction network to obtain the trained space-time prediction network; the specific formula of the joint loss function is: ； wherein, is a weight for the L1 loss function, is an input at a current time step, is an output at the current time step.

5. The method of claim 1, wherein, In step S5, obtaining the initial space-time feature and the initial space-time motion feature includes the following sub-steps: S501, the 5D tensor and the output of the past time state are combined, the retention amount of the 5D tensor is controlled through the gating mechanism and the attention mechanism, and the 5D tensor is stored in the time storage unit as the time memory feature; S502, the 5D tensor and the top layer space-time information of the past time are fused, the retention of the information is controlled through the attention mechanism and the gating mechanism, and the information is stored in the space-time storage unit as the space-time memory feature; S503, the time memory feature and the space-time memory feature are reduced in dimension through the convolution layer and modulated through the update gate to obtain the initial space-time feature, and the specific formula is as follows: ； S504, the initial space-time feature output by the space-time attention fusion unit is encoded, combined with the motion information of the past time as input, transmitted to the gradient highway unit, and the gradient feature information is obtained; S505, the motion perception module is used to perform difference operation on the transient change feature and the motion change trend of the past time to obtain the motion change feature; S506, the gradient feature information and the motion change feature are weighted and fused to obtain the space-time feature information with the motion change feature; S507, the space-time feature information with the motion change feature is bilinearly interpolated and decoded to obtain the initial space-time motion feature, and the specific formula is as follows: ； wherein, represents a motion filter, broadcast represents a broadcast operation, W m represents a weight of a motion filter, represents an input from a prediction block, concat represents a stacking operation in a channel direction, represents a filtered and interpolated motion change feature, represents a change operation of bilinear interpolation, represents a weight of a modulation gate.

6. The method of claim 1, wherein, In step S7, the space-time feature extracted by the fourth layer space-time attention fusion unit is decoded back to the state of each time step through the deconvolution method to obtain the predicted video sequence; the predicted video sequence is deconvoluted to map back to the corresponding single-frame predicted picture; at the same time, the space-time feature is transmitted to the top layer space-time attention fusion unit of the next time step along the horizontal direction and the bottom layer space-time attention fusion unit of the next time step along the zigzag direction, and flows into the next time step as one of the input information.

7. A video prediction system incorporating spatio-temporal memory features and motion perception, characterized in that, The video data set preprocessing module is used for preprocessing the acquired video data set and converting it into a 5D tensor. ​ The spatio-temporal attention fusion unit construction module is configured to fuse the time storage unit and the spatio-temporal storage unit through a gating mechanism to construct a complete spatio-temporal attention fusion unit. The motion gradient highway unit construction module is configured to introduce a decoding and modulation module to construct a complete motion gradient highway unit, and specifically comprises the following steps: Step 1: introducing an encoding module to encode the input and output features; Step 2: building a standard gradient highway unit; Step 3: building a motion perception module using a difference operation, and the specific formula is: ； wherein, is the motion trend learned at the current time step, is the motion trend learned at the past time step, is the motion change feature captured by the filter at the previous time step; Step 4: connecting the motion perception module with the gradient highway unit, and the specific formula is: ； wherein, is a motion change feature captured at the current time step, is a transformed input, is a control switch; Step 5: introducing a decoding and modulation module to construct a complete motion gradient highway unit, and the specific formula is: ； Furthermore ; wherein, denotes the decoded output, denotes the initial spatio-temporal motion feature captured at the current time step, denotes the hidden state of the spatio-temporal attention fusion unit at time t-1; The spatio-temporal prediction network construction module is configured to insert a motion gradient highway unit between layers of the spatio-temporal attention fusion unit, and directly connect through a high-speed channel to complete the construction of a spatio-temporal prediction network, and introduce a joint loss function to train the network to obtain a trained spatio-temporal prediction network. The initial spatio-temporal motion feature extraction module is configured to take the 5D tensor in the video dataset preprocessing module as the input of the trained spatio-temporal prediction network, input into the first layer spatio-temporal attention fusion unit, and obtain initial spatio-temporal features; the initial spatio-temporal features are adaptively extracted by a layer of motion gradient highway unit to obtain initial spatio-temporal motion features; The feature extraction module is configured to input the spatio-temporal features extracted in the previous spatio-temporal attention fusion unit and the spatio-temporal motion features extracted by the current motion gradient highway unit into the current spatio-temporal attention fusion unit for feature extraction. The single-frame prediction picture acquisition module is configured to repeatedly operate the feature extraction module twice to obtain the final spatio-temporal features in the fourth layer spatio-temporal attention fusion unit, output the final prediction sequence, and obtain the single-frame prediction picture corresponding to the prediction sequence through the deconvolution method. The predicted video frame acquisition module is configured to input the initial spatio-temporal features in the initial spatio-temporal motion feature extraction module and the final spatio-temporal features in the single-frame prediction picture acquisition module into the first layer spatio-temporal attention fusion unit of the trained spatio-temporal prediction network at the current time step, and take the 5D tensor in the video dataset preprocessing module as input information, and repeatedly operate the initial spatio-temporal motion feature extraction module, the feature extraction module and the single-frame prediction picture acquisition module again until n time steps of predicted video frames are obtained.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer-readable storage medium being characterized by, The computer program is executed by the processor to perform the method of any one of claims 1 to 6.

Images