A video prediction method and apparatus for reducing inter-frame error accumulation

Abstract

The application provides a video prediction method and device for reducing inter-frame error accumulation, wherein the method comprises: acquiring an image frame at a current time and m continuous image frames before the current time; processing the image frame at the current time and the m continuous image frames before the current time by using a space-time prediction model to obtain implicit states of k continuous image frames after the current time; the space-time prediction model adopts a parallel long short-term memory model with implicit state decoupling; processing the implicit states of the k continuous image frames after the current time by using a reconstruction layer to obtain the predicted k continuous image frames. The application can map the collected multiple observation videos to multiple prediction videos at one time, and avoid error accumulation caused by frame-by-frame recursive prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of spatio-temporal prediction, and in particular to a video prediction method and device for reducing inter-frame error accumulation. BACKGROUND

[0002] Currently, in the video prediction method at a traffic intersection, an autoregressive-based method and a non-autoregressive-based method are usually used.

[0003] The autoregressive-based spatio-temporal prediction method uses a traditional RNN architecture, and its prediction mode depends on recursively receiving the historical information of the previous frame, modeling the implicit state of the current frame, and passing it to the next frame, and the future frame is predicted by cyclically observing the frame. This explicit modeling method can ensure accurate prediction, but has problems of unstable gradient, low computational efficiency, and serious error accumulation.

[0004] The non-autoregressive-based method uses an encoder-decoder architecture, and a plurality of observed image sequences are input at one time, the spatio-temporal features are modeled by the spatio-temporal feature module in the encoder-decoder architecture, and all future frames are predicted at one time. Although this method supports multi-step joint prediction, it is easy to produce time sequence breakage or motion distortion due to the neglect of inter-frame dynamic coupling relationship. SUMMARY

[0005] Therefore, the present application provides a video prediction method and device for reducing inter-frame error accumulation to solve the problems of unstable gradient, low computational efficiency, and serious error accumulation in the existing video prediction method.

[0006] In a first aspect, the embodiments of the present application provide a video prediction method for reducing inter-frame error accumulation, comprising:

[0007] obtaining an image frame at a current time and m consecutive image frames before the current time;

[0008] processing the image frame at the current time and the m consecutive image frames before the current time by using a spatio-temporal prediction model to obtain the implicit state of k consecutive image frames after the current time; the spatio-temporal prediction model uses a parallel long short-term memory model with implicit state decoupling;

[0009] processing the implicit state of the k consecutive image frames after the current time by using a reconstruction layer to obtain the predicted k consecutive image frames.

[0010] In a possible implementation, the spatio-temporal prediction model comprises an initialization module, a processing module, and a parallel processing module.

[0011] The current time image frame and the continuous m image frames before the current time are processed by using a space-time prediction model to obtain the implicit state of the continuous k image frames after the current time; comprising:

[0012] An image frame sequence X containing the current time image frame and the continuous m image frames before the current time is set t,m+1 is: x t-m ,…x t ; wherein x t is the current time image frame, x t-m is the mth image frame before x t ;

[0013] The initialized predicted implicit state of the continuous k image frames is generated by using an initialization module;

[0014] The image frame sequence X is processed by using a processing module t,m+1 to obtain the candidate implicit state of the image frame sequence X t,m+1 and the filter gate parameter vector;

[0015] The initialized predicted implicit state of the continuous k image frames, the candidate implicit state of the image frame sequence X t,m+1 and the filter gate parameter vector are processed by using a parallel processing module to obtain the implicit state of the continuous k image frames after the current time.

[0016] In a possible implementation, the processing module comprises, connected in sequence: a linear projection layer, a SwinTransformer neural network and a logarithmic function;

[0017] The image frame sequence X is processed by using a processing module t,m+1 to obtain the candidate implicit state of the image frame sequence X t,m+1 and the filter gate parameter vector; comprising:

[0018] The image frame sequence X is processed by using a linear projection layer and a Swin Transformer neural network t,m+1 to obtain the candidate implicit state of the image frame sequence X t,m+1

[0019]

[0020] Wherein, LP(·) represents the linear projection layer; STB(·) represents the Swin Transformer neural network;

[0021] The candidate implicit state is operated by using a logarithmic function to obtain the filter gate parameter vector F t,m+1 :​

[0022]

[0023] wherein, ln(·) represents a logarithm function with base e.

[0024] In a possible implementation, the initialized predicted implicit states of the consecutive k image frames, the candidate implicit states of the image frame sequence X t,m+1 and the filter gate parameter vector are processed by using the parallelization processing module to obtain the implicit states of the consecutive k image frames after the current time; including:

[0025] Based on the filter gate parameter vector, a first parameter

[0026]

[0027] wherein, f t-i is the i-th component of the filter gate parameter vector F t,m+1 ;

[0028] A second parameter temp j is calculated.

[0029]

[0030] wherein, is the initialized predicted implicit states of the consecutive j image frames, 1≤j≤k; concat() is a concatenation function; and ⊙ represents a Hadamard product.

[0031] Based on the second parameter, a candidate implicit state of the consecutive k image frames is calculated.

[0032]

[0033] wherein, exp(·) is an exponential function with base e.

[0034] The predicted implicit states of the consecutive k image frames are generated by an exponential operation.

[0035]

[0036] In a possible implementation, the reconstruction layer is stacked by a plurality of deconvolution layers and a plurality of convolution layers.

[0037] In a possible implementation, the method further includes the steps of training the linear projection layer, the Swin Transformer neural network and the reconstruction layer.

[0038] In a second aspect, an embodiment of the present application provides a video prediction device for reducing error accumulation between frames, comprising:

[0039] an acquisition unit configured to acquire an image frame at a current time and m continuous image frames before the current time;

[0040] a prediction unit configured to process the image frame at the current time and the m continuous image frames before the current time by using a space-time prediction model to obtain implicit states of k continuous image frames after the current time; the space-time prediction model adopts a parallel long short-term memory model with implicit state decoupling;

[0041] a reconstruction unit configured to process the implicit states of the k continuous image frames after the current time by using a reconstruction layer to obtain the predicted k continuous image frames.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the method of the embodiment of the present application when executing the computer program.

[0043] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is run by a processor to implement the method of the embodiment of the present application.

[0044] The present application can map the collected multiple observation videos into multiple predicted videos at one time, and reduce error accumulation caused by frame-by-frame recursive prediction. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the specific embodiments of the present application or the prior art, the drawings needed in the description of the specific embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0046] Figure 1 a flowchart of the video prediction method for reducing error accumulation between frames provided by the embodiment of the present application;

[0047] Figure 2 a space-time prediction result example diagram of the prediction method provided by the embodiment of the present application on the Moving MNIST data set;

[0048] Figure 3 a space-time prediction result example diagram of the prediction method provided by the embodiment of the present application on the KTH data set;

[0049] Figure 4 The prediction method provided by the embodiment of the present application is compared with other methods in terms of calculation efficiency, and an example diagram is shown;

[0050] Figure 5 The prediction method provided by the embodiment of the present application is compared with other methods in terms of single-frame prediction results, and an example diagram is shown;

[0051] Figure 6 The functional structure diagram of the video prediction device for reducing the accumulation of inter-frame errors provided by the embodiment of the present application is shown;

[0052] Figure 7 The functional structure diagram of the electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0055] First, the design idea of the embodiment of the present application is briefly introduced.

[0056] The core goal of spatio-temporal prediction tasks is to accurately infer the future state of the system by analyzing the spatio-temporal dependencies in historical data. Such tasks face unique challenges: natural spatio-temporal processes exhibit complex non-stationary characteristics in both spatial and temporal dimensions. Specifically, in the spatial dimension, there is local dependency between adjacent pixels; in the temporal dimension, the joint distribution of these dependencies continuously changes dynamically. In-depth understanding and modeling of this high-order feature of spatio-temporal non-stationarity is of key significance to spatio-temporal prediction and video prediction tasks. In particular, in practical application scenarios such as climate modeling, autonomous driving, and traffic flow prediction, the system needs to capture both long-term trends and short-term fluctuations to achieve accurate spatio-temporal state prediction.

[0057] Exemplarily, in a video prediction task of a traffic scene, it is necessary to deduce the time sequence change of a future frame through the spatio-temporal features of historical consecutive frames. A typical scenario is to predict the dynamic evolution of a subsequent image frame based on an input sequence image (such as the previous T frames). This task needs to model both the intra-frame spatial structure features (such as object shape, texture distribution) and the inter-frame time evolution law (such as motion trajectory, state gradual change). For example, in intelligent traffic management, based on a sequence of video frames of a road intersection, a vehicle position probability heat map in the next 5 to 15 minutes is predicted for traffic signal control.

[0058] At present, the spatio-temporal prediction method based on deep neural network mainly includes two types: an autoregressive method and a non-autoregressive method. The autoregressive spatio-temporal prediction method uses a traditional RNN architecture, and its prediction mode depends on recursively receiving the historical information of the previous frame, modeling the implicit state of the current frame, and passing it to the next frame, and repeatedly predicting the future frame through the observation frame. This explicit modeling method can ensure accurate prediction, but has significant problems of computational efficiency and error accumulation; the non-autoregressive method uses an encoder-decoder architecture, which inputs a sequence of multiple observed image frames at one time, models the spatio-temporal features through the spatio-temporal feature module in the encoder-decoder architecture, and predicts all future frames at one time. Although this method supports multi-step joint prediction, it easily produces time sequence breaks or motion distortion due to neglecting the dynamic coupling relationship between frames.

[0059] In an intelligent urban traffic control system, spatio-temporal prediction based on video stream is a core link to realize dynamic signal optimization. For the prediction of a vehicle position probability heat map (physical quantity dimension: HxWx1, where H=1080, W=1920) of a road intersection monitoring video stream (resolution≥1920x1080, frame rate 25fps) in the next 5 to 15 frames (time span 1 to 3 minutes), a decision basis is provided for signal timing. This scenario faces three technical bottlenecks:

[0060] Feature loss caused by complex occlusion: the overlap rate of vehicles at the intersection is as high as 40%, and the traditional recursive model (such as ConvLSTM) is insufficient in local feature extraction, resulting in a false target proportion of more than 20% in the heat map;

[0061] Error accumulation caused by motion mutation: the vehicle's sudden acceleration / lane changing behavior makes the inter-frame motion distribution non-stationary, and the autoregressive model transmits the hidden state frame by frame, resulting in a prediction trajectory drift, and the peak signal-to-noise ratio (PSNR) of a 15-frame long sequence decays by more than 6dB;

[0062] Computing power bottleneck under real-time constraints: the single inference delay of the traditional sequence model is≥120ms, which cannot meet the 100ms level response requirement of the signal control system.

[0063] To this end, the present application combines the core ideas of autoregressive methods and non-autoregressive methods, parallelizes the autoregressive architecture by using parallel scan algorithm and decoupled implicit state, and designs a "multiple-input-multiple-output" architecture based on non-autoregressive methods during model training, aiming to reduce the error accumulation and computational time of autoregressive methods under the premise of explicit modeling of spatial and temporal dependencies.

[0064] The invention idea of the method of the present application is as follows:

[0065] Recursive formula conversion: convert the recursive formula of traditional LSTM (Long Short-Term Memory) into prefix sum calculation in logarithmic space. For the following form of recursive sequence: t = g t-1 x t-1 +h t , parallelization is achieved by the following transformation:

[0066]

[0067] Wherein, the operator and the operator calculate the cumulative product and cumulative sum of each element of the vector, respectively, and represents Hadamard product. g t-1 is the filter gate parameter, h t is the candidate implicit state, x0 represents the initial first frame image, x t represents the frame image at time t.

[0068] For the above formula, taking the logarithm of both sides of the equation can obtain:

[0069]

[0070] Wherein:

[0071] In particular, in the above formula, x0 and x t are replaced by the input frame sequence X 0:t-1 and the predicted frame sequence X t:K ,

[0072] Prefix sum calculation: cumulative sum of filter gate parameter logg 0:t-1 and candidate implicit state logh 1:t , the formula is:

[0073]

[0074]

[0075] Synchronous prediction: generate the implicit state or implicit feature of all predicted frames at once by exponential operation The formula is:

[0076]

[0077] where, represents the implicit features of all the predicted frame sequences, X 0:t-1 represents all the input frame sequences (observation frame sequences). By parallelizing the processing, the time complexity is reduced from O(n) to O(logn), significantly improving the training speed.

[0078] In summary, the prediction and generation of the automatic driving scene are faced with three challenges of untrusted generation, uncontrollable distribution, and unreliable understanding. The essence is the specific problems of spatiotemporal representation, content generation, and risk reasoning in the process of spatiotemporal modeling of traffic scenes, but the traditional RNN model has error accumulation and inefficient training due to sequence dependence, which is difficult to meet the complex needs of long-tail scenes. Therefore, the present application proposes a parallel SwinLSTM (Parallel SwinLSTM, PSL) model to realize efficient spatiotemporal prediction to support trusted traffic scene video prediction. Unlike the traditional architecture that predicts the next frame based on the current frame, PSL adopts a multi-input multi-output architecture, eliminates hidden state dependence through a parallel scanning algorithm, effectively reduces the impact of error accumulation, and reduces the error growth rate by 3-5 times compared to other methods on multiple benchmark datasets, solving the stability problem of traffic scene prediction. On the other hand, this parallel implementation greatly shortens the consumption of computing resources, and under the same conditions, only one-fourth of the time is used to achieve better generation results. In addition, the present application develops a PSG (Parallel SwinGRU) with a single gate and a single state lightweight design, without introducing any additional skills and complex strategies, PSL and PSG achieve the most advanced performance on four benchmark datasets. This new paradigm of "decoupling state dependence + global spatiotemporal features" provides high-fidelity generation and real-time risk reasoning capabilities for autonomous driving scenarios, supporting safe decision-making in extreme situations. In the future, the full-link capabilities of "spatiotemporal faithful generation-risk real-time deduction-decision closed-loop optimization" will be further realized.

[0079] The advantages of the present application are:

[0080] 1. Remove the explicit transmission dependence of the implicit state in the traditional LSTM:

[0081] In order to realize the parallelization of the LSTM architecture, the explicit transmission dependence of the implicit state must be removed, and the gate is merged: the input gate, the forget gate, and the output gate of the traditional LSTM are merged into the same filter gate F t, in order to realize numerical stability and parallelization, the above formula is converted to logarithmic space for implementation, canceling the candidate state range limit: removing the tanh function in traditional LSTM for limiting candidate state, directly outputting the unnormalized candidate hidden state through Swin Transformer neural network By simplifying the gating structure and canceling the value range limit, the model parameter amount is reduced by 30%, while avoiding the problem of gradient disappearance.

[0082] 2. Efficient parallel training significantly improves computing efficiency:

[0083] Through the innovative parallel scanning algorithm, the time complexity of the traditional recursive model is optimized from O(n) to O(logn) combined with the parallel computing capability of GPU, which greatly shortens the training time. Experiments show that on the Moving MNIST dataset, the single training cycle time is reduced from 320 seconds of traditional SwinLSTM to 98 seconds, with an efficiency improvement of 68%. At the same time, the memory occupancy is reduced by 40%, supporting large-scale data batch processing, providing a hardware-friendly solution for real-time prediction tasks.

[0084] 3. Suppress error accumulation and enhance long sequence prediction robustness:

[0085] The traditional recursive model causes error accumulation layer by layer due to the frame-by-frame transmission of hidden state, while the present application blocks the error propagation path by eliminating the dependence on explicit state. On the TaxiBJ dataset, the average error growth rate (AEGR) of 4-frame prediction is reduced from 0.473 to 0.335, and the long sequence prediction stability is significantly improved. For example, on the KTH dataset, the structural similarity (SSIM) is still 0.937 when predicting 40 frames, which is better than the traditional model of 0.703.

[0086] 4. Global-local joint modeling accurately captures complex dynamics:

[0087] Combining the window shifting mechanism of Swin Transformer with hierarchical feature fusion, the model can capture both local details (such as pedestrian limb movement) and global trends (such as traffic direction). On the Human3.6M dataset, the mean absolute error (MAE) is reduced from 10.4 to 5.66, demonstrating its efficient modeling capability for complex spatiotemporal dynamics. In addition, multi-scale feature fusion improves the target segmentation SSIM of Moving MNIST to 0.964, retaining more details.

[0088] 5. Flexible support for variable length prediction, adapt to diverse scenarios:

[0089] Based on the multiple-input multiple-output (MIMO) architecture, the model can predict a sequence of frames of any length at one time without adjustment. For example, 10 input frames can be flexibly output as 20 or 40 frame prediction results (as in the KTH dataset experiment), meeting the diversified needs of scenarios such as autonomous driving and weather prediction.

[0090] 6. Lightweight design and strong generalization capability to broaden application boundaries:

[0091] By merging the gating structure and removing the state range limit, the model parameter quantity is reduced by 30%. At the same time, it achieves the optimal performance in both synthetic data (Moving MNIST) and real scenarios (TaxiBJ, Human3.6M), proving strong cross-domain generalization capability and providing a reliable foundation for edge computing and cross-scenario deployment.

[0092] After introducing the application scenarios and design ideas of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0093] As shown in Figure 1 The present application provides a video prediction method for reducing the accumulation of inter-frame errors, which comprises the following steps:

[0094] Step 101: Obtain an image frame at the current time and m consecutive image frames before the current time;

[0095] For example, the image frames can come from traffic intersections or other traffic scenes.

[0096] Step 102: Process the image frame at the current time and the m consecutive image frames before the current time using a spatio-temporal prediction model to obtain the implicit state of the k consecutive image frames after the current time; the spatio-temporal prediction model uses a parallel long short-term memory model with implicit state decoupling;

[0097] Step 103: Process the implicit state of the k consecutive image frames after the current time using a reconstruction layer to obtain the predicted k consecutive image frames.

[0098] The present application constructs a spatio-temporal prediction model with a parallel LSTM architecture with implicit state decoupling, removes the explicit transmission dependency of the implicit state in the traditional LSTM, and realizes parallel computation of recursive sequences through a parallel scanning algorithm; simplifies the gating structure of LSTM, merges the input gate, the forget gate and the output gate into a single filtering gate, and cancels the value range limit of the candidate implicit state; the multiple frame observation data is mapped into multiple frame prediction results at one time, avoiding the error accumulation caused by frame-by-frame recursive prediction.

[0099] In some embodiments, the spatio-temporal prediction model comprises an initialization module, a processing module and a parallel processing module;

[0100] The space-time prediction model is used to process the current image frame and the continuous m image frames before the current time to obtain the implicit state of the continuous k image frames after the current time; comprising:

[0101] An image frame sequence X containing the current image frame and the continuous m image frames before the current time is set t,m+1 is: x t-m ,…x t ; wherein x t is the current image frame, x t-m is the mth image frame before x t ;

[0102] An initialized predicted implicit state of the continuous k image frames is generated by using an initialization module;

[0103] The image frame sequence X is processed by using a processing module t,m+1 to obtain the candidate implicit state of the image frame sequence X t,m+1 and the filter gate parameter vector;

[0104] The initialized predicted implicit state of the continuous k image frames, the candidate implicit state of the image frame sequence X t,m+1 and the filter gate parameter vector are processed by using a parallel processing module to obtain the implicit state of the continuous k image frames after the current time.

[0105] In some embodiments, the processing module comprises, connected in sequence: a linear projection layer, a Swin Transformer neural network and a logarithmic function;

[0106] The image frame sequence X is processed by using a processing module t,m+1 to obtain the candidate implicit state of the image frame sequence X t,m+1 and the filter gate parameter vector; comprising:

[0107] The image frame sequence X is processed by using a linear projection layer and a Swin Transformer neural network t,m+1 to obtain the candidate implicit state of the image frame sequence X t,m+1 :

[0108]

[0109] wherein LP(·) represents the linear projection layer; STB(·) represents the Swin Transformer neural network;

[0110] The candidate implicit state is operated by using a logarithmic function to obtain the filter gate parameter vector F t,m+1 :

[0111]

[0112] wherein, ln(·) represents a logarithm function with base e.

[0113] In particular, the number of levels of the Swin Transformer neural network is 12, the window size is 2x2, and the channel dimension is 1024, so as to balance the calculation efficiency and feature expression capability.

[0114] In some embodiments, the initialized predicted implicit state of the consecutive k image frames, the candidate implicit state of the image frame sequence X t,m+1 and the filtering gate parameter vector are processed by using the parallel processing module to obtain the implicit state of the consecutive k image frames after the current time; including:

[0115] Based on the filtering gate parameter vector, a first parameter

[0116]

[0117] wherein, f t-i is the i-th component of the filtering gate parameter vector F t,m+1 ;

[0118] A second parameter temp j is calculated:

[0119]

[0120] wherein, is the initialized predicted implicit state of the consecutive j image frames, 1≤j≤k; concat() is a concatenation function; and represents Hadamard product.

[0121] Based on the second parameter, a candidate implicit state of the predicted consecutive k image frames

[0122]

[0123] wherein, exp(·) is an exponential function with base e.

[0124] The predicted implicit state of the consecutive k image frames is generated by an exponential operation

[0125]

[0126] In some embodiments, the reconstruction layer is stacked by a plurality of deconvolution layers and a plurality of convolution layers.

[0127] In some embodiments, the method further comprises the step of training the linear projection layer, the Swin Transformer neural network, and the reconstruction layer.

[0128] In this embodiment, the mean square error (MSE) loss function is used for training.

[0129] The above method is verified on the Moving MNIST dataset, the Taxi BJ dataset, the KTH dataset, and the Human3.6M dataset, and exhibits superior performance on standard spatiotemporal prediction tasks and variable sequence prediction tasks. Some qualitative results are shown in Figure 2 and Figure 3 In addition, the outstanding contributions of the proposed method in reducing error accumulation and improving computational efficiency are comprehensively evaluated. Some results are shown in Figure 4 and Figure 5 .

[0130] Based on the above embodiments, the present embodiment provides a video prediction device for reducing inter-frame error accumulation. As shown in Figure 6 The video prediction device 200 provided in the present embodiment at least comprises:

[0131] An acquisition unit 201 is configured to acquire an image frame at a current time and m consecutive image frames before the current time.

[0132] A prediction unit 202 is configured to process the image frame at the current time and the m consecutive image frames before the current time by using a spatiotemporal prediction model to obtain implicit states of k consecutive image frames after the current time. The spatiotemporal prediction model adopts a parallel long short-term memory model with implicit state decoupling.

[0133] A reconstruction unit 203 is configured to process the implicit states of the k consecutive image frames after the current time by using a reconstruction layer to obtain predicted k consecutive image frames.

[0134] It should be noted that the principle of the video prediction device 200 provided in the present embodiment for reducing inter-frame error accumulation solves the technical problems similar to the method provided in the present embodiment. Therefore, the implementation of the video prediction device 200 provided in the present embodiment for reducing inter-frame error accumulation can be referred to the implementation of the method provided in the present embodiment, and the repeated parts will not be described herein.

[0135] Based on the above embodiments, the present embodiment further provides an electronic device. As shown in Figure 7As shown, the electronic device 300 provided by the embodiments of the present application at least includes a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the video prediction method for reducing frame error accumulation provided by the embodiments of the present application when executing the computer program.

[0136] The electronic device 300 provided by the embodiments of the present application can further include a bus 303 connecting different components (including the processor 301 and the memory 302). Wherein, the bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0137] The memory 302 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023.

[0138] The memory 302 can further include a program tool 3025 having a set of (at least one) program modules 3024, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0139] The electronic device 300 can also communicate with one or more external devices 304 (such as a keyboard, a remote control, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 300 (such as a phone, a computer, etc.), and / or with any devices (such as a router, a modem, etc.) that enable the electronic device 300 to communicate with one or more other electronic devices 300. Such communication can be carried out through an input / output (I / O) interface 305. And, the electronic device 300 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. As Figure 7 As shown, the network adapter 306 communicates with other modules of the electronic device 300 through the bus 303. It should be understood that although the network adapter 306 is shown as a distinct component from the processor 301, the network adapter 306 can be an integral part of the processor 301. Figure 7Other hardware and / or software modules can be used in conjunction with electronic device 300, as shown, including, but not limited to, microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems, etc.

[0140] It should be noted that, Figure 7 Electronic device 300 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0141] The embodiments of the present application further provide a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the video prediction method for reducing error accumulation between frames provided by the embodiments of the present application. Specifically, the executable program can be built-in or installed in electronic device 300, so that electronic device 300 can implement the video prediction method for reducing error accumulation between frames based on roadside perception provided by the embodiments of the present application by executing the built-in or installed executable program.

[0142] The video prediction method for reducing error accumulation between frames provided by the embodiments of the present application can also be implemented as a program product, and the program product includes program codes, and the program codes are used to make electronic device 300 execute the video prediction method for reducing error accumulation between frames provided by the embodiments of the present application when the program product can run on electronic device 300.

[0143] The program product provided by the embodiments of the present application can adopt any combination of one or more readable media, and the readable media can be readable signal media or readable storage media, and the readable storage media can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above, and more specifically, more specific examples (non-exhaustive list) of readable storage media include an electrical connection with one or more wires, a portable disc, a hard disk, a RAM, a ROM, an Erasable Programmable Read Only Memory (EPROM), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] The program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a computing device. However, the program product provided by the embodiments of the present application is not limited to this, and in the embodiments of the present application, the readable storage medium can be any tangible medium containing or storing programs, which can be used by or in combination with an instruction execution system, device or apparatus.

[0145] It should be noted that, although several units or sub-units of the apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0146] In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application is described in detail with reference to the embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of claims of the present application.

Claims

1. A method of video prediction that reduces the accumulation of inter-frame errors, characterized by , comprising: obtaining an image frame at a current time and m continuous image frames before the current time; processing the image frame at the current time and the m continuous image frames before the current time by using a space-time prediction model to obtain an implicit state of k continuous image frames after the current time; the space-time prediction model adopts a parallel long short-term memory model decoupled from the implicit state; processing the implicit state of the k continuous image frames after the current time by using a reconstruction layer to obtain the predicted k continuous image frames; the space-time prediction model comprises an initialization module, a processing module and a parallel processing module; processing the image frame at the current time and the m continuous image frames before the current time by using a space-time prediction model to obtain an implicit state of k continuous image frames after the current time; comprising: The image frame sequence comprises an image frame of a current moment and m image frames before the current moment For: ; wherein, is an image frame of a current moment, is the mth image frame before the current moment; generating an initialized implicit state of the predicted k continuous image frames by using the initialization module; Processing a sequence of image frames using a processing module to obtain a sequence of image frames candidate hidden states and filter gate parameter vectors The implicit states and image frame sequences of the initial prediction k consecutive image frames are processed using a parallel processing module. The candidate implicit states and the filter gate parameter vector are processed to obtain the implicit states of the k consecutive image frames after the current time. the processing module comprises a linear projection layer, a Swin Transformer neural network and a logarithmic function connected in sequence; Processing a sequence of image frames using a processing module to obtain a sequence of image frames of candidate implicit states and filter gate parameter vectors; comprising: Processing image frame sequences using linear projection layers and swin transformer neural networks :​​ wherein, represents a linear projection layer; represents a Swin Transformer neural network; Utilizing a logarithmic function on the candidate implicit states Performing operations to obtain a filter gate parameter vector : wherein denotes the natural logarithm function. The implicit states and image frame sequences of the initial prediction k consecutive image frames are processed using a parallel processing module. The candidate implicit states and filter gate parameter vectors are processed to obtain the implicit states of the k consecutive image frames after the current time step; including: Based on the filtered gate parameter vector, compute the first parameter : wherein, is the i-th component of the filtered door parameter vector is the i-th component of the filtered door parameter vector computing a second parameter : wherein, is the implicit state of the initialized prediction of the consecutive j image frames, is the stitching function; denotes the Hadamard product;​ based on the second parameter, compute a predicted candidate hidden state for the consecutive k image frames : wherein is an exponential function with base e; Generating predicted implicit states for the next k consecutive image frames by exponentiation : 。 2. The method of claim 1, wherein, the reconstruction layer is stacked by a plurality of deconvolution layers and a plurality of convolution layers.

3. The method of claim 2, wherein, The method further comprises the steps of training the linear projection layer, the Swin Transformer neural network and the reconstruction layer.

4. A video prediction apparatus that reduces accumulation of inter-frame errors, characterized by , comprising: an acquisition unit, configured to acquire an image frame at a current time and m continuous image frames before the current time; a prediction unit, configured to process the image frame at the current time and the m continuous image frames before the current time by using a space-time prediction model to obtain an implicit state of k continuous image frames after the current time; the space-time prediction model adopts a parallel long short-term memory model decoupled from the implicit state; a reconstruction unit, configured to process the implicit state of the k continuous image frames after the current time by using a reconstruction layer to obtain the predicted k continuous image frames; the space-time prediction model comprises an initialization module, a processing module and a parallel processing module; the prediction unit is specifically configured to: The image frame sequence comprises an image frame of a current moment and m image frames before the current moment For: ; wherein, is an image frame of a current moment, is the mth image frame before the current moment; generate an initialized implicit state of the predicted k continuous image frames by using the initialization module; Processing a sequence of image frames using a processing module to obtain a sequence of image frames candidate implicit states and filter gate parameter vectors; The implicit states and image frame sequences of the initial prediction k consecutive image frames are processed using a parallel processing module. The candidate implicit states and filter gate parameter vectors are processed to obtain the implicit states of k consecutive image frames after the current time. the processing module comprises a linear projection layer, a Swin Transformer neural network and a logarithmic function connected in sequence; Processing a sequence of image frames using a processing module to obtain a sequence of image frames of candidate implicit states and filter gate parameter vectors; comprising: Processing image frame sequences using linear projection layers and swin transformer neural networks Processing image frame sequences using linear projection layers and swin transformer neural networks Processing image frame sequences using linear projection layers and swin transformer neural networks Processing image frame sequences using linear projection layers and swin transformer neural networks wherein, represents a linear projection layer; represents a Swin Transformer neural network; Utilizing a logarithmic function on the candidate implicit states Performing operations to obtain a filter gate parameter vector : wherein denotes the natural logarithm function; The implicit states and image frame sequences of the initial prediction k consecutive image frames are processed using a parallel processing module. The candidate implicit states and filter gate parameter vectors are processed to obtain the implicit states of the k consecutive image frames after the current time step; including: Based on the filtered gate parameter vector, compute the first parameter : wherein, is the i-th component of the filtered gate parameter vector is the i-th component of the filtered gate parameter vector computing a second parameter : wherein, is the implicit state of the initialized prediction of the consecutive j image frames, is the stitching function; denotes the Hadamard product;​ based on the second parameter, compute a predicted candidate hidden state for the consecutive k image frames : wherein is an exponential function with base e; Generating predicted implicit states for the next k consecutive image frames by exponentiation : 。 5. An electronic device, comprising: comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implements the method of any one of claims 1-3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method of any one of claims 1-3.

Images