Video prediction method and device for reducing inter-frame error accumulation

Abstract

The invention provides a video prediction method and device for reducing inter-frame error accumulation. The method comprises the following steps: acquiring an image frame at the current moment and m continuous image frames before the current moment; processing the image frame at the current moment and m continuous image frames before the current moment by using a space-time prediction model to obtain implicit states of k continuous image frames after the current moment; the space-time prediction model adopts an implicit state decoupling parallel long and short term memory model; and processing the implicit states of k continuous image frames after the current moment by using the reconstruction layer to obtain k predicted continuous image frames. According to the invention, the collected multi-frame observation video can be mapped into the multi-frame prediction video at one time, and error accumulation caused by frame-by-frame recursion prediction is avoided.

Description

Technical Field

[0001] The present application relates to the field of spatiotemporal prediction technology, and in particular to a video prediction method and device for reducing inter-frame error accumulation. Background Art

[0002] Currently, in the video prediction methods for traffic intersections, autoregressive-based methods and non-autoregressive-based methods are usually used.

[0003] Autoregressive spatiotemporal prediction methods utilize a traditional RNN architecture. Their prediction model relies on recursively receiving historical information from the previous frame, modeling the implicit state of the current frame, and passing this information to the next frame, cyclically predicting future frames based on observed frames. This explicit modeling approach can ensure relatively accurate predictions, but it suffers from issues such as unstable gradients, low computational efficiency, and severe error accumulation.

[0004] The non-autoregressive method uses an encoder-decoder architecture to input a multi-frame image sequence at one time, models the spatiotemporal features through the spatiotemporal feature module in the encoder-decoder architecture, and predicts all future frames at one time. Although this method supports multi-step joint prediction, it ignores the dynamic coupling relationship between frames, which is prone to timing discontinuity or motion distortion. Summary of the Invention

[0005] In view of this, the present application provides a video prediction method and device for reducing inter-frame error accumulation, so as to solve the problems of gradient instability, low computational efficiency and serious error accumulation in existing video prediction methods.

[0006] In a first aspect, an embodiment of the present application provides a video prediction method for reducing inter-frame error accumulation, comprising:

[0007] Get the image frame at the current moment and the m consecutive image frames before the current moment;

[0008] Processing the image frame at the current moment and m consecutive image frames before the current moment using a spatiotemporal prediction model to obtain the implicit states of k consecutive image frames after the current moment; the spatiotemporal prediction model adopts a parallel long short-term memory model with implicit state decoupling;

[0009] The reconstruction layer is used to process the implicit states of the k consecutive image frames after the current moment to obtain the predicted k consecutive image frames.

[0010] In a possible implementation, the spatiotemporal prediction model includes: an initialization module, a processing module, and a parallel processing module;

[0011] The spatiotemporal prediction model is used to process the image frame at the current moment and the m consecutive image frames before the current moment to obtain the implicit state of the k consecutive image frames after the current moment; including:

[0012] Set the image frame sequence X that includes the image frame at the current moment and the m consecutive image frames before the current moment. t,m+1 is: x t-m ,…x t ; where x t is the image frame at the current moment, x t-m is x t The previous m-th image frame;

[0013] Using the initialization module to generate the implicit state of the initial predicted k consecutive image frames;

[0014] Use the processing module to process the image frame sequence X t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector of ;

[0015] The parallel processing module is used to initialize the implicit state of the predicted consecutive k image frames and the image frame sequence X t,m+1 The candidate implicit states and filter gate parameter vectors are processed to obtain the implicit states of k consecutive image frames after the current moment.

[0016] In one possible implementation, the processing module includes: a linear projection layer, a SwinTransformer neural network, and a logarithmic function connected in sequence;

[0017] Use the processing module to process the image frame sequence X t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector of ; including:

[0018] The image frame sequence X is processed using the linear projection layer and the Swin Transformer neural network. t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 Candidate implicit states of

[0019]

[0020] Where LP(·) represents the linear projection layer; STB(·) represents the Swin Transformer neural network;

[0021] Use the logarithmic function to calculate the candidate implicit state Perform the operation to obtain the filter gate parameter vector F t,m+1 :

[0022]

[0023] Wherein, ln(·) represents the logarithmic function with base e.

[0024] In a possible implementation, the parallel processing module is used to initialize the predicted implicit state of the consecutive k image frames, the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector are processed to obtain the implicit state of k consecutive image frames after the current moment; including:

[0025] Based on the filter gate parameter vector, calculate the first parameter

[0026]

[0027] Among them, f t-i is the filter gate parameter vector F t,m+1 The i-th component of ;

[0028] Calculate the second parameter temp j :

[0029]

[0030] in, is the implicit state of the initial predicted j consecutive image frames, 1≤j≤k; concat() is the concatenation function; ⊙ represents the Hadamard product;

[0031] Based on the second parameter, calculate the candidate implicit state of the predicted k consecutive image frames

[0032]

[0033] Where exp(·) is an exponential function with e as the base;

[0034] Generate the predicted implicit state of k consecutive image frames through exponential operation

[0035]

[0036] In a possible implementation, the reconstruction layer is formed by stacking multiple deconvolution layers and multiple convolution layers.

[0037] In a possible implementation, the method further includes the steps of training the linear projection layer, the Swin Transformer neural network, and the reconstruction layer.

[0038] In a second aspect, an embodiment of the present application provides a video prediction device for reducing inter-frame error accumulation, comprising:

[0039] An acquisition unit, configured to acquire an image frame at a current moment and m consecutive image frames before the current moment;

[0040] A prediction unit, configured to process the image frame at the current moment and m consecutive image frames before the current moment using a spatiotemporal prediction model to obtain implicit states of k consecutive image frames after the current moment; the spatiotemporal prediction model adopts a parallel long short-term memory model with implicit state decoupling;

[0041] The reconstruction unit is used to process the implicit states of k consecutive image frames after the current moment using the reconstruction layer to obtain predicted k consecutive image frames.

[0042] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method of the embodiment of the present application is implemented.

[0044] This application can map the collected multiple frames of observation video into multiple frames of prediction video at one time, reducing the error accumulation caused by frame-by-frame recursive prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 A flowchart of a video prediction method for reducing inter-frame error accumulation provided by an embodiment of the present application;

[0047] Figure 2 An example diagram of spatiotemporal prediction results of the prediction method provided in an embodiment of the present application on the Moving MNIST dataset;

[0048] Figure 3 An example diagram of spatiotemporal prediction results of the prediction method provided in an embodiment of the present application on the KTH dataset;

[0049] Figure 4 An example diagram comparing the prediction method provided in the embodiment of the present application with other methods in terms of computational efficiency;

[0050] Figure 5 An example diagram comparing the prediction method provided in the embodiment of the present application with other methods in terms of single-frame prediction results;

[0051] Figure 6 This is a functional structure diagram of a video prediction device for reducing inter-frame error accumulation provided by an embodiment of the present application;

[0052] Figure 7 This is a functional structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0055] First, a brief introduction to the design concept of the embodiments of the present application is given.

[0056] The core goal of spatiotemporal prediction tasks is to accurately infer the future state of a system by analyzing the spatiotemporal dependencies in historical data. This type of task faces unique challenges: natural spatiotemporal processes exhibit complex non-stationary characteristics in both spatial and temporal dimensions. Specifically, at the spatial level, there are local dependencies between adjacent pixels; in the temporal dimension, the joint distribution of these dependencies changes continuously and dynamically. A deep understanding and modeling of the high-order characteristics of this spatiotemporal non-stationarity is crucial for spatiotemporal and video prediction tasks. In particular, in practical application scenarios such as climate modeling, autonomous driving, and traffic flow forecasting, the system needs to capture both long-term trends and short-term fluctuations to achieve accurate spatiotemporal state predictions.

[0057] For example, in the video prediction task of traffic scenes, it is necessary to deduce the temporal changes of future frames through the spatiotemporal features of historical continuous frames. A typical scenario is to predict the dynamic evolution of subsequent image frames based on the input sequence image (such as the previous T frames). This task requires modeling both the spatial structural features within the frame (such as object morphology and texture distribution) and the temporal evolution laws between frames (such as motion trajectory and state gradient). For example, in smart traffic management, based on the intersection monitoring video frame sequence, the vehicle position probability heat map for the next 5 to 15 minutes is predicted for traffic light control.

[0058] Currently, spatiotemporal prediction methods based on deep neural networks are mainly divided into two categories: autoregressive-based methods and non-autoregressive-based methods. Autoregressive-based spatiotemporal prediction methods utilize the traditional RNN architecture. Their prediction model relies on recursively receiving historical information from the previous frame, modeling the implicit state of the current frame, and passing it to the next frame, cyclically predicting future frames through observed frames. This explicit modeling method can ensure relatively accurate predictions, but there are significant issues with computational efficiency and error accumulation. Non-autoregressive-based methods utilize an encoder-decoder architecture, inputting a multi-frame observed image sequence at one time, modeling the spatiotemporal features through the spatiotemporal feature module in the encoder-decoder architecture, and predicting all future frames at once. Although this method supports multi-step joint prediction, it ignores the dynamic coupling relationship between frames and is prone to temporal discontinuities or motion distortion.

[0059] In smart city traffic control systems, spatiotemporal prediction based on video streams is the core link in achieving dynamic signal optimization. A heat map (physical dimension: H×W×1, where H=1080 and W=1920) of vehicle position probabilities for the next 5 to 15 frames (a time span of 1 to 3 minutes) is generated based on the intersection monitoring video stream (resolution ≥1920×1080, frame rate 25fps) to provide a decision-making basis for signal timing. This scenario faces three technical bottlenecks:

[0060] Feature loss caused by complex occlusion: The vehicle overlap rate at intersections is as high as 40%. Traditional recursive models (such as ConvLSTM) are unable to extract local features, resulting in a false target ratio exceeding 20% ​​in the heat map.

[0061] Error accumulation caused by sudden motion changes: Rapid acceleration and lane changes make the inter-frame motion distribution non-stationary. The autoregressive model transfers hidden states frame by frame, causing the predicted trajectory to drift. The peak signal-to-noise ratio (PSNR) of a 15-frame sequence decreases by more than 6dB.

[0062] Computing bottleneck under real-time constraints: The single inference delay of traditional sequence models is ≥120ms, which cannot meet the 100ms response requirement of signal control systems.

[0063] To this end, this application combines the core ideas of autoregressive methods and non-autoregressive methods, realizes the parallelization of autoregressive architecture by using parallel scanning algorithm and decoupled implicit state, and designs a "multi-input-multi-output" architecture based on non-autoregressive methods during model training, aiming to reduce the error accumulation and computational time of autoregressive methods while ensuring explicit modeling of spatiotemporal dependencies.

[0064] The inventive ideas of the method of this application are as follows:

[0065] Recursive formula conversion: Convert the recursive formula of the traditional LSTM (Long Short-Term Memory) into a prefix sum calculation in logarithmic space. For the following recursive sequence: x t =g t-1 x t-1 +h t , parallelized by the following transformation:

[0066]

[0067] Among them, the operator Sum Operator Calculate the cumulative product and cumulative sum of each element of the vector respectively, ⊙ represents the Hadamard product. t-1 is the filter gate parameter, h t is a candidate implicit state, x0 represents the initial first frame image, x t Represents the frame image at time t.

[0068] For the above formula, taking the logarithm on both sides of the equal sign, we can get:

[0069]

[0070] in:

[0071] In particular, in the above formula, x0 and x t Replaced by the input frame sequence X 0:t-1 With the predicted frame sequence X t:K ,

[0072] Prefix and calculation: filter gate parameter logg 0:t-1 and candidate implicit state logh 1:t The cumulative summation formula is:

[0073]

[0074]

[0075] Synchronous prediction: Generate implicit states or implicit features of all predicted frames at once through exponential operations The formula is:

[0076]

[0077] in, represents the implicit features of all predicted frame sequences, X 0:t-1 Represents all input frame sequences (observation frame sequences). Through parallel processing, the time complexity is reduced from O(n) to O(logn), significantly improving the training speed.

[0078] In summary, the prediction and generation of autonomous driving scenarios face three major challenges: unreliable generation, uncontrollable distribution, and unreliable understanding. The essence of these challenges is the specific problems of spatiotemporal representation, content generation, and risk reasoning faced in the spatiotemporal modeling of traffic scenarios. However, the traditional RNN model suffers from error accumulation and inefficient training due to sequence dependency, making it difficult to meet the complex requirements of long-tail scenarios. To this end, this application proposes a parallel SwinLSTM (Parallel SwinLSTM, PSL) model to achieve efficient spatiotemporal prediction to support credible traffic scene video prediction. Unlike the traditional architecture that predicts the next frame based on the current frame, PSL adopts a multi-input and multi-output architecture, eliminates hidden state dependencies through a parallel scanning algorithm, effectively reduces the impact of error accumulation, and reduces the error growth rate by 3-5 times compared to other methods on multiple benchmark data sets, solving the problem of stability in traffic scene prediction. On the other hand, this parallel implementation greatly reduces the consumption of computing resources, and under the same conditions, better generation results can be achieved in only a quarter of the time. In addition, this application developed a single-gate, single-state lightweight PSG (Parallel SwinGRU). Without introducing any additional techniques or complex strategies, PSL and PSG achieved state-of-the-art performance on four benchmark datasets. This new paradigm of "decoupled state dependency + global spatiotemporal features" provides high-fidelity generation and real-time risk reasoning capabilities for autonomous driving scenarios, supporting safe decision-making in extreme situations. In the future, the full-link capability of "spatiotemporal fidelity generation - real-time risk deduction - decision closed-loop optimization" will be further realized.

[0079] The advantages of this application are:

[0080] 1. Remove the explicit transitive dependency of implicit states in traditional LSTM:

[0081] In order to achieve the parallelization of LSTM architecture, it is necessary to remove the explicit transitive dependency of hidden state and gate merging: the input gate, forget gate and output gate of traditional LSTM are merged into the same filter gate F. tIn order to achieve numerical stability and parallelization, the above formula is converted to logarithmic space for implementation, and the candidate state range restriction is cancelled: the tanh function used to restrict the candidate state in the traditional LSTM is removed, and the unnormalized candidate implicit state is directly output through the Swin Transformer neural network. By simplifying the gating structure and removing the value range restriction, the number of model parameters is reduced by 30% while avoiding the gradient vanishing problem.

[0082] 2. Efficient parallel training significantly improves computing efficiency:

[0083] Through an innovative parallel scanning algorithm, the time complexity of traditional recursive models is optimized from O(n) to O(logn). Combined with the parallel computing capabilities of GPUs, this significantly reduces training time. Experiments show that on the Moving MNIST dataset, a single training cycle time is reduced from 320 seconds for the traditional SwinLSTM to 98 seconds, a 68% efficiency improvement. Furthermore, graphics memory usage is reduced by 40%, supporting large-scale data batch processing and providing a hardware-friendly solution for real-time prediction tasks.

[0084] 3. Suppress error accumulation and enhance the robustness of long sequence prediction:

[0085] Traditional recursive models, which pass implicit states from frame to frame, cause errors to accumulate layer by layer. This application, however, eliminates explicit state dependencies and blocks the error propagation path. On the TaxiBJ dataset, the average error growth rate (AEGR) for predicting four frames dropped from 0.473 to 0.335, significantly improving the stability of long-sequence predictions. For example, when predicting 40 frames on the KTH dataset, the structural similarity (SSIM) remained at 0.937, surpassing the 0.703 of the traditional model.

[0086] 4. Global-local joint modeling accurately captures complex dynamics:

[0087] Combining the Swin Transformer's window shifting mechanism with hierarchical feature fusion, the model is able to simultaneously capture local details (such as pedestrian limb movements) and global trends (such as traffic flow direction). On the Human3.6M dataset, the mean absolute error (MAE) was reduced from 10.4 to 5.66, demonstrating its ability to effectively model complex spatiotemporal dynamics. Furthermore, multi-scale feature fusion improved the SSIM of object segmentation on the Moving MNIST dataset to 0.964, preserving even more detailed information.

[0088] 5. Flexible support for variable-length predictions to adapt to diverse scenarios:

[0089] Based on a multiple-input, multiple-output (MIMO) architecture, the model can predict frame sequences of any length in one go without requiring any adjustments. For example, a 10-frame input can flexibly output 20 or 40 frames of prediction results (as in the KTH dataset experiment), meeting the diverse needs of scenarios such as autonomous driving and weather forecasting.

[0090] 6. Lightweight design and strong generalization capabilities broaden application boundaries:

[0091] By incorporating a gated structure and removing state range restrictions, the model's parameters are reduced by 30%. Furthermore, it achieves optimal performance on both synthetic data (Moving MNIST) and real-world scenarios (TaxiBJ and Human3.6M), demonstrating its strong cross-domain generalization capabilities and providing a reliable foundation for edge computing and cross-scenario deployment.

[0092] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0093] like Figure 1 As shown, the embodiment of the present application provides a video prediction method for reducing inter-frame error accumulation, including:

[0094] Step 101: Acquire the image frame at the current moment and m consecutive image frames before the current moment;

[0095] For example, the image frame may be from a traffic intersection or other traffic scenes.

[0096] Step 102: Processing the image frame at the current moment and the m consecutive image frames before the current moment using a spatiotemporal prediction model to obtain the implicit states of the k consecutive image frames after the current moment; the spatiotemporal prediction model uses a parallel long short-term memory model with implicit state decoupling;

[0097] Step 103: Use the reconstruction layer to process the implicit states of the k consecutive image frames after the current moment to obtain predicted k consecutive image frames.

[0098] This application constructs a spatiotemporal prediction model of a parallel LSTM architecture with implicit state decoupling, removes the explicit transitive dependency of implicit states in traditional LSTM, and realizes parallel computation of recursive sequences through a parallel scanning algorithm; simplifies the gating structure of LSTM, merges the input gate, forget gate, and output gate into a single filter gate, and removes the value range restriction of candidate implicit states; maps multiple frames of input observation data into multiple frames of prediction results at one time to avoid error accumulation caused by frame-by-frame recursive prediction.

[0099] In some embodiments, the spatiotemporal prediction model includes: an initialization module, a processing module, and a parallel processing module;

[0100] The spatiotemporal prediction model is used to process the image frame at the current moment and the m consecutive image frames before the current moment to obtain the implicit state of the k consecutive image frames after the current moment; including:

[0101] Set the image frame sequence X that includes the image frame at the current moment and the m consecutive image frames before the current moment. t,m+1 is: x t-m ,…x t ; where x t is the image frame at the current moment, x t-m is x t The previous m-th image frame;

[0102] Using the initialization module to generate the implicit state of the initial predicted k consecutive image frames;

[0103] Use the processing module to process the image frame sequence X t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector of ;

[0104] The parallel processing module is used to initialize the implicit state of the predicted consecutive k image frames and the image frame sequence X t,m+1 The candidate implicit states and filter gate parameter vectors are processed to obtain the implicit states of k consecutive image frames after the current moment.

[0105] In some embodiments, the processing module includes: a linear projection layer, a Swin Transformer neural network, and a logarithmic function connected in sequence;

[0106] Use the processing module to process the image frame sequence X t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector of ; including:

[0107] The image frame sequence X is processed using the linear projection layer and the Swin Transformer neural network. t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 Candidate implicit states of

[0108]

[0109] Where LP(·) represents the linear projection layer; STB(·) represents the Swin Transformer neural network;

[0110] Use the logarithmic function to calculate the candidate implicit state Perform the operation to obtain the filter gate parameter vector F t,m+1 :

[0111]

[0112] Wherein, ln(·) represents the logarithmic function with base e.

[0113] In particular, the number of layers of the Swin Transformer neural network is 12, the window size is 2×2, and the channel dimension is 1024 to balance computational efficiency and feature expression capability.

[0114] In some embodiments, the parallel processing module is used to initialize the predicted implicit states of the consecutive k image frames, the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector are processed to obtain the implicit state of k consecutive image frames after the current moment; including:

[0115] Based on the filter gate parameter vector, calculate the first parameter

[0116]

[0117] Among them, f t-i is the filter gate parameter vector F t,m+1 The i-th component of ;

[0118] Calculate the second parameter temp j :

[0119]

[0120] in, is the implicit state of the initial predicted j consecutive image frames, 1≤j≤k; concat() is the concatenation function; ⊙ represents the Hadamard product;

[0121] Based on the second parameter, calculate the candidate implicit state of the predicted k consecutive image frames

[0122]

[0123] Where exp(·) is an exponential function with e as the base;

[0124] Generate the predicted implicit state of k consecutive image frames through exponential operation

[0125]

[0126] In some embodiments, the reconstruction layer is formed by stacking multiple deconvolution layers and multiple convolution layers.

[0127] In some embodiments, the method further includes the steps of training the linear projection layer, the Swin Transformer neural network, and the reconstruction layer.

[0128] In this embodiment, the mean square error (MSE) loss function is used for training.

[0129] This application verifies the above method on the Moving MNIST dataset, Taxi BJ dataset, KTH dataset, and Human3.6M dataset, and shows excellent performance in standard spatiotemporal prediction tasks and tasks with variable prediction sequences. Some qualitative results are shown in Figure 2 and Figure 3 In addition, the proposed method is comprehensively evaluated for its outstanding contribution in reducing error accumulation and improving computational efficiency. Some of the results are shown in Figure 4 and Figure 5 .

[0130] Based on the above embodiments, the present application provides a video prediction device that reduces inter-frame error accumulation. Figure 6 As shown, the video prediction device 200 for reducing inter-frame error accumulation provided by the embodiment of the present application at least includes:

[0131] An acquisition unit 201 is configured to acquire an image frame at a current moment and m consecutive image frames before the current moment;

[0132] A prediction unit 202 is configured to process the image frame at the current moment and m consecutive image frames before the current moment using a spatiotemporal prediction model to obtain the implicit states of k consecutive image frames after the current moment; the spatiotemporal prediction model adopts a parallel long short-term memory model with implicit state decoupling;

[0133] The reconstruction unit 203 is configured to process the implicit states of k consecutive image frames after the current moment using the reconstruction layer to obtain k consecutive predicted image frames.

[0134] It should be noted that the principle of solving the technical problem by the video prediction device 200 for reducing inter-frame error accumulation provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the video prediction device 200 for reducing inter-frame error accumulation provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0135] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 7As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the video prediction method for reducing inter-frame error accumulation provided in the embodiment of the present application is implemented.

[0136] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0137] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0138] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0139] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 7 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. Figure 7Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0140] It should be noted that Figure 7 The electronic device 300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0141] The present embodiment further provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the video prediction method for reducing inter-frame error accumulation provided in the present embodiment. Specifically, the executable program can be built into or installed in the electronic device 300. Thus, the electronic device 300 can implement the video prediction method for reducing inter-frame error accumulation based on roadside perception provided in the present embodiment by executing the built-in or installed executable program.

[0142] The video prediction method for reducing inter-frame error accumulation provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the video prediction method for reducing inter-frame error accumulation provided in the embodiment of the present application.

[0143] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] The program product provided in the embodiments of the present application may be a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiments of the present application is not limited thereto. In the embodiments of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0145] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0146] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0147] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of this application and are not intended to limit the scope of the present invention. Although this application has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application and should be encompassed by the claims of this application.

Claims

1. A video prediction method for reducing inter-frame error accumulation, characterized in that ,include: Get the image frame at the current moment and the m consecutive image frames before the current moment; Processing the image frame at the current moment and m consecutive image frames before the current moment using a spatiotemporal prediction model to obtain the implicit states of k consecutive image frames after the current moment; the spatiotemporal prediction model adopts a parallel long short-term memory model with implicit state decoupling; The reconstruction layer is used to process the implicit states of the k consecutive image frames after the current moment to obtain the predicted k consecutive image frames.

2. The method according to claim 1, characterized in that The spatiotemporal prediction model includes: an initialization module, a processing module and a parallel processing module; The spatiotemporal prediction model is used to process the image frame at the current moment and the m consecutive image frames before the current moment to obtain the implicit state of the k consecutive image frames after the current moment; including: Set the image frame sequence X that includes the image frame at the current moment and the m consecutive image frames before the current moment. t,m+1 is: x t-m ,…x t ; where x t is the image frame at the current moment, x t-m is x t The previous m-th image frame; Using the initialization module to generate the implicit state of the initial predicted k consecutive image frames; Use the processing module to process the image frame sequence X t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector of ; The parallel processing module is used to initialize the implicit state of the predicted consecutive k image frames and the image frame sequence X t,m+1 The candidate implicit states and filter gate parameter vectors are processed to obtain the implicit states of k consecutive image frames after the current moment.

3. The method according to claim 2, characterized in that The processing module includes: a linear projection layer, a Swin Transformer neural network and a logarithmic function connected in sequence; Use the processing module to process the image frame sequence X t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector of ; include: The image frame sequence X is processed using the linear projection layer and the Swin Transformer neural network. t,m+1 Processing is performed to obtain the image frame sequence X t,m+1 Candidate implicit states of Where LP(·) represents the linear projection layer; STB(·) represents the Swin Transformer neural network; Use the logarithmic function to calculate the candidate implicit state Perform the operation to obtain the filter gate parameter vector F t,m+1 : Wherein, ln(·) represents the logarithmic function with base e.

4. The method according to claim 3, characterized in that The parallel processing module is used to initialize the implicit state of the predicted consecutive k image frames and the image frame sequence X t,m+1 The candidate implicit state and filter gate parameter vector are processed to obtain the implicit state of k consecutive image frames after the current moment; including: Based on the filter gate parameter vector, calculate the first parameter Among them, f t-i is the filter gate parameter vector F t,m+1 The i-th component of ; Calculate the second parameter temp j : in, is the implicit state of the initial predicted j consecutive image frames, 1≤j≤k; concat() is the concatenation function; ⊙ represents the Hadamard product; Based on the second parameter, calculate the candidate implicit state of the predicted k consecutive image frames Where exp(·) is an exponential function with e as the base; Generate the predicted implicit state of k consecutive image frames through exponential operation 5. The method according to claim 3, characterized in that The reconstruction layer is formed by stacking multiple deconvolution layers and multiple convolution layers.

6. The method according to claim 5, characterized in that The method also includes the steps of training the linear projection layer, the SwinTransformer neural network and the reconstruction layer.

7. A video prediction device for reducing inter-frame error accumulation, characterized in that ,include: An acquisition unit, configured to acquire an image frame at a current moment and m consecutive image frames before the current moment; A prediction unit, configured to process the image frame at the current moment and m consecutive image frames before the current moment using a spatiotemporal prediction model to obtain implicit states of k consecutive image frames after the current moment; the spatiotemporal prediction model adopts a parallel long short-term memory model with implicit state decoupling; The reconstruction unit is used to process the implicit states of k consecutive image frames after the current moment using the reconstruction layer to obtain predicted k consecutive image frames.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images