Prediction method and device, electronic equipment and computer readable storage medium
Patent Information
- Application Number
- CN202280094951.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-29
- Publication Date
- 2024-11-29
AI Technical Summary
Traditional video prediction methods based on recurrent neural networks average the adjacent frames of the current video frame and all historical video frames equally when updating the hidden layer state, resulting in a decrease in video prediction accuracy and image quality.
By obtaining the historical hidden layer feature map of the video frame, the historical unit state and the hidden layer feature map of the previous frame, multiple rounds of reasoning and feature fusion are performed, and feature processing is performed step by step to enhance the hidden layer feature map of the current frame that depends on the previous frame. state, thereby improving inter-frame correlation.
The accuracy of video prediction and the image quality of predicted frames are improved, making the hidden layer feature map more accurately express the relationship between frames, and improving the effect of video frame prediction.
Smart Images

Figure CN119053973A_ABST
Abstract
Description
Prediction method, device, electronic device, and computer-readable storage medium Technical Field
[0001] The present application relates to the field of video processing based on artificial intelligence, and in particular to a prediction method, device, electronic device and computer-readable storage medium. Background Art
[0002] Video prediction involves predicting future video sequences based on existing ones. It has broad application prospects. In particular, in video coding, video prediction allows for the transmission of residuals between predicted and actual values, rather than the actual video frames themselves, after compression and encoding. This significantly improves coding efficiency. Furthermore, in meteorology, video prediction helps analyze typical spatiotemporal sequences—radar echoes. By predicting future radar echo patterns, we can effectively predict future precipitation. Furthermore, in the field of intelligent driving, video prediction technology can help machines predict road obstacles based on past scenarios, enabling effective predictions.
[0003] Typical video prediction methods include optical flow. This method calculates motion feature vectors from past video frames and performs extrapolated predictions based on these past motion feature vectors. Furthermore, with the rise of deep learning algorithms in recent years, neural network-based video prediction models have become increasingly common. Typical examples include methods based on convolutional neural networks, recurrent neural networks, and generative adversarial networks.
[0004] Currently, traditional video prediction methods based on recurrent neural networks use a long short-term memory (LSTM) network to average and fuse the video frame information from all historical time steps to update the hidden layer state. This LSTM hidden layer state corresponds to the current time step, and the predicted video frame corresponding to the current video frame is then predicted using the LSTM hidden layer state at the current time step. It can be seen that when updating the hidden layer state, the current method performs the same averaging on the adjacent frames of the current video frame and all historical video frames, which reduces the accuracy of the video prediction and the image quality of the predicted frame.
[0005] Summary of the Invention
[0006] The present application proposes a prediction method, device, electronic device and computer-readable storage medium, which can improve the accuracy of video prediction and the image quality of the predicted frame.
[0007] The technical solution of this application is achieved as follows:
[0008] In a first aspect, an embodiment of the present application provides a prediction method, the method comprising:
[0009] When predicting a current video frame among n video frames, obtaining the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map of the current video frame when predicting the previous video frame; the i-1-th hidden layer feature map is obtained by performing feature processing i-1 times on the current video frame; wherein i is an integer greater than or equal to 2; and n is an integer greater than or equal to 1;
[0010] Perform H rounds of inference and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map; wherein the inference result obtained in each round of inference is fused based on the i-th historical hidden layer feature map; H is an integer greater than or equal to 1;
[0011] Based on the i-th hidden layer feature map, continue the i+1-th feature processing until K-th feature processing is completed, and obtain the current predicted video frame corresponding to the current video frame; K is the total number of feature processing times and is an integer greater than or equal to 2.
[0012] In a second aspect, an embodiment of the present application provides a prediction device, comprising: an acquisition unit, configured to, when predicting a current video frame among n video frames, acquire an i-th historical hidden layer feature map, an i-th historical unit state, and an i-1-th hidden layer feature map of the current video frame when predicting a previous video frame; the i-1-th hidden layer feature map is obtained by performing i-1 feature processing on the current video frame; wherein i is an integer greater than or equal to 2; and n is an integer greater than or equal to 1;
[0013] The inference fusion part is configured to perform H rounds of inference and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map; wherein the inference result obtained in each round of inference is fused based on the i-th historical hidden layer feature map; H is an integer greater than or equal to 1;
[0014] The prediction part is configured to continue the i+1th feature processing based on the i-th hidden layer feature map until the K-th feature processing is completed, thereby obtaining the current predicted video frame corresponding to the current video frame; K is the total number of feature processing times and is an integer greater than or equal to 2.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device including a memory and a processor; wherein,
[0016] a memory for storing computer programs capable of running on the processor;
[0017] The processor is configured to execute the video prediction method as described in the first aspect when running a computer program.
[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the method described in the first aspect.
[0019] The embodiments of the present application provide a prediction method, device, electronic device, and computer-readable storage medium. During the i-th feature processing of the current video frame, inference is performed using the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map, and the inference result is fused with the i-th historical hidden layer feature map to obtain the i-th hidden layer feature map. This allows the currently updated hidden layer feature map to rely more on the adjacent hidden layer state of the previous time step, so that the inter-frame correlation is better expressed in the current hidden layer feature map, thereby improving the accuracy of feature processing, and further improving the accuracy of video frame prediction based on the hidden layer feature map obtained by each feature processing, as well as the quality of the predicted frame. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] FIG1 is a schematic diagram of an optional architecture of an in-vehicle system 100 that applies video frame prediction according to an embodiment of the present application;
[0021] FIG2 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0022] FIG3 is a schematic diagram of an optional structure of a neural network unit provided in an embodiment of the present application;
[0023] FIG4 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0024] FIG5 is a schematic diagram of an optional structure of a neural network architecture provided in an embodiment of the present application;
[0025] FIG6 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0026] FIG7 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0027] FIG8 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0028] FIG9 is a schematic diagram of an optional structure of a novel neural network unit provided in an embodiment of the present application;
[0029] FIG10 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0030] FIG11 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0031] FIG12 is a schematic diagram of an optional network architecture of a novel neural network unit stacking method provided in an embodiment of the present application;
[0032] FIG13 is a schematic diagram of an optional flow chart of a prediction method provided in an embodiment of the present application;
[0033] FIG14 is a schematic diagram of an optional network architecture of a novel neural network unit stacking method provided in an embodiment of the present application;
[0034] FIG15 is a block diagram of a video encoding system provided in an embodiment of the present application;
[0035] FIG16 is a block diagram of a video decoding system provided in an embodiment of the present application;
[0036] FIG17 is a schematic diagram of a neural network architecture applied in an actual scenario according to an embodiment of the present application;
[0037] FIG18 is a first schematic diagram comparing the video frame prediction effects of the prediction method provided by an embodiment of the present application and the prior art;
[0038] FIG19 is a second schematic diagram comparing the video frame prediction effects of the prediction method provided by an embodiment of the present application and the prior art;
[0039] FIG20a is a first schematic diagram showing a comparison of the prediction index data of the prediction method provided by the embodiment of the present application and the prior art;
[0040] FIG20 b is a second schematic diagram comparing the prediction index data of the prediction method provided by the embodiment of the present application and the prior art;
[0041] FIG20c is a third schematic diagram comparing the prediction index data of the prediction method provided by the embodiment of the present application and the prior art;
[0042] FIG20 d is a fourth schematic diagram comparing the prediction index data of the prediction method provided by the embodiment of the present application with that of the prior art;
[0043] FIG21 is a schematic diagram of the structure of a prediction device provided in an embodiment of the present application;
[0044] FIG22 is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0046] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0047] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0048] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.
[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.
[0050] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0051] 1) Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.
[0052] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.
[0053] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0054] 2) Receptive field: The receptive field defines the size of the area where the pixels on the feature map output by the convolutional neural network layer are mapped on the input image.
[0055] The solutions provided in the embodiments of this application involve technologies such as artificial intelligence video processing, which are specifically illustrated by the following embodiments.
[0056] Currently, when processing video prediction tasks for the current video frame, traditional long-short-term memory (LSTM) networks update the hidden layer state of the LSTM network by averaging and fusing the video frame information from all previous time steps to obtain the hidden layer state for the current time step. This hidden layer state is then used to predict the current input known video frame. It can be seen that the current LSTM hidden layer state update method, which uses averaging and fusing as a representation of the hidden layer state, is not accurate enough, thus reducing the accuracy of video frame prediction based on the hidden layer state and, in turn, the image quality of the predicted frame.
[0057] The embodiments of the present application provide a prediction method, apparatus, electronic device, and computer-readable storage medium that can improve the accuracy of video prediction and the image quality of the predicted video frames. The electronic device provided in the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and vehicle-mounted terminals, and can also be implemented as servers. Below, taking the vehicle-mounted scenario as an example, an exemplary application of the electronic device when it is implemented as a terminal is described.
[0058] Referring to Figure 1, Figure 1 is an optional architectural diagram of a vehicle-mounted system 100 that applies a video frame prediction method provided in an embodiment of the present application. As shown in Figure 1, the terminal of the embodiment of the present application may be a vehicle-mounted terminal, which is used to implement an automatic driving function using the prediction method provided in an embodiment of the present application. The vehicle-mounted terminal 400 (vehicle-mounted terminal 400-1 and vehicle-mounted terminal 400-2 are shown as examples) is connected to the server 200 via a network 300, and the network 300 may be a wide area network or a local area network, or a combination of the two.
[0059] The vehicle terminal 400 (such as the vehicle terminal 400 - 1 ) is used to capture a video of a road scene by using an image capture device while the vehicle is traveling, and obtain one or more video frames among n video frames.
[0060] The server 200 is used to send the video frames collected by other surrounding vehicle terminals (such as the vehicle terminal 400-2) to the vehicle terminal 400 as one or more video frames among n video frames, thereby realizing networked collaborative perception in autonomous driving.
[0061] The vehicle terminal 400 is also used to obtain the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map of the current video frame when predicting the current video frame among n video frames; the i-1-th hidden layer feature map is obtained by performing i-1 feature processing on the current video frame; wherein, i is an integer greater than or equal to 2; n is an integer greater than or equal to 1; H rounds of reasoning and feature fusion are performed on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map; wherein, the reasoning result obtained in each round of reasoning is fused based on the i-th historical hidden layer feature map; H is an integer greater than or equal to 1; based on the i-th hidden layer feature map, the i+1-th feature processing is continued until K feature processings are completed to obtain the current predicted video frame corresponding to the current video frame; K is the total number of feature processing times, and is an integer greater than or equal to 2.
[0062] The vehicle-mounted terminal 400 is also used to perform image prediction analysis on the current predicted video frame, such as road obstacle prediction or driving route prediction, and make vehicle control decisions based on the image analysis results to achieve automatic driving control of the vehicle.
[0063] It should be noted that the server 200 can also synchronize information from vehicles around the terminal 400 to the vehicle-mounted terminal 400, such as transmitting driving information of surrounding vehicles to the vehicle-mounted terminal 400, so that the terminal 400 can combine more comprehensive road environment images and information for decision analysis, and realize network-assisted information interaction in autonomous driving.
[0064] In some embodiments, the server 200 is also used to receive road scene video frames and vehicle driving information transmitted by the vehicle terminal 400, perform auxiliary decision analysis based on the road scene video frames and vehicle driving information of the vehicle terminal 400, and send the auxiliary decision results to the vehicle terminal 400, so that the vehicle terminal 400 can combine the auxiliary decision results of the server 200 with its own decision results to jointly determine the vehicle control decision and realize networked collaborative decision-making in autonomous driving.
[0065] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.
[0066] The prediction method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.
[0067] See Figure 2, which is an optional flow chart of the prediction method provided in an embodiment of the present application, which will be explained in conjunction with the steps shown in Figure 2.
[0068] S101. When predicting a current video frame among n video frames, obtain the i-th historical hidden layer feature map when predicting the previous video frame, the i-th historical unit state, and the i-1-th hidden layer feature map of the current video frame.
[0069] The embodiments of the present application are applicable to video frame prediction scenarios.
[0070] For example, in video encoding and decoding scenarios, it can be used for video encoding and decoding prediction, or in the meteorological field, future cloud image predictions can be made based on cloud images captured by satellites, and then weather forecasts, such as rainfall predictions, can be made based on the predicted cloud images; or in the field of smart transportation, on-board equipment can predict the upcoming road conditions based on currently collected road scene video frames, and then implement autonomous driving functions such as obstacle avoidance or driving according to navigation routes based on the predicted road scene video frames; or, a traffic management platform can predict traffic flow based on traffic flow video frames collected at intersections, etc. The specific selection will be based on actual circumstances and is not limited by the embodiments of this application.
[0071] In an embodiment of the present application, when processing a video frame prediction task, an electronic device can determine the time step for task processing based on the number of known frames and the number of frames to be predicted predefined for the prediction task. For example, for a prediction task that uses the first n frames of a video frame sequence to predict the next r frames, it can be determined that the task processing includes n+r-1 time steps. Here, n video frames are known preset video frames; n is an integer greater than or equal to 1; and r is an integer greater than or equal to 1.
[0072] In S101, for each of the n time steps, the electronic device determines the video frame corresponding to the current time step from the n video frames as the current video frame, and obtains historical state data representing the historical state, so as to use the current state data representing the current state obtained by processing the current video frame and combine it with the historical state data to predict the video frame of the next time step.
[0073] In an embodiment of the present application, the electronic device can perform K times of feature processing on the current video frame in each time step, and obtain the hidden state of the current video frame corresponding to the feature processing through each feature processing, that is, the hidden layer feature map, and the unit state. Among them, K is the total number of feature processing times; and is an integer greater than or equal to 2. Here, the hidden state represents the feature information obtained by the feature processing of the current video frame. The hidden layer feature map (feature map) of the embodiment of the present application can be understood as the expression of the hidden state in the feature map domain; the unit state represents the video frame feature information memorized or stored in each feature processing.
[0074] Thus, for the i-th feature processing of the current video frame in the current time step, the electronic device can obtain the hidden layer feature map and unit state obtained by performing the i-th feature processing on the previous video frame when predicting the previous video frame, as the i-th historical hidden layer feature map and the i-th historical unit state. The electronic device obtains the hidden layer feature map obtained by performing the i-1-th feature processing on the current video frame as the i-1-th hidden layer feature map. Here, the i-1-th hidden layer feature map is the hidden layer feature map obtained by the feature processing process before the i-th feature processing during the feature processing of the current video frame. Wherein, i is an integer greater than or equal to 2.
[0075] For the i-th feature processing process, the electronic device can use the i-th historical hidden layer feature map and the i-th historical unit state as historical state data, and the i-1-th hidden layer feature map of the current video frame as current state data to perform the current feature processing.
[0076] It should be noted that, for the first feature processing process in the K-time feature processing process, the electronic device obtains the first historical hidden layer feature map and the first historical unit state when predicting the previous video frame; uses the current video frame as the current state data; and processes the current video frame, the first historical hidden layer feature map, and the first historical unit state features to obtain the first hidden layer feature map of the current video frame. Thus, when i is 2, that is, during the second feature processing process, the first hidden layer feature map of the current video frame can be used as the i-1th hidden layer feature map to be obtained in the second feature processing process, and the second feature processing process is continued based on the first hidden layer feature map until the K-time feature processing is completed. Here, each feature processing can include H rounds of inference and feature fusion; H is an integer greater than or equal to 1.
[0077] S102. Perform H rounds of reasoning and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map; wherein the reasoning results obtained in each round of reasoning are fused based on the i-th historical hidden layer feature map.
[0078] In S102, the electronic device may perform H rounds of reasoning and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map. Each round of the H rounds includes a reasoning process and a feature fusion process.
[0079] In an embodiment of the present application, the electronic device can perform inference based on the previous state feature map H, the previous unit state C, and the current state feature map X to obtain an initial hidden layer feature map; and fuse the initial hidden layer feature map with the previous state feature map to obtain the current hidden layer feature map and the current unit state. Here, the electronic device can perform inference based on the historical state data represented by the previous state feature map H and the previous unit state C and the current state feature map X, and fuse the inference result, that is, the initial hidden layer feature map obtained by inference, with the previous state feature map H to further enhance the correlation between the historical state data and the current state data.
[0080] During the inference process in S102, the electronic device uses the i-th historical hidden layer feature map as the previous state feature map H, the i-th historical unit state as the previous unit state C, and the i-1-th hidden layer feature map as the current state feature map X. Based on the data representing the historical state and the data representing the current state, the electronic device infers the predicted state data for the next time step as the inference result. In S102, the inference result may be the i-th initial hidden layer feature map corresponding to the current time step.
[0081] In the fusion process in S102, the electronic device fuses the inference result, i.e., the i-th initial hidden layer feature map, with the previous state feature map, i.e., the i-th historical hidden layer feature map, to obtain the i-th hidden layer feature map. It can be understood that, compared to the i-th initial hidden layer feature map, the i-th hidden layer feature map includes feature expressions that are strongly correlated with the i-th historical hidden layer feature map, that is, the influence of the historical state information of the previous frame is further strengthened in the state information of the current video frame.
[0082] In some embodiments, the electronic device can use an LSTM network unit to implement the reasoning process. For example, the LSTM network unit shown in FIG3 can be used to implement the reasoning process in the i-th feature processing in the current time step t. As shown in FIG3, the electronic device can use the preset first weight W h , when predicting the previous video frame, that is, the i-th historical hidden layer feature map corresponding to the previous time step t-1 Perform weighted and convolution processing to obtain the first intermediate tensor; use the preset second weight W x , for the i-1th hidden layer feature map at the current time step t Perform weighted and convolution processing to obtain a second intermediate tensor. The electronic device concatenates the first intermediate tensor and the second intermediate tensor according to the number of channels to obtain a first tensor TE1, a second tensor TE2, a third tensor TE3, and a fourth tensor TE4. The electronic device can use a preset gating factor and, in combination with the i-th historical unit state, process the first tensor TE1, the second tensor TE2, the third tensor TE3, and the fourth tensor TE4 to obtain the i-th unit state corresponding to the current time step t, that is, the current video frame. and the i-th inference result
[0083] In some embodiments, the preset gating factors may include: an input gate, a forget gate, and an output gate. The electronic device uses the preset gating factors to process the first tensor TE1, the second tensor TE2, the third tensor TE3, and the fourth tensor TE4 to obtain the i-th unit state and the i-th inference result. The process can be shown in Figure 3. The electronic device performs S-curve activation (Sigmoid) and forget gate f on the first tensor TE1. t Processing, get the first update tensor; S curve activation and input gate i are performed on the second tensor TE2 t Processing is performed to obtain the second updated tensor; the third tensor TE3 is activated by hyperbolic tangent (tanh) to obtain the third updated tensor; the fourth tensor TE4 is activated by S curve and the output gate o is activated. t The electronic device calculates the third weight W according to the preset third weight W. c , fuse the first update tensor with the i-th history unit state The first intermediate state is obtained by merging the second updated tensor and the third updated tensor, such as by multiplication; the second intermediate state is obtained by merging the first intermediate state and the second intermediate state, such as by addition, to obtain the i-th unit state. The electronic device responds to the state of the i-th unit Perform hyperbolic tangent activation processing and fuse the processing result with the fourth update tensor to obtain the i-th inference result
[0084] Here, the first weight W is preset h , preset the second weight W x With the preset third weight W c The network parameters learned by the LSTM network unit through pre-network training.
[0085] In some embodiments, the above-mentioned state of the i-th unit is obtained The process can be realized by formula (1), as follows:
[0086]
[0087] In formula (1), g is the third update tensor; F t Forget gate f t The first updated tensor of the output; I t is the input gate i t The second updated tensor of the output. The state of the i-th unit Used to update the i-th hidden layer state in the next time step.
[0088] In some embodiments, the above-mentioned inference result i is obtained The process can be realized by formula (2), as follows:
[0089]
[0090] In formula (1), O t is the output gate o t The third updated tensor of the output.
[0091] According to FIG3, formula (1) and formula (2), it can be understood that for each current video frame after the first frame, the electronic device can obtain the historical hidden feature map and historical unit state of the previous time step. When the electronic device processes the first frame among n video frames, it can set the historical hidden feature map and historical unit state corresponding to the first frame to zero, or to other values. The specific selection is based on actual conditions and is not limited in the embodiments of this application.
[0092] Based on Figure 3, it can be seen that the initial hidden layer feature map In the above example, the correlation between the historical state information of the previous frame, that is, the historical hidden layer feature map, is not sufficiently expressed. If the initial hidden layer feature map of the inference process is directly based on Performing the next feature processing will reduce the accuracy of the hidden layer feature map obtained from the current feature processing. Based on this, the electronic device can fuse the inference result with the i-th historical hidden layer feature map to obtain the i-th hidden layer feature map. This allows the generation of each current hidden layer state to rely more on the historical hidden layer state of the previous time step, that is, the adjacent historical hidden layer states, thereby more accurately expressing the current hidden layer state and improving the accuracy of inter-frame video prediction. The fusion method will be further explained in the following embodiments.
[0093] S103 . Based on the i-th hidden layer feature map, continue to perform the i+1-th feature processing until K-th feature processing is completed, and obtain a current predicted video frame corresponding to the current video frame.
[0094] In S103, the electronic device uses the i-th hidden layer feature map as the data representing the current state for the next time, that is, the i+1-th feature processing, and continues the i+1-th feature processing with the same process until the K-th feature processing is completed, and obtains the K-th hidden layer feature map corresponding to the current video frame.
[0095] Furthermore, the electronic device can perform video frame prediction based on the K-th hidden layer feature map to obtain a current predicted video frame corresponding to the current video frame.
[0096] In some embodiments, for the prediction task of predicting the next r frames based on the previous n frames, after S103, the process may further include S104 and S105 as shown in FIG. 4 , as follows:
[0097] S104. When the current video frame is not the last frame among the n video frames, continue predicting the next video frame among the n video frames to obtain the next predicted video frame, until the prediction of the nth video frame is completed to obtain the nth predicted video frame.
[0098] In S104, when the current video frame corresponding to the current time step t is not the last frame among the n video frames, the electronic device can continue to predict the next video frame among the n video frames in the next time step t+1 to obtain the next predicted video frame, until the prediction of the nth video frame is completed, and the nth predicted video frame is obtained.
[0099] S105 , using the nth predicted video frame to predict the n+1th video frame to obtain the n+1th predicted video frame, until the n+rth predicted video frame is obtained based on the n+r-1th predicted video frame.
[0100] In S105, when n predicted video frames are predicted using n known video frames, in the next time step, the electronic device uses the nth predicted video frame, that is, the predicted video frame corresponding to the last known video frame, to predict the n+1th video frame and obtain the n+1th predicted video frame, until the n+rth predicted video frame is predicted based on the n+r-1th predicted video frame.
[0101] In some embodiments, the above process can be implemented based on the neural network architecture of FIG5 . The neural network architecture in FIG5 includes k network layers from bottom to top, corresponding to K times of feature processing in S103 . Each feature processing can be implemented by a separate feature processing unit of each network layer. The input frame sequence {X1, X2, ...X n} is equivalent to n video frames in S101. Taking the current video frame corresponding to the current time step t (t=2) as X2 as an example, in the first feature processing unit corresponding to X2 from bottom to top, the previous video frame is obtained, that is, the first historical hidden layer feature map corresponding to X1 With the first history unit state right And the current video frame X2 performs H rounds of reasoning and feature fusion to obtain the first hidden layer feature map of the current video frame X2 Continue based on the first hidden layer feature map The next feature processing is performed. During each feature processing process, for each feature processing unit, the i-th feature processing unit in each feature processing unit obtains the i-th historical hidden layer feature map and the i-th historical unit state from the i-th feature processing unit in the previous time step connected to the i-th feature processing unit; and obtains the i-1-th hidden layer feature map from the previous feature processing unit in the current time step connected to the i-th feature processing unit. In this way, in the i-th feature processing unit, H rounds of inference and feature fusion are performed on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map. The inference results obtained from each round of inference are fused with the i-th historical hidden layer feature map to obtain the i-th hidden layer feature map.
[0102] The electronic device performs K feature processing on the current video frame X2 using the neural network shown in FIG5 , using the same feature processing process as described above, to obtain the current predicted video frame Y3 corresponding to the current video frame X2 . Y3 represents the predicted video frame at the next time step t+1 (t+1=3) predicted based on the current video frame X2 .
[0103] As shown in FIG5 , the electronic device processes the input frame sequence {X1, X2, …X n}, and obtain the predicted video frame Y2 corresponding to X1; in the second time step, use X2 as the current video frame to perform video frame prediction, and obtain the predicted video frame Y3 corresponding to X2; until X n Perform video frame prediction and get X n The corresponding predicted video frame Y n+1 In the n+1th time step, the electronic device uses X n The corresponding predicted video frame Y n+1 As the current video frame, Y n+1 Perform video frame prediction to obtain predicted video frame Y n+2 Until the n+r-1th predicted video frame, that is, the predicted video frame Y n+r-1 , predict the n+rth predicted video frame Y n+r Until the time, the predicted frame sequence {Y n+1 , Y n+2 ,…Y n+r}, completing the video prediction task of predicting the next r video frames based on the first n video frames.
[0104] It can be understood that in the embodiment of the present application, during the i-th feature processing of the current video frame, reasoning is performed through the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map, and the reasoning result is fused with the i-th historical hidden layer feature map to obtain the i-th hidden layer feature map, so that the currently updated hidden layer feature map can be more dependent on the adjacent hidden layer state of the previous time step, so that the inter-frame correlation is better expressed in the current hidden layer feature map, thereby improving the accuracy of feature processing, and then improving the accuracy of video frame prediction based on the hidden layer feature map obtained by each feature processing, as well as the quality of the predicted frame.
[0105] In some embodiments, based on FIG. 2 , as shown in FIG. 6 , S102 may be implemented by executing the process of S1021 - S1023 , which will be described in conjunction with each step.
[0106] S1021. Infer the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the first round of initial hidden layer feature map.
[0107] In S1021, the electronic device performs a round of inference on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map in the first round of H rounds of processing, and uses the first round of inference results as the first round initial hidden layer feature map.
[0108] In some embodiments, the process of S1021 can be implemented using the LSTM network unit shown in Figure 3 to obtain the inference result as the first round of initial hidden layer feature map, which will not be repeated here.
[0109] S1022: Fusing the first-round initial hidden layer feature map with the i-th historical hidden layer feature map to obtain a first-round hidden layer feature map.
[0110] S1023. If H is 1, the first round of hidden layer feature map is used as the i-th hidden layer feature map.
[0111] In S1023, if H is 1, that is, each feature processing includes one round of feature processing and feature fusion, the electronic device uses the first round of hidden layer feature map as the i-th hidden layer feature map, and continues to perform the i+1-th feature processing based on the i-th hidden layer feature map.
[0112] In some embodiments, S1022 in FIG. 6 may be implemented by executing the process of S201 - S203 as shown in FIG. 7 , which will be described in conjunction with each step.
[0113] S201: Perform convolution and activation processing on the first round of initial hidden layer feature map to obtain feature offset, feature mask and residual compensation.
[0114] In S201, the electronic device may perform convolution processing on the first round of initial hidden layer feature maps in each coordinate axis direction to obtain a feature offset, perform convolution and activation processing on the first round of initial hidden layer feature maps to obtain a feature mask, and perform convolution processing on the first round of initial hidden layer feature maps to obtain residual compensation.
[0115] For example, the electronic device may generate the first round of initial hidden layer feature maps, such as the first round of inference Perform a kH*kW convolution to obtain a feature offset with a channel number of N*2*kH*kW. Here, kH*kW represents the number of convolution kernels; N represents the coordinate axis. N*2 represents the offset on the X-axis and Y-axis of the plane coordinate system respectively. The electronic device performs a kH*kW convolution on the initial hidden layer feature map of the first round, and then activates the convolution result, such as continuing to perform Sigmoid activation, to obtain a feature mask with a channel number of N*1*kH*kW. The electronic device performs a kH*kW convolution on the initial hidden layer feature map of the first round, and obtains a residual compensation with a channel number equal to the current layer hidden layer dimension. Here, the current layer hidden layer dimension refers to the hidden layer dimension of the current round of neural network unit for the current feature processing, such as the hidden layer dimension of the first round of neural network unit for the i-th feature processing.
[0116] In some embodiments, the electronic device may also perform a convolution and activation process on the first round of initial hidden layer feature map, and divide the convolution result into dimensions, thereby obtaining feature offsets, feature masks and residual compensation corresponding to different dimensions based on the convolution result.
[0117] S202: Perform deformable convolution processing on the i-th historical hidden layer feature map according to the feature offset and the feature mask to obtain a deformable convolution feature map.
[0118] In S202, the feature offset obtained in S201 can be used to offset each feature point position in the i-th historical hidden layer feature map along each coordinate axis, so that the network can better learn the feature point position corresponding to each feature point position after the deformable convolution. The feature mask can be used to add different weights to the offset of each feature point position to reduce the network's extraction of redundant feature offsets. Thus, the electronic device performs deformable convolution processing on the i-th historical hidden layer feature map based on the feature offset and the feature mask to obtain a deformable convolution feature map. By performing feature offset on the hidden layer features of the historical moment, the hidden layer features of the current moment are obtained.
[0119] Here, in standard convolution, the area where the convolution kernel acts is a rectangular area of the size of the convolution kernel, while the area where the deformable convolution acts in the embodiment of the present application is an irregular area. This is further explained in the following embodiment.
[0120] In some embodiments, based on FIG. 7 , as shown in FIG. 8 , the process of S202 may include S2021 to S2023 as follows:
[0121] S2021. Group the channels in the i-th historical hidden layer feature map to obtain at least one channel grouping.
[0122] S2022. Determine, according to the dimensions of the feature offset and the feature mask, a sub-feature offset and a sub-feature mask corresponding to each channel grouping of at least one group of channel groupings.
[0123] In the embodiment of the present application, the number of channels of the feature offset is N*2*kH*kW, and the number of channels of the feature mask is N*1*kH*kW. The electronic device can group the channels contained in the i-th historical hidden layer feature map, such as into N channel groups. In this way, each channel group can correspond to a dedicated sub-feature mask and 2D sub-feature offset in N*2*kH*kW, and each channel in the same group corresponds to the same sub-feature offset and sub-feature mask.
[0124] For example, if the i-th historical hidden layer feature map contains 64 channels, the electronic device may group the 64 channels into four groups, each of which contains 16 channels. The electronic device may process the 16 channels in each group using the same sub-feature offset and sub-feature mask.
[0125] S2023. Perform feature shifting on each historical feature point in each channel grouping through sub-feature shifting, and perform weighted convolution on the shifted features based on the sub-feature mask to obtain a deformable convolution feature map.
[0126] In S2023, for each channel in each channel grouping, each channel corresponds to a plurality of historical feature points. The electronic device superimposes an offset value corresponding to the historical feature point in the sub-feature offset on each historical feature point in each channel according to the sub-feature offset corresponding to each channel, and performs weighted convolution on each offset feature based on the mask value corresponding to each historical feature point in the sub-feature mask, thereby implementing deformable convolution and obtaining a deformable convolution map.
[0127] In some embodiments, the electronic device determines the target historical feature point set corresponding to the i-th historical hidden layer feature map for each target feature point in the deformable convolution feature map based on a preset receptive field size; then, for each target feature point in the deformable convolution feature map, the target historical feature point set is offset according to the sub-feature offset corresponding to each channel to obtain a corrected feature point set corresponding to each target feature point; and based on the sub-feature mask, the corrected feature point set is weighted convolved to obtain a target feature value corresponding to each target feature point; in this way, the electronic device can obtain a deformable convolution feature map in combination with the target feature value corresponding to each target feature point.
[0128] In some embodiments, for each target historical feature point in the target historical feature point set, the electronic device can determine the offset corresponding to each target historical feature point in the sub-feature offset; perform bilinear interpolation processing on each target feature point and each target historical feature point and the corresponding offset to obtain the corrected feature value corresponding to each target historical feature point in the target historical feature point set, as the corrected feature point set corresponding to each target feature point.
[0129] For example, for any target feature point p in the deformable convolution feature map output by the deformable convolution, its pixel value can be calculated by formula (3) as follows:
[0130]
[0131] In formula (3), y(p) represents the pixel value of the target feature point position P in the deformable convolution feature map. R is the receptive field, which is the area mapped by a feature point in the deformable convolution feature map on the i-th historical hidden layer feature map. n It can be the receptive field R, that is, the position of any target historical feature point in the mapping area corresponding to the target feature point P on the i-th historical hidden layer feature map, Δp n is the target historical feature point p n The corresponding offset in the sub-feature offset. Δm(p n ) is the target historical feature point p n The corresponding mask value in the sub-feature mask, w(p n ) is the weight of the deformable convolution, which can be determined by pre-training the deformable convolution network. n +Δp n ) is based on the target feature point p and the offset Δp n and the target historical feature point p n The resulting bilinear interpolation.
[0132] In some embodiments, x(p+p n +Δp n ) can be calculated as shown in formula (4):
[0133]
[0134] Here, p' in formula (4) is p+p in formula (3) n +Δp n , q represents the integer spatial position in the feature map; G is the bilinear interpolation kernel; the calculation method of G(q,p') can be shown in formula (5), as follows:
[0135] G(q,p')=g(q x ,p' x )·g(q y ,p' y ) (5)
[0136] In formula (5), p' x Characterization of p+p n +Δp n Position on the X axis, p' y Characterization of p+p n +Δp n Position on the Y axis; g(q x ,p' x ) and g(q y ,p' y The calculation process of ) can be shown as formula (6):
[0137] g(a,b)=max(0,1-|ab|) (6)
[0138] Here, a and b in formula (6) can be q in formula (5). x ,p' x or q y ,p' y The electronic device calculates g(q in formula (5) by formula (6) x ,p' x ) and g(q y ,p' y ) value. Then, based on formula (5), we can get x(p+p n +Δp n ), based on x(p+p n +Δp n ), and use formula (3) to obtain the pixel value y(p) at position P. The electronic device processes each feature point on the input feature map in the same way to obtain the pixel value corresponding to each feature point, and combines the pixel values corresponding to each feature point to obtain a deformable convolution feature map.
[0139] S203: Fusing the deformable convolution feature map with the residual compensation to obtain the first round of hidden layer feature map.
[0140] In S203, the electronic device performs deformable convolution on the i-th historical hidden layer feature map according to the feature offset and the feature mask to obtain a deformable convolution feature map, and fuses the deformable convolution feature map with the residual compensation obtained in S201 to obtain a first round of hidden layer feature map.
[0141] It should be noted that, for each round of feature fusion in the H rounds of processing, the fusion can be performed according to the method in S201 to S203, which will not be described in detail in this embodiment.
[0142] In some embodiments, the electronic device can implement the above-mentioned methods S201-S203 through the neural network unit structure shown in Figure 9. Figure 9 shows a new internal structure diagram of the LSTM network unit provided in an embodiment of the present application. For the i-th feature processing in the current time step t, each round of reasoning and fusion in the H rounds of reasoning in the i-th feature processing, Figure 9 includes the network unit part for reasoning as shown in Figure 3, which is used to implement the reasoning process in each round of reasoning and fusion process to obtain the initial hidden layer feature map. Then, using the 90 part of the new LSTM network unit, the initial hidden layer feature map Perform convolution to obtain residual compensation, feature offset and feature mask; use feature offset and feature mask to convolve the i-th historical hidden layer feature map Perform deformable convolution to obtain a deformable convolution feature map. Superimpose the deformable convolution feature map with the residual compensation to obtain the hidden layer feature map output by the inference and fusion process of the i-th feature processing in the current time step t.
[0143] It is understandable that the original convolutional long short-term memory network unit's method of directly obtaining the hidden layer state through convolution has been improved to obtain the feature offset and feature mask of deformable convolution from convolution, as well as residual compensation of the deformable convolution result. By performing deformable convolution on the hidden layer state of the previous time step through feature offset and feature mask, and adding residual compensation to obtain the hidden layer state of the current time step, the feature expression of the hidden layer state of the previous time step in the hidden layer state of the current time step is strengthened, and the influence of inter-frame correlation on the video prediction process is increased, thereby improving the video prediction accuracy and the quality of the predicted video frames.
[0144] In some embodiments, based on FIG. 2 , as shown in FIG. 10 , S102 may be implemented by executing processes of S1021 , S1022 , and S1024 , which will be described in conjunction with each step.
[0145] S1021. Infer the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the first round of initial hidden layer feature map.
[0146] S1022: Fusing the first-round initial hidden layer feature map with the i-th historical hidden layer feature map to obtain a first-round hidden layer feature map.
[0147] The processes of S1021 and S1022 have been described in the same steps above and will not be repeated here.
[0148] S1024. If H is greater than 1, then based on the first round hidden layer feature map, the i-th historical hidden layer feature map and the i-th historical unit state, continue with the next two rounds of reasoning and feature fusion until 2M rounds of reasoning and feature fusion are completed, and use the H-th round hidden layer feature map as the i-th hidden layer feature map.
[0149] In S1024, when H is greater than 1, that is, when the first round of reasoning and feature fusion is completed and the next round of reasoning and feature fusion is continued, the electronic device may perform reasoning and feature fusion in units of two rounds. That is, based on the first round hidden layer feature map, the i-th historical hidden layer feature map, and the i-th historical unit state, the next two rounds of reasoning and feature fusion are continued until 2M rounds of reasoning and feature fusion are completed, and the H-th round hidden layer feature map is used as the i-th hidden layer feature map. Wherein, M is an integer greater than or equal to 1, and 2M is equal to H-1.
[0150] In some embodiments, based on FIG. 10 , as shown in FIG. 11 , S1024 may be implemented by executing the process of S301 - S304 , which will be described in conjunction with each step.
[0151] S301: Obtain the first round of historical hidden layer feature maps when predicting the previous video frame.
[0152] In S301, when H is greater than 1, the electronic device performs three rounds of inference and feature fusion within a single feature processing. For the first round of inference and feature fusion, the electronic device can obtain the first round of hidden layer feature maps according to S1021-S1022. For the second round of inference and feature fusion, the electronic device obtains the first round of historical hidden layer feature maps when predicting the previous video frame as the historical state of the inference and feature fusion process. Here, the first round of historical hidden layer feature maps is obtained by performing the first round of inference and feature fusion during the i-th historical feature processing of the previous video frame.
[0153] S302: If M is equal to 1, perform inference and feature fusion on the first-round historical hidden layer feature map, the i-th historical hidden layer feature map, and the i-th historical unit state to obtain the second-round hidden layer feature map and the second-round unit state.
[0154] In S302, for the case where H is greater than 1 and M is equal to 1, that is, one feature processing includes three rounds of reasoning and feature fusion, the i-th historical hidden layer feature map and the i-th historical unit state are the hidden layer feature map and unit state output by the third round of reasoning and feature fusion in the i-th feature processing when predicting the previous video frame.
[0155] For the second round of reasoning and feature fusion in the i-th feature processing of the current video frame, the electronic device performs reasoning and feature fusion on the first round of historical hidden layer feature map, the i-th historical hidden layer feature map, and the i-th historical unit state obtained in S301 to obtain the second round of hidden layer feature map and the second round of unit state.
[0156] In S302, for the case where M is equal to 1, the previous state feature map H is the i-th historical hidden layer feature map, the previous unit state C is the i-th historical unit state, and the current state feature map X is the first-round historical hidden layer feature map, the electronic device performs inference and feature fusion on the previous state feature map H, the previous unit state C, and the current state feature map X, and the obtained current hidden layer feature map is the second-round hidden layer feature map, and the obtained current unit state is the second-round unit state.
[0157] Here, the process of the electronic device performing the second round of reasoning and feature fusion is similar to the process described in S201-S203, and will not be repeated here.
[0158] S303: Perform a third round of reasoning and feature fusion on the second round hidden layer feature map, the second round unit state, and the first round hidden layer feature map to obtain a third round hidden layer feature map.
[0159] In S303, when H is greater than 1 and M is equal to 1, for the third round of reasoning and feature fusion in the i-th feature processing, the electronic device performs a third round of reasoning and feature fusion on the second round hidden layer feature map, the second round unit state, and the first round hidden layer feature map to obtain the third round hidden layer feature map.
[0160] In S303, when the previous state feature map H is the second-round hidden layer feature map, the previous unit state C is the second-round unit state, and the current state feature map X is the first-round hidden layer feature map, the electronic device performs inference and feature fusion on the previous state feature map H, the previous unit state C, and the current state feature map X, and the obtained current hidden layer feature map is the third-round hidden layer feature map, and the obtained current unit state is the third-round unit state.
[0161] Here, the process of the electronic device performing the third round of reasoning and feature fusion is similar to the process described in S201-S203, and will not be repeated here.
[0162] S304: Use the third round of hidden layer feature map as the i-th hidden layer feature map.
[0163] In some embodiments, the electronic device can use the neural network structure shown in Figure 12 to complete each feature processing through three rounds of reasoning and feature fusion. Figure 12 shows a schematic diagram of the process of implementing each feature processing through a new stacking method of network units provided by an embodiment of the present application. t In the network part for video prediction, each layer of network units from bottom to top includes three inference fusion network units. In some embodiments, the inference fusion network unit can be a new AR-LSTM network unit as shown in Figure 9.
[0164] As shown in Figure 12, for each layer with three LSTM network units stacked, that is, each feature processing includes three rounds of reasoning and feature fusion, take the first feature processing, that is, i=1 as an example. The electronic device can process the first round of historical hidden layer feature map The i-th historical hidden layer feature map And the state of the i-th historical unit Perform inference and feature fusion to obtain the second round of hidden layer feature maps and the second round of unit states; the second round of hidden layer feature maps, the second round of unit states, and the first round of hidden layer feature maps Perform the third round of reasoning and feature fusion to obtain the third round of hidden layer feature map As the i-th hidden layer feature map. Here, the first round unit state Used to process video frame X in the next time step t+1 The first round of inference and feature fusion in the first feature processing during prediction.
[0165] It's understandable that during each feature processing, the hidden layer state generated by the first round of inference and fusion is refined by a combination of two rounds of inference and fusion before the first round of hidden layer feature maps enter the next round of feature processing. The refined hidden layer state is then passed to the next layer. This further decomposes the hidden layer state in the time dimension, enabling the refinement of the hidden layer state within a single time step, improving the accuracy of video prediction and the quality of the predicted video frames.
[0166] In some embodiments, based on FIG. 11 , after S301 , S305 - S310 may be performed as shown in FIG. 13 , as follows:
[0167] S305: If M is greater than 1, obtain the third round of historical hidden layer feature maps and the third round of historical unit states.
[0168] S306 , performing two rounds of reasoning and feature fusion based on the first round hidden layer feature map, the third round historical hidden layer feature map, and the third round historical unit state to obtain the second round hidden layer feature map and the third round hidden layer feature map.
[0169] In S305 , the third round of historical hidden layer feature maps and the third round of historical unit states are the hidden layer feature maps and unit states outputted by the third round of reasoning and feature fusion in the i-th feature processing when predicting the previous video frame.
[0170] In S306, the electronic device may perform inference and feature fusion on the first-round historical hidden layer feature map, the third-round historical hidden layer feature map, and the third-round historical unit state to obtain a second-round hidden layer feature map and a second-round unit state. Furthermore, a third round of inference and feature fusion may be performed on the second-round hidden layer feature map, the second-round unit state, and the first-round hidden layer feature map to obtain a third-round hidden layer feature map.
[0171] S307: Obtain the j-2th round hidden layer feature map, the j+1th round historical hidden layer feature map during the i-th historical feature processing, and the j+1th round historical unit state.
[0172] In S307, when M is greater than 1, that is, more than three rounds of reasoning and feature fusion are performed, based on the third round of reasoning and feature fusion completed in S306, the electronic device can continue with the fourth and fifth rounds of reasoning and feature fusion until 2M rounds of reasoning and feature fusion are completed.
[0173] In S307, j represents the number of rounds of inference and feature fusion in the i-th feature processing, and j is an even number greater than 3 and less than 2M+1 or H. For the j-th round of inference and feature fusion, the electronic device obtains the j-2-th round hidden layer feature map, the j+1-th round historical hidden layer feature map during the i-th historical feature processing, and the j+1-th round historical unit state.
[0174] S308. Perform the j-th round of reasoning and feature fusion on the j-2th round hidden layer feature map, the j+1th round historical hidden layer feature map, and the j+1th round historical unit state to obtain the j-th round hidden layer feature map and the j-th round unit state.
[0175] In S308, when the previous state feature map H is the j+1th round historical hidden layer feature map, the previous unit state C is the j+1th round historical unit state, and the current state feature map X is the j-2th round hidden layer feature map, the electronic device performs inference and feature fusion based on the previous state feature map H, the previous unit state C, and the current state feature map X, and the obtained current hidden layer feature map is the jth round hidden layer feature map, and the obtained current unit state is the jth round unit state.
[0176] In S308, for the j-th round of reasoning and feature fusion, the electronic device performs the j-th round of reasoning and feature fusion on the j-2-th round hidden layer feature map, the j+1-th round historical hidden layer feature map, and the j+1-th round historical unit state, obtains the j-th round hidden layer feature map and the j-th round unit state, completes the j-th round of reasoning and feature fusion, and continues the j+1-th round of reasoning and feature fusion in every two rounds of reasoning and feature fusion.
[0177] S309: Obtain the j-1th round hidden layer feature map.
[0178] S310, perform the j+1th round of reasoning and feature fusion on the jth round hidden layer feature map, the jth round unit state, and the j-1th round hidden layer feature map to obtain the j+1th round hidden layer feature map, until 2M rounds of reasoning and feature fusion are completed, and use the obtained Hth round hidden layer feature map as the i-th hidden layer feature map.
[0179] In S309 and S310, when the previous state feature map H is the j-th round hidden layer feature map, the previous unit state C is the j-th round unit state, and the current state feature map X is the j-1-th round hidden layer feature map, the electronic device performs inference and feature fusion based on the previous state feature map H, the previous unit state C, and the current state feature map X, and the obtained current hidden layer feature map is the j+1-th round hidden layer feature map; the obtained current unit state is the j+1-th round unit state.
[0180] In S310, for the j+1th round of reasoning and feature fusion, the electronic device obtains the j-1th round hidden layer feature map, performs the j+1th round of reasoning and feature fusion on the jth round hidden layer feature map, the jth round unit state, and the j-1th round hidden layer feature map, and obtains the j+1th round hidden layer feature map, until 2M rounds of reasoning and feature fusion are completed, and the obtained Hth round hidden layer feature map is used as the i-th hidden layer feature map.
[0181] For example, the feature processing process when M is greater than 1 can be shown in Figure 14. Figure 14 shows a schematic diagram of the feature processing process when M is greater than 1 using 2M+1(H) stacked adaptive kernel recursion LSTM (AR-LSTM) network units. In Figure 14, the internal structure of each AR-LSTM network unit can be shown in Figure 9.
[0182] As shown in Figure 14, for the current video frame X t For the i-th feature processing, taking i equal to 1 as an example, the electronic device can obtain the first round of hidden layer state feature map through the network unit AR-LSTM-1; the electronic device obtains the second round of hidden layer state feature map and the third round of hidden layer state feature map through the network units AR-LSTM-2 and AR-LSTM-3 according to the above S304-S306. Taking j=4 as an example, for the fourth round of reasoning and feature fusion, the electronic device obtains the j-2th round (second round) hidden layer feature map through the network unit AR-LSTM-4 The j+1th round (fifth historical round) of historical hidden layer feature map during the i-th historical feature processing And the historical unit status of round j+1 (fifth historical round) Electronic device’s hidden layer feature map for round j-2 The j+1th round of historical hidden layer feature map And the j+1th round of historical unit status Perform the jth round of reasoning and feature fusion to obtain the jth round hidden layer feature map and the j-th round unit state Complete the jth round (fourth round) of reasoning and feature fusion.
[0183] As shown in Figure 14, for the j+1th round (fifth round) of reasoning and feature fusion, the electronic device obtains the j-1th round (third round) hidden layer feature map For the jth round (fourth round) hidden layer feature map Unit status in round j (fourth round) And the j-1th round hidden layer feature map Perform the j+1th round of reasoning and feature fusion to obtain the j+1th round hidden layer feature map (fifth round), until 2M rounds of reasoning and feature fusion are completed, and the obtained Hth round (i.e. 2M+1 round) hidden layer feature map is obtained. As the i-th hidden layer feature map.
[0184] It is understandable that in the embodiment of the present application, the electronic device can perform multiple hidden layer state refinements in one feature processing to further improve the accuracy of video frame prediction.
[0185] It should be noted that in the embodiment of the present application, traditional LSTM units can also be used to stack network units in the stacking manner as shown in Figure 12 to obtain a new type of stacked neural network, and then use the stacking method provided in the embodiment of the present application to use multiple network units in one feature processing to refine the hidden layer state multiple times, thereby improving the accuracy of the hidden layer state and thus improving the accuracy of video frame prediction based on the hidden layer state.
[0186] 15 and 16 , the following describes a scenario in which the prediction method according to an embodiment of the present application is applied to actual video encoding and decoding.
[0187] Refer to Figure 15, which shows an example of a block diagram of a video coding system provided by an embodiment of the present application; as shown in Figure 15, the video coding system 10 includes a transform and quantization unit 101, an intra-frame estimation unit 102, an intra-frame prediction unit 103, a motion compensation unit 104, a motion estimation unit 105, an inverse transform and inverse quantization unit 106, a filter control analysis unit 107, a filtering unit 108, an encoding unit 109 and a decoded image cache unit 110, etc., wherein the filtering unit 108 can implement deblocking filtering and sample adaptive offset (SAO) filtering, and the encoding unit 109 can implement header information encoding and context-based adaptive binary arithmetic coding (CABAC).For the input original video signal, a video coding block can be obtained by dividing it into coding tree units (CTUs). Then, the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transform and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to perform intra-frame prediction on the video coding block; specifically, the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to determine the intra-frame prediction mode to be used to encode the video coding block; the motion compensation unit 104 and the motion estimation unit 105 are used to perform inter-frame prediction coding on the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information; the motion estimation performed by the motion estimation unit 105 is the process of generating a motion vector, which can estimate the motion of the video coding block, and the motion compensation unit 104 then calculates the motion vector based on the motion vector determined by the motion estimation unit 105. After determining the intra-frame prediction mode, the intra-frame prediction unit 103 is further configured to provide the selected intra-frame prediction data to the encoding unit 109, and the motion estimation unit 105 also sends the calculated motion vector data to the encoding unit 109. In addition, the inverse transform and inverse quantization unit 106 is configured to reconstruct the video coding block and reconstruct a residual block in the pixel domain. The reconstructed residual block is subjected to the filter control analysis unit 107 and the filtering unit 108 to remove the block effect artifacts. The reconstructed residual block is then added to a predictive block in the frame of the decoded image buffer unit 110 to generate a reconstructed video coding block. The encoding unit 109 is configured to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-frame prediction mode, and output the code stream of the video signal. The decoded image buffer unit 110 is configured to store the reconstructed video coding block for prediction reference. As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image buffer unit 110 .
[0188] The prediction method provided in the embodiment of the present application can be applied to the inter-frame prediction encoding process of the motion compensation unit 104 and the motion estimation unit 105 in Figure 15.
[0189] Refer to Figure 16, which shows an example of a block diagram of the composition of a video decoding system provided in an embodiment of the present application; as shown in Figure 16, the video decoding system 20 includes a decoding unit 201, an inverse transform and inverse quantization unit 202, an intra-frame prediction unit 203, a motion compensation unit 204, a filtering unit 205 and a decoded image cache unit 206, etc., wherein the decoding unit 201 can implement header information decoding and CABAC decoding, and the filtering unit 205 can implement deblocking filtering and SAO filtering. After the input video signal is encoded as shown in FIG2A, a code stream of the video signal is output; the code stream is input to the video decoding system 20 and first passes through the decoding unit 201 to obtain decoded transform coefficients; the transform coefficients are processed by the inverse transform and inverse quantization unit 202 to generate residual blocks in the pixel domain; the intra-frame prediction unit 203 can be used to generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and data from the previously decoded blocks of the current frame or picture; the motion compensation unit 204 determines the prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses The prediction information is used to generate a predictive block for the video decoding block being decoded; a decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 202 with the corresponding predictive block generated by the intra-frame prediction unit 203 or the motion compensation unit 204; the decoded video signal passes through the filtering unit 205 to remove blocking artifacts, thereby improving video quality; the decoded video block is then stored in the decoded image buffer unit 206, which stores reference images used for subsequent intra-frame prediction or motion compensation, and is also used for outputting the video signal, thereby obtaining the restored original video signal.
[0190] The prediction method provided in the embodiment of the present application can be applied to the inter-frame prediction decoding process of the motion compensation unit 204 in Figure 16.
[0191] It should be noted that, in the process of inter-frame prediction, when predicting any video frame, the prediction method provided by this application can be used to implement the prediction. The electronic device in the above embodiment also corresponds to an encoder and a decoder.
[0192] In some embodiments of the present application, for example, as shown in FIG17 , in order to realize the video prediction task of predicting the next r frames based on the known first n frames, the network model provided in the embodiment of the present application corresponds to n+r-1 time steps. During the processing of each time step, the network model processes the input frame by stacking a multi-layer network. Each layer of the multi-layer network contains three new long short-term memory network units, namely AR-LSTM, and the three AR-LSTM network units are stacked according to the new cyclic neural network unit stacking structure (Recurrent in Recurrent) as shown in FIG17 . The hidden layer dimensions of the AR-LSTM network units of each layer from bottom to top can be set separately according to the computing power of the neural network model and the preset accuracy requirements.
[0193] As shown in Figure 17, at a given time step t, the input frame is the current video frame X t The first new long short-term memory network unit (AR-LSTM-1) in layer 1 will be input first. The previous hidden layer state of AR-LSTM-1, i.e. the first round of historical hidden layer feature map, is The cell state, i.e. the first round of historical cell state, is The updated hidden layer state of AR-LSTM-1, i.e. the first round of hidden layer feature map, is The cell state, that is, the first round of cell state, is The electronic device combines the second round of inference and feature fusion in the first feature processing at the previous time step t-1 to obtain the historical second round hidden layer feature map Second round unit status with history The previous hidden layer and cell states of the second new long short-term memory network unit (AR-LSTM-2) are input into AR-LSTM-2 to obtain the updated hidden layer state and cell state of AR-LSTM-2 (not shown). It should be noted that before the start of the first time step, the hidden layer state and cell state of all AR-LSTM network units are reset to zero.
[0194] The electronic device inputs the updated hidden layer state and cell state of AR-LSTM-2 as the previous state of the third new long short-term memory network unit (AR-LSTM-3) into AR-LSTM-3, and obtains the updated hidden layer state and cell state of AR-LSTM-3. and This is the current feature processing process, that is, the hidden layer state after the three AR-LSTM units in the current layer are refined. Electronic devices will This is used as the input for the second layer, undergoing a second round of feature processing. The internal structure of each AR-LSTM unit in Figure 17 can be seen as in Figure 9. Each AR-LSTM unit performs inference and feature fusion on the current video frame or the refined hidden layer state of the previous layer, as well as the hidden layer state and unit state of the previous time step, outputting an updated hidden layer feature map. The next round of AR-LSTM unit processing is then performed based on this updated hidden layer feature map.
[0195] As shown in Figure 17, for the second layer and each layer above, the input of each layer is the hidden layer state after the final refinement of the previous layer. The transmission and update method of the hidden layer and cell state of each layer, that is, the feature processing method is the same as the first layer. The electronic device processes the current video frame X through each layer of the network. t Perform feature processing until it is completed, and convolve the hidden layer state of the last feature processing to obtain the current video frame X t The corresponding current prediction frame Y t+1 .
[0196] It can be understood that in each feature processing unit of the embodiment of the present application, such as the AR-LSTM unit, the generation of each hidden layer state is based on the deformable convolution of the hidden layer state of the past time step, so that the generation of each new hidden layer state is more dependent on the hidden layer state of the previous time step, so that the correlation between video frames can be better expressed in the video prediction task. In addition, the embodiment of the present application further splits the time dimension through a new stacking method, thereby realizing the further refinement of the hidden layer state within a single time step, and obtaining a more accurate hidden layer state expression for video frame prediction. Therefore, the video prediction method provided by the embodiment of the present application has higher prediction accuracy for both single-frame and multi-frame video inter-frame prediction than the video prediction method of the convolutional long short-term memory network, and can effectively improve quality problems such as blurring and ghosting caused by video frame edges or occlusions.
[0197] In some embodiments, a comparison of the prediction effects of the embodiments of the present application with the current predictive recurrent neural network (PredRNN) and the current convolutional LSTM (ConvLSTM) network model can be shown in Figures 18 and 19. Figures 18 and 19 are schematic diagrams of the prediction effects of using the Moving MNIST image set to continuously predict 10 frames using the AR-LSTM, PredRNN, and ConvLSTM network models, respectively, using the prediction method of the embodiments of the present application.
[0198] As shown in FIG18 , when the above network model predicts 10 input video frames from time t=1 to time t=10, its annotated prediction frames can be shown as the video frames from time t=11 to time t=20 in FIG18 . It can be seen that, first of all, the AR-LSTM network model provided in the embodiment of the present application is more accurate in predicting the position of the digits. Taking the prediction of the input video frame at time t=9 in FIG18 as an example, the input video frame at time t=9 corresponds to the annotated prediction frame at time t=19. As shown in FIG18 , in the annotated prediction frame at time t=19, the number 1 should overlap at the right edge of the number 8. In the video frame predicted by the AR-LSTM network model provided in the embodiment of the present application, the thickness and relative position of the numbers 1 and 8 are closest to the annotated prediction frame at time t=19. In the video frame predicted by the PredRNN, the number 1 overlaps in the middle of the number 8, so the position relationship is inaccurate; and in the video frame predicted by the ConvLSTM, the number 1 is obviously too thin, so the thickness is inaccurate, that is, the edge prediction is inaccurate.
[0199] Furthermore, taking the above-mentioned network models as an example, the input video frame at time t = 10 in Figure 18 is predicted by each of the above-mentioned network models. The input video frame at time t = 10 corresponds to the annotated prediction frame at time t = 20. In the prediction frame at time t = 20 of the AR-LSTM network model provided in the embodiment of the present application, the edge of the number 8 is very clear and the shape of the number is more accurate; however, in the prediction frame at time t = 20 of the PredRNN and ConvLSTM, the edge of the number 8 is relatively blurred and deformed.
[0200] In some embodiments, as shown in FIG19 , for the 10 input video frames at times t=1 to t=10, the annotated prediction frames may be as shown in the video frames from t=11 to t=20 in FIG19 . FIG19 shows a comparison of the video frame prediction effects of the AR-LSTM network model of the embodiment of the present application and the PredRNN and ConvLSTM models when the image overlaps or occlusions in the input video frames are high. In FIG19 , for the prediction frames obtained by the above three network models for the input video frames from t=4 to t=7, it can be clearly seen that the AR-LSTM network model of the embodiment of the present application can still predict relatively clear digital edge contours and digital shapes when there is overlap or occlusion in the image.
[0201] In summary, the prediction method in the embodiment of the present application is significantly superior to the PredRNN and ConvLSTM network models in terms of the visual effect performance of image prediction and the image quality of the predicted frame.
[0202] In some embodiments, the applicant compares the AR-LSTM network model, PredRNN and ConvLSTM using the prediction method in the embodiment of the present application based on several indicators such as mean square error (MSE), peak signal to noise ratio (PSNR), structural similarity (SSIM) and perceptual similarity (LPIPS), as shown in Figures 20a, 20b, 20c and 20d. Among them, the MSE indicator and the LPIPS indicator are reverse evaluation indicators, that is, the smaller the MSE data and the LPIPS data, the better the performance of the model. The PSNR indicator and the SSIM indicator are positive evaluation indicators, that is, the larger the PSNR data and the SSIM data, the better the performance of the model.
[0203] See Figure 20a, which shows a comparison of the mean squared error (MSE) metrics for video frame prediction using the AR-LSTM network model, PredRNN, and ConvLSTM, as provided in this embodiment. As shown in Figure 20a, the mean squared error (MSE) of the AR-LSTM network model provided in this embodiment is lower than that of the PredRNN and ConvLSTM network models at each prediction time point.
[0204] See Figure 20b, which shows a comparison of peak signal-to-noise ratio (PSNR) metrics for video frame prediction using the AR-LSTM network model, PredRNN, and ConvLSTM, as provided in this embodiment. As shown in Figure 20b, the AR-LSTM network model provided in this embodiment has higher PSNRs than the PredRNN and ConvLSTM network models at all prediction time points.
[0205] See Figure 20c, which shows a comparison of structural similarity index data for the AR-LSTM network model, PredRNN, and ConvLSTM video frame predictions provided by the embodiment of this application. As shown in Figure 20c, the structural similarity values of the AR-LSTM network model provided by the embodiment of this application are generally higher than those of the PredRNN and ConvLSTM network models.
[0206] See Figure 20d, which shows a comparison of perceptual similarity metrics for video frame prediction using the AR-LSTM network model, PredRNN, and ConvLSTM, as provided in this embodiment. As shown in Figure 20d, the AR-LSTM network model provided in this embodiment achieves lower perceptual similarity values at each prediction time point than the PredRNN and ConvLSTM network models.
[0207] Below, combined with Table 1, the data in the above-mentioned mean square error, peak signal-to-noise ratio, structural similarity and perceptual similarity indicators are further compared.
[0208] Table 1
[0209] Model MSE (↓) PSNR (↑) SSIM (↑) LPIPS (↓) ConvLSTM 65.318.520.8330.141 PredRNN 57.819.220.8670.104 AR-LSTM 50.219.840.8770.084
[0210] Combined with Table 1, it can be seen that the AR-LSTM network model using the prediction method in the embodiment of the present application has an MSE data and LPIPS data of 50.2 and 0.084, respectively, which are significantly lower than those of the traditional PredRNN and ConvLSTM networks. At the same time, the SSIM data and LPIPS data are 0.877 and 19.84, respectively, which are significantly higher than those of the traditional PredRNN and ConvLSTM networks. Therefore, the AR-LSTM network model provided in the embodiment of the present application also outperforms the traditional neural network prediction model in terms of the data performance of the above-mentioned indicators.
[0211] The present application also provides a prediction device for use in electronic devices. FIG21 is a schematic diagram of the structure of the prediction device provided in an embodiment of the present application; as shown in FIG21 , the prediction device 1 includes:
[0212] The acquisition part 11 is configured to acquire, when predicting a current video frame among n video frames, the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map of the current video frame when predicting the previous video frame; the i-1-th hidden layer feature map is obtained by performing feature processing i-1 times on the current video frame; wherein i is a positive integer greater than or equal to 2;
[0213] The reasoning and fusion part 12 is configured to perform H rounds of reasoning and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map; H is an integer greater than or equal to 1;
[0214] The prediction part 13 is configured to continue the i+1th feature processing based on the i-th hidden layer feature map until the K-th feature processing is completed, and obtain the current predicted video frame corresponding to the current video frame; K is the total number of feature processing times and is an integer greater than or equal to 2.
[0215] In some embodiments, the inference and fusion part 12 is also used to infer the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain a first round of initial hidden layer feature map; the first round of initial hidden layer feature map is fused with the i-th historical hidden layer feature map to obtain a first round of hidden layer feature map; if H is greater than 1, then based on the first round of hidden layer feature map, the i-th historical hidden layer feature map and the i-th historical unit state, the next two rounds of reasoning and feature fusion are continued until 2M rounds of reasoning and feature fusion are completed, and the H-th round hidden layer feature map is used as the i-th hidden layer feature map; wherein M is an integer greater than or equal to 1, and 2M is equal to H-1.
[0216] In some embodiments, the inference fusion part 12 is further configured to fuse the first round of initial hidden layer feature map with the i-th historical hidden layer feature map to obtain the first round of hidden layer feature map. If H is 1, the first round of hidden layer feature map is used as the i-th hidden layer feature map.
[0217] In some embodiments, the reasoning and fusion part 12 is further configured to obtain the first round of historical hidden layer feature maps when predicting the previous video frame; the first round of historical hidden layer feature maps are obtained by performing the first round of reasoning and feature fusion in the i-th historical feature processing of the previous video frame; if M is equal to 1, the first round of historical hidden layer feature maps, the i-th historical hidden layer feature maps, and the i-th historical unit states are subjected to reasoning and feature fusion to obtain the second round of hidden layer feature maps and the second round of unit states; the second round of hidden layer feature maps, the second round of unit states, and the first round of hidden layer feature maps are subjected to the third round of reasoning and feature fusion to obtain the third round of hidden layer feature maps; and the third round of hidden layer feature maps are used as the i-th hidden layer feature maps.
[0218] In some embodiments, the reasoning and fusion part 12 is further configured to obtain the first round of historical hidden layer feature maps when predicting the previous video frame, if M is greater than 1, obtain the third round of historical hidden layer feature maps and the third round of historical unit states; perform two rounds of reasoning and feature fusion based on the first round of hidden layer feature maps, the third round of historical hidden layer feature maps and the third round of historical unit states to obtain the second round of hidden layer feature maps and the third round of hidden layer feature maps; obtain the j-2th round of hidden layer feature maps, the j+1th round of historical hidden layer feature maps during the i-th historical feature processing, and the j+1th round of historical unit states; j is greater than 3 and less than 2 An even number of M+1 or H; performing the j-2th round hidden layer feature map, the j+1th round historical hidden layer feature map, and the j+1th round historical unit state to obtain the j-th round hidden layer feature map and the j-th round unit state; obtaining the j-1th round hidden layer feature map; performing the j+1th round reasoning and feature fusion on the j-2th round hidden layer feature map, the j+1th round unit state, and the j-1th round hidden layer feature map to obtain the j+1th round hidden layer feature map, until 2M rounds of reasoning and feature fusion are completed, and the obtained H-th round hidden layer feature map is used as the i-th hidden layer feature map.
[0219] In some embodiments, the inference and fusion part 12 is further configured to perform inference and feature fusion on the first round of historical hidden layer feature maps, the third round of historical hidden layer feature maps, and the third round of historical unit states to obtain the second round of hidden layer feature maps and the second round of unit states; and perform a third round of inference and feature fusion on the second round of hidden layer feature maps, the second round of unit states, and the first round of hidden layer feature maps to obtain the third round of hidden layer feature maps.
[0220] In some embodiments, the inference fusion part 12 is further configured to perform inference and feature fusion on the previous state feature map H, the previous unit state C, and the current state feature map X to obtain the current hidden layer feature map and the current unit state.
[0221] In some embodiments, the inference fusion part 12 is further configured to perform inference based on the previous state feature map H, the previous unit state C, and the current state feature map X to obtain the initial hidden layer feature map; and fuse the initial hidden layer feature map with the previous state feature map to obtain the current hidden layer feature map and the current unit state.
[0222] In some embodiments, M is equal to 1, the previous state feature map H is the i-th historical hidden layer feature map, the previous unit state C is the i-th historical unit state, and the current state feature map X is the first-round historical hidden layer feature map, the current hidden layer feature map is the second-round hidden layer feature map; and the current unit state is the second-round unit state;
[0223] When M is greater than 1, the previous state feature map H is the third round of historical hidden layer feature map, the previous unit state C is the third round of historical unit state, and the current state feature map X is the first round of hidden layer feature map, the current hidden layer feature map is the second round of hidden layer feature map; and the current unit state is the second round of unit state;
[0224] When the previous state feature map H is the second round hidden layer feature map, the previous unit state C is the second round unit state, and the current state feature map X is the first round hidden layer feature map, the current hidden layer feature map is the third round hidden layer feature map; and the current unit state is the third round unit state;
[0225] When the previous state feature map H is the j+1th round historical hidden layer feature map, the previous unit state C is the j+1th round historical unit state, and the current state feature map X is the j-2th round hidden layer feature map, the current hidden layer feature map is the jth round hidden layer feature map; the current unit state is the jth round unit state;
[0226] When the previous state feature map H is the j-th round hidden layer feature map, the previous unit state C is the j-th round unit state, and the current state feature map X is the j-1-th round hidden layer feature map, the current hidden layer feature map is the j+1-th round hidden layer feature map; and the current unit state is the j+1-th round unit state.
[0227] In some embodiments, the inference fusion part 12 is further configured to perform convolution and activation processing on the first round of initial hidden layer feature maps to obtain feature offset, feature mask and residual compensation; perform deformable convolution processing on the i-th historical hidden layer feature map according to the feature offset and the feature mask to obtain a deformable convolution feature map; and fuse the deformable convolution feature map with the residual compensation to obtain the first round of hidden layer feature maps.
[0228] In some embodiments, the inference fusion part 12 is further configured to perform convolution processing on the first round of initial hidden layer feature maps in each coordinate axis direction to obtain the feature offset; perform convolution and activation processing on the first round of initial hidden layer feature maps to obtain the feature mask; and perform convolution processing on the first round of initial hidden layer feature maps to obtain the residual compensation.
[0229] In some embodiments, the inference fusion part 12 is further configured to group the channels in the i-th historical hidden layer feature map to obtain at least one group of channel groupings; determine the sub-feature offset and sub-feature mask corresponding to each group of channel groupings of the at least one group of channel groupings according to the dimension of the feature offset and the feature mask; perform feature offset on each historical feature point in each group of channel groupings through the sub-feature offset, and perform weighted convolution on the offset features based on the sub-feature mask to obtain the deformable convolution feature map.
[0230] In some embodiments, the inference fusion part 12 is further configured to determine the target historical feature point set corresponding to the i-th historical hidden layer feature map for each target feature point in the deformable convolution feature map according to a preset receptive field size; for each target feature point, the target historical feature point set is offset according to the sub-feature offset corresponding to each channel to obtain a corrected feature point set corresponding to each target feature point; based on the sub-feature mask, the corrected feature point set is weighted convolved to obtain the target feature value corresponding to each target feature point; and the deformable convolution feature map is obtained by combining the target feature value corresponding to each target feature point.
[0231] In some embodiments, the reasoning fusion part 12 is also configured to determine, for each target historical feature point in the target historical feature point set, the offset corresponding to each target historical feature point in the sub-feature offset; perform bilinear interpolation processing on each target feature point and each target historical feature point and the corresponding offset to obtain the corrected feature value corresponding to each target historical feature point in the target historical feature point set, as the corrected feature point set corresponding to each target feature point.
[0232] In some embodiments, the inference and fusion part 12 is further configured to obtain the first historical hidden layer feature map and the first historical unit state when predicting the previous video frame before obtaining the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map when predicting the previous video frame; perform H rounds of inference and feature fusion on the current video frame, the first historical hidden layer feature map, and the first historical unit state to obtain the first hidden layer feature map of the current video frame.
[0233] In some embodiments, the inference fusion part 12 is further configured to use a preset first weight to perform weighted and convolution processing on the i-th historical hidden layer feature map to obtain a first intermediate tensor, and use a preset second weight to perform weighted and convolution processing on the i-1th hidden layer feature map to obtain a second intermediate tensor; splice the first intermediate tensor and the second intermediate tensor to obtain a first tensor, a second tensor, a third tensor and a fourth tensor; use a preset gating factor, combined with the i-th historical unit state, to process the first tensor, the second tensor, the third tensor and the fourth tensor to obtain the first round of initial hidden layer feature map.
[0234] In some embodiments, the prediction part 12 is further configured to continue the i+1th feature processing based on the i-th hidden layer feature map until K times of feature processing are completed, and after obtaining the current predicted video frame corresponding to the current video frame, when the current video frame is not the last frame of the n video frames, continue to predict the next video frame in the n video frames to obtain the next predicted video frame, until the prediction of the n-th video frame is completed, and the n-th predicted video frame is obtained; use the n-th predicted video frame to predict the n+1-th video frame to obtain the n+1-th predicted video frame, until the n+r-1-th predicted video frame is predicted based on the n+r-1-th predicted video frame; r is an integer greater than or equal to 1.
[0235] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.
[0236] In some embodiments, the present application also provides an electronic device. FIG22 is a schematic diagram of an optional structure of an electronic device provided in an embodiment of the present application. As shown in FIG22 , electronic device 2 includes a memory 22 and a processor 23. The memory 22 and processor 23 are connected via a communication bus 24. The memory 22 is used to store executable instructions. The processor 23 is used to execute the executable instructions stored in the memory 22 to implement the methods provided in the embodiments of the present application, for example, the prediction method provided in the embodiments of the present application.
[0237] An embodiment of the present application provides a computer-readable storage medium storing executable prediction instructions for causing the processor 23 to execute and implement the method provided by the embodiment of the present application, for example, the prediction method provided by the embodiment of the present application.
[0238] In some embodiments of the present application, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
[0239] In some embodiments of the present application, executable prediction instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0240] As an example, the executable prediction instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (for example, files storing one or more modules, subroutines, or code portions).
[0241] As an example, executable prediction instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.
[0242] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.
[0243] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.
[0244] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0245] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0246] The above description is merely a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Industrial Applicability
[0247] The embodiments of the present application provide a prediction method, device, electronic device, and computer-readable storage medium. In the process of performing the i-th feature processing on the current video frame in the video prediction task, reasoning is performed through the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map, and the reasoning result is fused with the i-th historical hidden layer feature map to obtain the i-th hidden layer feature map. This allows the hidden layer feature map obtained by the current hidden layer update to be more dependent on the adjacent hidden layer state of the previous time step, so that the inter-frame correlation is better expressed in the hidden layer feature map, thereby improving the accuracy of feature processing, and further improving the accuracy of video frame prediction based on the hidden layer feature map obtained by feature processing, as well as the quality of the predicted frame. In addition, in each feature processing, the hidden layer state generated by the first round of reasoning and fusion will be refined by a combination of two rounds of reasoning and fusion before the first round of hidden layer feature map enters the next feature processing. The refined hidden layer state will be passed to the upper layer again. This further splits the time dimension and refines the hidden layer state within a single time step, further improving the accuracy of video prediction and the quality of the predicted video frames.
Claims
1. A prediction method comprising: When predicting a current video frame among n video frames, obtaining the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map of the current video frame when predicting the previous video frame; the i-1-th hidden layer feature map is obtained by performing feature processing i-1 times on the current video frame; wherein i is an integer greater than or equal to 2; and n is an integer greater than or equal to 1; Perform H rounds of inference and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map; wherein the inference result obtained in each round of inference is fused based on the i-th historical hidden layer feature map; H is an integer greater than or equal to 1; Based on the i-th hidden layer feature map, continue the i+1-th feature processing until K-th feature processing is completed, and obtain the current predicted video frame corresponding to the current video frame; K is the total number of feature processing times and is an integer greater than or equal to 2.
2. The method according to claim 1, wherein The performing H rounds of reasoning and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map includes: Inferring the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain a first-round initial hidden layer feature map; Fusing the first-round initial hidden layer feature map with the i-th historical hidden layer feature map to obtain a first-round hidden layer feature map; If H is greater than 1, then based on the first round of hidden layer feature map, the i-th historical hidden layer feature map and the i-th historical unit state, continue with the next two rounds of reasoning and feature fusion until 2M rounds of reasoning and feature fusion are completed, and the H-th round hidden layer feature map is used as the i-th hidden layer feature map; where M is an integer greater than or equal to 1, and 2M is equal to H-1.
3. The method according to claim 2, wherein: After fusing the first-round initial hidden layer feature map with the i-th historical hidden layer feature map to obtain the first-round hidden layer feature map, the method further includes: If H is 1, the first round of hidden layer feature map is used as the i-th hidden layer feature map.
4. The method according to claim 2, wherein: The method of continuing the next two rounds of reasoning and feature fusion based on the first round hidden layer feature map, the i-th historical hidden layer feature map, and the i-th historical unit state until 2M rounds of reasoning and feature fusion are completed, and using the H-th round hidden layer feature map as the i-th hidden layer feature map, includes: Obtaining a first-round historical hidden layer feature map when predicting the previous video frame; the first-round historical hidden layer feature map is obtained by performing a first-round reasoning and feature fusion in the i-th historical feature processing of the previous video frame; If M is equal to 1, the first round of historical hidden layer feature maps, the i-th historical hidden layer feature maps, and the i-th historical unit state are inferred and feature fused to obtain the second round of hidden layer feature maps and the second round of unit states; Performing a third round of reasoning and feature fusion on the second round hidden layer feature map, the second round unit state, and the first round hidden layer feature map to obtain a third round hidden layer feature map; The third round of hidden layer feature map is used as the i-th hidden layer feature map.
5. The method according to claim 4, wherein After obtaining the first round of historical hidden layer feature maps when predicting the previous video frame, the method further includes: If M is greater than 1, obtain the third round of historical hidden layer feature maps and the third round of historical unit states; Perform two rounds of reasoning and feature fusion based on the first round hidden layer feature map, the third round historical hidden layer feature map, and the third round historical unit state to obtain the second round hidden layer feature map and the third round hidden layer feature map; Obtain the j-2th round hidden layer feature map, the j+1th round historical hidden layer feature map during the i-th historical feature processing, and the j+1th round historical unit state; j is an even number greater than 3 and less than 2M+1 or H; Performing the j-th round of reasoning and feature fusion on the j-2th round hidden layer feature map, the j+1th round historical hidden layer feature map, and the j+1th round historical unit state to obtain the j-th round hidden layer feature map and the j-th round unit state; Get the j-1th round hidden layer feature map; Perform the j+1th round of reasoning and feature fusion on the j-th round hidden layer feature map, the j-th round unit state, and the j-1th round hidden layer feature map to obtain the j+1th round hidden layer feature map, until 2M rounds of reasoning and feature fusion are completed, and the obtained H-th round hidden layer feature map is used as the i-th hidden layer feature map.
6. The method according to claim 5, wherein: The method of performing two rounds of reasoning and feature fusion based on the first round hidden layer feature map, the third round historical hidden layer feature map, and the third round historical unit state to obtain the second round hidden layer feature map and the third round hidden layer feature map includes: Performing inference and feature fusion on the first-round historical hidden layer feature map, the third-round historical hidden layer feature map, and the third-round historical unit state to obtain the second-round hidden layer feature map and the second-round unit state; A third round of reasoning and feature fusion is performed on the second-round hidden layer feature map, the second-round unit state, and the first-round hidden layer feature map to obtain the third-round hidden layer feature map.
7. The method according to any one of claims 1 to 6, wherein: Perform inference and feature fusion on the previous state feature map H, the previous unit state C, and the current state feature map X to obtain the current hidden layer feature map and the current unit state, including: Reasoning based on the previous state feature map H, the previous unit state C, and the current state feature map X to obtain the initial hidden layer feature map; The initial hidden layer feature map is fused with the previous state feature map to obtain the current hidden layer feature map and the current unit state.
8. The method according to claim 7, wherein: When M is equal to 1, the previous state feature map H is the i-th historical hidden layer feature map, the previous unit state C is the i-th historical unit state, and the current state feature map X is the first-round historical hidden layer feature map, the current hidden layer feature map is the second-round hidden layer feature map; and the current unit state is the second-round unit state; When M is greater than 1, the previous state feature map H is the third round of historical hidden layer feature map, the previous unit state C is the third round of historical unit state, and the current state feature map X is the first round of hidden layer feature map, the current hidden layer feature map is the second round of hidden layer feature map; and the current unit state is the second round of unit state; When the previous state feature map H is the second round hidden layer feature map, the previous unit state C is the second round unit state, and the current state feature map X is the first round hidden layer feature map, the current hidden layer feature map is the third round hidden layer feature map; and the current unit state is the third round unit state; When the previous state feature map H is the j+1th round historical hidden layer feature map, the previous unit state C is the j+1th round historical unit state, and the current state feature map X is the j-2th round hidden layer feature map, the current hidden layer feature map is the jth round hidden layer feature map; the current unit state is the jth round unit state; When the previous state feature map H is the j-th round hidden layer feature map, the previous unit state C is the j-th round unit state, and the current state feature map X is the j-1-th round hidden layer feature map, the current hidden layer feature map is the j+1-th round hidden layer feature map; and the current unit state is the j+1-th round unit state.
9. The method according to claim 2, wherein: The fusing the first-round initial hidden layer feature map with the i-th historical hidden layer feature map to obtain the first-round hidden layer feature map includes: Perform convolution and activation processing on the first round of initial hidden layer feature maps to obtain feature offset, feature mask and residual compensation; Performing a deformable convolution process on the i-th historical hidden layer feature map according to the feature offset and the feature mask to obtain a deformable convolution feature map; The deformable convolution feature map is fused with the residual compensation to obtain the first round hidden layer feature map.
10. The method according to claim 9, wherein: The convolution and activation processing is performed on the first round of initial hidden layer feature maps to obtain feature offset, feature mask and residual compensation, including: Performing convolution processing on the first round of initial hidden layer feature maps in each coordinate axis direction to obtain the feature offset; Performing convolution and activation processing on the first round of initial hidden layer feature maps to obtain the feature mask; The first round of initial hidden layer feature maps are convolved to obtain the residual compensation.
11. The method according to claim 9 or 10, wherein: The step of performing deformable convolution processing on the i-th historical hidden layer feature map according to the feature offset and the feature mask to obtain a deformable convolution feature map includes: Grouping the channels in the i-th historical hidden layer feature map to obtain at least one channel grouping; Determining, according to the feature offset and the dimension of the feature mask, a sub-feature offset and a sub-feature mask corresponding to each channel grouping of the at least one channel grouping; Through the sub-feature offset, feature offset is performed on each historical feature point in each channel group, and weighted convolution is performed on the offset features based on the sub-feature mask to obtain the deformable convolution feature map.
12. The method according to claim 11, wherein The method of performing feature shifting on each historical feature point in each channel grouping by the sub-feature shifting and performing weighted convolution on the shifted features based on the sub-feature mask to obtain the deformable convolution feature map includes: Determine, according to a preset receptive field size, a target historical feature point set corresponding to each target feature point in the deformable convolution feature map in the i-th historical hidden layer feature map; For each target feature point, offset the target historical feature point set according to the sub-feature offset corresponding to each channel to obtain a revised feature point set corresponding to each target feature point; Based on the sub-feature mask, weighted convolution is performed on the modified feature point set to obtain a target feature value corresponding to each target feature point; The deformable convolution feature map is obtained by combining the target feature value corresponding to each target feature point.
13. The method according to claim 12, wherein: For each target feature point, offsetting the target historical feature point set according to the sub-feature offset corresponding to each channel to obtain a revised feature point set corresponding to each target feature point includes: For each target historical feature point in the target historical feature point set, determining an offset corresponding to each target historical feature point in the sub-feature offset; Bilinear interpolation is performed on each target feature point, each target historical feature point and the corresponding offset to obtain a corrected feature value corresponding to each target historical feature point in the target historical feature point set, as the corrected feature point set corresponding to each target feature point.
14. The method according to any one of claims 1 to 10, 12 or 13, wherein: Before obtaining the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map of the current video frame when predicting the previous video frame, the method further includes: Get the first historical hidden layer feature map and the first historical unit state when predicting the previous video frame; Perform H rounds of reasoning and feature fusion on the current video frame, the first historical hidden layer feature map, and the first historical unit state to obtain a first hidden layer feature map of the current video frame.
15. The method according to any one of claims 2 to 10, 12 or 13, wherein: The reasoning on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain a first round of initial hidden layer feature map includes: Using a preset first weight, weighting and convolving the i-th historical hidden layer feature map to obtain a first intermediate tensor, and using a preset second weight, weighting and convolving the i-1-th hidden layer feature map to obtain a second intermediate tensor; Concatenate the first intermediate tensor and the second intermediate tensor to obtain a first tensor, a second tensor, a third tensor, and a fourth tensor; The first tensor, the second tensor, the third tensor, and the fourth tensor are processed using a preset gating factor in combination with the i-th historical unit state to obtain the first-round initial hidden layer feature map.
16. The method according to any one of claims 1 to 10, 12 or 13, wherein: The method further comprises: continuing the i+1th feature processing based on the i-th hidden layer feature map until the Kth feature processing is completed, and obtaining a current predicted video frame corresponding to the current video frame. When the current video frame is not the last frame among the n video frames, continue predicting the next video frame among the n video frames to obtain the next predicted video frame, until the prediction of the nth video frame is completed, to obtain the nth predicted video frame; The nth predicted video frame is used to predict the n+1th video frame to obtain the n+1th predicted video frame, until the n+rth predicted video frame is predicted based on the n+r-1th predicted video frame; r is an integer greater than or equal to 1.
17. A video frame prediction device, comprising: The acquisition part is configured to, when predicting a current video frame among n video frames, obtain the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map of the current video frame when predicting the previous video frame; the i-1-th hidden layer feature map is obtained by performing feature processing i-1 times on the current video frame; wherein i is an integer greater than or equal to 2; and n is an integer greater than or equal to 1; The inference fusion part is configured to perform H rounds of inference and feature fusion on the i-th historical hidden layer feature map, the i-th historical unit state, and the i-1-th hidden layer feature map to obtain the i-th hidden layer feature map; wherein the inference result obtained in each round of inference is fused based on the i-th historical hidden layer feature map; H is an integer greater than or equal to 1; The prediction part is configured to continue the i+1th feature processing based on the i-th hidden layer feature map until the K-th feature processing is completed, thereby obtaining the current predicted video frame corresponding to the current video frame; K is the total number of feature processing times and is an integer greater than or equal to 2.
18. An electronic device comprising: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 16 by executing executable instructions stored in the memory.
19. A computer-readable storage medium storing executable instructions for causing a processor to execute the instructions to implement the method according to any one of claims 1 to 16.
