Video representation method, video reconstruction method and device

Abstract

The invention provides a video representation method and device and a video reconstruction method and device, and relates to the technical field of video coding and decoding. Comprising the following steps: creating an information grid corresponding to each video frame of a target video sequence; circularly taking each video frame as the current video frame to execute the following steps until the training stopping condition is met: acquiring a first hidden state according to the information grid corresponding to the current video frame, the first recurrent neural network and the hidden state of the previous video frame of the current video frame; obtaining a reconstructed video frame according to the first hidden state and the reconstruction network; adjusting at least one of parameters of the information grid, the first recurrent neural network and the reconstruction network according to the current video frame, the reconstruction video frame and a preset loss function; and when the training stopping condition is satisfied, generating representation data of a target video sequence according to the information grid corresponding to each video frame, the first recurrent neural network and the reconstruction network. Some embodiments of the invention are used for improving the video representation efficiency.

Description

Technical Field

[0001] Some embodiments of this application relate to the field of video encoding and decoding technology. More specifically, they relate to a video representation method, a video reconstruction method, and an apparatus. Background Technology

[0002] Neural Representations for Videos (NeRV) is a method for parameterizing videos. It learns a feature by passing the temporal index t of a video frame through a Multilayer Perceptron (MLP) network, and then feeds this feature into a reconstruction network consisting of convolutional and upsampling modules to obtain the reconstructed frame. This method fits the video into a neural network model.

[0003] NeRV can represent video as parameters of a neural network, transforming video encoding into fitting the video into a neural network model, thus converting the video compression problem into a model compression problem. However, videos are typically composed of a series of consecutive video frames, with neighboring frames being visually very similar, and video frames may also contain many repetitive patterns or textures. Therefore, videos have a lot of redundant information in both the temporal and spatial dimensions. NeRV only fits video frames and indices together through an MLP network, which cannot effectively utilize the spatiotemporal redundancy of the video. Furthermore, the MLP network has a large number of parameters, so the efficiency of video representation needs to be improved. Summary of the Invention

[0004] Exemplary embodiments of this application provide a video representation method, a video reconstruction method, and an apparatus for improving video representation efficiency.

[0005] The technical solutions provided by some embodiments of this application are as follows:

[0006] Firstly, some embodiments of this application provide a video representation method, including:

[0007] Create an information grid corresponding to each video frame of the target video sequence;

[0008] The following steps are performed iteratively, using each video frame of the target video sequence as the current video frame, until the training stop condition is met:

[0009] Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained.

[0010] Based on the first hidden state of the current video frame and the reconstruction network, obtain the reconstructed video frame corresponding to the current video frame;

[0011] Based on the current video frame, the reconstructed video frame corresponding to the current video frame, and the preset loss function, at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network is adjusted.

[0012] When the training stop condition is met, representation data of the target video sequence is generated based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network.

[0013] Secondly, some embodiments of this application provide a video reconstruction method, including:

[0014] Obtain the representation data of the target video sequence;

[0015] Based on the representation data of the target video sequence, obtain the information grid, the first recurrent neural network, and the reconstruction network corresponding to each video frame of the target video sequence;

[0016] The following steps are performed, taking each video frame of the target video sequence as the current video frame, to obtain the reconstructed video frames corresponding to each video frame of the target video sequence:

[0017] Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained.

[0018] The reconstructed video frame corresponding to the current video frame is obtained based on the first hidden state of the current video frame and the reconstruction network.

[0019] Thirdly, some embodiments of this application provide a video display apparatus, including:

[0020] The module is used to create information grids corresponding to each video frame of the target video sequence;

[0021] The training module is used to iteratively execute the following steps, taking each video frame of the target video sequence as the current video frame, until the training stop condition is met:

[0022] Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained.

[0023] Based on the first hidden state of the current video frame and the reconstruction network, obtain the reconstructed video frame corresponding to the current video frame;

[0024] Based on the current video frame, the reconstructed video frame corresponding to the current video frame, and the preset loss function, at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network is adjusted.

[0025] The generation module is used to generate representation data of the target video sequence based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network when the training stop condition is met.

[0026] Fourthly, some embodiments of this application provide a video reconstruction apparatus, including:

[0027] Acquisition unit, used for representation data of the target video sequence;

[0028] The processing unit is configured to obtain, based on the representation data of the target video sequence, the information grid, the first recurrent neural network, and the reconstruction network corresponding to each video frame of the target video sequence;

[0029] The reconstruction unit is configured to take each video frame of the target video sequence as the current video frame and perform the following steps to obtain the reconstructed video frames corresponding to each video frame of the target video sequence:

[0030] Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained.

[0031] The reconstructed video frame corresponding to the current video frame is obtained based on the first hidden state of the current video frame and the reconstruction network.

[0032] Fifthly, some embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video representation method of the first aspect or the video reconstruction method of the second aspect when executing the computer program.

[0033] Sixthly, some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video representation method of the first aspect or the video reconstruction method of the second aspect.

[0034] In a seventh aspect, some embodiments of this application provide a chip including a processor and a memory, the memory being used to store programs or instructions executable on the processor, and the processor being used to execute the programs or instructions to cause the video representation method of the first aspect or the video reconstruction method of the second aspect to be executed.

[0035] Eighthly, some embodiments of this application provide a computer program product that, when run on a computer, causes the computer to implement the video representation method of the first aspect or the video reconstruction method of the second aspect.

[0036] As can be seen from the above technical solutions, the video representation methods provided in the above embodiments, when representing a target video sequence, first create an information grid corresponding to each video frame of the target video sequence, then iteratively use each video frame of the target video sequence as the current video frame, and obtain the first hidden state of the current video frame based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network. Based on the first hidden state of the current video frame and the reconstruction network, obtain the reconstructed video frame corresponding to the current video frame, and adjust at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network based on the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function. When the training stop condition is met, the representation data of the target video sequence is generated based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network. Since the video representation methods provided in the above embodiments are based on recurrent neural networks for video representation, and recurrent neural networks have memory functions and can retain information through their recurrent structure, thereby capturing long-term dependencies in the video sequence, the above embodiments can effectively utilize the spatiotemporal redundancy information of the video. Compared to NeRV, which cannot effectively utilize the spatiotemporal redundancy information of video and has a large number of parameters in MLP networks, the above-mentioned embodiments can effectively utilize the spatiotemporal redundancy information of video without using MLP networks. Therefore, the above-mentioned embodiments can improve the representation efficiency of video. Attached Figure Description

[0037] To more clearly illustrate the implementation methods in some embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0038] Figure 1The diagram shows a schematic representation of the network model of the video representation and video reconstruction methods provided in some embodiments.

[0039] Figure 2 A schematic diagram of the structure of a first recurrent neural network provided in some embodiments is shown;

[0040] Figure 3 Schematic diagrams of the super-resolution network structure are shown in some embodiments;

[0041] Figure 4 A schematic diagram of the structure of the NeRV block in some embodiments is shown;

[0042] Figure 5 The diagram shows a schematic representation of the network model of the video representation and video reconstruction methods provided in some embodiments.

[0043] Figure 6 The flowcharts of the video representation method in some embodiments are shown;

[0044] Figure 7 The diagram shows a schematic representation of the network model of the video representation method provided in some embodiments;

[0045] Figure 8 The diagram shows a schematic representation of the network model of the video representation method provided in some embodiments;

[0046] Figure 9 The diagram shows a schematic representation of the network model of the video representation method provided in some embodiments;

[0047] Figure 10 The diagram shows a schematic representation of the network model of the video representation method provided in some embodiments;

[0048] Figure 11 The following flowcharts illustrate the steps of the video reconstruction method in some embodiments of this application;

[0049] Figure 12 The present application shows a schematic diagram of the structure of a video display device in some embodiments;

[0050] Figure 13 The following is a flowchart illustrating the steps of a video reconstruction apparatus in some embodiments of this application;

[0051] Figure 14 A schematic diagram of the hardware structure of an electronic device in some embodiments of this application is shown. Detailed Implementation

[0052] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below. In order to make the purpose and implementation of this application clearer, the exemplary embodiments of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0053] This application is not intended to limit the implementation of the invention. Unless otherwise stated, these terms should be understood in their ordinary and common sense.

[0054] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0055] The use of phrases such as "some implementations" or "some embodiments" in the specification indicates that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when describing a specific feature, structure, or characteristic in connection with an embodiment, it is considered that implementing that feature, structure, or characteristic in connection with other implementations (whether explicitly described herein or not) is within the knowledge of those skilled in the art.

[0056] Implicit Neural Representation (INR) is a method for parameterizing signals. Traditional signal representations are typically discrete; for example, images are discrete grids of pixels, audio signals are discrete samples of amplitude, and 3D shapes are usually parameterized as voxels, point clouds, or grids. In contrast, INR can parameterize signals as continuous functions. Because INR can parameterize signals as continuous functions, the memory required for parameterizing the signal is independent of spatial resolution, proportional only to the complexity of the hidden signal, and it can be sampled at arbitrary spatial resolution, thus obtaining finer signal details.

[0057] 3D Neural Radiance Fields (3D NeRF) is a computer vision technique used to generate high-quality 3D reconstructed models. 3DNeRF utilizes deep learning to extract the geometric shape and texture information of objects from images viewed from multiple perspectives. This information is then used to generate a continuous 3D radiance field that can describe the color and density of any point in the scene in any direction, thus enabling the creation of highly realistic 3D models at any angle and distance.

[0058] Inspired by INR and 3DNeRF, INR representation technology has been extensively researched and developed for application in the field of 2D video. NeRV is one such research result that applies INR technology to 2D video.

[0059] NeRV represents video as parameters of a neural network, transforming video encoding into fitting the video into a neural network model, and converting the video compression problem into a model compression problem. It achieves performance comparable to conventional video compression methods. NeRV provides a new perspective and solution for video representation, compression, and other video-related tasks, and compared to traditional and learning-based video compression methods, it features a simpler training process and higher decoding speed.

[0060] In NeRV, the temporal index t of a video frame is learned as a feature by a Multilayer Perceptron (MLP) network. This feature is then fed into a subsequent reconstruction network consisting of convolutional and upsampling modules to obtain the reconstructed frame. This method fits the video into the entire neural network model. Specifically, the frame index t at each time step is first normalized. The normalized value is then processed by a positional encoding formula to obtain a vector similar to a Fourier transform, which is used to preserve the high-frequency information brought about by the index changing over time. This vector is then used as the feature of the current frame and input into the subsequent reconstruction network.

[0061] Videos are typically composed of a series of consecutive video frames (images). Neighboring video frames are visually very similar, resulting in significant temporal redundancy. Furthermore, video frames may contain numerous repeating patterns or textures, adding to the spatial redundancy. However, NeRV simply fits video frames and indices together using an MLP network, failing to effectively utilize this spatiotemporal redundancy. Additionally, the large number of parameters in an MLP network further hinders the efficiency of the neural representation of video.

[0062] To improve the efficiency of video representation, some embodiments of this application provide the following technical solutions:

[0063] Figure 1 This is a schematic diagram of the structure of a network model that can be used to perform video representation methods and video reconstruction methods, provided for some embodiments of this application.

[0064] Reference Figure 1 In some embodiments, the network model that can be used to perform video representation methods and video reconstruction methods may include: residual grids11 corresponding to each video frame of the video sequence.

[0065] The residual grid 11 can be used to store the residual of a video frame. The residual refers to the difference between the original video frame and the predicted frame (or reference frame). In representing video, representing features for each video frame would result in a large number of network parameters, which is detrimental to compression. Therefore, some embodiments of this application represent the information of each video frame using sparse residuals, replacing the original frame information grid with a learnable residual grid, thereby making the features of the video frames more spatially sparse and reducing the amount of data that needs to be transmitted or stored.

[0066] In some embodiments, a small-sized, multi-channel learnable information grid can be modeled for each video frame of the video sequence. During network training, the parameters in the information grid are continuously optimized through gradient backpropagation to obtain the sparse residuals of the long-term dependent features of the final video relative to the recurrent network. Each vertex of this learnable grid stores the multi-channel residuals corresponding to its spatial location. Compared to MLP networks, which have spatial positional relationships, this is beneficial for learning and fitting 2D signals like video frames. The information grid representing the residuals is then input into subsequent network structures for feature processing of the current video frame.

[0067] Reference Figure 1 In some embodiments, the network model used to perform the video representation method and the video reconstruction method may include a feature fusion module 12. The feature fusion module 12 can be used to add and fuse the residual of the current video frame represented by the residual grid and the hidden state of the previous video frame (hereinafter referred to as "the previous video frame") obtained by the first recurrent neural network to obtain the features of the current video frame.

[0068] The residual of the current video frame is represented as R. t The hidden state of the previous video frame obtained by the first recurrent neural network is represented as follows: The features of the current video frame are represented as follows The operation performed by feature fusion module 12 can be represented as:

[0069]

[0070] Based on the characteristics of recurrent neural networks (RNNs), they can learn hidden states that contain spatiotemporal information of the entire video sequence. When reconstructing the current video frame, the hidden state h of the previous video frame is... t-1 It includes the spatiotemporal relationships of all frames preceding the current video frame, and with the addition of residuals, it can represent the characteristics of the current video frame, effectively utilizing the spatiotemporal information between video sequences.

[0071] Reference Figure 1In some embodiments, the network model used to perform the video representation method and the video reconstruction method may include a first recurrent neural network 13. The first recurrent neural network 13 can be used to obtain a first hidden state of the current video frame based on the features of the current video frame and the hidden state of the current video frame obtained by the first recurrent neural network.

[0072] Recurrent Neural Networks (RNNs) are a type of neural network architecture suitable for processing sequential data. RNNs possess memory capabilities, retaining information through their recurrent structure, thus enabling them to capture long-term dependencies in video sequences.

[0073] As mentioned above, NeRV only fits video frames and indices together using an MLP network, which fails to effectively utilize the spatiotemporal redundancy information of the video, and the MLP network has a large number of parameters. Some of the above embodiments represent the video based on recurrent neural networks. Therefore, these embodiments can utilize the time-series learning capabilities of recurrent neural networks to process video sequences, thereby effectively utilizing the redundant spatiotemporal information of the video. Simultaneously, by avoiding the use of an MLP network, the number of parameters in the network model can be reduced.

[0074] In some embodiments, the first recurrent neural network may be a Long Short-Term Memory (LSTM) network.

[0075] In some embodiments, the first recurrent neural network 13 may be a gated recurrent unit (GRU).

[0076] GRU is a special type of recurrent neural network that controls the flow of information through a gate mechanism, which can include update gates and reset gates. GRU works by combining the current input, the previous hidden state, and the outputs of the two gates at each timestamp to calculate the current hidden state, thus implementing a cyclical update process. It features fast training speed and low memory usage.

[0077] In some embodiments, the first recurrent neural network 13 can be a gated recurrent unit obtained by replacing the linear layers in the GRU with convolutional layers, denoted as ConvGRU.

[0078] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include an update gate 201. The update gate 201 can be used to obtain update gate coefficients based on the hidden state of the previous video frame obtained by the first recurrent neural network and the features of the current video frame.

[0079] Reference Figure 2 In some embodiments, the update gate 201 may include an update gate state convolutional layer 2011. The update gate state convolutional layer 2011 can be used to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the update gate hidden state.

[0080] The hidden state of the previous video frame obtained by the first recurrent neural network is represented as: The updated hidden state of the door is represented as α. zh The operation performed by the gate-updating convolutional layer 2011 can be represented as:

[0081]

[0082] In some embodiments, update the door hiding state α zh The hidden state h of the previous video frame obtained by the first recurrent neural network r-1 They are the same size.

[0083] During the training of the network model, the weight coefficients W of the convolution kernel of the gate state convolutional layer 2011 are updated. zh It will be updated through backpropagation (an algorithm used to calculate the error gradient in a neural network). Specifically, the weight coefficients W of the convolutional kernels in the updated gate state convolutional layer 2011 are updated. zh The hidden state h of the previous video frame obtained by the first recurrent neural network is used to characterize it. t-1 The different dimensions affect the output of the gate-updating convolutional layer 2011 (updating the hidden state α of the gate). zh The extent of its impact.

[0084] Reference Figure 2 In some embodiments, the update gate 201 may further include an update gate feature convolutional layer 2012. The update gate feature convolutional layer 2012 can be used to perform convolution operations on the features of the current video frame to obtain update gate features.

[0085] The features of the current video frame are represented as The update gate feature is represented as α zf The operation performed by the gate feature convolutional layer 2012 can be represented as:

[0086]

[0087] In some embodiments, the update gate feature is α zf Features of the current video frame They are the same size.

[0088] During the training of the network model, the weight coefficients W of the convolution kernel of the gate feature convolutional layer 2012 are updated. zf It will also be updated through backpropagation of the network. Specifically, the weight coefficients W of the convolutional kernels in the updated gate feature convolutional layer 2012 are... zf Features used to characterize the current video frame The different dimensions affect the output of the gate-updating convolutional layer 2011 (updating the hidden state α of the gate). zh The extent of its impact.

[0089] Reference Figure 2 In some embodiments, the update gate 201 may further include an update gate fusion layer 2013. The update gate fusion layer 2013 can be used to superimpose the update gate hidden state, update gate features, and update gate bias to obtain update gate fusion features.

[0090] Let the hidden state of the update gate be represented as α. zh The updated gate feature is represented as α. zf The updated gate bias is represented as b. z The updated gate fusion feature is represented as F. z The operation performed by the update gate fusion layer 2013 can be represented as:

[0091]

[0092] Reference Figure 2 In some embodiments, the update gate 201 may further include an update gate activation function layer 2014. The update gate activation function layer 2014 can be used to perform a nonlinear transformation on the update gate fusion features to obtain the update gate coefficients.

[0093] In some embodiments, the activation function of the update gate activation function layer 2014 can be a sigmoid activation function, so that the value of the update gate coefficient is between 0 and 1.

[0094] The update gate fusion feature is represented as F. z The updated gate coefficient is represented as z. t The nonlinear transformation operation of the update gate activation function layer is represented as σ. z (), then the operation performed by the update gate activation function layer 2014 can be represented as:

[0095]

[0096] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a reset gate 202. The reset gate 202 can be used to obtain the reset gate coefficient of the current video frame based on the hidden state of the previous video frame obtained by the first recurrent neural network and the features of the current video frame.

[0097] Reference Figure 2 In some embodiments, the reset gate 202 may include a reset gate state convolutional layer 2021. The reset gate state convolutional layer 2021 can be used to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the reset gate hidden state.

[0098] The hidden state of the previous video frame obtained by the first recurrent neural network is represented as: Resetting the hidden state of the door is represented by α. rh The operation performed by the gate-resetting convolutional layer 2021 can be represented as:

[0099]

[0100] In some embodiments, the door hiding state α is reset. rh The hidden state h of the previous video frame obtained by the first recurrent neural network t-1 They are the same size.

[0101] During the training of the network model, the weight coefficients W of the convolutional kernels of the gate-state resetting convolutional layer 2021 are... rh It will be updated through backpropagation in the network. Specifically, the weight coefficients W of the convolutional kernel in the reset gate state convolutional layer 2021... rh The hidden state h of the previous video frame obtained by the first recurrent neural network is used to characterize it. t-1 Different dimensions affect the output of the reset gate state convolutional layer 2021 (reset gate hidden state α) rh The extent of its impact.

[0102] Reference Figure 2 In some embodiments, the reset gate 202 may further include a reset gate feature convolutional layer 2022. The reset gate feature convolutional layer 2022 can be used to perform convolution operations on the features of the current video frame to obtain reset gate features.

[0103] The features of the current video frame are represented as The reset door feature is represented by α. rf The operation performed by the reset gate feature convolutional layer 2022 can be represented as:

[0104]

[0105] Reference Figure 2 In some embodiments, the reset door 202 may further include a reset door fusion layer 2023. The reset door fusion layer 2023 can be used to add and fuse the reset door hidden state, reset door features, and reset door bias to obtain reset door fusion features.

[0106] Let α represent the reset door's hidden state. rh The reset gate feature is represented by α. rf The gate offset reset is represented as b. r The reset gate fusion feature is represented as F. r The operation performed by resetting the gate fusion layer 2023 can be represented as:

[0107]

[0108] Reference Figure 2 In some embodiments, the reset gate 202 may further include a reset gate activation function layer 2024. The reset gate activation function layer 2024 can be used to perform a nonlinear transformation on the reset gate fusion features to obtain the reset gate coefficients.

[0109] In some embodiments, the activation function of the reset gate activation function layer 2024 can be a sigmoid activation function, so that the value of the reset gate coefficient is between 0 and 1.

[0110] The reset gate fusion feature is represented as F. r The reset gate coefficient is represented by r. t The nonlinear transformation operation of the reset gate activation function layer is represented as σ. r (), then the operation performed by the reset gate activation function layer 2024 can be expressed as:

[0111]

[0112] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a first multiplier 203. The first multiplier 203 can be used to multiply the reset gate coefficients element-wise with the hidden state of the previous video frame obtained by the first recurrent neural network to obtain a first feature.

[0113] Let the reset gate coefficient be denoted as r. t The first feature is denoted as A, and the element-wise multiplication is denoted as ⊙. The hidden state of the previous video frame obtained by the first recurrent neural network is denoted as... The operation performed by the first multiplier 203 can be represented as:

[0114]

[0115] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a first adder 204. The first adder 204 can be used to superimpose the first feature with the feature of the current video frame to obtain a second feature.

[0116] Let the first feature be denoted as A, and the feature of the current video frame be denoted as... If the second feature is represented by B, then the operation performed by the first adder 204 can be represented as:

[0117]

[0118] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a convolutional unit 205. The convolutional unit 205 may be used to perform a convolution operation on the second feature to obtain a third feature.

[0119] If the second feature is denoted as B and the third feature as C, then the operation performed by convolutional unit 205 can be represented as:

[0120]

[0121] In some embodiments, the third feature has the same dimensions as the second feature.

[0122] During the training of the network model, the weight coefficients W of the convolutional kernel of convolutional unit 205 hh It will be updated through backpropagation of the network model. Specifically, the weight coefficients W of the convolutional kernel in convolutional unit 205... hh This is used to characterize the degree of influence of features of different dimensions on the output (third feature) of convolutional unit 205.

[0123] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a second adder 206. The second adder 206 can be used to superimpose a third feature with an update gate bias to obtain a fourth feature.

[0124] Let the third feature be represented by C, the fourth feature by D, and the update gate bias by b. z Then the operation performed by the second adder 206 can be represented as:

[0125]

[0126] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include an activation unit 207. The activation unit 207 may be used to perform a nonlinear transformation on the fourth feature to obtain candidate hidden states.

[0127] In some embodiments, the activation function of the activation unit 207 is the hyperbolic tangent function.

[0128] The candidate hidden state is represented as The fourth feature is denoted as D, and the nonlinear transformation operation of activation unit 207 is denoted as tanh(). Therefore, the operation performed by activation unit 207 can be expressed as:

[0129]

[0130] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a second multiplier 208. The second multiplier 208 can be used to multiply the hidden state of the previous video frame obtained by the first recurrent neural network element-wise with the update gate coefficients to obtain the fifth feature.

[0131] The hidden state of the previous video frame obtained by the first recurrent neural network is represented as: The update gate coefficient is represented as z t If the fifth feature is denoted as E, then the operation performed by the second multiplier 208 can be represented as:

[0132]

[0133] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a third multiplier 209. The third multiplier 209 can be used to multiply the candidate hidden state element-wise with the complement of the update gate coefficient to obtain a sixth feature.

[0134] The candidate hidden state is represented as The update gate coefficient is represented as z t If the sixth feature is denoted as F, then the operation performed by the third multiplier 209 can be represented as:

[0135]

[0136] Reference Figure 2 In some embodiments, the first recurrent neural network 13 may include a third adder 210. The third adder 210 can be used to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0137] Let h be the first hidden state of the current video frame. t If the fifth feature is denoted as E and the sixth feature as F, then the operation performed by the third adder 210 can be represented as:

[0138]

[0139] Reference Figure 1In some embodiments, the network model used to perform the video representation method and the video reconstruction method may include a super-resolution network 14. The super-resolution network 14 can be used to process the first hidden state of the current video frame to obtain the reconstructed features of the current video frame.

[0140] The first hidden state of the current video frame is represented as The reconstructed features of the current video frame are represented as follows If the operation of super-resolution network 14 is represented by NeRV(), then the operation performed by super-resolution network 14 can be represented as:

[0141]

[0142] Reference Figure 3 In some embodiments, the super-resolution network 14 consists of multiple cascaded NeRV blocks 30. Figure 3 The super-resolution network 14 is illustrated as consisting of five cascaded NeRV blocks. However, in other embodiments, the super-resolution network 14 can also consist of other numbers of NeRV blocks, such as four cascaded NeRV blocks or six cascaded NeRV blocks. The more NeRV blocks that make up the super-resolution network 14, the smaller the upsampling factor of a single NeRV block can be, resulting in smoother feature changes and reducing the likelihood of jagged edges or blurring. However, the computational cost of upsampling is relatively high, and the number of model parameters in the super-resolution network 14 will also increase accordingly. Conversely, the fewer NeRV blocks that make up the super-resolution network 14, the larger the upsampling factor of a single NeRV block is required, resulting in a relatively lower computational cost and fewer model parameters in the super-resolution network 14. However, the feature changes are relatively drastic, making it prone to jagged edges or blurring. Therefore, in practical applications, the number of NeRV blocks that make up the super-resolution network 14 can be determined by considering the computational cost of upsampling, the number of model parameters in the super-resolution network, and the quality requirements for the re-added video frames.

[0143] In some embodiments, the multiple NeRV blocks 30 constituting the super-resolution network 14 are all used to upsample the input features along the feature channel dimension, and the upsampling factor depends on the factor required to upsample the features along the spatial dimension. For example, if the factor required to upsample the input features along both the height and width is s, then the NeRV block 30 can be used to upsample the input features along the feature channel dimension by s. 2 For example, if the input features need to be upsampled by a factor of s1 in height and by a factor of s2 in width, then NeRV block 30 is used to upsample the input features by a factor of s1*s2 in the feature channel dimension.

[0144] In some embodiments, the super-resolution network 14 consists of five cascaded NeRV blocks, each of which is used to upsample the input features from the feature channel dimension, and the upsampling factors are 25, 4, 4, 4, and 4 respectively.

[0145] The super-resolution network 14 is composed of multiple NeRV blocks stacked together. Pixels at different positions can share the convolution kernels in the NeRV blocks, so the super-resolution network can efficiently reconstruct video frames.

[0146] Reference Figure 4 In some embodiments, the NeRV block may include a convolutional layer 41. The convolutional layer 41 can be used to perform upsampling convolution processing on the input features of the NeRV block to obtain the output features of the convolutional layer 41. Wherein, when the NeRV block is a first-level NeRV block, the input features of the NeRV block are the first hidden state of the current video frame. When the NeRV block is a level n NeRV block, the input features of the NeRV block are the output features of the (n-1)th level NeRV block, where n is an integer greater than or equal to 2.

[0147] The input features of the NeRV block are represented as The output features of convolutional layer 41 are represented as follows If the upsampling convolution is represented by Conv(), then the operation performed by convolutional layer 41 can be represented as:

[0148]

[0149] The values ​​of S for the first to fifth level NeRV blocks are 5, 2, 2, 2, 2, respectively.

[0150] The above embodiments progressively upsample small-sized features using five NeRV blocks, while continuously increasing the number of feature channels during convolution. Therefore, these embodiments can enable the feature map to contain more information, thereby improving the quality of the reconstructed video frames.

[0151] Reference Figure 4 In some embodiments, the NeRV block may include a pixel shuffle layer 42. The pixel shuffle layer 42 can be used to perform pixel shuffle on the output features of the convolutional layer 41 to obtain the output features of the pixel shuffle layer 42.

[0152] The output features of convolutional layer 41 are represented as follows The pixel shuffle operation is represented as Pixelshuffle(), and the output features of pixel shuffle layer 42 are represented as... The operation performed by pixel shuffling layer 42 can be represented as:

[0153]

[0154] Reference Figure 4 In some embodiments, the NeRV block may include an activation function layer 43. The activation function layer 43 can be used to perform a nonlinear transformation on the output features of the pixel shuffling layer 42 to obtain the output features of the NeRV block.

[0155] In some embodiments, the activation function of activation function layer 43 is the sigmoid activation function.

[0156] The output features of pixel shuffling layer 42 are represented as follows: The output features of the NeRV block are represented as follows: If the nonlinear transformation operation of activation function layer 43 is represented by σ(), then the operation performed by activation function layer 43 can be represented as:

[0157]

[0158] Reference Figure 1 In some embodiments, the network model used to perform the video representation method and the video reconstruction method may include: a second recurrent neural network 15.

[0159] In some embodiments, the second recurrent neural network 15 may also be a GRU.

[0160] The structure of the second recurrent neural network 15 can be similar to... Figure 2 The structure of the first recurrent neural network 13 shown is the same, and will not be described in detail here to avoid redundancy.

[0161] The difference between the second recurrent neural network 15 and the first recurrent neural network 13 lies in the fact that the input of the first recurrent neural network 13 is the residual R of the current video frame. t The hidden state of the previous video frame obtained by the first recurrent neural network. The input to the second recurrent neural network 15 is the reconstructed features of the current video frame. The hidden state of the previous video frame obtained by the second recurrent neural network 15

[0162] Represent the reconstructed features of the current video frame as The hidden state of the previous video frame obtained by the second recurrent neural network 15 is represented as follows: The second hidden state of the current video frame output by the second recurrent neural network 15 is represented as follows: The processing operation of the second recurrent neural network 15 is represented as ConvGRU_2(), and the operation performed by the second recurrent neural network 15 can be represented as:

[0163]

[0164] The super-resolution network 14 only processes the features of the current video frame. As a result, the reconstructed video frame does not take into account the temporal correlation between video frames as a sequence. Therefore, some of the above embodiments set a recurrent neural network (second recurrent neural network) after the super-resolution network 14. This recurrent neural network can establish long-term dependencies between video frames, compensate for the spatiotemporal details between video frames, obtain more temporal correlations between adjacent video frames, and thus improve the quality of the reconstructed video frame.

[0165] Reference Figure 1 In some embodiments, the network model used to perform the video representation method and the video reconstruction method may include an output module 16. The output module 16 can be used to determine the second hidden state of the current video frame. Reconstruct the current video frame.

[0166] The reconstruction operation of output module 16 is represented as hand(), and the reconstructed video frame of the current video frame is represented as... The operation performed by output module 16 can be represented as:

[0167]

[0168] In some implementations, the output module 16 can be a convolutional layer.

[0169] In some embodiments, based on Figure 1 The video representation method of the network model shown may include the following steps a to i:

[0170] Step a: Construct the residual grid corresponding to each video frame in the video sequence and initialize the residual grid corresponding to each video frame.

[0171] In some embodiments, the height of the residual grid corresponding to each video frame can be 9, the width can be 16, and the number of feature dimensions stored in each grid vertex is C0. That is, the residual grid corresponding to the video frame with index t can be represented as follows:

[0172] In some embodiments, initializing the residual grid corresponding to each video frame may include: initializing the parameters in the residual grid to 0.

[0173] Step b: Obtain the residual corresponding to the current video frame.

[0174] During the training of the network model, given the temporal index of a video frame in a video sequence, the corresponding residual feature of dimension C0*9*16 can be found. This is used to represent the residual of the current frame. The residual feature is then fed into the subsequent grid structure for further feature extraction and reconstruction.

[0175] Step c: Obtain the hidden state of the previous video frame from the first recurrent neural network. The residual R with the current video frame t Add them together to obtain the features of the current video frame.

[0176] That is, the operation represented by the above equation (1) is performed through the feature fusion module.

[0177] It should be noted that when performing step c above on the first video frame of the video sequence (the current video frame is the first video frame of the video sequence), since the hidden state of the previous video frame obtained by the first recurrent neural network does not exist at this time, a learnable grid can be used. Let this represent the hidden state of the previous video frame obtained by the first recurrent neural network, and let this learnable grid be... It will be updated via backpropagation during subsequent training.

[0178] In some embodiments, the initial update gate bias b of the first recurrent neural network z and initial reset gate bias b r All parameters are non-zero. During residual mesh initialization, the parameters in the residual mesh are initialized to 0, so the first recurrent neural network can update the gate bias b. z and reset door offset b r It is initialized to a non-zero parameter to prevent the network model from failing to learn useful information when the input is zero, thus slowing down the learning speed.

[0179] Step d: Based on the characteristics of the current video frame The hidden state of the previous video frame obtained by the first recurrent neural network. Get the first hidden state of the current video frame

[0180] That is, the operations represented by equations (2) to (17) above are executed through the first recurrent neural network.

[0181] Step e: Based on the first hidden state of the current video frame Obtain the reconstructed features of the current video frame

[0182] That is, the operations represented by equations (18) to (21) above are performed through the super-resolution network.

[0183] Step f: Based on the reconstruction features of the current video frame The hidden state of the previous video frame obtained by the second recurrent neural network. Get the second hidden state of the current video frame

[0184] That is, the operation represented by the above equation (22) is performed through the second recurrent neural network.

[0185] Step g: Based on the second hidden state of the current video frame Reconstruct the video frame from the current video frame

[0186] That is, the operation represented by the above equation (23) is executed through the output module.

[0187] Step h: Based on the current video frame and the reconstructed video frame of the current video frame. The system calculates the loss value using a preset loss function and adjusts the parameters of the residual grid, the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module of the current frame based on the loss value.

[0188] In some embodiments, the preset loss function is a loss function obtained by combining the L1 loss function and the Structural Similarity Index (SSIM) loss function.

[0189] In some embodiments, the preset loss function is as follows:

[0190]

[0191] Where T is the total number of video frames in the video sequence, f θ (t) represents the reconstructed video frame of the current video frame, v t For the current video frame (frame truth value), α is a hyperparameter used to balance the weights of the L1 loss component and the SSIM loss component.

[0192] In some embodiments, α is 0.7. That is, the weight of the L1 loss component is 0.7, and the weight of the SSIM loss component is 0.3.

[0193] In some embodiments, Adaptive Moment Estimation (Adam) is selected as the optimizer, the batch size is set to 1, and the initial learning rate is 5e. -4 And a cosine annealing learning rate adjustment method is adopted.

[0194] Adam is an optimization algorithm used to train deep learning models. On one hand, unlike traditional gradient descent algorithms that update parameters solely based on the current gradient, Adam incorporates a momentum algorithm, considering not only the current gradient but also the cumulative effect of previous gradients when updating parameters. On the other hand, Adam also incorporates RMSProp (Root Mean Square Propagation, an adaptive learning rate optimization algorithm), which adjusts the learning rate by maintaining an exponentially weighted moving average of the squared gradient. Therefore, Adam can adaptively adjust the learning rate for each parameter.

[0195] Step i: Take the next video frame after the current video frame as the current video frame, and repeat steps b to h until the training stop condition is met. Then, generate the representation data of the video sequence based on the residual grid, the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module corresponding to each video frame of the video sequence.

[0196] In some embodiments, the training stop condition is that the training iterations of each video frame in the video sequence reach a preset number. For example, the training stop condition is that the training iterations of each video frame in the video sequence reach 300.

[0197] In some embodiments, the representation data of the video sequence may include: a first recurrent neural network, a super-resolution network, a second recurrent neural network, and various parameters in the output module, including: trainable parameters and non-trainable parameters.

[0198] In some embodiments, the representation data of the video sequence may include: a first recurrent neural network, a super-resolution network, a second recurrent neural network, and various trainable parameters in the output module.

[0199] When the representation data of the video sequence only includes the trainable parameters of the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module, the non-trainable parameters of the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module are first determined by default values, etc., and then the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module are reconstructed based on the parameters of the non-trainable parameters of the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module.

[0200] In some embodiments, based on Figure 1The video reconstruction method of the network model shown may include the following steps 1 to 9:

[0201] Step 1: Obtain the representation data of the video sequence.

[0202] The representation data of the video sequence may include: residual grids corresponding to each video frame of the video sequence, a first recurrent neural network, a super-resolution network, a second recurrent neural network, and an output module.

[0203] Step 2: Obtain the residual R corresponding to the current video frame based on the residual grid corresponding to each video frame in the video sequence. t .

[0204] Step 3: Calculate the hidden state of the previous video frame obtained from the first recurrent neural network. The residual R with the current video frame t Add them together to obtain the features of the current video frame.

[0205] That is, the operation represented by the above equation (1) is performed through the feature fusion module.

[0206] Step 4: Based on the characteristics of the current video frame The hidden state of the previous video frame obtained by the first recurrent neural network. Get the first hidden state of the current video frame

[0207] That is, the operations represented by equations (2) to (17) above are executed through the first recurrent neural network.

[0208] Step 5: Based on the first hidden state of the current video frame Obtain the reconstructed features of the current video frame

[0209] That is, the operations represented by equations (18) to (21) above are performed through the super-resolution network.

[0210] Step 6: Based on the reconstruction features of the current video frame The hidden state of the previous video frame obtained by the second recurrent neural network. Get the second hidden state of the current video frame

[0211] That is, the operation represented by the above equation (22) is performed through the second recurrent neural network.

[0212] Step 7: Based on the second hidden state of the current video frame Reconstruct the video frame from the current video frame

[0213] That is, the operation represented by the above equation (23) is executed through the output module.

[0214] Step 8: Take the next video frame after the current video frame as the current video frame, and repeat steps 2 to 7 until the reconstructed video frames of each video frame in the video sequence are obtained.

[0215] Peak signal-to-noise ratio (PSNR) was used as an image quality evaluation metric to test the video representation and video reconstruction methods provided in the above embodiments. Network models of different sizes were obtained by changing the number of channels in the residual grid and the number of channels in the NeRV block, and the results were compared under different sizes. Figure 1 The representational power of the network model shown was tested at sizes of 1.5M, 2.5M, 3.2M (NeRV-S), 6.3M (NeRV-M), and 12.5M (NeRV-L), and the test results are shown in Table 1.

[0216] Table 1. Test Results and Comparison of PSNR Index

[0217]

[0218] As shown in Table 1. Based on the test results and comparisons, compared to NeRV, Figure 1 The network model shown improves PSNR across different sizes.

[0219] Reference Figure 5 In some embodiments, the network model used to perform the video representation method may also include only: a residual grid 11, a first recurrent neural network 13, a super-resolution network 14, and an output module 16. The operations performed by the residual grid 11, the first recurrent neural network 13, and the super-resolution network 14 are similar to those in... Figure 1 The same applies to the output module 16, which reconstructs the current video frame based on the reconstruction features of the current video frame output by the super-resolution network 14.

[0220] The reconstruction operation of output module 16 is represented as hand(), and the reconstructed video frame of the current video frame is represented as... but Figure 5 The operation performed by output module 16 can be represented as:

[0221]

[0222] Some embodiments of this application provide a video representation method, see reference Figure 6 The video representation method may include the following steps:

[0223] S61. Create an information grid corresponding to each video frame of the target video sequence.

[0224] In some embodiments, the information grid corresponding to each video frame can have a height of 9, a width of 16, and the number of feature dimensions stored in each grid vertex is C0.

[0225] In some embodiments, after creating the information grid corresponding to each video frame of the target video sequence, the method further includes: initializing the information grid corresponding to each video frame of the target video sequence.

[0226] The following steps S62 to S63 are performed iteratively, using each video frame of the target video sequence as the current video frame, until the training stop condition is met:

[0227] S62. Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, obtain the first hidden state of the current video frame.

[0228] In some embodiments, step S62 (obtaining the first hidden state of the current video frame based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network) includes the following steps (1) and (2):

[0229] Step 1: Obtain the features of the current video frame based on the content grid corresponding to the current video frame.

[0230] Step (2): Based on the features of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, obtain the first hidden state of the current video frame.

[0231] Reference Figure 7 When the implementation of step S62 includes steps (1) and (2) above, the network model used to execute the video representation method provided in the above embodiments includes: an information grid 71 corresponding to each video frame, a first recurrent neural network 72, and a reconstruction network 73. The information grid corresponding to each video frame is used to represent the features of each video frame.

[0232] In some embodiments, step S62 (obtaining the first hidden state of the current video frame based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network) includes the following steps ① to ③:

[0233] Step 1: Obtain the residual of the current video frame based on the information grid corresponding to the current video frame.

[0234] Step 2: Superimpose the residual of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the features of the current video frame.

[0235] Step 3: Based on the features of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, obtain the first hidden state of the current video frame.

[0236] Reference Figure 8 When the implementation of step S62 includes steps ① to ③, the network model used to execute the video representation method provided in the above embodiments includes: an information grid 81 corresponding to each video frame, a feature fusion module 82, a first recurrent neural network 83, and a reconstruction network 84. The information grid corresponding to each video frame is used to represent the residuals of each video frame, and the feature fusion module 82 is used to superimpose the residual of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the features of the current video frame.

[0237] Compared to Figure 7 In the illustrated embodiment, the information grid corresponding to each video frame is used to represent the features of each video frame. Figure 8 In the illustrated embodiment, the information grid corresponding to each video frame is used to represent the residual of each video frame, therefore Figure 8 In the illustrated embodiment, the information grid has sparser grid parameters, which can reduce the amount of data representing the target video sequence.

[0238] Reference Figure 2 In some embodiments, the first recurrent neural network includes:

[0239] Update gate 201 can be used to obtain update gate coefficients based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0240] Reset gate 202 can be used to obtain reset gate coefficients based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0241] The first multiplier 203 can be used to multiply the reset gate coefficients element-wise with the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the first feature;

[0242] The first adder 204 can be used to superimpose the first feature with the feature of the current video frame to obtain the second feature;

[0243] Convolutional unit 205 can be used to perform a convolution operation on the second feature to obtain the third feature;

[0244] The second adder 206 can be used to superimpose the third feature with the update gate bias to obtain the fourth feature;

[0245] The activation unit 207 can be used to perform a nonlinear transformation on the fourth feature to obtain the candidate hidden state;

[0246] The second multiplier 208 can be used to multiply the hidden state of the previous video frame obtained by the first recurrent neural network with the update gate coefficient element by element to obtain the fifth feature;

[0247] The third multiplier 209 can be used to multiply the candidate hidden state by the complement of the update gate coefficient element by element to obtain the sixth feature;

[0248] The second adder 210 can be used to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0249] Reference Figure 2 In some embodiments, the update gate 201 includes:

[0250] The updated gate state convolutional layer 2011 can be used to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the updated gate hidden state.

[0251] The Update Gate Feature Convolutional Layer 2012 can be used to perform convolution operations on the features of the current video frame to obtain the Update Gate features;

[0252] The update gate fusion layer 2013 can be used to overlay the update gate hidden state, the update gate features, and the update gate bias to obtain the update gate fusion features;

[0253] The update gate activation function layer 2014 can be used to perform a nonlinear transformation on the update gate fusion features to obtain the update gate coefficients.

[0254] Reference Figure 2 In some embodiments, the reset door 202 includes:

[0255] The reset gate state convolutional layer 2021 can be used to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the reset gate hidden state.

[0256] The reset gate feature convolutional layer 2022 can be used to perform convolution operations on the features of the current video frame to obtain the reset gate features;

[0257] The reset door fusion layer 2023 can be used to overlay the reset door hidden state, the reset door features, and the reset door offset to obtain the reset door fusion features;

[0258] The reset gate activation function layer 2024 can be used to perform a nonlinear transformation on the reset gate fusion features to obtain the reset gate coefficients.

[0259] In some of the above embodiments, both the update gate and the reset gate use convolutional layers to perform convolution operations on the hidden state of the previous video frame and the features of the current video frame obtained by the first recurrent neural network. Therefore, the above embodiments can enable the first recurrent neural network to have a better ability to learn spatial information features, thereby improving the quality of the final reconstructed video.

[0260] S63. Obtain the reconstructed video frame corresponding to the current video frame based on the first hidden state of the current video frame and the reconstruction network.

[0261] Reference Figure 9 In some embodiments, in Figure 8 Based on the illustrated embodiment, the reconstructed network 84 includes:

[0262] The super-resolution network 91 can be used to process the first hidden state of the current video frame to obtain the reconstructed features of the current video frame;

[0263] The output module 92 can be used to obtain the reconstructed video frame corresponding to the current video frame based on the reconstruction features of the current video frame.

[0264] That is, step S63 above includes: processing the first hidden state of the current video frame to obtain the reconstruction features of the current video frame, and obtaining the reconstructed video frame corresponding to the current video frame based on the reconstruction features of the current video frame.

[0265] Reference Figure 10 In some embodiments, in Figure 8 Based on the illustrated embodiment, the reconstructed network 84 includes:

[0266] Super-resolution network 101 can be used to process the first hidden state of the current video frame in order to obtain the reconstructed features of the current video frame;

[0267] The second recurrent neural network 102 can be used to obtain the second hidden state of the current video frame based on the reconstruction features of the current video frame and the hidden state of the previous video frame obtained by the second recurrent neural network model.

[0268] The output module 103 can be used to obtain the reconstructed video frame corresponding to the current video frame based on the second hidden state of the current video frame.

[0269] That is, step S63 above includes: processing the first hidden state of the current video frame to obtain the reconstruction features of the current video frame; obtaining the second hidden state of the current video frame based on the reconstruction features of the current video frame and the hidden state of the previous video frame obtained by the second recurrent neural network model; and obtaining the reconstructed video frame corresponding to the current video frame based on the reconstruction features of the current video frame.

[0270] because Figure 10 In the embodiments shown, after obtaining the reconstruction features of the current video frame, the reconstruction features of the current video frame are further processed by a second recurrent neural network. Therefore, the above embodiments can utilize the recurrent neural network to obtain more spatiotemporal correlations between adjacent frames in time sequence, and capture more spatiotemporal detail information at the video frame level, thereby improving the quality of the final reconstructed video frame.

[0271] In some embodiments, refer to Figure 3 The super-resolution network 14 includes: a five-level cascaded NeRV block 30.

[0272] In some embodiments, the upsampling factors of the five cascaded NeRV blocks are 25, 4, 4, 4, and 4, respectively.

[0273] In some embodiments, refer to Figure 4 The NeRV block 30 includes:

[0274] Convolutional layer 41 can be used to perform upsampling convolution processing on the input features of the NeRV block to obtain the output features of the convolutional layer;

[0275] The pixel shuffling layer 42 can be used to shuffle the output features of the convolutional layer to obtain the output features of the pixel shuffling layer.

[0276] The activation function layer 43 can be used to perform nonlinear transformation on the output features of the pixel shuffling layer to obtain the output features of the NeRV block.

[0277] In some embodiments, the output module consists of a single convolutional layer.

[0278] In some embodiments, the network structure of the second recurrent neural network is the same as that of the first recurrent neural network.

[0279] S64. Based on the current video frame, the reconstructed video frame corresponding to the current video frame, and the preset loss function, adjust at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network.

[0280] In some embodiments, the preset loss function is:

[0281]

[0282] Where T is the number of video frames in the target video sequence, f θ (t) represents the reconstructed video frame of the current video frame, v t For the current video frame, α is a hyperparameter, ||f θ (t)-v t ||1 is f θ (t) and v t The L1 loss value, SSIM(f θ (t),v t ) is f θ (t) and v t The SSIM loss value is the structural similarity index.

[0283] After executing step S64 above, determine whether the training stop condition is met. If the training stop condition is not met, return to step S61 and execute steps S61 to S64 again. If the training stop condition is met, execute the following step S65:

[0284] S65. Generate representation data of the target video sequence based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network.

[0285] In some embodiments, generating representation data of the target video sequence based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network includes: obtaining the information grid corresponding to each video frame of the target video sequence, the trainable network parameters of the first recurrent neural network, and the trainable network parameters of the reconstruction network; and compressing the information grid corresponding to each video frame of the target video sequence, the trainable network parameters of the first recurrent neural network, and the trainable network parameters of the reconstruction network to obtain the representation data of the target video sequence.

[0286] The video representation method provided in some embodiments of this application, when representing a target video sequence, first creates an information grid corresponding to each video frame of the target video sequence. Then, it iteratively uses each video frame of the target video sequence as the current video frame. Based on the information grid corresponding to the current video frame, a first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, a first hidden state of the current video frame is obtained. Based on the first hidden state of the current video frame and a reconstruction network, a reconstructed video frame corresponding to the current video frame is obtained. Based on the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function, at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network is adjusted. When the training stop condition is met, representation data of the target video sequence is generated based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network. Since the video representation method provided in the above embodiments is based on recurrent neural networks for video representation, and recurrent neural networks have memory functions and can retain information through their recurrent structure, thereby capturing long-term dependencies in the video sequence, the above embodiments can effectively utilize the spatiotemporal redundancy information of the video. Compared to NeRV, which cannot effectively utilize the spatiotemporal redundancy information of video and has a large number of parameters in MLP networks, the above-mentioned embodiments can effectively utilize the spatiotemporal redundancy information of video without using MLP networks. Therefore, the above-mentioned embodiments can improve the representation efficiency of video.

[0287] Some embodiments of this application provide a video reconstruction method, referring to... Figure 11 The video reconstruction method may include the following steps:

[0288] S111. Obtain the representation data of the target video sequence.

[0289] In some embodiments, obtaining representation data of a target video sequence includes receiving representation data of a target video sequence sent by a media distribution server.

[0290] S112. Based on the representation data of the target video sequence, obtain the information grid, the first recurrent neural network, and the reconstruction network corresponding to each video frame of the target video sequence.

[0291] In some embodiments, the representation data of the target video sequence includes: based on the representation data of the target video sequence, the information grid corresponding to each video frame of the target video sequence, the network parameters of the first recurrent neural network, and the network parameters of the reconstruction network; constructing the first recurrent neural network based on the network parameters of the first recurrent neural network; and constructing the reconstruction network based on the network parameters of the reconstruction network.

[0292] Each video frame of the target video sequence is used as the current video frame to perform the following steps S113 and S114 to obtain the reconstructed video frames corresponding to each video frame of the target video sequence:

[0293] S113. Obtain the first hidden state of the current video frame based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0294] The implementation method of step S113, the structure of the network model corresponding to different implementation methods, and the structure of the first recurrent neural network can be referred to the analysis and explanation of step S62 above. To avoid redundancy, they will not be described in detail here.

[0295] S114. Obtain the reconstructed video frame corresponding to the current video frame based on the first hidden state of the current video frame and the reconstruction network.

[0296] The implementation method of step S114, the structure of the network model corresponding to different implementation methods, and the structure of the reconstructed network can be referred to the analysis and explanation of step S63 above. To avoid redundancy, they will not be described in detail here.

[0297] The video reconstruction methods provided in the above embodiments, after obtaining the representation data of the target video sequence, first obtain the information grid, first recurrent neural network, and reconstruction network corresponding to each video frame of the target video sequence based on the representation data of the target video sequence. Then, each video frame of the target video sequence is taken as the current video frame, and the first hidden state of the current video frame is obtained based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network. Finally, the reconstructed video frame corresponding to the current video frame is obtained based on the first hidden state of the current video frame and the reconstruction network, thereby obtaining the reconstructed video frames corresponding to each video frame of the target video sequence. Therefore, the video reconstruction methods provided in the above embodiments can reconstruct the video represented by the video representation methods provided in the above embodiments. Thus, the above embodiments can improve the video representation efficiency while ensuring that the video can be reconstructed.

[0298] Reference Figure 12 Some embodiments of this application provide a video display device 1200, which may include:

[0299] The creation module 121 can be used to create information grids corresponding to each video frame of the target video sequence;

[0300] Training module 122 can be used to iteratively execute the following steps, taking each video frame of the target video sequence as the current video frame, until the training stop condition is met:

[0301] Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained.

[0302] Based on the first hidden state of the current video frame and the reconstruction network, obtain the reconstructed video frame corresponding to the current video frame;

[0303] Based on the current video frame, the reconstructed video frame corresponding to the current video frame, and the preset loss function, at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network is adjusted.

[0304] The generation module 123 can be used to generate representation data of the target video sequence based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network when the training stop condition is met.

[0305] In some embodiments, the training module 122 is specifically used to obtain the residual of the current video frame based on the information grid corresponding to the current video frame; superimpose the residual of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the features of the current video frame; and obtain the first hidden state of the current video frame based on the features of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0306] In some embodiments, the first recurrent neural network may include:

[0307] The update gate can be used to obtain the update gate coefficient based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0308] The reset gate can be used to obtain the reset gate coefficient based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0309] The first multiplier can be used to multiply the reset gate coefficients element-wise with the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the first feature;

[0310] The first adder can be used to superimpose the first feature with the feature of the current video frame to obtain the second feature;

[0311] A convolutional unit can be used to perform a convolution operation on the second feature to obtain the third feature;

[0312] The second adder can be used to superimpose the third feature with the update gate bias to obtain the fourth feature;

[0313] The activation unit can be used to perform a nonlinear transformation on the fourth feature to obtain the candidate hidden state;

[0314] The second multiplier can be used to multiply the hidden state of the previous video frame obtained by the first recurrent neural network with the update gate coefficient element-wise to obtain the fifth feature;

[0315] The third multiplier can be used to multiply the candidate hidden state element-wise with the complement of the update gate coefficient to obtain the sixth feature;

[0316] The second adder can be used to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0317] In some embodiments, the update gate may include:

[0318] The updated gate state convolutional layer can be used to perform convolution operations on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the updated gate hidden state.

[0319] The update gate feature convolutional layer can be used to perform convolution operations on the features of the current video frame to obtain the update gate features;

[0320] The update gate fusion layer can be used to overlay the update gate hidden state, the update gate features, and the update gate bias to obtain the update gate fusion features;

[0321] The update gate activation function layer can be used to perform nonlinear transformations on the update gate fusion features to obtain the update gate coefficients.

[0322] In some embodiments, the reset door may include:

[0323] The reset gate state convolutional layer can be used to perform convolution operations on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the reset gate hidden state.

[0324] The reset gate feature convolutional layer can be used to perform convolution operations on the features of the current video frame to obtain the reset gate features;

[0325] The reset door fusion layer can be used to overlay the reset door hidden state, the reset door features, and the reset door offset to obtain the reset door fusion features;

[0326] The reset gate activation function layer can be used to perform nonlinear transformations on the reset gate fusion features to obtain the reset gate coefficients.

[0327] In some embodiments, the network reconstruction may include:

[0328] Super-resolution networks can be used to process the first hidden state of the current video frame to obtain the reconstructed features of the current video frame;

[0329] The output module can be used to obtain the reconstructed video frame corresponding to the current video frame based on the reconstruction features of the current video frame.

[0330] In some embodiments, the network reconstruction may include:

[0331] Super-resolution networks can be used to process the first hidden state of the current video frame in order to obtain the reconstructed features of the current video frame.

[0332] The second recurrent neural network can be used to obtain the second hidden state of the current video frame based on the reconstruction features of the current video frame and the hidden state of the previous video frame obtained by the second recurrent neural network model.

[0333] The output module can be used to obtain the reconstructed video frame corresponding to the current video frame based on the second hidden state of the current video frame.

[0334] In some embodiments, the super-resolution network may include: NeRV blocks of neural representations of multi-level cascaded videos.

[0335] In some embodiments, the super-resolution network may include: a five-level cascaded NeRV block;

[0336] The upsampling factors of the five cascaded NeRV blocks are 25, 4, 4, 4, and 4, respectively.

[0337] In some embodiments, the NeRV block may include:

[0338] The convolutional layer can be used to perform upsampling convolution processing on the input features of the NeRV block to obtain the output features of the convolutional layer;

[0339] A pixel shuffling layer can be used to shuffle the output features of the convolutional layer to obtain the output features of the pixel shuffling layer.

[0340] The activation function layer can be used to perform nonlinear transformations on the output features of the pixel shuffling layer to obtain the output features of the NeRV block.

[0341] In some embodiments, the output module consists of a single convolutional layer.

[0342] In some embodiments, the network structure of the second recurrent neural network is the same as that of the first recurrent neural network.

[0343] In some embodiments, the preset loss function is:

[0344]

[0345] Where T is the number of video frames in the target video sequence, f θ (t) represents the reconstructed video frame of the current video frame, v t For the current video frame, α is a hyperparameter, ||f θ (t)-v t ||1 is f θ (t) and v t The L1 loss value,

[0346] SSIM(f θ (t),v t ) is f θ (t) and v t The SSIM loss value is the structural similarity index.

[0347] The video display device provided in the above embodiments can execute the video display method provided in any of the above embodiments, and the implementation principle and the achieved technical effect are the same. To avoid redundancy, it will not be described in detail again.

[0348] Reference Figure 13 Some embodiments of this application provide a video reconstruction apparatus 1300, which may include:

[0349] The acquisition unit 131 can be used to acquire representation data of the target video sequence;

[0350] The processing unit 132 can be used to obtain the information grid, the first recurrent neural network, and the reconstruction network corresponding to each video frame of the target video sequence based on the representation data of the target video sequence.

[0351] The reconstruction unit 133 can be used to perform the following steps, taking each video frame of the target video sequence as the current video frame, to obtain the reconstructed video frames corresponding to each video frame of the target video sequence:

[0352] Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained.

[0353] The reconstructed video frame corresponding to the current video frame is obtained based on the first hidden state of the current video frame and the reconstruction network.

[0354] In some embodiments, the reconstruction unit 133 is specifically configured to obtain the residual of the current video frame based on the information grid corresponding to the current video frame; superimpose the residual of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the features of the current video frame; and obtain the first hidden state of the current video frame based on the features of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0355] In some embodiments, the first recurrent neural network may include:

[0356] The update gate can be used to obtain the update gate coefficient based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0357] The reset gate can be used to obtain the reset gate coefficient based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network.

[0358] The first multiplier can be used to multiply the reset gate coefficients element-wise with the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the first feature;

[0359] The first adder can be used to superimpose the first feature with the feature of the current video frame to obtain the second feature;

[0360] A convolutional unit can be used to perform a convolution operation on the second feature to obtain the third feature;

[0361] The second adder can be used to superimpose the third feature with the update gate bias to obtain the fourth feature;

[0362] The activation unit can be used to perform a nonlinear transformation on the fourth feature to obtain the candidate hidden state;

[0363] The second multiplier can be used to multiply the hidden state of the previous video frame obtained by the first recurrent neural network with the update gate coefficient element-wise to obtain the fifth feature;

[0364] The third multiplier can be used to multiply the candidate hidden state element-wise with the complement of the update gate coefficient to obtain the sixth feature;

[0365] The second adder can be used to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0366] In some embodiments, the update gate may include:

[0367] The updated gate state convolutional layer can be used to perform convolution operations on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the updated gate hidden state.

[0368] The update gate feature convolutional layer can be used to perform convolution operations on the features of the current video frame to obtain the update gate features;

[0369] The update gate fusion layer can be used to overlay the update gate hidden state, the update gate features, and the update gate bias to obtain the update gate fusion features;

[0370] The update gate activation function layer can be used to perform nonlinear transformations on the update gate fusion features to obtain the update gate coefficients.

[0371] In some embodiments, the reset door may include:

[0372] The reset gate state convolutional layer can be used to perform convolution operations on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the reset gate hidden state.

[0373] The reset gate feature convolutional layer can be used to perform convolution operations on the features of the current video frame to obtain the reset gate features;

[0374] The reset door fusion layer can be used to overlay the reset door hidden state, the reset door features, and the reset door offset to obtain the reset door fusion features;

[0375] The reset gate activation function layer can be used to perform nonlinear transformations on the reset gate fusion features to obtain the reset gate coefficients.

[0376] In some embodiments, the network reconstruction may include:

[0377] Super-resolution networks can be used to process the first hidden state of the current video frame to obtain the reconstructed features of the current video frame;

[0378] The output module can be used to obtain the reconstructed video frame corresponding to the current video frame based on the reconstruction features of the current video frame.

[0379] In some embodiments, the network reconstruction may include:

[0380] Super-resolution networks can be used to process the first hidden state of the current video frame in order to obtain the reconstructed features of the current video frame.

[0381] The second recurrent neural network can be used to obtain the second hidden state of the current video frame based on the reconstruction features of the current video frame and the hidden state of the previous video frame obtained by the second recurrent neural network model.

[0382] The output module can be used to obtain the reconstructed video frame corresponding to the current video frame based on the second hidden state of the current video frame.

[0383] In some embodiments, the super-resolution network may include: NeRV blocks of neural representations of multi-level cascaded videos.

[0384] In some embodiments, the super-resolution network may include: a five-level cascaded NeRV block;

[0385] The upsampling factors of the five cascaded NeRV blocks are 25, 4, 4, 4, and 4, respectively.

[0386] In some embodiments, the NeRV block may include:

[0387] The convolutional layer can be used to perform upsampling convolution processing on the input features of the NeRV block to obtain the output features of the convolutional layer;

[0388] A pixel shuffling layer can be used to shuffle the output features of the convolutional layer to obtain the output features of the pixel shuffling layer.

[0389] The activation function layer can be used to perform nonlinear transformations on the output features of the pixel shuffling layer to obtain the output features of the NeRV block.

[0390] In some embodiments, the output module consists of a single convolutional layer.

[0391] The video reconstruction apparatus provided in the above embodiments can perform the video reconstruction method provided in any of the above embodiments, and the implementation principle and the achieved technical effect are the same. To avoid redundancy, it will not be described in detail again.

[0392] Some embodiments of this application provide an electronic device that may include: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the video representation method or the video reconstruction method described in any of the above embodiments.

[0393] Figure 14 An exemplary configuration block diagram of a display device 1400 from some of the above embodiments is shown. Figure 14 As shown, the display device 1400 may include at least one of the following: a tuner / demodulator 141, a communicator 142, a detector 143, an external device interface 144, a processor 145, a display 146, an audio output interface 147, a memory 148, a power supply 149, and a user interface.

[0394] In some embodiments, processor 145 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0395] The display 146 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the processor output, and a user interface for displaying video content, image content, menu control interface, and user control UI.

[0396] The display 146 may be a liquid crystal display, an organic light-emitting diode (OLED) display, or a projection display, etc.

[0397] The communicator 142 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 1400 can send and receive control signals and data signals with the control device or server through the communicator 142.

[0398] The user interface can be used to receive control signals input by the user through a control device (such as an infrared remote control) or by touch or gesture.

[0399] Detector 143 can be used to acquire signals from the external environment or to interact with the external environment. For example, detector 143 may include a light receiver, a sensor for acquiring ambient light intensity; or, detector 143 may include an image acquisition device, such as a camera, for acquiring external environmental scenes, user attributes, or user interaction gestures; or, detector 143 may include a sound acquisition device, such as a microphone, for receiving external sounds.

[0400] The external device interface 144 may include, but is not limited to, one or more of the following: High-Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), Composite Video Broadcast Signal (CVBS), Universal Serial Bus (USB), RGB (Red, Green, Blue) port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0401] The tuner / demodulator 141 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. In some embodiments, the processor 145 and the tuner / demodulator 141 may be located in different separate devices, that is, the tuner / demodulator 141 may also be located in an external device of the main device where the processor 145 is located, such as an external set-top box.

[0402] Processor 145 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Processor 145 controls the overall operation of display device 1400. For example, in response to receiving a user command to select a UI object to display on display 146, processor 145 can execute operations related to the object selected by the user command.

[0403] In some embodiments, processor 145 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0404] Some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video representation method or the video reconstruction method described in any of the above embodiments.

[0405] In a seventh aspect, some embodiments of this application provide a chip including a processor and a memory, the memory being used to store programs or instructions executable on the processor, and the processor being used to execute the programs or instructions to cause the video representation method or the video reconstruction method described in any of the above embodiments to be executed.

[0406] Eighthly, some embodiments of this application provide a computer program product that, when run on a computer, enables the computer to implement the video representation method or the video reconstruction method described in any of the above embodiments.

[0407] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0408] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A video representation method, characterized in that, include: Create an information grid corresponding to each video frame of the target video sequence; The following steps are performed iteratively, using each video frame of the target video sequence as the current video frame, until the training stop condition is met: Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained. Based on the first hidden state of the current video frame and the reconstruction network, obtain the reconstructed video frame corresponding to the current video frame; Based on the current video frame, the reconstructed video frame corresponding to the current video frame, and the preset loss function, at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network is adjusted. When the training stop condition is met, representation data of the target video sequence is generated based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network.

2. The method according to claim 1, characterized in that, The step of obtaining the first hidden state of the current video frame based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network includes: Obtain the residual of the current video frame based on the information grid corresponding to the current video frame; The residual of the current video frame is superimposed with the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the features of the current video frame. Based on the features of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained.

3. The method according to claim 2, characterized in that, The first recurrent neural network includes: The update gate is used to obtain the update gate coefficient based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network. The reset gate is used to obtain the reset gate coefficient based on the features of the current video frame and the hidden state of the previous video frame obtained by the first recurrent neural network. The first multiplier is used to multiply the reset gate coefficient element-wise with the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the first feature; The first adder is used to superimpose the first feature with the feature of the current video frame to obtain the second feature; A convolutional unit is used to perform a convolution operation on the second feature to obtain the third feature; The second adder is used to superimpose the third feature with the update gate bias to obtain the fourth feature; An activation unit is used to perform a nonlinear transformation on the fourth feature to obtain a candidate hidden state; The second multiplier is used to multiply the hidden state of the previous video frame obtained by the first recurrent neural network by the update gate coefficient element-wise to obtain the fifth feature; The third multiplier is used to multiply the candidate hidden state element-wise with the complement of the update gate coefficient to obtain the sixth feature; The third adder is used to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

4. The method according to claim 3, characterized in that, The update gate includes: The updated gate state convolutional layer is used to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the updated gate hidden state. The update gate feature convolutional layer is used to perform convolution operations on the features of the current video frame to obtain the update gate features; An update gate fusion layer is used to overlay the update gate hidden state, the update gate features, and the update gate bias to obtain update gate fusion features; An update gate activation function layer is used to perform a nonlinear transformation on the update gate fusion features to obtain the update gate coefficients.

5. The method according to claim 3, characterized in that, The reset door includes: The reset gate state convolutional layer is used to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain the reset gate hidden state. The reset gate feature convolutional layer is used to perform convolution operations on the features of the current video frame to obtain the reset gate features; A reset door fusion layer is used to overlay the reset door hidden state, the reset door features, and the reset door offset to obtain the reset door fusion features; The reset gate activation function layer is used to perform a nonlinear transformation on the reset gate fusion features to obtain the reset gate coefficients.

6. The method according to claim 1, characterized in that, The reconstructed network includes: Super-resolution networks are used to process the first hidden state of the current video frame in order to obtain the reconstructed features of the current video frame. The output module is used to obtain the reconstructed video frame corresponding to the current video frame based on the reconstruction features of the current video frame.

7. The method according to claim 11, characterized in that, The reconstructed network includes: Super-resolution networks are used to process the first hidden state of the current video frame in order to obtain the reconstructed features of the current video frame. The second recurrent neural network is used to obtain the second hidden state of the current video frame based on the reconstruction features of the current video frame and the hidden state of the previous video frame obtained by the second recurrent neural network model. The output module is used to obtain the reconstructed video frame corresponding to the current video frame based on the second hidden state of the current video frame.

8. The method according to claim 6 or 7, characterized in that, The super-resolution network includes: NeRV blocks, which are cascaded neural representations of videos.

9. The method according to claim 8, characterized in that, The super-resolution network includes: five cascaded NeRV blocks; The upsampling factors of the five cascaded NeRV blocks are 25, 4, 4, 4, and 4, respectively.

10. The method according to claim 8, characterized in that, The NeRV block includes: A convolutional layer is used to perform upsampling convolution processing on the input features of the NeRV block to obtain the output features of the convolutional layer; A pixel shuffling layer is used to shuffle the output features of the convolutional layer to obtain the output features of the pixel shuffling layer. An activation function layer is used to perform a nonlinear transformation on the output features of the pixel shuffling layer to obtain the output features of the NeRV block.

11. The method according to claim 6 or 7, characterized in that, The output module consists of a single convolutional layer.

12. The method according to claim 7, characterized in that, The network structure of the second recurrent neural network is the same as that of the first recurrent neural network.

13. The method according to claim 11, characterized in that, The preset loss function is: Where T is the number of video frames in the target video sequence, f θ (t) represents the reconstructed video frame of the current video frame, v t For the current video frame, α is a hyperparameter, ||f θ (t)-v t ||1 is f θ (t) and v t The L1 loss value, SSIM(f θ (t), v t ) is f θ (t) and v t The SSIM loss value is the structural similarity index.

14. A video reconstruction method, characterized in that, include: Obtain the representation data of the target video sequence; Based on the representation data of the target video sequence, obtain the information grid, the first recurrent neural network, and the reconstruction network corresponding to each video frame of the target video sequence; The following steps are performed, taking each video frame of the target video sequence as the current video frame, to obtain the reconstructed video frames corresponding to each video frame of the target video sequence: Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained. The reconstructed video frame corresponding to the current video frame is obtained based on the first hidden state of the current video frame and the reconstruction network.

15. A video display device, characterized in that, include: The module is used to create information grids corresponding to each video frame of the target video sequence; The training module is used to iteratively execute the following steps, taking each video frame of the target video sequence as the current video frame, until the training stop condition is met: Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained. Based on the first hidden state of the current video frame and the reconstruction network, obtain the reconstructed video frame corresponding to the current video frame; Based on the current video frame, the reconstructed video frame corresponding to the current video frame, and the preset loss function, at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network, and the parameters of the reconstruction network is adjusted. The generation module is used to generate representation data of the target video sequence based on the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network when the training stop condition is met.

16. A video reconstruction apparatus, characterized in that, include: Acquisition unit, used for representation data of the target video sequence; The processing unit is configured to obtain, based on the representation data of the target video sequence, the information grid, the first recurrent neural network, and the reconstruction network corresponding to each video frame of the target video sequence; The reconstruction unit is configured to take each video frame of the target video sequence as the current video frame and perform the following steps to obtain the reconstructed video frames corresponding to each video frame of the target video sequence: Based on the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame obtained by the first recurrent neural network, the first hidden state of the current video frame is obtained. The reconstructed video frame corresponding to the current video frame is obtained based on the first hidden state of the current video frame and the reconstruction network.

17. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the method as described in any one of claims 1-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to perform the method according to any one of claims 1-14.

19. A chip, characterized in that, The chip includes a processor and a memory, the memory being used to store programs or instructions that can run on the processor, and the processor being used to execute the programs or instructions to cause the method of any one of claims 1-14 to be performed.

Images