Video representation method, video reconstruction method, and apparatus

Abstract

The present application relates to the technical field of video encoding / decoding, and provides a video representation method, a video construction method, and an apparatus. The video representation method comprises: creating an information grid corresponding to each video frame of a target video sequence; and cyclically using each video frame as a current video frame to execute the following steps until a training stop condition is met: acquiring a first hidden state on the basis of the information grid corresponding to the current video frame, a first recurrent neural network, and a hidden state of a video frame previous to the current video frame; acquiring a reconstructed video frame on the basis of the first hidden state and a reconstruction network; on the basis of the current video frame, the reconstructed video frame, and a preset loss function, adjusting at least one of the parameter of the information grid, the parameter of the first recurrent neural network, and the parameter of the reconstruction network; and when the training stop condition is met, generating representation data of the target video sequence on the basis of the information grid corresponding to each video frame, the first recurrent neural network, and the reconstruction network. Some embodiments of the present application are used for improving the representation efficiency of videos.

Description

Video representation method, video reconstruction method and device

[0001] The present application claims priority to the Chinese patent application No. 202410993701.3, filed on July 23, 2024, and entitled "Video representation method, video reconstruction method and device", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] Some embodiments of the present application relate to the technical field of video coding. More specifically, the present application relates to a video representation method, a video reconstruction method and a device. BACKGROUND

[0003] Neural Representations for Videos (NeRV) is a method of parameterizing a video, which learns a feature of a video frame with a time index t through a Multilayer Perceptron (MLP) network, and then sends the feature to a reconstruction network composed of convolution and upsampling modules, and finally obtains a reconstructed frame. Through this method, the video is fitted into a neural network model.

[0004] NeRV can represent a video as a parameter of a neural network, convert video coding into fitting a video into a neural network model, and thus convert the video compression problem into a model compression problem. However, a video is usually composed of a series of consecutive video frames, adjacent video frames are very similar in vision, and the video frame can also contain many repeated patterns or textures, so the video has a lot of redundant information in the time dimension and the space dimension. However, NeRV only fits the video frame and the index together through the MLP network, and cannot effectively utilize the temporal and spatial redundancy information of the video. Moreover, the parameter amount of the MLP network is large, and thus the efficiency of the video representation needs to be improved. SUMMARY

[0005] Exemplary embodiments of the present application provide a video representation method, a video reconstruction method and a device for improving the efficiency of video representation.

[0006] Some embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, some embodiments of the present application provide a video representation method, comprising:

[0008] creating an information grid corresponding to each video frame of a target video sequence;

[0009] cycling through each video frame of the target video sequence as a current video frame to execute the following steps until a stop training condition is met:

[0010] obtain a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network;

[0011] obtain a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network;

[0012] adjust at least one of the information grid corresponding to the current video frame, parameters of the first recurrent neural network, and parameters of the reconstruction network according to the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function;

[0013] when the stop training condition is met, generate representation data of the target video sequence according to the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network.

[0014] In a second aspect, some embodiments of the present application provide a video reconstruction method, comprising:

[0015] obtain representation data of a target video sequence;

[0016] obtain an information grid corresponding to each video frame of the target video sequence, a first recurrent neural network, and a reconstruction network according to the representation data of the target video sequence;

[0017] respectively take each video frame of the target video sequence as a current video frame to execute the following steps to obtain a reconstructed video frame corresponding to each video frame of the target video sequence:

[0018] obtain a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network;

[0019] obtain a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network.

[0020] In a third aspect, some embodiments of the present application provide a video representation device, comprising:

[0021] a creating module configured to create an information grid corresponding to each video frame of a target video sequence;

[0022] a training module configured to execute the following steps by taking each video frame of the target video sequence as a current video frame in a loop until a stop training condition is met:

[0023] obtain a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network;

[0024] obtain a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network;

[0025] adjust at least one of the information grid corresponding to the current video frame, parameters of the first recurrent neural network, and parameters of the reconstruction network according to the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function;

[0026] generate, when the stop training condition is met, representation data of the target video sequence according to the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network.

[0027] In a fourth aspect, some embodiments of the present application provide a video reconstruction apparatus, comprising:

[0028] an obtaining unit configured to obtain representation data of a target video sequence;

[0029] a processing unit configured to obtain, according to the representation data of the target video sequence, an information grid corresponding to each video frame of the target video sequence, a first recurrent neural network, and a reconstruction network;

[0030] a reconstruction unit configured to perform the following steps on each video frame of the target video sequence as a current video frame to obtain a reconstructed video frame corresponding to each video frame of the target video sequence:

[0031] obtain a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network;

[0032] obtain a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network.

[0033] In a fifth aspect, some embodiments of the present application provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the video representation method of the first aspect or the video reconstruction method of the second aspect.

[0034] In a sixth aspect, some embodiments of the present application provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a computing device, causes the computing device to implement the video representation method of the first aspect or the video reconstruction method of the second aspect.

[0035] In a seventh aspect, some embodiments of the present application provide a chip, comprising a processor and a memory, the memory being configured to store a program or instructions executable on the processor, and the processor being configured to execute the program or instructions, so that the video representation method of the first aspect or the video reconstruction method of the second aspect is executed.

[0036] In an eighth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, causes the computer to implement the video representation method of the first aspect or the video reconstruction method of the second aspect.

[0037] According to the above technical solutions, the video representation method provided by some embodiments above can first create information grids corresponding to each video frame of a target video sequence, then loop to take each video frame of the target video sequence as a current video frame, and obtain a first hidden state of the current video frame according to the information grid corresponding to the current video frame, a first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network, obtain a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and a reconstruction network, and adjust at least one of the information grid corresponding to the current video frame, parameters of the first recurrent neural network, and parameters of the reconstruction network according to the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function. When a stop training condition is met, representation data of the target video sequence is generated according to the information grids corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network. Since the video representation method provided by some embodiments above is based on a recurrent neural network for video representation, the recurrent neural network has a memory function and can retain information through its recurrent structure, thereby capturing long-term dependencies in a video sequence, so some embodiments above can effectively utilize the spatiotemporal redundancy information of the video. Compared with NeRV, which cannot effectively utilize the spatiotemporal redundancy information of the video, and the MLP network, which has a large number of parameters, some embodiments above can effectively utilize the spatiotemporal redundancy information of the video without using the MLP network, so some embodiments above can improve the representation efficiency of the video. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the implementation manners in the related art, the accompanying drawings needed to be used in the embodiments or related art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0039] FIG. 1 shows a structural schematic diagram of a network model of a video representation method and a video reconstruction method according to some embodiments;

[0040] FIG. 2 shows a structural schematic diagram of a first recurrent neural network according to some embodiments;

[0041] FIG. 3 shows a structural schematic diagram of a super-resolution network according to some embodiments;

[0042] FIG. 4 shows a structural schematic diagram of a NeRV block according to some embodiments;

[0043] FIG. 5 shows a structural schematic diagram of a network model of a video representation method and a video reconstruction method according to some embodiments;

[0044] FIG. 6 shows a step flowchart of a video representation method according to some embodiments;

[0045] FIG. 7 shows a structural schematic diagram of a network model of a video representation method according to some embodiments;

[0046] FIG. 8 shows a structural schematic diagram of a network model of a video representation method according to some embodiments;

[0047] FIG. 9 shows a structural schematic diagram of a network model of a video representation method according to some embodiments;

[0048] FIG. 10 shows a structural schematic diagram of a network model of a video representation method according to some embodiments;

[0049] FIG. 11 shows a step flowchart of a video reconstruction method according to some embodiments of the present application;

[0050] FIG. 12 shows a structural schematic diagram of a video representation apparatus according to some embodiments of the present application;

[0051] FIG. 13 shows a step flowchart of a video reconstruction apparatus according to some embodiments of the present application;

[0052] FIG. 14 shows a hardware structural schematic diagram of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0053] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and in order to make the purpose and embodiments of the present application more clear, the exemplary embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, not all the embodiments.

[0054] The embodiments of the present application are not intended to be limited. Unless otherwise specified, these terms should be understood in accordance with their ordinary and customary meanings.

[0055] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclusively include, for example, a product or device that contains a list of components without necessarily being limited to the components clearly listed, but can include other components not clearly listed or inherent to such products or devices.

[0056] The description in the specification refers to "some implementations", "some embodiments" and the like, indicating that the described implementation or embodiment can include a specific feature, structure or property, but can not necessarily include this specific feature, structure or property in each embodiment. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure or property is described in connection with an embodiment, it is considered that it is within the knowledge of those skilled in the art to realize such feature, structure or property in connection with other implementations, whether described herein or not.

[0057] Implicit Neural Representation (INR) is a method of parameterizing signals. Traditional signal representation is usually discrete, for example: an image is a discrete grid of pixels, an audio signal is a discrete sample of amplitude, and a three-dimensional shape is usually parameterized as a voxel, a point cloud or a mesh; while INR can parameterize signals as continuous functions. Since INR can parameterize signals as continuous functions, the memory required to parameterize signals is independent of spatial resolution, and is only proportional to the complexity of the hidden signal, and can be sampled at any spatial resolution to obtain more detailed signal details.

[0058] 3D Neural Radiance Fields (3D NeRF) is a computer vision technology that can be used to generate high-quality three-dimensional reconstruction models. 3D NeRF uses deep learning technology to extract the geometric shape and texture information of an object from multiple views of images, and then uses this information to generate a continuous three-dimensional radiation field that can describe the color and density of any point in the scene in any direction, so as to present a highly realistic three-dimensional model at any angle and distance.

[0059] Inspired by INR and 3DNeRF, the INR representation technique is applied to the field of 2D video and has been widely researched and developed. NeRV is one of the research results of applying INR technology to 2D video.

[0060] NeRV represents video as the parameters of a neural network, converts video encoding into fitting video into a neural network model, and converts video compression into model compression. The performance of NeRV can be comparable to that of conventional video compression methods. The proposal of NeRV provides a new perspective and solution for video representation, compression and other video-related tasks. Compared with traditional video compression methods and learning-based video compression methods, this method has a simpler training process and higher decoding speed.

[0061] In NeRV, the time index t of the video frame is learned by a multilayer perceptron (MLP) network to obtain a feature, and then the feature is sent to a subsequent reconstruction network composed of convolution and upsampling modules, and finally a reconstructed frame is obtained. By this method, the video is fitted into the entire neural network model. Specifically, first, the index t of each time frame is normalized, and the normalized value is processed by a position encoding formula to obtain a set of vectors similar to Fourier transform, which is used to preserve the high-frequency information brought by the change of the index with time. Then, the vector is input as the feature of the current time frame into the subsequent reconstruction network.

[0062] A video is usually composed of a series of continuous video frames (images), and adjacent video frames are very similar in vision. Therefore, there is a lot of redundant information in the time dimension of the video. In addition, a video frame may also contain many repeated patterns or textures, and there is also a lot of redundant information in the spatial dimension of the video. However, NeRV only fits the video frame and the index together through the MLP network, and cannot effectively utilize the temporal and spatial redundancy information of the video. At the same time, the parameter amount of the MLP network is large, so the efficiency of the neural representation of the video needs to be improved.

[0063] In order to improve the efficiency of the video representation, some embodiments of the present application provide the following technical solutions:

[0064] FIG. 1 is a structural schematic diagram of a network model that can be used to execute a video representation method and a video reconstruction method according to some embodiments of the present application.

[0065] Referring to FIG. 1, in some embodiments, the network model that can be used to execute a video representation method and a video reconstruction method can include: residual grids 11 corresponding to each video frame of a video sequence.

[0066] The residual grid 11 can be used to save the residual of a video frame. The residual refers to the difference between the original video frame and the predicted frame (or referred to as the reference frame). In the process of representing a video, if the features of each video frame are represented, a large number of network parameters will be brought, which is not conducive to compression. Therefore, some embodiments of the present application represent the information of each video frame by sparse residual, and replace the original frame information grid representing the video with a learnable residual grid, so that the features of the video frame are more sparse in space, thereby reducing the amount of data to be transmitted or stored.

[0067] In some embodiments, a small-size multi-channel learnable information grid can be modeled for each video frame of a video sequence, and the parameters in the information grid are continuously optimized through gradient backpropagation during network training, so as to obtain the sparse residual of the final video relative to the long-term dependence features established by the recurrent network. The learnable grid stores multi-channel residuals corresponding to the spatial positions at each vertex, which has a spatial positional relationship compared with the MLP network, and is conducive to the learning and fitting of the 2D signal of the video frame. Then, the information grid representing the residual is input into the subsequent network structure for feature processing of the current video frame.

[0068] Referring to FIG. 1, in some embodiments, the network model for performing the video representation method and the video reconstruction method can include a feature fusion module 12. The feature fusion module 12 can be used to add and fuse the residual of the current video frame represented by the residual grid and the hidden state of the previous video frame (hereinafter referred to as “previous video frame”) of the current video frame obtained by the first recurrent neural network, to obtain the feature of the current video frame.

[0069] The residual of the current video frame is represented as R t The hidden state of the previous video frame obtained by the first recurrent neural network is represented as The feature of the current video frame is represented as The operation performed by the feature fusion module 12 can be represented as:

[0070] According to the characteristics of the recurrent neural network, the recurrent neural network can learn the hidden state containing the space-time information of the entire video sequence. In the reconstruction of the current video frame, the hidden state h t-1 of the previous video frame contains the space-time relationship of all frames before the current video frame, and after adding the residual, the feature of the current video frame can be represented, and the space-time information between the video sequences is effectively utilized.

[0071] Referring to FIG. 1, in some embodiments, the network model for performing the video representation method and the video reconstruction method can comprise a first recurrent neural network 13. The first recurrent neural network 13 can be configured to obtain a first hidden state of a current video frame according to a feature of the current video frame and a hidden state of the current video frame obtained by the first recurrent neural network.

[0072] A recurrent neural network (RNN) is a neural network architecture suitable for processing sequence data. The recurrent neural network has a memory function and can retain information through its recurrent structure, so that the recurrent neural network can capture long-term dependencies in a video sequence.

[0073] As described above, NeRV only fits the video frames and the index together through the MLP network, and there is a problem that the spatiotemporal redundancy information of the video cannot be effectively utilized and the parameter amount of the MLP network is large. In some embodiments described above, the video is represented based on the recurrent neural network, so the video sequence can be processed by the time series learning ability of the recurrent neural network in some embodiments described above, so as to effectively utilize the redundancy spatiotemporal information of the video. At the same time, since the MLP network is avoided, the parameter amount of the network model can also be reduced.

[0074] In some embodiments, the first recurrent neural network can be a long short-term memory (LSTM).

[0075] In some embodiments, the first recurrent neural network 13 can be a gated recurrent unit (GRU).

[0076] The GRU is a special recurrent neural network that controls the flow of information through a gating mechanism, which can include two gating mechanisms, an update gate and a reset gate. The working principle of the GRU is that the current input, the previous hidden state and the output of the two gates are combined at each timestamp to calculate the current hidden state to realize a recurrent update process, which has the characteristics of fast training speed and small memory occupation.

[0077] In some embodiments, the first recurrent neural network 13 can be a gated recurrent unit obtained by replacing the linear layer in the GRU with a convolutional layer, denoted as ConvGRU.

[0078] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can comprise an update gate 201. The update gate 201 can be configured to obtain an update gate coefficient according to a hidden state of a previous video frame obtained by the first recurrent neural network and a feature of a current video frame.

[0079] Referring to FIG. 2, in some embodiments, the update gate 201 can include an update gate state convolutional layer 2011. The update gate state convolutional layer 2011 can be configured to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain an update gate hidden state.

[0080] Let the hidden state of the previous video frame obtained by the first recurrent neural network be denoted as h The update gate hidden state is denoted as a zh The operation performed by the update gate state convolutional layer 2011 can be represented as:

[0081] In some embodiments, the update gate hidden state a zh has the same size as the hidden state h t-1 of the previous video frame obtained by the first recurrent neural network.

[0082] In the training process of the network model, the weight coefficients W zh of the convolution kernel of the update gate state convolutional layer 2011 are updated through backpropagation of the network. The weight coefficients W zh of the convolution kernel of the update gate state convolutional layer 2011 are used to represent the influence of different dimensions of the hidden state h t-1 of the previous video frame obtained by the first recurrent neural network on the output (update gate hidden state a zh ) of the update gate state convolutional layer 2011.

[0083] Referring to FIG. 2, in some embodiments, the update gate 201 can also include an update gate feature convolutional layer 2012. The update gate feature convolutional layer 2012 can be configured to perform a convolution operation on the features of the current video frame to obtain update gate features.

[0084] Let the features of the current video frame be denoted as f The update gate features are denoted as a zf The operation performed by the update gate feature convolutional layer 2012 can be represented as:

[0085] In some embodiments, the update gate features a zf have the same size as the features f of the current video frame.

[0086] In the training process of the network model, the weight coefficients W zf of the convolution kernel of the update gate feature convolutional layer 2012 are also updated through backpropagation of the network. The weight coefficients W zfFeatures for characterizing a current video frame the influence degree of different dimensions of the features for the current video frame on the output (reset gate hidden state zh ) of the reset gate state convolutional layer 2021.

[0087] Referring to FIG. 2, in some embodiments, the reset gate 202 can further include a reset gate activation function layer 2024. The reset gate activation function layer 2024 can be configured to perform a non-linear transformation on the reset gate fusion features to obtain the reset gate coefficient.

[0088] Let the reset gate hidden state be denoted as zh , the reset gate features be denoted as zf , the reset gate bias be denoted as z , and the reset gate fusion features be denoted as z , the operation performed by the reset gate activation function layer 2024 can be represented as:

[0089] Referring to FIG. 2, in some embodiments, the reset gate 202 can further include a reset gate activation function layer 2024. The reset gate activation function layer 2024 can be configured to perform a non-linear transformation on the reset gate fusion features to obtain the reset gate coefficient.

[0090] In some embodiments, the activation function of the reset gate activation function layer 2024 can be a sigmoid activation function, so that the value of the reset gate coefficient is between 0 and 1.

[0091] Let the reset gate fusion features be denoted as z , the reset gate coefficient be denoted as t , and the non-linear transformation operation of the reset gate activation function layer be denoted as z (), the operation performed by the reset gate activation function layer 2024 can be represented as:

[0092] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a reset gate 202. The reset gate 202 can be configured to obtain a reset gate coefficient for a current video frame based on a hidden state of a previous video frame obtained by the first recurrent neural network and features for the current video frame.

[0093] Referring to FIG. 2, in some embodiments, the reset gate 202 can include a reset gate state convolutional layer 2021. The reset gate state convolutional layer 2021 can be configured to perform a convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain a reset gate hidden state.

[0094] Let the hidden state of the previous video frame obtained by the first recurrent neural network be denoted as The reset gate hidden state is denoted as a rh The operation performed by the reset gate state convolutional layer 2021 can be represented as:

[0095] In some embodiments, the reset gate hidden state a rh is of the same size as the hidden state h t-1 of the previous video frame obtained by the first recurrent neural network.

[0096] In the training process of the network model, the weight coefficients W rh of the convolutional kernel of the reset gate state convolutional layer 2021 are updated through backpropagation of the network. The weight coefficients W rh of the convolutional kernel of the reset gate state convolutional layer 2021 are used to represent the influence of different dimensions of the hidden state h t-1 of the previous video frame obtained by the first recurrent neural network on the output (the reset gate hidden state a rh ) of the reset gate state convolutional layer 2021.

[0097] Referring to FIG. 2, in some embodiments, the reset gate 202 can further include a reset gate feature convolutional layer 2022. The reset gate feature convolutional layer 2022 can be used to perform convolutional operation on the features of the current video frame to obtain reset gate features.

[0098] The features of the current video frame are denoted as The reset gate features are denoted as a rf The operation performed by the reset gate feature convolutional layer 2022 can be represented as:

[0099] Referring to FIG. 2, in some embodiments, the reset gate 202 can further include a reset gate fusion layer 2023. The reset gate fusion layer 2023 can be used to add and fuse the reset gate hidden state, the reset gate features, and the reset gate bias to obtain reset gate fusion features.

[0100] The reset gate hidden state is denoted as a rh The reset gate features are denoted as a rf The reset gate bias is denoted as b r The reset gate fusion features are denoted as F r The operation performed by the reset gate fusion layer 2023 can be represented as:

[0101] Referring to FIG. 2, in some embodiments, the reset gate 202 can further include a reset gate activation function layer 2024. The reset gate activation function layer 2024 can be used to perform nonlinear transformation on the reset gate fusion features to obtain the reset gate coefficients.

[0102] In some embodiments, the activation function of the reset gate activation function layer 2024 can be a sigmoid activation function, so that the value of the reset gate coefficient is between 0 and 1.

[0103] The reset gate fusion feature is represented as F r The reset gate coefficient is represented as r t The nonlinear transformation operation of the reset gate activation function layer is represented as σ r (), the operation performed by the reset gate activation function layer 2024 can be represented as:

[0104] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a first multiplier 203. The first multiplier 203 can be configured to multiply the reset gate coefficient and the hidden state of the previous video frame obtained by the first recurrent neural network element by element to obtain a first feature.

[0105] The reset gate coefficient is represented as r t The first feature is represented as A, the element-wise multiplication is represented as, and the hidden state of the previous video frame obtained by the first recurrent neural network is represented as The operation performed by the first multiplier 203 can be represented as:

[0106] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a first adder 204. The first adder 204 can be configured to superimpose the first feature and the feature of the current video frame to obtain a second feature.

[0107] The first feature is represented as A, and the feature of the current video frame is represented as The second feature is represented as B, and the operation performed by the first adder 204 can be represented as:

[0108] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a convolution unit 205. The convolution unit 205 can be configured to perform a convolution operation on the second feature to obtain a third feature.

[0109] The second feature is represented as B, and the third feature is represented as C. The operation performed by the convolution unit 205 can be represented as:

[0110] In some embodiments, the third feature has the same size as the second feature.

[0111] In the training process of the network model, the weight coefficient W hhThe convolutional unit 205 is updated through back propagation of the network model. The weight coefficient W of the convolution kernel of the convolutional unit 205 is updated according to the following formula: W = W - lr * dL / dW hh The convolutional unit 205 is updated through back propagation of the network model. The weight coefficient W of the convolution kernel of the convolutional unit 205 is updated according to the following formula: W = W - lr * dL / dW

[0112] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a second adder 206. The second adder 206 can be configured to superimpose the third feature and an update gate bias to obtain a fourth feature.

[0113] The third feature is denoted as C, the fourth feature is denoted as D, and the update gate bias is denoted as b z The operation performed by the second adder 206 can be represented as follows:

[0114] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include an activation unit 207. The activation unit 207 can be configured to perform a nonlinear transformation on the fourth feature to obtain a candidate hidden state.

[0115] In some embodiments, the activation function of the activation unit 207 is a hyperbolic tangent function.

[0116] The candidate hidden state is denoted as The fourth feature is denoted as D, and the nonlinear transformation operation of the activation unit 207 is denoted as tanh(). The operation performed by the activation unit 207 can be represented as follows:

[0117] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a second multiplier 208. The second multiplier 208 can be configured to multiply the hidden state of the previous video frame obtained by the first recurrent neural network and an update gate coefficient element by element to obtain a fifth feature.

[0118] The hidden state of the previous video frame obtained by the first recurrent neural network is denoted as The update gate coefficient is denoted as z t The fifth feature is denoted as E. The operation performed by the second multiplier 208 can be represented as follows:

[0119] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a third multiplier 209. The third multiplier 209 can be configured to multiply the candidate hidden state and the complement of the update gate coefficient element by element to obtain a sixth feature.

[0120] The candidate hidden state is denoted as The update gate coefficient is denoted as z t The sixth feature is denoted as F, and the operation performed by the third multiplier 209 can be represented as:

[0121] Referring to FIG. 2, in some embodiments, the first recurrent neural network 13 can include a third adder 210. The third adder 210 can be configured to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0122] The first hidden state of the current video frame is denoted as h t The fifth feature is denoted as E, the sixth feature is denoted as F, and the operation performed by the third adder 210 can be represented as:

[0123] Referring to FIG. 1, in some embodiments, the network model for performing the video representation method and the video reconstruction method can include a super-resolution network 14. The super-resolution network 14 can be configured to process the first hidden state of the current video frame to obtain a reconstructed feature of the current video frame.

[0124] The first hidden state of the current video frame is denoted as The reconstructed feature of the current video frame is denoted as f t C,H,W The operation of the super-resolution network 14 is denoted as NeRV(), and the operation performed by the super-resolution network 14 can be represented as:

[0125] Referring to FIG. 3, in some embodiments, the super-resolution network 14 is composed of a plurality of cascaded NeRV blocks 30. In FIG. 3, the super-resolution network 14 is exemplified by being composed of five cascaded NeRV blocks, but in other embodiments, the super-resolution network 14 can also be composed of other numbers of NeRV blocks, for example, the super-resolution network 14 can also be composed of four cascaded NeRV blocks, and for another example, the super-resolution network 14 can also be composed of six cascaded NeRV blocks. The more the number of NeRV blocks that compose the super-resolution network 14, the smaller the up-sampling multiple of a single NeRV block can be, the relatively smoother the change of the feature is, and the problems such as jaggies or blurring are less likely to occur, but the calculation amount of the up-sampling is relatively large, and the model parameters of the super-resolution network 14 will also increase accordingly; on the contrary, the fewer the number of NeRV blocks that compose the super-resolution network 14, the larger the up-sampling multiple of a single NeRV block needs to be, the calculation amount of the up-sampling is relatively small, and the model parameters of the super-resolution network 14 will also decrease accordingly, but the change of the feature is relatively sharp, and the problems such as jaggies or blurring are more likely to occur, so in actual use, the number of NeRV blocks that compose the super-resolution network 14 can be set by comprehensively considering the calculation amount of the up-sampling, the amount of model parameters of the super-resolution network, and the quality requirement of the reconstructed video frame.

[0126] In some embodiments, each of the plurality of NeRV blocks 30 constituting the super-resolution network 14 is configured to up-sample the input features in the feature channel dimension, and the up-sampling factor depends on the factor by which the input features need to be up-sampled in the spatial dimension. For example, if the input features need to be up-sampled by a factor of s in both the height and the width, the NeRV block 30 can be configured to up-sample the input features by a factor of s in the feature channel dimension. 2 For example, if the input features need to be up-sampled by a factor of s1 in the height and by a factor of s2 in the width, the NeRV block 30 can be configured to up-sample the input features by a factor of s1*s2 in the feature channel dimension.

[0127] In some embodiments, the super-resolution network 14 is composed of five cascaded NeRV blocks, each of which is configured to up-sample the input features in the feature channel dimension, and the up-sampling factors are 25, 4, 4, 4, and 4 in sequence.

[0128] The super-resolution network 14 is stacked with a plurality of NeRV blocks, and different positions of pixels can share the convolution kernels in the NeRV blocks, so that the super-resolution network can efficiently reconstruct the video frames.

[0129] Referring to FIG. 4, in some embodiments, the NeRV block can include a convolution layer 41. The convolution layer 41 can be configured to perform up-sampling convolution processing on the input features of the NeRV block to obtain output features of the convolution layer 41. Wherein, when the NeRV block is a first-level NeRV block, the input features of the NeRV block are the first hidden state of the current video frame When the NeRV block is an n-level NeRV block, the input features of the NeRV block are the output features of the (n-1)-level NeRV block, and n is an integer greater than or equal to 2.

[0130] Let the input features of the NeRV block be denoted as f t c,h,w , and the output features of the convolution layer 41 be denoted as The up-sampling convolution processing is denoted as Conv(), and the operation performed by the convolution layer 41 can be represented as:

[0131] Wherein, the values of S for the first to fifth-level NeRV blocks are 5, 2, 2, 2, and 2 in sequence.

[0132] Some of the above embodiments gradually up-sample the small-size features through five NeRV blocks, and at the same time, the number of feature channels is also constantly increased in the convolution process. Therefore, some of the above embodiments can make the feature map contain more information, thereby improving the quality of the reconstructed video frames.

[0133] Referring to FIG. 4, in some embodiments, the NeRV block can include a pixel shuffle layer 42. The pixel shuffle layer 42 can be configured to perform a pixel shuffle operation on the output feature of the convolution layer 41 to obtain the output feature of the pixel shuffle layer 42.

[0134] The output feature of the convolution layer 41 is denoted as f The pixel shuffle operation is denoted as Pixelshuffle(), and the output feature of the pixel shuffle layer 42 is denoted as f t c,h*s,w*s The operation performed by the pixel shuffle layer 42 can be denoted as:

[0135] Referring to FIG. 4, in some embodiments, the NeRV block can include an activation function layer 43. The activation function layer 43 can be configured to perform a nonlinear transformation on the output feature of the pixel shuffle layer 42 to obtain the output feature of the NeRV block.

[0136] In some embodiments, the activation function of the activation function layer 43 is a sigmoid activation function.

[0137] The output feature of the pixel shuffle layer 42 is denoted as f t c,h*s,w*s The output feature of the NeRV block is denoted as f t C,H,W The nonlinear transformation operation of the activation function layer 43 is denoted as σ(), and the operation performed by the activation function layer 43 can be denoted as:

[0138] Referring to FIG. 1, in some embodiments, the network model for performing the video representation method and the video reconstruction method can include a second recurrent neural network 15.

[0139] In some embodiments, the second recurrent neural network 15 can also be a GRU.

[0140] The structure of the second recurrent neural network 15 can be the same as that of the first recurrent neural network 13 shown in FIG. 2, and thus will not be described in detail here.

[0141] The difference between the operations performed by the second recurrent neural network 15 and the first recurrent neural network 13 lies in that the input of the first recurrent neural network 13 is the residual error R t of the current video frame and the hidden state h of the previous video frame obtained by the first recurrent neural network, while the input of the second recurrent neural network 15 is the reconstructed feature f t C,H,W of the current video frame and the hidden state h

[0142] Let the reconstructed feature of the current video frame be denoted as f t C,H,W Let the hidden state of the previous video frame obtained by the second recurrent neural network 15 be denoted as Let the second hidden state of the current video frame output by the second recurrent neural network 15 be denoted as Let the processing operation of the second recurrent neural network 15 be denoted as ConvGRU_2(), then the operation performed by the second recurrent neural network 15 can be denoted as:

[0143] The super-resolution network 14 only processes the features of the current video frame, so that the reconstructed video frame does not take into account the temporal correlation between video frames as a sequence, therefore, some embodiments described above are provided behind the super-resolution network 14, a recurrent neural network (second recurrent neural network) is set up, which can establish long-time dependence between video frames, compensate for the space-time detail information between video frames, obtain more temporal correlation between adjacent video frames, and thus improve the quality of the reconstructed video frame.

[0144] Referring to FIG. 1, in some embodiments, the network model for performing the video representation method and the video reconstruction method can include an output module 16. The output module 16 can be used to reconstruct the current video frame according to the second hidden state of the current video frame

[0145] Let the reconstruction operation of the output module 16 be denoted as hand(), and the reconstructed video frame of the current video frame be denoted as Then the operation performed by the output module 16 can be denoted as:

[0146] In some embodiments, the output module 16 can be a convolutional layer.

[0147] In some embodiments, the video representation method based on the network model shown in FIG. 1 can include the following steps a to i:

[0148] Step a, constructing a residual grid corresponding to each video frame in the video sequence, and initializing the residual grid corresponding to each video frame.

[0149] In some embodiments, the height of the residual grid corresponding to each video frame can be 9, the width can be 16, and the number of feature dimensions stored at each grid vertex can be C0. That is, the residual grid corresponding to the video frame with index t can be denoted as

[0150] ​In some embodiments, initializing the residual grid corresponding to each video frame may include: initializing the parameters in the residual grid to 0.

[0151] Step b: Obtain the residual corresponding to the current video frame.

[0152] During the training of the network model, given the temporal index of a video frame in a video sequence, the corresponding residual feature of dimension C0*9*16 can be found. This is used to represent the residual of the current frame. The residual feature is then fed into the subsequent grid structure for further feature extraction and reconstruction.

[0153] Step c: Obtain the hidden state of the previous video frame from the first recurrent neural network. The residual R with the current video frame t Add them together to obtain the features of the current video frame.

[0154] That is, the operation represented by the above equation (1) is performed through the feature fusion module.

[0155] It should be noted that when performing step c above on the first video frame of the video sequence (the current video frame is the first video frame of the video sequence), since the hidden state of the previous video frame obtained by the first recurrent neural network does not exist at this time, a learnable grid can be used. Let this represent the hidden state of the previous video frame obtained by the first recurrent neural network, and let this learnable grid be... It will be updated via backpropagation during subsequent training.

[0156] In some embodiments, the initial update gate bias b of the first recurrent neural network z and initial reset gate bias b r All parameters are non-zero. During residual mesh initialization, the parameters in the residual mesh are initialized to 0, so the first recurrent neural network can update the gate bias b. z and reset door offset b r It is initialized to a non-zero parameter to prevent the network model from failing to learn useful information when the input is zero, thus slowing down the learning speed.

[0157] Step d: Based on the characteristics of the current video frame The hidden state of the previous video frame obtained by the first recurrent neural network. Get the first hidden state of the current video frame

[0158] That is, the operations represented by equations (2) to (17) above are executed through the first recurrent neural network.

[0159] Step e, obtaining a first hidden state of the current video frame according to the first hidden state of the previous video frame Step f, obtaining a reconstructed feature f of the current video frame t C,H,W .

[0160] That is, the operations represented by the above formulas (18)-(21) are performed by the super-resolution network.

[0161] Step f, obtaining a first hidden state of the current video frame according to the first hidden state of the previous video frame t C,H,W and the hidden state of the previous video frame obtained by the second recurrent neural network Step g, obtaining a second hidden state of the current video frame according to the first hidden state of the current video frame

[0162] That is, the operation represented by the above formula (22) is performed by the second recurrent neural network.

[0163] Step g, obtaining a second hidden state of the current video frame according to the first hidden state of the current video frame Step h, obtaining a reconstructed video frame of the current video frame according to the second hidden state of the current video frame

[0164] That is, the operation represented by the above formula (23) is performed by the output module.

[0165] Step h, obtaining a reconstructed video frame of the current video frame according to the second hidden state of the current video frame and a preset loss function, and adjusting parameters of the residual grid of the current frame, the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module according to the loss value.

[0166] In some embodiments, the preset loss function is a loss function obtained by combining an L1 loss function and a Structural Similarity Index (SSIM) loss function.

[0167] In some embodiments, the preset loss function is as follows:

[0168] wherein T is the total number of video frames of the video sequence, f θ (t) is the reconstructed video frame of the current video frame, v t is the current video frame (frame true value), and a is a hyperparameter used to balance the weights of the L1 loss component and the SSIM loss component.

[0169] In some embodiments, a is 0.7. That is, the weight of the L1 loss component is 0.7, and the weight of the SSIM loss component is 0.3.

[0170] In some embodiments, the Adaptive Moment Estimation (Adam) is selected as the optimizer, the batch size is set to 1, and the initial learning rate is 5e -4 and the cosine annealing learning rate adjustment method is adopted.

[0171] Adam is an optimization algorithm for training deep learning models. On the one hand, unlike traditional gradient descent algorithms that only update parameters based on current gradients, Adam combines the Momentum algorithm, which considers not only the current gradient but also the cumulative effect of previous gradients when updating parameters. On the other hand, Adam also combines RMSProp, an optimization algorithm that can adjust the learning rate by maintaining an exponentially weighted moving average of the square of the gradient, so Adam can adaptively adjust the learning rate of each parameter.

[0172] Step i, the next video frame of the current video frame is taken as the current video frame, and the above steps b-h are executed in a loop until the stop training condition is met, and the representation data of the video sequence is generated according to the residual grid corresponding to each video frame of the video sequence, the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module.

[0173] In some embodiments, the stop training condition is that the number of training times of each video frame of the video sequence reaches a preset number. For example, the stop training condition is that the number of training times of each video frame of the video sequence reaches 300 times.

[0174] In some embodiments, the representation data of the video sequence can include each parameter in the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module, including trainable parameters (Trainable Parameters) and non-trainable parameters (Non-Trainable Parameters).

[0175] In some embodiments, the representation data of the video sequence can include each trainable parameter (Trainable Parameters) in the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module.

[0176] When the representation data of the video sequence only includes the trainable parameters of the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module, the non-trainable parameters of the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module are first determined by default values, etc., and then the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module are reconstructed based on the parameters of the non-trainable parameters of the first recurrent neural network, the super-resolution network, the second recurrent neural network, and the output module.

[0177] In some embodiments, the video reconstruction method based on the network model shown in Figure 1 may include the following steps 1 to 9:

[0178] Step 1: Obtain the representation data of the video sequence.

[0179] The representation data of the video sequence may include: residual grids corresponding to each video frame of the video sequence, a first recurrent neural network, a super-resolution network, a second recurrent neural network, and an output module.

[0180] Step 2: Obtain the residual R corresponding to the current video frame based on the residual grid corresponding to each video frame in the video sequence. t .

[0181] Step 3: Calculate the hidden state of the previous video frame obtained from the first recurrent neural network. The residual R with the current video frame t Add them together to obtain the features of the current video frame.

[0182] That is, the operation represented by the above equation (1) is performed through the feature fusion module.

[0183] Step 4: Based on the characteristics of the current video frame The hidden state of the previous video frame obtained by the first recurrent neural network. Get the first hidden state of the current video frame

[0184] That is, the operations represented by equations (2) to (17) above are executed through the first recurrent neural network.

[0185] Step 5: Based on the first hidden state of the current video frame Obtain the reconstructed features f of the current video frame t C,H,W .

[0186] That is, the operations represented by equations (18) to (21) above are performed through the super-resolution network.

[0187] Step 6: Based on the reconstruction features f of the current video framet C,H,W and the hidden state of the previous video frame obtained by the second recurrent neural network obtaining a second hidden state of the current video frame

[0188] That is, the operation represented by the above formula (22) is performed by the second recurrent neural network.

[0189] Step 7, obtaining a second hidden state of the current video frame according to the second hidden state of the current video frame obtaining a reconstructed video frame of the current video frame

[0190] That is, the operation represented by the above formula (23) is performed by the output module.

[0191] Step 8, performing the above steps 2-7 cyclically with the next video frame of the current video frame as the current video frame until the reconstructed video frames of each video frame in the video sequence are obtained.

[0192] The video representation method and the video reconstruction method provided by some embodiments are tested by taking the peak signal-to-noise ratio (PSNR) as an image quality evaluation index, and different sizes of network models are obtained by changing the number of channels of the residual grid and the number of channels of the NeRV block, the representation capabilities of the network model shown in FIG. 1 under different sizes are compared, and tests are performed on 1.5M, 2.5M, 3.2M (NeRV-S), 6.3M (NeRV-M) and 12.5M (NeRV-L) sizes respectively, and the test results are shown in Table 1.

[0193] Table 1 Test results and comparison of PSNR index

[0194] As shown in Table 1. From the test results and comparison, the PSNR of the network model shown in FIG. 1 under different sizes is improved compared with NeRV.

[0195] Referring to FIG. 5, in some embodiments, the network model for performing the video representation method can also only include: a residual grid 11, a first recurrent neural network 13, a super-resolution network 14 and an output module 16. Wherein, the operations performed by the residual grid 11, the first recurrent neural network 13 and the super-resolution network 14 are the same as in FIG. 1, and the output module 16 reconstructs the current video frame according to the reconstructed feature of the current video frame output by the super-resolution network 14.

[0196] The reconstruction operation of the output module 16 is represented as hand(), and the reconstructed video frame of the current video frame is represented as The executed operations of the output module 16 in FIG. 5 can be represented as follows:

[0197] Some embodiments of the present application provide a video representation method, which can include the following steps with reference to FIG. 6:

[0198] S61, creating an information grid corresponding to each video frame of a target video sequence.

[0199] In some embodiments, the height of the information grid corresponding to each video frame can be 9, the width can be 16, and the number of feature dimensions stored at each grid vertex can be C0.

[0200] In some embodiments, after creating the information grid corresponding to each video frame of the target video sequence, the method further includes initializing the information grid corresponding to each video frame of the target video sequence.

[0201] The loop executes the following steps S62-S63 with each video frame of the target video sequence as a current video frame until a stop training condition is met:

[0202] S62, obtaining a first hidden state of the current video frame according to the information grid corresponding to the current video frame, a first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network.

[0203] In some embodiments, the above step S62 (obtaining a first hidden state of the current video frame according to the information grid corresponding to the current video frame, a first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network) includes the following steps 1 and 2:

[0204] Step 1, obtaining a feature of the current video frame according to the content grid corresponding to the current video frame.

[0205] Step 2, obtaining a first hidden state of the current video frame according to the feature of the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network.

[0206] With reference to FIG. 7, when the implementation of the above step S62 includes the above steps 1 and 2, a network model for executing the video representation method provided by some embodiments includes an information grid 71 corresponding to each video frame, a first recurrent neural network 72, and a reconstruction network 73. Among them, the information grid corresponding to each video frame is used to represent the feature of each video frame.

[0207] In some embodiments, the step S62 (obtaining the first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network) comprises the following steps ① to ③:

[0208] Step ①: obtaining the residual of the current video frame according to the information grid corresponding to the current video frame.

[0209] Step ②: superimposing the residual of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network to obtain the feature of the current video frame.

[0210] Step ③: obtaining the first hidden state of the current video frame according to the feature of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0211] Referring to FIG. 8, when the implementation mode of the step S62 comprises the steps ① to ③, the network model for executing the video representation method provided by some embodiments comprises an information grid 81 corresponding to each video frame, a feature fusion module 82, a first recurrent neural network 83, and a reconstruction network 84. The information grid corresponding to each video frame is respectively used to represent the residual of each video frame, and the feature fusion module 82 is used to superimpose the residual of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network to obtain the feature of the current video frame.

[0212] Compared with the embodiment shown in FIG. 7, in which the information grid corresponding to each video frame is respectively used to represent the feature of each video frame, in the embodiment shown in FIG. 8, the information grid corresponding to each video frame is respectively used to represent the residual of each video frame, so that the grid parameters of the information grid in the embodiment shown in FIG. 8 are more sparse, and the data amount of the target video sequence can be reduced.

[0213] Referring to FIG. 2, in some embodiments, the first recurrent neural network comprises:

[0214] The update gate 201 is used to obtain an update gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0215] The reset gate 202 is used to obtain a reset gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0216] The first multiplier 203 is configured to multiply the reset gate coefficient and the hidden state of the previous video frame obtained by the first recurrent neural network to obtain a first feature;

[0217] The first adder 204 is configured to superimpose the first feature and the feature of the current video frame to obtain a second feature;

[0218] The convolution unit 205 is configured to perform convolution operation on the second feature to obtain a third feature;

[0219] The second adder 206 is configured to superimpose the third feature and the update gate bias to obtain a fourth feature;

[0220] The activation unit 207 is configured to perform nonlinear transformation on the fourth feature to obtain a candidate hidden state;

[0221] The second multiplier 208 is configured to multiply the hidden state of the previous video frame obtained by the first recurrent neural network and the update gate coefficient element by element to obtain a fifth feature;

[0222] The third multiplier 209 is configured to multiply the candidate hidden state and the complement of the update gate coefficient element by element to obtain a sixth feature;

[0223] The second adder 210 is configured to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0224] Referring to FIG. 2, in some embodiments, the update gate 201 includes:

[0225] The update gate state convolution layer 2011 is configured to perform convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain an update gate hidden state;

[0226] The update gate feature convolution layer 2012 is configured to perform convolution operation on the feature of the current video frame to obtain an update gate feature;

[0227] The update gate fusion layer 2013 is configured to superimpose the update gate hidden state, the update gate feature, and an update gate bias to obtain an update gate fusion feature;

[0228] The update gate activation function layer 2014 is configured to perform nonlinear transformation on the update gate fusion feature to obtain the update gate coefficient.

[0229] Referring to FIG. 2, in some embodiments, the reset gate 202 includes:

[0230] The reset gate state convolutional layer 2021 can be configured to perform convolutional operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain a reset gate hidden state.

[0231] The reset gate feature convolutional layer 2022 can be configured to perform convolutional operation on the feature of the current video frame to obtain a reset gate feature.

[0232] The reset gate fusion layer 2023 can be configured to superimpose the reset gate hidden state, the reset gate feature and a reset gate bias to obtain a reset gate fusion feature.

[0233] The reset gate activation function layer 2024 can be configured to perform nonlinear transformation on the reset gate fusion feature to obtain the reset gate coefficient.

[0234] In some embodiments described above, the update gate and the reset gate both use the convolutional layer to perform convolutional operation on the hidden state of the previous video frame obtained by the first recurrent neural network and the feature of the current video frame, so that the first recurrent neural network can have better ability to learn spatial information features, thereby improving the quality of the final reconstructed video.

[0235] S63, obtaining a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network.

[0236] Referring to FIG. 9, in some embodiments, the reconstruction network 84 shown in FIG. 8 includes:

[0237] The super-resolution network 91 can be configured to process the first hidden state of the current video frame to obtain a reconstructed feature of the current video frame.

[0238] The output module 92 can be configured to obtain a reconstructed video frame corresponding to the current video frame according to the reconstructed feature of the current video frame.

[0239] That is, the step S63 includes: processing the first hidden state of the current video frame to obtain a reconstructed feature of the current video frame, and obtaining a reconstructed video frame corresponding to the current video frame according to the reconstructed feature of the current video frame.

[0240] Referring to FIG. 10, in some embodiments, the reconstruction network 84 shown in FIG. 8 includes:

[0241] The super-resolution network 101 can be configured to process the first hidden state of the current video frame to obtain a reconstructed feature of the current video frame.

[0242] The second recurrent neural network 102 can be used to obtain a second hidden state of the current video frame according to the reconstructed feature of the current video frame and a hidden state of a previous video frame of the current video frame obtained by the second recurrent neural network model.

[0243] The output module 103 can be used to obtain a reconstructed video frame corresponding to the current video frame according to the second hidden state of the current video frame.

[0244] That is, the step S63 includes: processing the first hidden state of the current video frame to obtain a reconstructed feature of the current video frame, obtaining a second hidden state of the current video frame according to the reconstructed feature of the current video frame and a hidden state of a previous video frame of the current video frame obtained by the second recurrent neural network model, and obtaining a reconstructed video frame corresponding to the current video frame according to the reconstructed feature of the current video frame.

[0245] Since the current video frame is processed by the second recurrent neural network after the reconstructed feature of the current video frame is obtained in the embodiment shown in FIG. 10, some of the above embodiments can obtain more spatio-temporal correlation between adjacent frames in time sequence by using the recurrent neural network, capture more spatio-temporal detail information at the video frame level scale, and further improve the quality of the finally reconstructed video frame.

[0246] In some embodiments, referring to FIG. 3, the super-resolution network 14 includes a five-level cascaded NeRV block 30.

[0247] In some embodiments, the up-sampling multiples of the five-level cascaded NeRV block are 25, 4, 4, 4, and 4 in sequence.

[0248] In some embodiments, referring to FIG. 4, the NeRV block 30 includes:

[0249] The convolution layer 41 can be used to perform up-sampling convolution processing on the input feature of the NeRV block to obtain an output feature of the convolution layer;

[0250] The pixel shuffle layer 42 can be used to perform pixel shuffling on the output feature of the convolution layer to obtain an output feature of the pixel shuffle layer;

[0251] The activation function layer 43 can be used to perform nonlinear transformation on the output feature of the pixel shuffle layer to obtain an output feature of the NeRV block.

[0252] In some embodiments, the output module is composed of a convolution layer.

[0253] In some embodiments, the network structure of the second recurrent neural network is the same as that of the first recurrent neural network.

[0254] S64, adjusting at least one of the information grid corresponding to the current video frame, the parameters of the first recurrent neural network and the parameters of the reconstruction network according to the current video frame, the reconstructed video frame corresponding to the current video frame and the preset loss function.

[0255] In some embodiments, the preset loss function is:

[0256] wherein T is the number of video frames of the target video sequence, f θ (t) is the reconstructed video frame of the current video frame, v t is the current video frame, a is a hyperparameter, and ||f θ (t)-v t ||1 is the L1 loss value of f θ (t) and v t , and SSIM(f θ (t), v t ) is the structural similarity index SSIM loss value of f θ (t) and v t .

[0257] After performing the above step S64, it is judged whether the stop training condition is met. If the stop training condition is not met, the above steps S61 to S64 are performed again. If the stop training condition is met, the following step S65 is performed:

[0258] S65, generating representation data of the target video sequence according to the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network and the reconstruction network.

[0259] In some embodiments, generating representation data of the target video sequence according to the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network and the reconstruction network comprises: obtaining the information grid corresponding to each video frame of the target video sequence, the trainable network parameters of the first recurrent neural network and the trainable network parameters of the reconstruction network; and compressing the information grid corresponding to each video frame of the target video sequence, the trainable network parameters of the first recurrent neural network and the trainable network parameters of the reconstruction network to obtain the representation data of the target video sequence.

[0260] Some embodiments of the video representation method provided by the present application first create information grids corresponding to each video frame of a target video sequence when representing the target video sequence, then loop each video frame of the target video sequence as a current video frame, and obtain a first hidden state of the current video frame according to the information grid corresponding to the current video frame, a first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network, obtain a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and a reconstruction network, and adjust at least one of the information grid corresponding to the current video frame, parameters of the first recurrent neural network, and parameters of the reconstruction network according to the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function. When a stop training condition is met, generate representation data of the target video sequence according to the information grids corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network. Since the video representation method provided by some embodiments of the present application is based on a recurrent neural network for video representation, the recurrent neural network has a memory function and can retain information through its loop structure, thereby capturing long-term dependencies in video sequences. Therefore, some embodiments of the present application can effectively utilize the spatiotemporal redundancy information of the video. Compared with NeRV, which cannot effectively utilize the spatiotemporal redundancy information of the video, and the MLP network, which has a large number of parameters, some embodiments of the present application can effectively utilize the spatiotemporal redundancy information of the video without using the MLP network. Therefore, some embodiments of the present application can improve the representation efficiency of the video.

[0261] Some embodiments of the present application provide a video reconstruction method, which can include the following steps with reference to FIG. 11:

[0262] S111, obtaining representation data of a target video sequence.

[0263] In some embodiments, obtaining the representation data of the target video sequence includes receiving the representation data of the target video sequence sent by a media distribution server.

[0264] S112, obtaining information grids corresponding to each video frame of the target video sequence, a first recurrent neural network, and a reconstruction network according to the representation data of the target video sequence.

[0265] In some embodiments, the representation data of the target video sequence includes information grids corresponding to each video frame of the target video sequence, network parameters of the first recurrent neural network, and network parameters of the reconstruction network according to the representation data of the target video sequence, constructing the first recurrent neural network according to the network parameters of the first recurrent neural network, and constructing the reconstruction network according to the network parameters of the reconstruction network.

[0266] respectively, as a current video frame, and the following steps S113 and S114 are performed to obtain a reconstructed video frame corresponding to each video frame of the target video sequence:

[0267] S113, obtaining a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network.

[0268] The implementation of step S113, the structure of the network model corresponding to different implementations, and the structure of the first recurrent neural network can refer to the analysis description of step S62 above. To avoid redundancy, details are not described here.

[0269] S114, obtaining a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network.

[0270] The implementation of step S114, the structure of the network model corresponding to different implementations, and the structure of the reconstruction network can refer to the analysis description of step S63 above. To avoid redundancy, details are not described here.

[0271] The video reconstruction method provided by some of the above embodiments first obtains an information grid corresponding to each video frame of a target video sequence, a first recurrent neural network, and a reconstruction network according to representation data of the target video sequence after obtaining the representation data of the target video sequence. Then, each video frame of the target video sequence is respectively taken as a current video frame, and a first hidden state of the current video frame is obtained according to an information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network. In addition, a reconstructed video frame corresponding to the current video frame is obtained according to the first hidden state of the current video frame and the reconstruction network. Thus, the reconstructed video frame corresponding to each video frame of the target video sequence is obtained. Therefore, the video reconstruction method provided by some of the above embodiments can reconstruct the video represented by the video representation method provided by some of the above embodiments. Therefore, some of the above embodiments can improve the representation efficiency of the video while ensuring that the video can be reconstructed.

[0272] Referring to FIG. 12, some embodiments of the present application provide a video representation device 1200, which can include:

[0273] The creation module 121 can be used to create an information grid corresponding to each video frame of a target video sequence.

[0274] The training module 122 can be configured to cyclically execute the following steps on each video frame of the target video sequence as a current video frame until a stop training condition is met:

[0275] According to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network, a first hidden state of the current video frame is obtained.

[0276] According to the first hidden state of the current video frame and the reconstruction network, a reconstructed video frame corresponding to the current video frame is obtained.

[0277] According to the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function, at least one of the information grid corresponding to the current video frame, parameters of the first recurrent neural network, and parameters of the reconstruction network is adjusted.

[0278] The generation module 123 can be configured to, when the stop training condition is met, generate representation data of the target video sequence according to the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network.

[0279] In some embodiments, the training module 122 is specifically configured to obtain a residual of the current video frame according to the information grid corresponding to the current video frame; superimpose the residual of the current video frame and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network to obtain a feature of the current video frame; and obtain the first hidden state of the current video frame according to the feature of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0280] In some embodiments, the first recurrent neural network can include:

[0281] The update gate can be configured to obtain an update gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0282] The reset gate can be configured to obtain a reset gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0283] The first multiplier can be configured to multiply the reset gate coefficient and the hidden state of the previous video frame obtained by the first recurrent neural network element by element to obtain a first feature.

[0284] The first adder can be configured to superimpose the first feature and the feature of the current video frame to obtain a second feature.

[0285] a convolution unit, configured to perform convolution operation on the second feature to obtain a third feature;

[0286] a second adder, configured to superimpose the third feature and an update gate bias to obtain a fourth feature;

[0287] an activation unit, configured to perform nonlinear transformation on the fourth feature to obtain a candidate hidden state;

[0288] a second multiplier, configured to multiply the hidden state of the previous video frame obtained by the first recurrent neural network and the update gate coefficient element by element to obtain a fifth feature;

[0289] a third multiplier, configured to multiply the candidate hidden state and the complement of the update gate coefficient element by element to obtain a sixth feature;

[0290] a second adder, configured to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0291] In some embodiments, the update gate can include:

[0292] an update gate state convolution layer, configured to perform convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain an update gate hidden state;

[0293] an update gate feature convolution layer, configured to perform convolution operation on the feature of the current video frame to obtain an update gate feature;

[0294] an update gate fusion layer, configured to superimpose the update gate hidden state, the update gate feature and an update gate bias to obtain an update gate fusion feature;

[0295] an update gate activation function layer, configured to perform nonlinear transformation on the update gate fusion feature to obtain the update gate coefficient.

[0296] In some embodiments, the reset gate can include:

[0297] a reset gate state convolution layer, configured to perform convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain a reset gate hidden state;

[0298] a reset gate feature convolution layer, configured to perform convolution operation on the feature of the current video frame to obtain a reset gate feature;

[0299] a reset gate fusion layer, configured to superimpose the reset gate hidden state, the reset gate feature and a reset gate bias to obtain a reset gate fusion feature;

[0300] The reset gate activation function layer is configured to perform nonlinear transformation on the reset gate fusion feature to obtain the reset gate coefficient.

[0301] In some embodiments, the reconstruction network comprises:

[0302] The super-resolution network is configured to process the first hidden state of the current video frame to obtain the reconstruction feature of the current video frame.

[0303] The output module is configured to obtain the reconstructed video frame corresponding to the current video frame according to the reconstruction feature of the current video frame.

[0304] In some embodiments, the reconstruction network comprises:

[0305] The super-resolution network is configured to process the first hidden state of the current video frame to obtain the reconstruction feature of the current video frame.

[0306] The second recurrent neural network is configured to obtain the second hidden state of the current video frame according to the reconstruction feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the second recurrent neural network model.

[0307] The output module is configured to obtain the reconstructed video frame corresponding to the current video frame according to the second hidden state of the current video frame.

[0308] In some embodiments, the super-resolution network comprises a plurality of cascaded NeRV blocks of video.

[0309] In some embodiments, the super-resolution network comprises five cascaded NeRV blocks.

[0310] The up-sampling multiples of the five cascaded NeRV blocks are 25, 4, 4, 4, and 4 in sequence.

[0311] In some embodiments, the NeRV block comprises:

[0312] The convolution layer is configured to perform up-sampling convolution processing on the input feature of the NeRV block to obtain the output feature of the convolution layer.

[0313] The pixel shuffle layer is configured to perform pixel shuffling on the output feature of the convolution layer to obtain the output feature of the pixel shuffle layer.

[0314] The activation function layer is configured to perform nonlinear transformation on the output feature of the pixel shuffle layer to obtain the output feature of the NeRV block.

[0315] In some embodiments, the output module is composed of a convolution layer.

[0316] In some embodiments, the network structure of the second recurrent neural network is the same as the network structure of the first recurrent neural network.

[0317] In some embodiments, the preset loss function is:

[0318] wherein, T is the number of video frames of the target video sequence, f θ (t) is a reconstructed video frame of the current video frame, v t is the current video frame, a is a hyperparameter, and ||f θ (t)-v t ||1 is the L1 loss value of f θ (t) and v t , and SSIM(f θ (t), v t ) is the structural similarity index SSIM loss value of f θ (t) and v t .

[0319] The video representation apparatus provided in the above embodiments can perform the video representation method provided in any of the above embodiments, and the implementation principle and the achieved technical effects are the same. To avoid redundancy, no longer detailed description is made.

[0320] Referring to FIG. 13, some embodiments of the present application provide a video reconstruction apparatus 1300, which can include:

[0321] The acquisition unit 131 can be configured to acquire representation data of a target video sequence.

[0322] The processing unit 132 can be configured to acquire, according to the representation data of the target video sequence, an information grid corresponding to each video frame of the target video sequence, a first recurrent neural network, and a reconstruction network.

[0323] The reconstruction unit 133 can be configured to execute the following steps by taking each video frame of the target video sequence as a current video frame, respectively, to acquire a reconstructed video frame corresponding to each video frame of the target video sequence.

[0324] According to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network, a first hidden state of the current video frame is acquired.

[0325] According to the first hidden state of the current video frame and the reconstruction network, a reconstructed video frame corresponding to the current video frame is acquired.

[0326] In some embodiments, the reconstruction unit 133 is specifically configured to obtain a residual of the current video frame according to the information grid corresponding to the current video frame; superimpose the residual of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network to obtain a feature of the current video frame; and obtain the first hidden state of the current video frame according to the feature of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0327] In some embodiments, the first recurrent neural network can include:

[0328] The update gate is configured to obtain an update gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0329] The reset gate is configured to obtain a reset gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

[0330] The first multiplier is configured to multiply the reset gate coefficient and the hidden state of the previous video frame obtained by the first recurrent neural network element by element to obtain a first feature.

[0331] The first adder is configured to superimpose the first feature and the feature of the current video frame to obtain a second feature.

[0332] The convolution unit is configured to perform a convolution operation on the second feature to obtain a third feature.

[0333] The second adder is configured to superimpose the third feature and an update gate bias to obtain a fourth feature.

[0334] The activation unit is configured to perform a nonlinear transformation on the fourth feature to obtain a candidate hidden state.

[0335] The second multiplier is configured to multiply the hidden state of the previous video frame obtained by the first recurrent neural network and the update gate coefficient element by element to obtain a fifth feature.

[0336] The third multiplier is configured to multiply the candidate hidden state and the complement of the update gate coefficient element by element to obtain a sixth feature.

[0337] The second adder is configured to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

[0338] In some embodiments, the update gate can include:

[0339] The update gate state convolution layer is configured to perform convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain an update gate hidden state.

[0340] The update gate feature convolution layer is configured to perform convolution operation on the feature of the current video frame to obtain an update gate feature.

[0341] The update gate fusion layer is configured to superimpose the update gate hidden state, the update gate feature and an update gate bias to obtain an update gate fusion feature.

[0342] The update gate activation function layer is configured to perform nonlinear transformation on the update gate fusion feature to obtain the update gate coefficient.

[0343] In some embodiments, the reset gate can include:

[0344] The reset gate state convolution layer is configured to perform convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain a reset gate hidden state.

[0345] The reset gate feature convolution layer is configured to perform convolution operation on the feature of the current video frame to obtain a reset gate feature.

[0346] The reset gate fusion layer is configured to superimpose the reset gate hidden state, the reset gate feature and a reset gate bias to obtain a reset gate fusion feature.

[0347] The reset gate activation function layer is configured to perform nonlinear transformation on the reset gate fusion feature to obtain the reset gate coefficient.

[0348] In some embodiments, the reconstruction network can include:

[0349] The super-resolution network is configured to process the first hidden state of the current video frame to obtain a reconstruction feature of the current video frame.

[0350] The output module is configured to obtain a reconstructed video frame corresponding to the current video frame according to the reconstruction feature of the current video frame.

[0351] In some embodiments, the reconstruction network can include:

[0352] The super-resolution network is configured to process the first hidden state of the current video frame to obtain a reconstruction feature of the current video frame.

[0353] The second recurrent neural network is configured to obtain a second hidden state of the current video frame according to the reconstruction feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the second recurrent neural network model.

[0354] The output module is configured to obtain a reconstructed video frame corresponding to the current video frame according to the second hidden state of the current video frame.

[0355] In some embodiments, the super-resolution network comprises a plurality of cascaded NeRV blocks.

[0356] In some embodiments, the super-resolution network comprises five cascaded NeRV blocks.

[0357] The up-sampling multiples of the five cascaded NeRV blocks are 25, 4, 4, 4, and 4, respectively.

[0358] In some embodiments, the NeRV block comprises:

[0359] The convolution layer is configured to perform up-sampling convolution processing on the input feature of the NeRV block to obtain an output feature of the convolution layer.

[0360] The pixel shuffle layer is configured to perform pixel shuffling on the output feature of the convolution layer to obtain an output feature of the pixel shuffle layer.

[0361] The activation function layer is configured to perform non-linear transformation on the output feature of the pixel shuffle layer to obtain an output feature of the NeRV block.

[0362] In some embodiments, the output module comprises a convolution layer.

[0363] The video reconstruction apparatus provided in the above embodiments can perform the video reconstruction method provided in any of the above embodiments, and the implementation principle and the achieved technical effects are the same. To avoid redundancy, no further detailed description is given.

[0364] Some embodiments of the present application provide an electronic device, which can include a memory and a processor, the memory storing a computer program, and the processor being configured to implement the video representation method of any of the above embodiments or the video reconstruction method of any of the above embodiments when executing the computer program.

[0365] FIG. 14 illustrates a configuration block diagram of a display device 1400 in some embodiments described above. As shown in FIG. 14, the display device 1400 can include at least one of a tuner and demodulator 141, a communicator 142, a detector 143, an external device interface 144, a processor 145, a display 146, an audio output interface 147, a memory 148, a power supply 149, and a user interface.

[0366] In some embodiments, the processor 145 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM (random access memory), a ROM (read-only memory), a first interface to an n-th interface available for input / output, a communication bus, and the like.

[0367] The display 146 includes a display screen component available for presenting a picture, and a driving component for driving the image display, a component available for receiving an image signal originating from the processor output, and displaying video content, image content, and a menu operation interface, and a user operation UI interface.

[0368] The display 146 can be a liquid crystal display, an organic light emitting diode (OLED) display, a projection display, and the like.

[0369] The communicator 142 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 1400 can transmit and receive control signals and data signals with a control device or a server through the communicator 142.

[0370] The user interface is available for receiving a control signal input by a user through a control device (such as an infrared remote controller, etc.) or a touch or gesture, and the like.

[0371] The detector 143 is available for collecting signals of an external environment or interaction with the outside. For example, the detector 143 includes a light receiver, and a sensor available for collecting ambient light intensity; or the detector 143 includes an image collector, such as a camera, which can be used to collect an external environment scene, user attributes, or user interaction gestures; or the detector 143 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0372] The external device interface 144 can include, but is not limited to, any one or more of a High-Definition Multimedia Interface (HDMI), an analog or digital High-Definition Component input interface (Component), a Composite Video Broadcast Signal (CVBS) input interface, a Universal Serial Bus (USB) input interface, an RGB (Red, Green, Blue) port, and the like. It can also be a composite input / output interface formed by a plurality of the above interfaces.

[0373] The tuner-demodulator 141 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals. In some embodiments, the processor 145 and the tuner-demodulator 141 can be located in different separate devices, i.e., the tuner-demodulator 141 can also be in an external device of the main device in which the processor 145 is located, such as an external set-top box, etc.

[0374] The processor 145 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The processor 145 controls the overall operation of the display device 1400. For example, in response to receiving a user command for selecting a UI object displayed on the display 146, the processor 145 can perform an operation related to the object selected by the user command.

[0375] In some embodiments, the processor 145 includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), a RAM (Random Access Memory), a ROM (Read-Only Memory), a first to an n-th interface for input / output, a communication bus, and the like.

[0376] Some embodiments of the present application provide a computer readable storage medium having stored thereon a computer program, which, when executed by a computing device, causes the computing device to implement the video representation method of any of the above embodiments or the video reconstruction method of any of the above embodiments.

[0377] In a seventh aspect, some embodiments of the present application provide a chip, comprising a processor and a memory, the memory is configured to store a program or instructions executable on the processor, and the processor is configured to execute the program or instructions, so that the video representation method of any of the above embodiments or the video reconstruction method of any of the above embodiments is executed.

[0378] In an eighth aspect, some embodiments of the present application provide a computer program product, when the computer program product is executed on a computer, so that the computer implements the video representation method of any of the above embodiments or the video reconstruction method of any of the above embodiments.

[0379] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0380] In order to facilitate explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A method of video representation, characterized by, The method comprises: creating information grids corresponding to each video frame of a target video sequence; performing the following steps on each video frame of the target video sequence as a current video frame in a loop until a stop training condition is met: obtaining a first hidden state of the current video frame according to an information grid corresponding to the current video frame, a first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network; obtaining a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and a reconstruction network; adjusting at least one of the information grid corresponding to the current video frame, parameters of the first recurrent neural network, and parameters of the reconstruction network according to the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function; when the stop training condition is met, generating representation data of the target video sequence according to the information grids corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network.

2. The method of claim 1, wherein, The method comprises: obtaining a residual of the current video frame according to the information grid corresponding to the current video frame; superimposing the residual of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network to obtain a feature of the current video frame; obtaining the first hidden state of the current video frame according to the feature of the current video frame, the first recurrent neural network, and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network.

3. The method of claim 2, wherein, The first recurrent neural network comprises: an update gate configured to obtain an update gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network; a reset gate configured to obtain a reset gate coefficient according to the feature of the current video frame and the hidden state of the previous video frame of the current video frame obtained by the first recurrent neural network; a first multiplier configured to multiply the reset gate coefficient and the hidden state of the previous video frame obtained by the first recurrent neural network element by element to obtain a first feature; a first adder configured to superimpose the first feature and the feature of the current video frame to obtain a second feature; a convolution unit configured to perform a convolution operation on the second feature to obtain a third feature; a second adder configured to superimpose the third feature and an update gate bias to obtain a fourth feature; an activation unit configured to perform a nonlinear transformation on the fourth feature to obtain a candidate hidden state; a second multiplier configured to multiply the hidden state of the previous video frame obtained by the first recurrent neural network and the update gate coefficient element by element to obtain a fifth feature; a third multiplier configured to multiply the candidate hidden state and a complement of the update gate coefficient element by element to obtain a sixth feature; a third adder configured to superimpose the fifth feature and the sixth feature to obtain the first hidden state of the current video frame.

4. The method of claim 3, wherein, The update gate comprises: an update gate state convolution layer configured to perform convolution operation on a hidden state of a previous video frame obtained by the first recurrent neural network to obtain an update gate hidden state; an update gate feature convolution layer configured to perform convolution operation on a feature of the current video frame to obtain an update gate feature; an update gate fusion layer configured to superimpose the update gate hidden state, the update gate feature, and an update gate bias to obtain an update gate fusion feature; an update gate activation function layer configured to perform nonlinear transformation on the update gate fusion feature to obtain the update gate coefficient.

5. The method of claim 3, wherein, The reset gate comprises: a reset gate state convolution layer configured to perform convolution operation on the hidden state of the previous video frame obtained by the first recurrent neural network to obtain a reset gate hidden state; a reset gate feature convolution layer configured to perform convolution operation on the feature of the current video frame to obtain a reset gate feature; a reset gate fusion layer configured to superimpose the reset gate hidden state, the reset gate feature, and a reset gate bias to obtain a reset gate fusion feature; a reset gate activation function layer configured to perform nonlinear transformation on the reset gate fusion feature to obtain the reset gate coefficient.

6. The method of claim 1, wherein, The reconstruction network comprises: a super-resolution network configured to process a first hidden state of the current video frame to obtain a reconstruction feature of the current video frame; an output module configured to obtain a reconstructed video frame corresponding to the current video frame according to the reconstruction feature of the current video frame.

7. The method of claim 1, wherein, The reconstruction network comprises: a super-resolution network configured to process a first hidden state of the current video frame to obtain a reconstruction feature of the current video frame; a second recurrent neural network configured to obtain a second hidden state of the current video frame according to the reconstruction feature of the current video frame and a hidden state of a previous video frame of the current video frame obtained by the second recurrent neural network model; an output module configured to obtain a reconstructed video frame corresponding to the current video frame according to the second hidden state of the current video frame.

8. The method according to claim 6 or 7, characterized in that, The super-resolution network comprises a plurality of cascaded NeRV blocks.

9. The method of claim 8, wherein, The super-resolution network comprises five cascaded NeRV blocks. The up-sampling multiples of the five cascaded NeRV blocks are 25, 4, 4, 4, and 4 in sequence.

10. The method of claim 8, wherein, The NeRV block comprises: a convolution layer configured to perform up-sampling convolution processing on an input feature of the NeRV block to obtain an output feature of the convolution layer; a pixel shuffle layer configured to perform pixel shuffle on the output feature of the convolution layer to obtain an output feature of the pixel shuffle layer; an activation function layer configured to perform nonlinear transformation on the output feature of the pixel shuffle layer to obtain an output feature of the NeRV block.

11. The method of claim 6 or 7, wherein, The output module is composed of a convolution layer.

12. The method of claim 7, wherein, The network structure of the second recurrent neural network is the same as that of the first recurrent neural network.

13. The method of claim 1, wherein, The preset loss function is: where T is the number of video frames of the target video sequence, f θ (t) is the reconstructed video frame of the current video frame, v t is the current video frame, a is a hyperparameter, and ||f θ (t) - v t ||1 is the L1 loss value of f θ (t) and v t . SSIM(f θ (t), v t ) is the structural similarity index SSIM loss value of f θ (t) and v t .

14. A method of video reconstruction, characterized by, comprises: obtaining representation data of a target video sequence; obtaining information grids corresponding to each video frame of the target video sequence, a first recurrent neural network, and a reconstruction network according to the representation data of the target video sequence; respectively taking each video frame of the target video sequence as a current video frame to perform the following steps to obtain a reconstructed video frame corresponding to each video frame of the target video sequence: obtaining a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network; obtaining a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network.

15. A video display device, characterized by comprising: a creating module configured to create an information grid corresponding to each video frame of a target video sequence; a training module configured to repeatedly perform the following steps by taking each video frame of the target video sequence as a current video frame until a stop training condition is met: obtaining a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network; obtaining a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network; adjusting at least one of the information grid corresponding to the current video frame, parameters of the first recurrent neural network, and parameters of the reconstruction network according to the current video frame, the reconstructed video frame corresponding to the current video frame, and a preset loss function; a generating module configured to generate representation data of the target video sequence according to the information grid corresponding to each video frame of the target video sequence, the first recurrent neural network, and the reconstruction network when the stop training condition is met.

16. A video reconstruction apparatus, characterized by comprising: comprising: an obtaining unit configured to obtain representation data of a target video sequence; a processing unit configured to obtain an information grid corresponding to each video frame of the target video sequence, a first recurrent neural network, and a reconstruction network according to the representation data of the target video sequence; a reconstructing unit configured to respectively take each video frame of the target video sequence as a current video frame to perform the following steps to obtain a reconstructed video frame corresponding to each video frame of the target video sequence: obtaining a first hidden state of the current video frame according to the information grid corresponding to the current video frame, the first recurrent neural network, and a hidden state of a previous video frame of the current video frame obtained by the first recurrent neural network; obtaining a reconstructed video frame corresponding to the current video frame according to the first hidden state of the current video frame and the reconstruction network.

17. An electronic device, comprising: comprising: a memory and a processor, the memory storing a computer program, and the processor being configured to implement the method of any one of claims 1-14 when executing the computer program.

18. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and when the computer program is executed by a computing device, the computing device implements the method of any one of claims 1-14.

19. A chip, characterized by The chip comprises a processor and a memory, the memory is used to store a program or instruction which can run on the processor, and the processor is used to execute the program or instruction to make the method of any one of claims 1-14 be executed.

Images