Video coding method and system based on implicit neural video representation

Abstract

The invention provides a video coding method and system based on implicit neural video representation, and belongs to the technical field of video coding. The method comprises the following steps: acquiring local and all hidden layer state features of a current video frame at a previous moment; compensating the global hidden layer state feature at the previous moment based on the mixed residual feature extracted by the residual grid, and processing the compensated global hidden layer state feature by using a coupling mapping RNN (Recurrent Neural Network) to obtain a local content feature and a background feature; acquiring global information features based on the first mapping RNN module; and inputting the global feature and the global hidden layer state feature at the previous moment into a second mapping RNN module, acquiring the global hidden layer state feature at the current moment, and inputting the global hidden layer state feature into an up-sampling module to reconstruct a video frame to obtain a reconstructed video. The video inter-frame time domain information processing capability is effectively enhanced, the decomposition of global and local motion among the video frames is realized, and the reconstruction quality of the video frames is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video encoding, and in particular to a video encoding method and system based on implicit neural video representation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of networking and digitalization, artificial intelligence has made great achievements in various fields and achieved a series of breakthroughs. In the field of video, due to the improvement of equipment and the development of related industries, more and more high-quality videos have participated in people's daily lives and played an important role in information interaction. Therefore, a video encoding method that takes into account both video compression efficiency and video reconstruction quality is needed.

[0004] Implicit neural representations for videos (NeRV) is an implicit neural representation method that processes video signals frame by frame. It fits the video signal into an implicit neural network, converts the video compression problem into a neural network compression problem, and can use common model compression methods such as model pruning, model quantization, and entropy coding to compress the implicit neural network that fits the video information, thereby achieving the effect of video compression. Compared with traditional video coding methods and learning-based video coding methods, implicit neural video representation methods have faster decoding speeds and higher video compression efficiency. However, there are some problems with current implicit neural video representation methods. When processing video sequences, they do not fully consider the long-term spatiotemporal relationship between frames, and have poor processing capabilities for temporal information between video frames; they cannot effectively decompose the global and local motion between video frames; they cannot model regular information and irregular residual information in the video separately, and there is still much room for improvement in video representation quality and compression efficiency. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a video encoding method and system based on implicit neural video representation, which utilizes a coupled mapping RNN (Coup led WarpRNN) network to effectively process the time domain information between video frames, and at the same time, the global motion information and local motion information in the video are mapped and enhanced respectively, which can better process the motion information between video frames, model the regular information and irregular information in the video signal, and thus effectively improve the video reconstruction quality and compression efficiency.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a video encoding method based on implicit neural video representation;

[0008] A video coding method based on implicit neural video representation, comprising:

[0009] Obtain the local hidden layer state features and the global hidden layer state features of the current video frame at the previous moment;

[0010] Extracting mixed residual features in the video frame based on the mixed residual grid, adding the mixed residual features to the global hidden layer state features of the previous moment of the current video frame to obtain the compensated global hidden layer state features;

[0011] The compensated global hidden layer state features are input into the coupled mapping RNN module for iterative update to obtain the reconstructed video frame;

[0012] Among them, the coupled mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features;

[0013] Input the local content features and the local hidden layer state features of the previous moment of the current video frame into the first mapping RNN module to obtain the local hidden layer state features of the current moment; aggregate the local hidden layer state features of the current moment and the background features to obtain the global information features; input the global features and the global hidden layer state features of the previous moment of the current video frame into the second mapping RNN module to obtain the global hidden layer state features of the current moment;

[0014] The global hidden layer state features at the current moment are input into the upsampling module to reconstruct the video frame, and finally the reconstructed video is obtained.

[0015] As a further technical solution, the hybrid residual grid is:

[0016] R G ∈R L*C*H*W

[0017] Where C, H, and W are the temporal resolution, number of channels, height, and width of the grid, respectively.

[0018] As a further technical solution, the coupled mapping RNN module uses convolution to process the compensated global hidden layer state features. The specific process of obtaining local content features and background features is as follows:

[0019] The compensated global hidden layer state features are input into the convolutional layer to generate a soft mask to decompose the compensated global hidden layer state features into local content features. and background characteristics The formula is:

[0020]

[0021] In the formula, is the global hidden layer state feature after compensation; R t is the mixed residual feature; is the global hidden layer state feature after compensation; W l is a soft mask; * indicates element-by-element multiplication.

[0022] As a further technical solution, the local hidden layer state characteristics at the current moment are:

[0023]

[0024] In the formula, is the local hidden layer state feature at the current moment; f g () represents the processing operation of the mapping RNN module; P t () represents the motion information generation module; warp() represents the deformation mapping operation.

[0025] As a further technical solution, the global information feature is specifically:

[0026]

[0027] In the formula, It is a global information feature.

[0028] As a further technical solution, the global hidden layer state characteristics at the current moment are:

[0029]

[0030] In the formula, is the global hidden layer state feature.

[0031] As a further technical solution, the global hidden layer state feature at the current moment is input into the upsampling module to reconstruct the video frame, and the process of finally obtaining the reconstructed video is as follows:

[0032] Perform spatial projection on the global hidden layer state features to generate appearance features; use the upsampling module to reconstruct the video frame based on the appearance features; specifically:

[0033]

[0034] In the formula, represents the reconstructed video, f NB () represents the upsampling module, which is composed of multiple layers of convolution and upsampling operations stacked together; P s Represents spatial projection, which is achieved through a convolutional layer.

[0035] A second aspect of the present invention provides a video coding system based on implicit neural video representation.

[0036] A video coding system based on implicit neural video representation, comprising:

[0037] The hidden layer state feature acquisition module is configured to: acquire the local hidden layer state feature and the global hidden layer state feature of the previous moment of the current video frame;

[0038] The residual feature compensation module is configured to: extract a mixed residual feature in the video frame based on the mixed residual grid, add the mixed residual feature to the global hidden layer state feature of the previous moment of the current video frame, and obtain the compensated global hidden layer state feature;

[0039] The video frame reconstruction module is configured to: input the compensated global hidden layer state features into the coupled mapping RNN module for iterative update to obtain the reconstructed video frame;

[0040] Among them, the coupled mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features;

[0041] Input the local content features and the local hidden layer state features of the previous moment of the current video frame into the first mapping RNN module to obtain the local hidden layer state features of the current moment; aggregate the local hidden layer state features of the current moment and the background features to obtain the global information features; input the global features and the global hidden layer state features of the previous moment of the current video frame into the second mapping RNN module to obtain the global hidden layer state features of the current moment;

[0042] The global hidden layer state features at the current moment are input into the upsampling module to reconstruct the video frame, and finally the reconstructed video is obtained.

[0043] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a video encoding method based on implicit neural video representation as described in the first aspect of the present invention.

[0044] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a video encoding method based on implicit neural video representation as described in the first aspect of the present invention are implemented.

[0045] One or more of the above technical solutions have the following beneficial effects:

[0046] (1) The present invention decomposes the video into regular, structured information and irregular information, proposes a multi-scale motion representation and compensation module based on Coupled WarpRNN, and further integrates time domain processing into the neural network based on global and local motion decomposition, effectively enhancing the time domain information processing capability between video frames and realizing the decomposition of global and local motion between video frames.

[0047] (2) The present invention also uses a hybrid residual grid and network reuse to represent the remaining irregular motion and appearance information, providing more detailed motion and appearance information for video frame reconstruction, greatly improving the quality of video reconstruction.

[0048] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 This is a flow chart of the method of the first embodiment.

[0051] Figure 2 It is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0052] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0053] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0054] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0055] Implicit Neural Representation (INR) is a representation method based on neural networks. This method uses deep neural networks as function approximators to map input parameters to the feature space of the object. Neural Radiance Fields (NeRF) uses the idea of ​​INR to extract the geometric shape and texture information of the object from images from multiple perspectives using neural networks, and then generates a continuous three-dimensional radiation field, so that it can present highly realistic three-dimensional models at any angle and distance, achieving outstanding results. The Implicit Neural Representations for Videos (NeRV) method is based on methods such as neural radiance fields. It is an implicit neural representation method that processes video signals frame by frame. It fits the video signal into an implicit neural network. Given a video sequence V consisting of T frames of images, The implicit neural video representation network will model the mapping from index t to the corresponding video frame:

[0056]

[0057] Implicit neural video representation converts the video compression problem into a neural network compression problem, and can use commonly used model compression methods such as model pruning, model quantization and entropy coding to compress the implicit neural network that fits the video information, thereby achieving the effect of video compression. However, there are some problems with the implicit neural video representation method. When processing video sequences, the long-term spatiotemporal relationship between frames is not fully considered, and the regular information and irregular residual information in the video are not modeled separately. There is still a lot of room for improvement in video representation quality and compression efficiency. In response to the above problems, the present invention processes the time domain information between video frames through a coupled mapping RNN (Coup led WarpRNN) module, and at the same time, the global motion information and local motion information in the video are mapped and enhanced respectively, and the regularity information in the modeling video signal is modeled. Using a mixed residual feature grid, the residual feature is interpolated from the mixed residual feature grid according to the index of the video frame to supplement the irregular information that is not easy to learn by the neural network. Finally, in the decoding stage, the features of the obtained video frame are reconstructed by the upsampling module, and finally the reconstructed video frame is obtained. The time domain information processing capability between video frames is effectively enhanced, and the decomposition of global and local motion between video frames is realized, thereby improving the reconstruction quality of video frames. The present invention will be described in detail below in conjunction with embodiments.

[0058] Embodiment 1

[0059] This embodiment discloses a video encoding method based on implicit neural video representation. Figure 1As shown, a video encoding method based on implicit neural video representation includes:

[0060] Step S1, obtaining the local hidden layer state features and the global hidden layer state features of the current video frame at the previous moment;

[0061] In a recurrent neural network (RNN), the hidden state represents the network’s “memory” or “summary” of previously seen data when processing sequence data. Specifically, the hidden state is a vector that is updated at each time step to contain a compressed representation of all the information in the sequence so far. For a video frame at time t, the hidden state h t-1 It can represent the features of the video frame at the previous moment. There is a motion relationship between the input at this moment. The motion information can be extracted through the convolution layer, mapped to the current frame features, and then processed by the recurrent neural network. For the first frame without a previous hidden state, a learnable tensor is used to represent the hidden state. It can be regarded as an anchor frame, maintaining the main structure of the video and used to predict all frames. The video frame feature h at the current moment t As shown below:

[0062]

[0063] In the formula, f g () is the processing operation of the mapping RNN module. , Represents the input at the current moment, which can be regarded as h here t-1 .P t () represents the motion information generation module, and warp() represents the deformation mapping operation.

[0064] Based on this, in step S1, the features of the video frame at the current moment are predicted by acquiring the local hidden layer state features and the global hidden layer state features of the current video frame at the previous moment.

[0065] Step S2, extracting mixed residual features in the video frame based on the mixed residual grid, adding the mixed residual features to the global hidden layer state features at the previous moment, and obtaining compensated global hidden layer state features;

[0066] In the process of video frame reconstruction, it is necessary not only to provide efficient representation of the regular and structured information in the video, but also to process the irregular information in the video, so as to provide more detailed motion and appearance information in the process of video frame reconstruction. Although neural networks can provide efficient representation of regular and structured information in videos, relying entirely on neural networks may lead to the loss of irregular information, resulting in a significant decrease in video frame reconstruction performance.

[0067] Since the motion information and appearance information in the video are highly correlated in the lossy representation, in this embodiment, these irregular information are modeled together as a learnable hybrid grid structure, and the hybrid residual grid (MRG) is used to learn the irregular motion and appearance information that the neural network cannot effectively represent. MRG is constructed along the time dimension. For a video containing T frames, it can be constructed as

[0068] R G ∈R L*C*H*W

[0069] Where C, H, and W are the time dimension resolution, number of channels, height, and width of the grid, respectively. In order to improve the efficiency of representation, L is usually set to be smaller than the length of the sequence. When the index t of a frame is specified, the mixed residual feature R at the corresponding moment t By using bilinear interpolation from the mixed residual grid R G It is obtained by temporal interpolation.

[0070] The obtained mixed residual features are first processed by a convolution layer, and then added to the global hidden layer state features to obtain the compensated global hidden layer state features, and input into the coupled mapping RNN module. By using the mixed residual features for compensation, more detailed motion and appearance information can be provided for subsequent time processing and frame reconstruction. In this embodiment, although the hidden layer can be learned time by time only by relying on the coupled mapping RNN module to iterate itself, the reconstruction effect is poor. By introducing a mixed residual feature grid to supplement additional information, the mixed residual feature grid and the coupled mapping RNN module are retrained together. Since the mixed residual feature grid is more inclined to learn some irregular information, the coupled mapping RNN module is a neural network structure that is easier to learn regular information. Training the two together can provide more detailed motion and appearance information for video frame reconstruction, greatly improving the quality of video reconstruction.

[0071] Step S3, inputting the compensated global hidden layer state features into the coupled mapping RNN module for iterative update to obtain the reconstructed video frame;

[0072] Since the convolutional gated recurrent unit has fewer parameters and better performance when processing video data, the coupled mapping RNN module uses the convolutional gated recurrent unit (ConvGRU) as the basic RNN network. The formula for the current video frame feature can be expanded as:

[0073]

[0074]

[0075] In the formula, z t and r tRepresent the update gate and reset gate in the recurrent neural network, W z ,W r and W h Represent the corresponding convolutional network weights, is an intermediate state of the hidden layer, used to update the hidden layer, σ represents the Sigmoid activation function, and * is element-by-element multiplication.

[0076] Furthermore, the coupled mapping RNN module includes two mapping RNN (WarpRNN) modules, wherein the first mapping RNN module is used to learn information related to local motion, and the second mapping RNN module is used to process global motion information. and Respectively represent the local hidden layer state features and the compensated global hidden layer state features learned by the two mapping RNN modules at the current time t. Specifically, step S3 also includes:

[0077] Step S31, the coupled mapping RNN module uses convolution to process the compensated global hidden layer state features and learns a soft mask W l Used to separate the local moving content from the background information in the compensated global hidden layer state features to obtain the local content features and background characteristics Among them, local content features and background characteristics They are:

[0078]

[0079] In the formula, is the global hidden layer state feature after compensation; R t is the mixed residual feature; is the global hidden layer state feature of the previous moment of the current video frame; W l is a soft mask; * indicates element-by-element multiplication.

[0080] Step S32: Since the mapping RNN module only processes information related to local motion, the local hidden layer state features of the previous moment of the current video frame are mapped to and local content features Input into the first mapping RNN module to obtain the local hidden layer state features at the current moment Specifically:

[0081]

[0082] The local hidden layer state features at the current moment and background characteristics Aggregate to obtain global information features The formula is:

[0083]

[0084] The global feature and the global hidden layer state features of the previous moment of the current video frame Input into the second mapping RNN module to obtain the global hidden layer state characteristics at the current moment The formula is:

[0085]

[0086] Step S3, inputting the global hidden layer state feature at the current moment into the upsampling module to reconstruct the video frame, and finally obtaining the reconstructed video;

[0087] Among them, in step S3, the global feature A spatial projection process is performed to generate appearance features, because in the mapping RNN module, appearance and motion information are represented together. Finally, the frame reconstruction is completed by stacking multiple upsampling, as follows:

[0088]

[0089] in represents the reconstructed image, f NB () represents the upsampling module, which is composed of multiple layers of convolution and upsampling operations stacked together. s For spatial projection, it can be achieved through a convolutional layer.

[0090] Embodiment 2

[0091] This embodiment discloses a video coding system based on implicit neural video representation;

[0092] like Figure 2 As shown, a video coding system based on implicit neural video representation includes:

[0093] The hidden layer state feature acquisition module is configured to: acquire the local hidden layer state feature and the global hidden layer state feature of the previous moment of the current video frame;

[0094] The residual feature compensation module is configured to: extract a mixed residual feature in the video frame based on the mixed residual grid, add the mixed residual feature to the global hidden layer state feature of the previous moment of the current video frame, and obtain the compensated global hidden layer state feature;

[0095] The video frame reconstruction module is configured to: input the compensated global hidden layer state features into the coupled mapping RNN module for iterative update to obtain the reconstructed video frame;

[0096] Among them, the coupled mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features;

[0097] Input the local content features and the local hidden layer state features of the previous moment of the current video frame into the first mapping RNN module to obtain the local hidden layer state features of the current moment; aggregate the local hidden layer state features of the current moment and the background features to obtain the global information features; input the global features and the global hidden layer state features of the previous moment of the current video frame into the second mapping RNN module to obtain the global hidden layer state features of the current moment;

[0098] The global hidden layer state features at the current moment are input into the upsampling module to reconstruct the video frame, and finally the reconstructed video is obtained.

[0099] Embodiment 3

[0100] The purpose of this embodiment is to provide a computer-readable storage medium.

[0101] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a video encoding method based on implicit neural video representation as described in Example 1.

[0102] Embodiment 4

[0103] The purpose of this embodiment is to provide an electronic device.

[0104] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a video encoding method based on implicit neural video representation as described in Example 1 are implemented.

[0105] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0106] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0107] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A video coding method based on implicit neural video representation, characterized in that: include: Obtain the local hidden layer state features and the global hidden layer state features of the current video frame at the previous moment; Extracting mixed residual features in the video frame based on the mixed residual grid, adding the mixed residual features to the global hidden layer state features of the previous moment of the current video frame to obtain the compensated global hidden layer state features; The compensated global hidden layer state features are input into the coupled mapping RNN module for iterative update to obtain the reconstructed video frame; Among them, the coupled mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features; Input the local content features and the local hidden layer state features of the previous moment of the current video frame into the first mapping RNN module to obtain the local hidden layer state features of the current moment; aggregate the local hidden layer state features of the current moment and the background features to obtain the global information features; input the global features and the global hidden layer state features of the previous moment of the current video frame into the second mapping RNN module to obtain the global hidden layer state features of the current moment; The global hidden layer state features at the current moment are input into the upsampling module to reconstruct the video frame, and finally the reconstructed video is obtained.

2. A video coding method based on implicit neural video representation as claimed in claim 1, characterized in that: The mixed residual grid is: R G ∈R L*C*H*W Where C, H, and W are the temporal resolution, number of channels, height, and width of the grid, respectively.

3. The video coding method based on implicit neural video representation according to claim 1, characterized in that: The coupled mapping RNN module uses convolution to process the compensated global hidden layer state features. The specific process of obtaining local content features and background features is as follows: The compensated global hidden layer state features are input into the convolutional layer to generate a soft mask to decompose the compensated global hidden layer state features into local content features. and background characteristics The formula is: In the formula, is the global hidden layer state feature after compensation; R t is the mixed residual feature; is the global hidden layer state feature after compensation; W l is a soft mask; * indicates element-by-element multiplication.

4. The video coding method based on implicit neural video representation according to claim 1, characterized in that: The local hidden layer state characteristics at the current moment are: In the formula, is the local hidden layer state feature at the current moment; f g () represents the processing operation of the mapping RNN module; P t () represents the motion information generation module; warp() represents the deformation mapping operation.

5. The video coding method based on implicit neural video representation according to claim 1, characterized in that: The global information features are specifically: In the formula, It is a global information feature.

6. A video coding method based on implicit neural video representation as claimed in claim 1, characterized in that: The global hidden layer state characteristics at the current moment are: In the formula, is the global hidden layer state feature.

7. A video coding method based on implicit neural video representation as claimed in claim 1, characterized in that: The global hidden layer state features at the current moment are input into the upsampling module to reconstruct the video frame. The process of finally obtaining the reconstructed video is as follows: Perform spatial projection on the global hidden layer state features to generate appearance features; use the upsampling module to reconstruct the video frame based on the appearance features; specifically: In the formula, represents the reconstructed video, f NB () represents the upsampling module, which is composed of multiple layers of convolution and upsampling operations stacked together; P s Represents spatial projection, which is achieved through a convolutional layer.

8. A video coding system based on implicit neural video representation, characterized in that include: The hidden layer state feature acquisition module is configured to: acquire the local hidden layer state feature and the global hidden layer state feature of the previous moment of the current video frame; The residual feature compensation module is configured to: extract a mixed residual feature in the video frame based on the mixed residual grid, add the mixed residual feature to the global hidden layer state feature of the previous moment of the current video frame, and obtain the compensated global hidden layer state feature; The video frame reconstruction module is configured to: input the compensated global hidden layer state features into the coupled mapping RNN module for iterative update to obtain the reconstructed video frame; Among them, the coupled mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features; Input the local content features and the local hidden layer state features of the previous moment of the current video frame into the first mapping RNN module to obtain the local hidden layer state features of the current moment; aggregate the local hidden layer state features of the current moment and the background features to obtain the global information features; input the global features and the global hidden layer state features of the previous moment of the current video frame into the second mapping RNN module to obtain the global hidden layer state features of the current moment; The global hidden layer state features at the current moment are input into the upsampling module to reconstruct the video frame, and finally the reconstructed video is obtained.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the video encoding method based on implicit neural video representation as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the video encoding method based on implicit neural video representation as described in any one of claims 1-7 are implemented.

Images