A video encoding method and system based on implicit neural video representation

Abstract

The application provides a video coding method and system based on an implicit neural video representation, and belongs to the technical field of video coding; the method comprises the following steps: acquiring local and overall hidden layer state features of a previous moment of a current video frame; compensating the overall hidden layer state features of the previous moment based on hybrid residual features extracted by a residual grid; processing the compensated overall hidden layer state features by using a coupled mapping RNN network to acquire local content features and background features; acquiring global information features based on a first mapping RNN module; inputting the global features and the overall hidden layer state features of the previous moment into a second mapping RNN module to acquire the overall hidden layer state features of the current moment and inputting the overall hidden layer state features of the current moment into an up-sampling module to reconstruct the video frame, so that a reconstructed video is obtained. The method effectively enhances the processing capability of the time domain information between video frames, realizes the decomposition of the global and local motion between video frames, and improves the video frame reconstruction quality.

Description

Technical Field

[0001] This invention belongs to the field of video coding technology, and in particular relates to a video coding method and system based on implicit neural video representation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of networking and digitalization, artificial intelligence has shone brightly in various fields, achieving a series of groundbreaking accomplishments. In the video field, due to advancements in equipment and the development of related industries, more and more high-quality videos are participating in people's daily lives, playing an important role in information interaction. Therefore, video coding methods that simultaneously balance video compression efficiency and video reconstruction quality are needed.

[0004] Neural Representations for Videos (NeRV) is a frame-by-frame implicit neural representation method for video signals. It fits the video signal into an implicit neural network, transforming the video compression problem into a neural network compression problem. It can utilize common model compression methods such as model pruning, model quantization, and entropy coding to compress the implicit neural network that fits the video information, thereby achieving video compression. Compared with traditional video coding methods and learning-based video coding methods, NeRV offers faster decoding speeds and higher video compression efficiency. However, current NeRV methods have some limitations: they do not fully consider long-term inter-frame spatiotemporal relationships when processing video sequences, resulting in poor processing capabilities for inter-frame temporal information; they cannot effectively decompose global and local motion between video frames; and they cannot separately model regular and irregular residual information in the video. Therefore, there is still significant room for improvement in video representation quality and compression efficiency. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a video coding method and system based on implicit neural video representation. By utilizing a Coupled Warp RNN network, the temporal information between video frames can be effectively processed. At the same time, the global motion information and local motion information in the video are mapped and enhanced respectively, which can better process the motion information between video frames, model the regular and irregular information in the video signal, and thus effectively improve the video reconstruction quality and compression efficiency.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention provides a video coding method based on implicit neural video representation;

[0008] A video coding method based on implicit neural video representation, comprising:

[0009] Obtain the local hidden layer state features and global hidden layer state features of the previous moment in the current video frame;

[0010] Based on the hybrid residual mesh, the hybrid residual features in the video frame are extracted, and the hybrid residual features are added to the global hidden layer state features of the previous moment of the current video frame to obtain the compensated global hidden layer state features.

[0011] The compensated global hidden layer state features are input into the coupled mapping RNN module for iterative updating to obtain the reconstructed video frames;

[0012] Among them, the coupling mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features;

[0013] The local content features and the local hidden layer state features of the previous moment of the current video frame are input into the first mapping RNN module to obtain the local hidden layer state features of the current moment; the local hidden layer state features of the current moment and the background features are aggregated to obtain global information features; the global features and the global hidden layer state features of the previous moment of the current video frame are input into the second mapping RNN module to obtain the global hidden layer state features of the current moment.

[0014] The current global hidden layer state features are input into the upsampling module to reconstruct video frames, and finally the reconstructed video is obtained.

[0015] As a further technical solution, the hybrid residual mesh is:

[0016]

[0017] In the formula, L, C, H, and W are the temporal resolution, number of channels, height, and width of the grid, respectively.

[0018] As a further technical solution, the coupling mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features. The specific process is as follows:

[0019] The compensated global hidden layer state features are input into the convolutional layer to generate a soft mask, which decomposes the compensated global hidden layer state features into local content features. and background features The formula is:

[0020]

[0021]

[0022]

[0023] In the formula, The compensated global hidden layer state features; It is a mixed residual characteristic; The global hidden layer state features of the previous time step in the current video frame; It is a soft mask; This indicates element-wise multiplication.

[0024] As a further technical solution, the local hidden layer state characteristics at the current moment are as follows:

[0025]

[0026] In the formula, This represents the local hidden layer state characteristics at the current moment; This represents the processing operations of the mapping RNN module; Represents the motion information generation module; This represents a transformation mapping operation.

[0027] As a further technical solution, the global information feature specifically includes:

[0028]

[0029] In the formula, This is a global information feature.

[0030] As a further technical solution, the current global hidden layer state characteristics are as follows:

[0031]

[0032] In the formula, This represents the state features of the global hidden layer.

[0033] As a further technical solution, the current global hidden layer state features are input into the upsampling module to reconstruct video frames, and the process of finally obtaining the reconstructed video is as follows:

[0034] Spatial projection is performed on the global hidden layer state features to generate appearance features; the appearance features are then reconstructed from video frames using an upsampling module; specifically:

[0035]

[0036] In the formula, The video represents the post-reconstruction period. The upsampling module is composed of multiple layers of convolution and upsampling operations stacked together. This represents spatial projection, achieved through a convolutional layer.

[0037] A second aspect of the present invention provides a video coding system based on implicit neural video representation.

[0038] A video coding system based on implicit neural video representation, comprising:

[0039] The hidden layer state feature acquisition module is configured to: acquire the local hidden layer state features and global hidden layer state features of the previous moment of the current video frame;

[0040] The residual feature compensation module is configured to: extract the hybrid residual features in the video frame based on the hybrid residual grid, add the hybrid residual features to the global hidden layer state features of the previous moment of the current video frame, and obtain the compensated global hidden layer state features.

[0041] The video frame reconstruction module is configured to input the compensated global hidden layer state features into the coupled mapping RNN module for iterative updates to obtain the reconstructed video frames.

[0042] Among them, the coupling mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features;

[0043] The local content features and the local hidden layer state features of the previous moment of the current video frame are input into the first mapping RNN module to obtain the local hidden layer state features of the current moment; the local hidden layer state features of the current moment and the background features are aggregated to obtain global information features; the global features and the global hidden layer state features of the previous moment of the current video frame are input into the second mapping RNN module to obtain the global hidden layer state features of the current moment.

[0044] The current global hidden layer state features are input into the upsampling module to reconstruct video frames, and finally the reconstructed video is obtained.

[0045] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a video coding method based on implicit neural video representation as described in the first aspect of the present invention.

[0046] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a video coding method based on implicit neural video representation as described in the first aspect of the present invention.

[0047] The above one or more technical solutions have the following beneficial effects:

[0048] (1) This invention decomposes video into regular, structured information and irregular information, proposes a multi-scale motion representation and compensation module based on Coupled WarpRNN, and further integrates temporal processing into a neural network based on global and local motion decomposition, which effectively enhances the ability to process temporal information between video frames and realizes the decomposition of global and local motion between video frames.

[0049] (2) The present invention also uses hybrid residual mesh and network reuse to represent the remaining irregular motion and appearance information, providing more detailed motion and appearance information for video frame reconstruction, which greatly improves the quality of video reconstruction.

[0050] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0051] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0052] Figure 1 This is a flowchart of the method in the first embodiment.

[0053] Figure 2 This is a system structure diagram of the second embodiment. Detailed Implementation

[0054] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0055] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0056] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0057] Implicit Neural Representation (INR) is a neural network-based representation method that uses a deep neural network as a function approximator to map input parameters to the feature space of an object. Neural Radiance Fields (NeRF) utilizes the idea of ​​INR, extracting the geometric shape and texture information of an object from images from multiple perspectives using a neural network, and then generating a continuous 3D radiance field. This allows for the representation of highly realistic 3D models at arbitrary angles and distances, achieving remarkable results. Neural Representations for Videos (NeRV), building upon methods such as NeRF, is an implicit neural representation method that processes video signals frame by frame, fitting the video signal into an implicit neural network. Given a video sequence... Depend on Frame image composition, Implicit neural video representation networks will model indexes. Mapping to corresponding video frames:

[0058]

[0059] Implicit neural video representation transforms the video compression problem into a neural network compression problem. It can utilize common model compression methods such as model pruning, model quantization, and entropy coding to compress the implicit neural network fitting video information, thereby achieving video compression. However, implicit neural video representation methods have some limitations. They do not fully consider long-term inter-frame spatiotemporal relationships when processing video sequences, and they do not separately model regular and irregular residual information in the video, leaving significant room for improvement in video representation quality and compression efficiency. To address these issues, this invention uses a Coupled Warp RNN module to process the temporal information between video frames, while simultaneously mapping and enhancing global and local motion information in the video to model the regular information in the video signal. A hybrid residual feature grid is used to interpolate residual features from the hybrid residual feature grid based on the video frame index, supplementing the irregular information that neural networks struggle to learn. Finally, in the decoding stage, the features of the obtained video frames are reconstructed by an upsampling module to obtain the reconstructed video frames. This invention effectively enhances the processing capability of temporal information between video frames and realizes the decomposition of global and local motion between video frames, thereby improving the reconstruction quality of video frames. The invention will now be described in detail with reference to specific embodiments.

[0060] Example 1

[0061] This embodiment discloses a video coding method based on implicit neural video representation, such as... Figure 1 As shown, a video coding method based on implicit neural video representation includes:

[0062] Step S1: Obtain the local hidden layer state features and global hidden layer state features of the previous moment of the current video frame;

[0063] In recurrent neural networks (RNNs), the hidden state represents the network's "memory" or "summary" of previously seen data as it processes sequential data. Specifically, the hidden state is a vector that is updated at each time step to contain a compressed representation of all information from the sequence up to that point. Video frames at any given time, hidden layer state Features representing the video frame at the previous moment have a motion relationship with the input at this moment. Motion information can be extracted using convolutional layers, mapped onto the features of the current frame, and then processed by a recurrent neural network. For the first frame without a previous hidden state, a learnable tensor is used to represent the hidden state. This can be considered an anchor frame, preserving the main structure of the video and used to predict all frames. The video frame features at the current moment... As shown in the following formula:

[0064]

[0065] In the formula, For the processing operations of the mapping RNN module, The input representing the current moment can be viewed here as . Represents the motion information generation module. This represents a transformation mapping operation.

[0066] Based on this, in step S1, the features of the current video frame are predicted by obtaining the local hidden layer state features and global hidden layer state features of the previous moment of the current video frame.

[0067] Step S2: Extract the hybrid residual features from the video frame based on the hybrid residual grid, and add the hybrid residual features to the global hidden layer state features of the previous time step to obtain the compensated global hidden layer state features.

[0068] In the process of video frame reconstruction, it is necessary not only to provide efficient representations of the regular and structured information in the video, but also to process the irregular information in the video to provide more detailed motion and appearance information during the video frame reconstruction process. Although neural networks can provide efficient representations of the regular and structured information in the video, relying entirely on neural networks may lead to the loss of irregular information, resulting in a significant deterioration in video frame reconstruction performance.

[0069] Since motion and appearance information in video are highly correlated in lossy representations, this embodiment models these irregularities together as a learnable hybrid mesh structure, employing a hybrid residual grid (MRG) to learn irregular motion and appearance information that neural networks cannot effectively represent. The MRG is constructed along the time dimension; for a video containing T frames, it can be constructed as follows:

[0070]

[0071] Where L, C, H, and W represent the temporal resolution, number of channels, height, and width of the grid, respectively. To improve representation efficiency, L is typically set to be less than the sequence length. When specifying the index of a frame... At that time, the mixed residual characteristics at the corresponding time point From the mixed residual mesh using bilinear interpolation It is obtained by interpolating over time.

[0072] The obtained hybrid residual features are first processed through a convolutional layer, then added to the global hidden layer state features to obtain the compensated global hidden layer state features, which are then input into the Coupled Mapping RNN module. By utilizing the hybrid residual features for compensation, more detailed motion and appearance information can be provided for subsequent temporal processing and frame reconstruction. In this embodiment, although the hidden layer can be learned time-by-time by the Coupled Mapping RNN module alone, its reconstruction effect is poor. By introducing a hybrid residual feature grid to supplement additional information, the hybrid residual feature grid and the Coupled Mapping RNN module are retrained together. Since the hybrid residual feature grid tends to learn some irregular information, and the Coupled Mapping RNN module is a neural network structure that is more likely to learn regular information, training them together can provide more detailed motion and appearance information for video frame reconstruction, greatly improving the quality of video reconstruction.

[0073] Step S3: Input the compensated global hidden layer state features into the coupled mapping RNN module for iterative updates to obtain the reconstructed video frames;

[0074] Because Convolutional Gated Recurrent Units (ConvGRUs) have fewer parameters and better performance when processing video data, the Coupled Mapping RNN module uses ConvGRUs as the basic RNN network. Therefore, the formula for the video frame features at the current moment can be expanded as follows:

[0075]

[0076]

[0077]

[0078]

[0079] In the formula, and These represent the update gate and reset gate in a recurrent neural network, respectively. and These represent the corresponding convolutional network weights. This is an intermediate state of the hidden layer, used to update the hidden layer. Represents the Sigmoid activation function. It is element-wise multiplication.

[0080] Furthermore, the coupled mapping RNN module includes two mapping RNN (WarpRNN) modules, where the first mapping RNN module is used to learn information related to local motion, while the second mapping RNN module is used to process global motion information. In this embodiment, and They represent the current time. The two mapping RNN modules learn the local hidden layer state features and the compensated global hidden layer state features, respectively. Specifically, step S3 also includes:

[0081] Step S31: The coupling mapping RNN module uses convolution to process the compensated global hidden layer state features and learns a soft mask. This is used to separate local motion content from background information in the compensated global hidden layer state features, and obtain local content features. and background features Among them, local content features and background features They are respectively:

[0082]

[0083]

[0084]

[0085] In the formula, The compensated global hidden layer state features; It is a mixed residual characteristic; The global hidden layer state features of the previous time step in the current video frame; It is a soft mask; This indicates element-wise multiplication.

[0086] Step S32: Since the mapping RNN module only processes information related to local motion, it maps the local hidden layer state features of the previous time step of the current video frame. and local content features The input is fed into the first mapping RNN module to obtain the local hidden layer state features at the current time step. Specifically:

[0087]

[0088] The local hidden layer state features at the current moment and background features Aggregate to obtain global information features The formula is:

[0089]

[0090] The global features Features of the global hidden layer state at the previous time step of the current video frame The input is fed into the second mapping RNN module to obtain the global hidden layer state features at the current time step. The formula is:

[0091] .

[0092] Step S3: Input the current global hidden layer state features into the upsampling module to reconstruct the video frames, and finally obtain the reconstructed video.

[0093] In step S3, global features are processed. Spatial projection processing is performed to generate appearance features because appearance and motion information are represented together in the mapping RNN module. Finally, frame reconstruction is completed by stacking multiple upsampling operations, as shown in the equation:

[0094]

[0095] in This represents the image after reconstruction. The upsampling module is composed of multiple layers of convolution and upsampling operations stacked together. Spatial projection can be achieved using a convolutional layer.

[0096] Example 2

[0097] This embodiment discloses a video coding system based on implicit neural video representation;

[0098] like Figure 2 As shown, a video coding system based on implicit neural video representation includes:

[0099] The hidden layer state feature acquisition module is configured to: acquire the local hidden layer state features and global hidden layer state features of the previous moment of the current video frame;

[0100] The residual feature compensation module is configured to: extract the hybrid residual features in the video frame based on the hybrid residual grid, add the hybrid residual features to the global hidden layer state features of the previous moment of the current video frame, and obtain the compensated global hidden layer state features.

[0101] The video frame reconstruction module is configured to input the compensated global hidden layer state features into the coupled mapping RNN module for iterative updates to obtain the reconstructed video frames.

[0102] Among them, the coupling mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features;

[0103] The local content features and the local hidden layer state features of the previous moment of the current video frame are input into the first mapping RNN module to obtain the local hidden layer state features of the current moment; the local hidden layer state features of the current moment and the background features are aggregated to obtain global information features; the global features and the global hidden layer state features of the previous moment of the current video frame are input into the second mapping RNN module to obtain the global hidden layer state features of the current moment.

[0104] The current global hidden layer state features are input into the upsampling module to reconstruct video frames, and finally the reconstructed video is obtained.

[0105] Example 3

[0106] The purpose of this embodiment is to provide a computer-readable storage medium.

[0107] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a video coding method based on implicit neural video representation as described in Embodiment 1.

[0108] Example 4

[0109] The purpose of this embodiment is to provide an electronic device.

[0110] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a video coding method based on implicit neural video representation as described in Embodiment 1.

[0111] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0112] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0113] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A video coding method based on implicit neural video representation, characterized in that, include: Obtain the local hidden layer state features and global hidden layer state features of the previous moment in the current video frame; Based on the hybrid residual mesh, the hybrid residual features in the video frame are extracted, and the hybrid residual features are added to the global hidden layer state features of the previous moment of the current video frame to obtain the compensated global hidden layer state features. The compensated global hidden layer state features are input into the coupled mapping RNN module for iterative updating to obtain the reconstructed video frames; Among them, the coupling mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features; The local content features and the local hidden layer state features of the previous moment of the current video frame are input into the first mapping RNN module to obtain the local hidden layer state features at the current moment; the local hidden layer state features at the current moment and the background features are aggregated to obtain global information features; the global information features and the global hidden layer state features of the previous moment of the current video frame are input into the second mapping RNN module to obtain the global hidden layer state features at the current moment. The current global hidden layer state features are input into the upsampling module to reconstruct video frames, and finally the reconstructed video is obtained.

2. The video coding method based on implicit neural video representation as described in claim 1, characterized in that, The hybrid residual mesh is: In the formula, L, C, H, and W represent the temporal resolution, number of channels, height, and width of the grid, respectively.

3. The video coding method based on implicit neural video representation as described in claim 1, characterized in that, The coupling mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features. The specific process is as follows: The compensated global hidden layer state features are input into the convolutional layer to generate a soft mask, which decomposes the compensated global hidden layer state features into local content features. and background features The formula is: In the formula, The compensated global hidden layer state features; It is a mixed residual characteristic; The global hidden layer state features of the previous time step in the current video frame; It is a soft mask; This indicates element-wise multiplication.

4. The video coding method based on implicit neural video representation as described in claim 1, characterized in that, The local hidden layer state characteristics at the current moment are: In the formula, This represents the local hidden layer state characteristics at the current moment; This represents the processing operations of the first mapping RNN module; This represents the local hidden layer state features of the previous time step in the current video frame; For local content features; Represents the motion information generation module; This represents a transformation mapping operation.

5. The video coding method based on implicit neural video representation as described in claim 1, characterized in that, The specific features of the global information are: In the formula, Features of global information; This represents the local hidden layer state characteristics at the current moment; Background features.

6. The video coding method based on implicit neural video representation as described in claim 1, characterized in that, The current global hidden layer state features are as follows: In the formula, This represents the global hidden layer state features; This represents the processing operations of the second mapping RNN module; () represents a transformation mapping operation; The global hidden layer state features of the previous time step in the current video frame; This is a global information feature.

7. The video coding method based on implicit neural video representation as described in claim 1, characterized in that, The process of inputting the current global hidden layer state features into the upsampling module to reconstruct video frames and finally obtaining the reconstructed video is as follows: Spatial projection is performed on the global hidden layer state features to generate appearance features; the appearance features are then reconstructed from video frames using an upsampling module; specifically: In the formula, The video represents the post-reconstruction period. The upsampling module is composed of multiple layers of convolution and upsampling operations stacked together. This represents spatial projection, achieved through a convolutional layer; This represents the state features of the global hidden layer.

8. A video coding system based on implicit neural video representation, characterized in that, include: The hidden layer state feature acquisition module is configured to: acquire the local hidden layer state features and global hidden layer state features of the previous moment of the current video frame; The residual feature compensation module is configured to: extract the hybrid residual features in the video frame based on the hybrid residual grid, add the hybrid residual features to the global hidden layer state features of the previous moment of the current video frame, and obtain the compensated global hidden layer state features. The video frame reconstruction module is configured to input the compensated global hidden layer state features into the coupled mapping RNN module for iterative updates to obtain the reconstructed video frames. Among them, the coupling mapping RNN module uses convolution to process the compensated global hidden layer state features to obtain local content features and background features; The local content features and the local hidden layer state features of the previous moment of the current video frame are input into the first mapping RNN module to obtain the local hidden layer state features at the current moment; the local hidden layer state features at the current moment and the background features are aggregated to obtain global information features; the global information features and the global hidden layer state features of the previous moment of the current video frame are input into the second mapping RNN module to obtain the global hidden layer state features at the current moment. The current global hidden layer state features are input into the upsampling module to reconstruct video frames, and finally the reconstructed video is obtained.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in a video coding method based on implicit neural video representation as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the video coding method based on implicit neural video representation as described in any one of claims 1-7.

Images