A deep video intelligent interframe coding method

Abstract

The application discloses a kind of deep video intelligent interframe coding method, method includes: key frame and non-key frame division are carried out to depth video frame, to judge the category of depth video frame, i.e. depth key frame or depth non-key frame;For depth key frame, using video codec is encoded and decoded, and the reconstructed depth key frame is obtained;For depth non-key frame, the encoding process of depth non-key frame is directly skipped at the encoding end, at the decoding end, according to the adjacent reconstructed depth key frame in time domain, high-quality depth non-key frame is generated through non-key frame generation network;Edge consistency loss is constructed, and the edge of depth frame is optimized by minimizing the distortion of reconstructed edge map;The trained edge-aware non-key frame generation network model is integrated into MIV coding platform to generate reconstructed depth non-key frame.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and video coding, and in particular to a deep video intelligent inter-frame coding method. Background Technology

[0002] Immersive video can display scene content observed from different positions and perspectives within a certain range, providing viewers with a more immersive visual experience. In recent years, immersive video has been widely researched and applied in fields such as live sports broadcasts, entertainment, and immersive education. To efficiently encode immersive video, the international standardization organization MPEG-I launched the MPEG immersive video (MIV) standard. The MIV standard can effectively compress 3D scene representations captured from multiple viewpoints, where each viewpoint contains color video and a corresponding depth video. Unlike color video with complex textures, depth video contains smoother content. The similar smooth content between depth video frames contains a large amount of temporal redundancy, necessitating the research of efficient inter-frame coding methods for depth video to further improve coding efficiency.

[0003] As a crucial component of the video coding framework, inter-frame prediction technology aims to utilize and reduce temporal redundancy in video. To achieve more accurate inter-frame prediction, a hierarchical B-frame prediction structure is typically employed, using previously encoded video frames from lower temporal layers as temporal references for the current frame to be encoded. Since the reconstruction quality of lower temporal layer video frames affects the inter-frame prediction of higher temporal layer video frames, lower quantization parameters are usually used to encode these frames to ensure higher reference quality. However, higher temporal layer video frames have less impact on the video coding process, yet they still employ the same coding methods as lower temporal layer frames, hindering improvements in coding efficiency. Considering that depth video frames contain a significant amount of smooth content, this smooth content can be easily predicted using temporal reference frames. Furthermore, depth video frames are not used for direct display in immersive video systems, and the smooth content of depth video can tolerate a certain degree of distortion.

[0004] In recent years, benefiting from the powerful feature extraction capabilities of deep learning technology, deep learning-based color video coding techniques have been widely studied and have achieved significant success. Inspired by this, scholars have focused on exploring deep learning-based deep video coding techniques to improve the coding efficiency of deep videos. Among these, Li et al. proposed a multi-resolution coding method for deep videos, reducing coding bitrate consumption by encoding low-resolution deep coding blocks and using convolutional neural networks to upsample the encoded low-resolution coding blocks to the original resolution. Zhang et al. proposed using convolutional neural networks to generate more accurate intra-frame prediction results for deep videos. These methods can effectively reduce spatial redundancy in deep videos and generate high-quality reconstructions. However, deep learning-based deep video coding techniques are still in their early stages, lacking methods to reduce temporal redundancy in deep videos, and there is still considerable room for improvement in deep video coding efficiency. Summary of the Invention

[0005] Since depth video contains a large amount of smooth content, acceptable quality depth video frames can be directly generated based on temporal reference frames, improving coding efficiency. This invention provides an intelligent inter-frame coding method for depth video, which reduces bit consumption by skipping the coding of non-key frames at the encoding end, and designs an edge-aware non-key frame generation network to reconstruct high-quality non-key frames at the decoding end, as detailed below:

[0006] A deep video intelligent inter-frame coding method, the method comprising:

[0007] Depth video frames are divided into keyframes and non-keyframes to determine the category of depth video frames, i.e., depth keyframes or depth non-keyframes.

[0008] For depth keyframes, a video codec is used for encoding and decoding to obtain the reconstructed depth keyframes; for depth non-keyframes, the encoding process of depth non-keyframes is skipped at the encoding end, and high-quality depth non-keyframes are generated at the decoding end based on the temporally adjacent reconstructed depth keyframes through a non-keyframe generation network.

[0009] An edge consistency loss is constructed to optimize the edges of depth frames by minimizing the distortion of the reconstructed edge map; the trained edge-aware non-key frame generation network model is integrated into the MIV encoding platform to generate reconstructed depth non-key frames.

[0010] The non-keyframes are the depth frames of the highest time layer; the depth frames of the remaining time layers are classified as depth keyframes.

[0011] Furthermore, the non-keyframe generation network consists of a distortion-based generation module and an edge-aware refinement module;

[0012] Based on a distortion-based generation module, coarse depth non-keyframes are generated by leveraging the temporal correlation between depth keyframes and depth non-keyframes. Design an edge-aware refinement module to refine coarse depth non-keyframes, making the edges of depth non-keyframes clear.

[0013] The edge-aware refinement module is as follows:

[0014] From the cascaded residual generator using a U-Net-based residual generator and Generated in The residual R1 between the original depth non-keyframe D1 and the original depth non-keyframe is used to obtain Missing edge details;

[0015] Using R1 compensation To obtain the final high-quality reconstructed frames, the edge details are improved. The expression is as follows:

[0016]

[0017] Where RG(·) represents a residual generator based on U-Net.

[0018] Furthermore, the edge consistency loss is:

[0019] Using the edge extraction unit from Extracting the reconstructed edge map The true edge map B1 is calculated from D1 using the classic Sobel edge detection filter, and the edge consistency loss L is calculated using the mean absolute error. B .

[0020] The beneficial effects of the technical solution provided by this invention are:

[0021] 1. This invention designs a deep video intelligent inter-frame coding method, which improves coding efficiency by effectively removing temporal redundancy;

[0022] 2. This invention designs an edge-aware non-key frame generation network to directly generate high-quality depth non-key frames. The designed network utilizes temporal correlation to generate video frames and refines the object boundaries of the reconstructed frames to reduce synthetic viewpoint distortion.

[0023] 3. This invention integrates the proposed keyframe coding method into the MIV reference software TMIV10.0. Compared with the MIV reference software TMIV10.0, the proposed method can effectively improve coding efficiency. Attached Figure Description

[0024] Figure 1A flowchart of a deep video intelligent inter-frame coding method;

[0025] Figure 2 This is a schematic diagram of the structure of a flow estimation unit based on U-Net;

[0026] Figure 3 This is a schematic diagram of the edge extraction unit. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0028] To overcome the shortcomings of existing technologies, embodiments of the present invention provide a deep video intelligent inter-frame coding method, see [link to relevant documentation]. Figure 1 The method includes the following steps:

[0029] I. Key Frame and Non-Key Frame Division in Depth Video

[0030] Design a method to classify depth video frames into keyframes and non-keyframes, to determine the category of depth video frames, i.e., depth keyframes or depth non-keyframes.

[0031] In a layered B-frame architecture, reconstructed depth frames from lower layers serve as references for predicting higher-layer depth frames, while reconstructed depth frames from the highest temporal layer are not used for predicting other depth frames. Therefore, since the reconstruction quality of the highest temporal layer depth frame typically does not affect inter-frame prediction of other temporal layer depth frames, this embodiment of the invention classifies the highest temporal layer depth frame as a depth non-critical frame. Furthermore, depth frames from other temporal layers are classified as depth critical frames because distortion in these depth frames can affect inter-frame prediction. The formula is expressed as follows:

[0032]

[0033] Among them, C i Let l be the category of the depth frame at time i. i l represents the time layer where the depth frame at time i is located. max This indicates the highest time level, with "key" representing a keyframe and "non-key" representing a non-keyframe.

[0034] II. Depth Keyframe and Depth Non-Keyframe Encoding

[0035] After obtaining the category of the depth video frame, different encoding methods are used to compress the depth video frame according to its category.

[0036] For depth keyframes, traditional video codecs are used for encoding and decoding to obtain the reconstructed depth keyframes. For depth non-keyframes, the encoding process for depth non-keyframes is skipped at the encoding end, thus eliminating the need to encode the motion vectors and residuals of depth non-keyframes, effectively saving the bitrate required for encoding. At the decoding end, high-quality depth non-keyframes are directly generated through a non-keyframe generation network based on temporally adjacent reconstructed depth keyframes.

[0037] III. Constructing a Non-Key Frame Generation Network

[0038] The input to the constructed non-keyframe generation network is the temporally nearest forward reconstruction depth keyframe. and backward reconstruction depth keyframes The output is the reconstructed depth non-keyframes. The constructed non-keyframe generation network consists of a distortion-based generation module and an edge-aware refinement module.

[0039] Specifically, a warp-based generation module is first used to generate coarse depth non-keyframes by leveraging the temporal correlation between depth keyframes and depth non-keyframes. Subsequently, an edge-aware refinement module was designed to refine the coarse depth non-keyframes, thereby making the edges of the depth non-keyframes clearer.

[0040] In the warp-based generation module, optical flow F = {F0, F2} is first estimated using a U-Net-based flow estimation unit to model the temporal correlation between reconstructed depth keyframes and current depth non-keyframes. Here, F0 represents the optical flow from the forward-reconstructed depth keyframe to the current depth non-keyframe, and F2 represents the optical flow from the backward-reconstructed depth keyframe to the current depth non-keyframe. Furthermore, the flow estimation unit simultaneously outputs a weight map V, representing the pixel values ​​of the forward-reconstructed depth keyframes warped according to the optical flow. The proportion in the matrix. Then, based on the estimated optical flow and weight map, the reconstructed depth keyframes of the network input are warped to obtain... The formula is expressed as follows:

[0041]

[0042]

[0043] Where [·] denotes a cascading operation, FE is a flow estimation unit based on U-Net, and Warping(·) represents a warping operation implemented using Adaptive Collaboration of Flows (AdaCof). Figure 2As shown, the flow estimation unit based on U-Net consists of nine stacked base blocks (numbered from left to right as the first base block, second base block, ..., ninth base block) and an output head. Each base block comprises three stacked convolutional layers, and the output head comprises three sub-blocks, each consisting of four stacked convolutional layers and one upsampling layer. To accelerate network convergence, the output features of the fifth base block are added to the upsampled output of the sixth base block, the fourth base block to the upsampled output of the seventh base block, the third base block to the upsampled output of the eighth base block, and the second base block to the upsampled output of the ninth base block.

[0044] Subsequently, the edge-aware refinement module was designed to... This involves refining the image to further improve the quality of depth-non-keyframes in edge regions. Specifically, this is achieved first by using a U-Net-based residual generator to refine the image from a cascaded... and Generated in The residual R1 between the original depth non-keyframe D1 and the original depth non-keyframe is used to obtain The missing edge details are addressed. The basic component of the U-Net-based residual generator is a gated convolutional block, which consists of three stacked gated convolutional layers. The residual generator comprises seven stacked gated convolutional blocks (from left to right: first gated convolutional block, second gated convolutional block, ..., to seventh gated convolutional block). To generate more accurate edge details, the features output by the second gated convolutional block are added to the features output by the fourth gated convolutional block, and the features output by the first gated convolutional block are added to the features output by the fifth gated convolutional block. R1 compensation is then applied. To obtain the final high-quality reconstructed frames, the edge details are improved. The formula is expressed as follows:

[0045]

[0046] Where RG(·) represents a residual generator based on U-Net.

[0047] IV. Constructing Edge Consistency Loss

[0048] An edge consistency loss is constructed to optimize the edges of depth frames by minimizing the distortion of the reconstructed edge map. First, an edge extraction unit is used to extract the edges from the depth frame. Extracting the reconstructed edge map Among them, such as Figure 3As shown, the edge extraction unit is a U-Net structure containing 10 stacked convolutional layers and one sigmoid layer. To better extract the edge map, the output features of the fourth convolutional layer are concatenated with the upsampled output features of the fifth convolutional layer, the output features of the third convolutional layer are concatenated with the upsampled output features of the sixth convolutional layer, the output features of the second convolutional layer are concatenated with the upsampled output features of the seventh convolutional layer, and the output features of the first convolutional layer are concatenated with the upsampled output features of the seventh convolutional layer.

[0049] Subsequently, the ground truth B1 of the edge map is calculated from D1 using the classic edge detection filter Sobel. Then, the edge consistency loss L is calculated using the mean absolute error. B The formula is expressed as follows:

[0050]

[0051] Furthermore, embodiments of the present invention also use reconstruction loss L. B The quality of non-keyframes in the constrained reconstruction depth is expressed by the following formula:

[0052]

[0053] Finally, the total loss L, which includes the joint reconstruction loss and the edge consistency loss, is expressed as follows:

[0054] L = L R +αL B

[0055] Where α represents the weight of the edge consistency loss, Represents the edge map of the reconstruction. This indicates a depth-non-keyframe for reconstruction.

[0056] V. Training a Non-Key Frame Generation Network for Edge Awareness

[0057] In this embodiment of the invention, reconstruction loss and edge consistency loss are used to train the non-key frame generation network to obtain an edge-aware non-key frame generation network model.

[0058] First, a warp-based generative module is trained using a reconstruction loss function. After the warp-based generative module converges, its parameters are fixed, and the weight α is set to 1. Then, an edge-aware refinement module and an edge extraction unit are trained using reconstruction loss and edge consistency loss.

[0059] VI. Integration into the MIV encoding platform

[0060] The trained edge-aware non-keyframe generation network model was integrated into the MIV encoding platform TMIV 10.0 to generate reconstructed depth non-keyframes.

[0061] At the encoding end, a frame classification flag is used to indicate the category of the current depth-coded frame, i.e., a depth keyframe or a depth non-keyframe. Furthermore, since the base viewpoint video is packed into the atlas along the vertical direction in the MIV standard, the height of the encoded atlas differs from the original base viewpoint height. Based on this, embodiments of the present invention encode the original height value of the base viewpoint into the bitstream, thereby enabling the decoding end to obtain the reconstructed depth keyframe of a single base viewpoint from the atlas based on the height and width of the base viewpoint.

[0062] At the decoding end, a traditional decoder is used to decode depth keyframes to obtain reconstructed depth keyframes. Then, an edge-aware non-keyframe generation network model is used to generate reconstructed depth non-keyframes for all base viewpoints on a viewpoint-by-view basis.

[0063] To provide high-quality temporal reference, the forward and backward reconstructed depth keyframes that are temporally closest to the target depth non-keyframes are selected as inputs to the edge-aware non-keyframe generation network model.

[0064] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep video intelligent inter-frame coding method, characterized in that, The method includes: Depth video frames are divided into keyframes and non-keyframes to determine their category, i.e., depth keyframes or depth non-keyframes. For depth keyframes, a video codec is used for encoding and decoding to obtain the reconstructed depth keyframes. For depth non-keyframes, the encoding process of the depth non-keyframes is skipped at the encoding end, and at the decoding end, the temporally nearest forward-reconstructed depth keyframes are used. and backward reconstruction depth keyframes As input to the non-keyframe generation network, the output is the reconstructed depth non-keyframe. The non-keyframes are the depth frames of the highest time layer; the depth frames of the remaining time layers are classified as depth keyframes. The non-keyframe generation network consists of a distortion-based generation module and an edge-aware refinement module. Based on a distortion-based generation module, coarse depth non-keyframes are generated by leveraging the temporal correlation between depth keyframes and depth non-keyframes. Design an edge-aware refinement module to refine coarse depth non-keyframes, making the edges of depth non-keyframes clear. An edge consistency loss is constructed to optimize the edges of reconstructed depth non-key frames by minimizing the distortion of the reconstructed edge map. The trained edge-aware non-key frame generation network model is integrated into the MIV encoding platform. At the encoding end, a frame classification flag is used to indicate the category of the current depth encoded frame, i.e., depth key frame or depth non-key frame. At the decoding end, a traditional decoder is used to decode the depth key frames to obtain the reconstructed depth key frames. The edge-aware non-key frame generation network model is then used to generate reconstructed depth non-key frames for all basic viewpoints on a viewpoint-by-view basis. The edge-aware refinement module is as follows: From the cascaded residual generator using a U-Net-based residual generator , and Generated in and original depth non-keyframes residuals between to obtain Missing edge details; use compensate To obtain the final high-quality reconstructed frames, the edge details are improved. The expression is as follows: ； in, This represents a residual generator based on U-Net.

2. The method for intelligent inter-frame coding of deep video according to claim 1, characterized in that, The edge consistency loss is: Using the edge extraction unit from Extracting the reconstructed edge map Using the classic edge detection filter Sobel from The true value of the edge map is calculated in the middle. Marginal consistency loss is calculated using mean absolute error. .

Images