Ultra-low bit rate face video hybrid compression method and system based on bidirectional frame prediction

Abstract

This invention discloses a bidirectional frame prediction method for facial videos, comprising: sampling three frames of facial video in the time domain; obtaining key point heat maps corresponding to each of the three frames; and reconstructing an intermediate frame image by combining the three frames and the key point heat maps. This invention facilitates the synthesis of smooth and fluent videos in the time domain, enhancing the viewer's subjective experience. The invention also discloses an extremely low-bitrate hybrid compression method for facial videos based on this method. This method combines the advantages of traditional encoding techniques for pixel-level precision restoration with the advantages of deep learning techniques for high-quality reconstruction based on simplified information, achieving the goal of synthesizing high-quality facial videos at extremely low bit rates.

Description

Technical Field

[0001] The present invention relates to the field of face video compression, and in particular to an extremely low bit rate face video hybrid compression method and system based on bidirectional frame prediction. Background Art

[0002] Video calls are a common way for people to communicate in daily life, and their use cases are increasing. However, in the case of poor network quality, video calls are often forced to be interrupted, seriously affecting the user's call experience. Therefore, how to achieve stable ultra-low bitrate video calls under poor network quality is a research hotspot, and research on facial video compression technology is becoming increasingly important. Existing facial video compression methods can be divided into two categories: traditional encoding methods and deep learning-based methods.

[0003] Traditional video coding methods offer the advantage of pixel-accurate restoration, with VVC being the most advanced standard. Compared to HEVC, VVC reduces bitrate by approximately 50% while maintaining comparable visual quality. However, these traditional methods, such as VVC, do not differentiate between video content and instead compress all videos in the same manner, despite the fact that facial videos have greater potential for compression.

[0004] Deep learning has great potential for facial video compression due to its ability to generate information based on simplified information. Researchers have proposed several deep learning-based methods for facial video compression. These methods often transmit partial video frames as reference frames, then combine the transmitted simplified facial representation with deep learning methods to synthesize complete facial video frames. This significantly reduces the video bitrate while ensuring high-quality reconstruction of the video image. Some of these methods use static reference frames, making it difficult to maintain good performance under scene changes or large pose changes, and thus unable to achieve high-fidelity restoration. Others add synthesized frames to the reference frame pool, which easily leads to error accumulation and prevents pixel-level precision restoration, which happens to be the strength of traditional coding methods. Furthermore, most existing deep learning-based facial video compression methods have difficulty dynamically adjusting the bitrate, limiting their practicality. Summary of the Invention

[0005] The present invention provides a bidirectional frame prediction method for face videos and a very low bit rate face video hybrid compression method and system.

[0006] According to a first aspect of the present invention, the present invention provides a bidirectional frame prediction method for face videos, comprising:

[0007] Sample three frames of face video in the time domain;

[0008] Obtaining key point heat maps corresponding to each of the three frames of image;

[0009] The three frames of images and the key point heat map are combined to reconstruct an intermediate frame image.

[0010] Preferably, the collecting of three frames of facial video in the time domain includes:

[0011] The first and last frames are collected as key frame images, denoted as Fk1 and Fk2;

[0012] Use Fk1 and Fk2 to predict the intermediate frame between the two frames, that is, the non-key frame image Ft;

[0013] The obtaining of the key point heat maps corresponding to the three frames of images includes:

[0014] For the non-key frame image Ft between two key frames, the key point coordinates extracted by the key point extractor based on convolutional neural network are used to generate the key point heat map Ht;

[0015] For the key frame images Fk1 and Fk2, a key point extractor based on a convolutional neural network is used to obtain key point heat maps Hk1 and Hk2;

[0016] Combining the three frames of images and the key point heat map to reconstruct an intermediate frame image includes:

[0017] The key frame image Fk1, the key point heat map Hk1 of the key frame image Fk1, and the key point heat map Ht of the non-key frame image Ft are input into a dense motion network based on a convolutional neural network to obtain a dense optical flow Tk1 and an occlusion map Ok1;

[0018] Input the key frame image Fk1, the dense optical flow Tk1 and the occlusion map Ok1 into a generator based on a convolutional neural network to obtain a non-key frame image Ft1' reconstructed using the key frame image Fk1;

[0019] The key frame image Fk2, the key point heat map Hk2 of the key frame image Fk2, and the key point heat map Ht of the non-key frame image Ft are input into a dense motion network based on a convolutional neural network to obtain a dense optical flow Tk2 and an occlusion map Ok2;

[0020] Input the key frame image Fk2, the dense optical flow Tk2 and the occlusion map Ok2 into a generator based on a convolutional neural network to obtain a non-key frame image Ft2' reconstructed using the key frame image Fk2;

[0021] Input the key point heat map Ht of the non-key frame image Ft, the key point heat map Hk1 of the key frame image Fk1, and the key point heat map Hk2 of the key frame image Fk2 into the mask predictor based on the convolutional neural network to obtain the face mask Mt;

[0022] The face mask Mt is weighted and applied to the non-key frame reconstructed images Ft1 ′ and Ft2 ′ to obtain a final reconstructed non-key frame reconstructed image Ft′.

[0023] Preferably, the acquisition of the key point heat maps Ht, Hk1 and Hk2 includes:

[0024] The non-key frame image Ft and the key frame images Fk1 and Fk2 are respectively used as inputs of a key point extractor based on a convolutional neural network;

[0025] The key point extractor extracts sparse key point information of the input image through a convolutional neural network and generates key point heat maps Ht, Hk1, and Hk2 corresponding to the input image.

[0026] Preferably, the key frame image Fk1, the dense optical flow Tk1 and the occlusion map Ok1 are input into a generator based on a convolutional neural network to obtain a non-key frame image Ft1' reconstructed using the key frame image Fk1.

[0027] or,

[0028] The key frame image Fk2, dense optical flow Tk2 and occlusion map Ok2 are input into the generator based on convolutional neural network to obtain the non-key frame image Ft2' reconstructed using the key frame image Fk2;

[0029] include:

[0030] The generator network extracts features of the keyframe image;

[0031] Use dense optical flow Tk1 to distort the features of key frame images;

[0032] Multiply the distorted features with the occlusion map;

[0033] The final non-keyframe image is obtained by decoding the features after distortion and occlusion processing.

[0034] Preferably, the acquisition of the non-key frame reconstructed image Ft' includes:

[0035] The mask predictor uses the key point heat maps Hk1, Hk2 and Ht of the key frame images Fk1, Fk2 and the non-key frame image Ft to predict a face mask Mt;

[0036] Mt represents the contribution of the non-key frame images reconstructed using two key frame images to the final non-key frame reconstructed image;

[0037] The size of Mt is the same as the reconstructed image, and the weighting formula is as follows:

[0038] F′ t =M t *F′ t1 +(1-M t )*F′ t2 .

[0039] According to a second aspect of the present invention, a method for hybrid compression of face videos at an extremely low bit rate is provided, comprising:

[0040] Get stream 1 and stream 2 from the complete video sequence at the sender;

[0041] Transmitting the code stream 1 and the code stream 2 to a receiving end;

[0042] The code stream 1 is decoded to obtain a reconstructed key frame;

[0043] Performing lossless decoding on the bitstream 2 to obtain key point data of non-key frames, combining the reconstructed key frames, and reconstructing the non-key frames using the bidirectional frame prediction method;

[0044] The reconstructed key frames and the reconstructed non-key frames are arranged in time sequence to obtain a final video sequence.

[0045] Preferably, the step of obtaining the bitstream 1 and the bitstream 2 from the complete video sequence at the transmitting end includes:

[0046] For the complete video sequence at the transmitter, set the sampling interval N, that is, sample one frame every N frames as a key frame, regard the sampled key frame as a new video sequence and encode it using the VVenC encoder to obtain bitstream 1;

[0047] The key point extractor based on the convolutional neural network is used to detect several Uint8 type key points of each non-key frame that have not been sampled, and the detected Uint8 type key point data is losslessly encoded to obtain code stream 2.

[0048] Preferably, the lossless coding method, i.e., using intra-frame prediction to eliminate spatial redundancy and using inter-frame prediction to eliminate temporal redundancy, includes:

[0049] For the first non-key frame, intra-frame prediction is used to remove spatial redundancy, that is, the (N-1)th key point is used to predict the Nth key point;

[0050] Inter-frame prediction uses the key points in the (K-1)th frame to predict the corresponding key points in the Kth frame, and encodes the residual of the key point data values ​​between the (K-1)th frame and the Kth frame.

[0051] Preferably, the lossless decoding of the bitstream 2 to obtain key point data of non-key frames, combining the reconstructed key frames, and reconstructing the non-key frames using the bidirectional frame prediction method includes:

[0052] After receiving the code stream 2, the receiving end decodes it to obtain the key point data of the Uint8 type of the non-key frame;

[0053] The bidirectional frame prediction method predicts the image of the corresponding non-key frame based on the key point data of the two adjacent reconstructed key frames and the decoded middle non-key frame, thereby completing the reconstruction of the non-key frame.

[0054] According to a third aspect of the present invention, there is provided an extremely low bit rate face video hybrid compression system, comprising:

[0055] The sending module samples the key frames from the complete video sequence, then encodes the sampled key frames using VVenC to obtain the bitstream 1, and extracts the key point data of the non-key frames and performs lossless encoding to obtain the bitstream 2;

[0056] The receiving module decodes the received bitstream 1 using the VVenC decoder to obtain a reconstructed key frame; decodes the received bitstream 2 to obtain key point data of non-key frames, and then reconstructs the non-key frames using the bidirectional frame prediction method in combination with the reconstructed key frames;

[0057] Finally, the reconstructed key frames and the reconstructed non-key frames are arranged in time sequence to obtain a final video sequence.

[0058] Preferably, the sending module includes:

[0059] A key frame sampling unit, wherein the key frame sampling unit samples the complete video sequence at a sampling interval N, that is, sampling one frame as a key frame every N frames;

[0060] A video encoding unit, wherein the video encoding unit regards the sampled key frames as a new video sequence and encodes it using a VVenC encoder to obtain a bit stream 1;

[0061] A key point extraction unit, wherein the key point extraction unit uses the key point extractor based on the convolutional neural network to extract a plurality of Uint8 type key points of each non-key frame;

[0062] A key point encoding unit encodes the extracted key point data of Uint8 type using the lossless encoding scheme to obtain a code stream 2.

[0063] Preferably, the receiving module includes:

[0064] A video decoding unit, wherein the video decoding unit decodes the bit stream 1 using a VVenC decoder to obtain a reconstructed key frame;

[0065] A key point decoding unit, which decodes the code stream 2 to obtain key point data of the Uint8 type of non-key frame;

[0066] A non-key frame reconstruction unit is configured to reconstruct the non-key frame using the key point data of the decoded non-key frame in combination with the reconstructed key frame using the bidirectional frame prediction method.

[0067] A sorting unit arranges the reconstructed key frames and the reconstructed non-key frames in time sequence to obtain a final video sequence.

[0068] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0069] In an embodiment of the present invention, a bidirectional frame prediction method for face videos uses key points from two preceding and succeeding key frames and a non-key frame in the middle to jointly predict the non-key frame in the middle. This method solves the temporal jitter problem between non-key frames and key frames during unidirectional frame prediction, facilitates the synthesis of smooth and fluent videos in the temporal domain, and enhances the viewer's subjective experience.

[0070] A hybrid compression method for extremely low-bitrate facial videos in an embodiment of the present invention combines the advantages of traditional coding techniques for pixel-level precision restoration with the advantages of deep learning techniques for high-quality reconstruction based on simplified information, achieving the goal of synthesizing high-quality facial videos at extremely low bitrates.

[0071] The hybrid facial video compression system based on a bidirectional frame prediction method for facial videos in this embodiment of the present invention can dynamically adjust the keyframe sampling interval or the QP value used when encoding keyframes using VVenC based on network conditions, thereby achieving dynamic bitrate adjustment. Furthermore, the system is applicable to any existing encoder or configuration to meet different encoding requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0073] Figure 1 Schematic diagram of a flow chart of a bidirectional frame prediction method for face videos according to an embodiment of the present invention;

[0074] Figure 2 Schematic diagram of the structure of a bidirectional frame prediction method for face videos according to an embodiment of the present invention;

[0075] Figure 3 Schematic diagram of the process of the extremely low bit rate face video hybrid compression method according to an embodiment of the present invention;

[0076] Figure 4 Schematic diagram of the architecture of the extremely low bit rate face video hybrid compression method according to an embodiment of the present invention;

[0077] Figure 5 A schematic diagram of the components and architecture of an extremely low bit rate face video hybrid compression system according to an embodiment of the present invention;

[0078] Figure 6 This is a performance comparison chart of the extremely low bit rate face video hybrid compression method and VVenC in an embodiment of the present invention;

[0079] Figure 7 A schematic diagram showing the effect of a facial image frame synthesized by the bidirectional frame prediction method according to an embodiment of the present invention;

[0080] Figure 8 This is a comparison chart of the generation results of the extremely low bit rate face video hybrid compression method and VVenC according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0082] To address the shortcomings of existing technologies, a hybrid coding scheme combining traditional coding techniques with deep generative technology is urgently needed. This approach leverages the advantages of traditional coding for pixel-accurate restoration and deep learning for high-quality reconstruction based on minimal information, thereby enabling high-quality reconstruction of facial videos at extremely low bitrates. Furthermore, dynamically adjusting the bitrate based on real-time network bandwidth is a major challenge that needs to be addressed.

[0083] Based on the above reasons, the present invention provides an embodiment, a bidirectional frame prediction method for face videos, comprising:

[0084] Sample three frames of face video in the time domain;

[0085] Obtain the key point heat maps corresponding to each of the three frames of images;

[0086] Combining the three frames of images and the key point heat map, the non-key frame images corresponding to the three frames of images are reconstructed.

[0087] Figure 1 Schematic diagram of a flow chart of a bidirectional frame prediction method for face videos further optimized based on the above embodiment. Figure 2 Schematic diagram of the structure of the bidirectional frame prediction method for face videos according to this embodiment.

[0088] Bidirectional frame prediction methods for face videos include:

[0089] S101: For a face video composed of a series of continuous frames, three frames of images are sampled in the time domain, with the first and last two frames being key frame images, denoted as F k1 、F k2 , use F k1 and F k2 The prediction interval is the intermediate frame between two frames, that is, the non-key frame image F t For the non-key frame image F between two key frames t , using the key point coordinates extracted by the designed key point extractor based on convolutional neural network to generate the key point heat map H t For the key frame image F k1 、F k2 , also use the key point extractor based on convolutional neural network to obtain the key point heat map H k1 、H k2 ;

[0090] S102: Using key frame image F k1 For example, the key frame image F k1 , key frame image F k1 Key point heat map H k1 And the non-keyframe image F t Key point heat map H t Input into the dense motion network based on convolutional neural network to obtain dense optical flow T k1 and occlusion map O k1 . Then the key frame image F k1 , dense optical flow T k1 and occlusion map O k1 Input into the generator based on convolutional neural network to obtain the key frame image F k1 Reconstructed non-keyframe image F t1 '. Similarly, we can get the key frame image F k2 Get the reconstructed non-key frame image F t2 ';

[0091] S103: At the same time, the non-key frame image F t Key point heat map H t , key frame image F k1 Key point heat map H k1 and key frame image F k2 Key point heat map H k2 Input into the mask predictor based on convolutional neural network to obtain the face mask M t . Mask the face M t The weighted effect is to use the key frame image F k1 、F k2 The obtained non-keyframe reconstructed image F t1 '、F t2 'The final reconstructed non-key frame reconstructed image F is obtained t '.

[0092] In a preferred embodiment of the present invention, S101 is performed, and the key point extractor based on the convolutional neural network is a variant of the UNet model. The input of the key point extractor is any face image, and the output is a number of Uint8 type key point data extracted from the image. The number of key points extracted from each frame image is user-defined. Typically, the number of key points can be set to 10, that is, 10 key points are extracted from each frame image. In this step, the key frame image F is obtained using the key point extractor. k1 、F k2 and non-keyframe image F t The keypoint coordinates are then used to generate a keypoint heatmap, which serves as the input for subsequent modules. This step ultimately results in three keypoint heatmaps for two keyframe images and one non-keyframe image.

[0093] In a preferred embodiment of the present invention, S102 is performed, and the input of the dense motion network based on the convolutional neural network is the key frame image F k1 , key frame image F k1 Key point heat map H k1 And the non-keyframe image F t Key point heat map H t The dense motion network estimates the dense optical flow T from the non-keyframe image to the keyframe image k1 and occlusion map O k1 .

[0094] When using keyframe images to generate non-keyframe images, some areas of the image can be distorted from the keyframe images, while other areas need to be generated using subsequent generator modules. k1 The features used to distort the key frame image. k1The occluded part represents the area that needs to be generated by the generator, and the unoccluded part represents the area that can be obtained by distorting the key frame image.

[0095] The generator network based on convolutional neural network first extracts the features of key frame images, and then uses dense optical flow T k1 Distort the features of the key frame image, multiply the distorted features with the occlusion map, and finally decode the features after distortion and occlusion to obtain the final non-key frame image. This step requires the use of the key frame image F k1 and F k2 The reconstructed non-key frame images F are obtained respectively t1 ' and F t2 '.

[0096] In a preferred embodiment of the present invention, S103 is performed, and the mask predictor based on the convolutional neural network uses the key frame image F k1 、F k2 and non-keyframe image F t Key point heat map H k1 、H k2 and H t Predict a face mask M t . M t Representation of the non-key frame image F reconstructed by two key frame images t1 ' and F t2 'Reconstruct the final non-key frame image F t 'Contribution level. t The size of is the same as the reconstructed image, and the weighting formula is as follows:

[0097] F′ t =M t *F′ t1 +(1-M t )*F′ t2

[0098] Based on the same inventive concept, the embodiment of the invention also provides a very low bit rate face video hybrid compression method, the flow chart of the method is as follows: Figure 3 Shown, including:

[0099] S301: For the complete video sequence at the transmitter, key frames are sampled at a certain sampling interval N, and these key frames are encoded using the VVenC encoder to obtain a bitstream 1;

[0100] S302: For non-key frames that have not been sampled, a key point extractor based on a convolutional neural network is used to detect several key points of each non-key frame, and the detected key points are losslessly encoded to obtain bitstream 2;

[0101] S303: Transmitting code stream 1 and code stream 2 to the receiving end;

[0102] S304: Decode the bitstream 1 using the VVenC decoder to obtain a reconstructed key frame;

[0103] S305: Losslessly decode the bitstream 2 to obtain key point data of non-key frames, combine the reconstructed key frames, and reconstruct the non-key frames using a bidirectional frame prediction method;

[0104] S306: Arrange the reconstructed key frames and the reconstructed non-key frames in time sequence to obtain a final video sequence.

[0105] Figure 4 This is a schematic diagram of the architecture of the ultra-low bitrate hybrid facial video compression method according to an embodiment of the present invention. In this embodiment, a hybrid compression scheme combining traditional coding and deep learning is employed for a facial video to be compressed at the transmitter. Specifically, all video frames are divided into keyframes and non-keyframes. Keyframes are encoded using the traditional VVenC encoding method, and then non-keyframes are reconstructed using deep learning methods based on the key points of the non-keyframes.

[0106] As a preferred embodiment, S301 is performed, where a frame is sampled at a certain sampling interval N as a key frame, the sampled key frame is regarded as a new video sequence, and is encoded using the VVenC encoder to obtain a bitstream 1. VVenC is an open source implementation of the latest coding standard VVC. The hybrid coding method allows VVenC to be replaced with any traditional video coding method such as H.265 and H.264.

[0107] As a preferred embodiment, S302 is performed. For non-key frames that have not been sampled, the key point data of these non-key frames is transmitted at the transmitting end instead of transmitting the complete video frame. At the receiving end, the non-key frames are reconstructed based on the key point data of these non-key frames to achieve the purpose of reducing the bit rate. Specifically, a key point extractor based on a convolutional neural network is used to extract several Uint8-type key point data for each non-key frame. The extracted Uint8-type key point data is losslessly encoded to obtain bitstream 2.

[0108] The lossless coding method uses intra-frame prediction to eliminate spatial redundancy and inter-frame prediction to eliminate temporal redundancy. Specifically, for the first non-key frame, intra-frame prediction is used to remove spatial redundancy, that is, the (N-1)th key point is used to predict the Nth key point. Inter-frame prediction uses the key point in the (K-1)th frame to predict the corresponding key point in the Kth frame, which means that the key point value of the Kth frame is not directly encoded, but the residual of the key point data value between the (K-1)th frame and the Kth frame is encoded. Since the key points between adjacent frames change little and the residual value is small, it is well in line with the characteristics of exponential Golomb coding (a variable-length coding algorithm), so the zero-order exponential Golomb coding algorithm is used to encode the residual value. In addition, an adaptive binary arithmetic coding algorithm (an entropy coding algorithm) is used to further reduce the bit rate.

[0109] As a preferred embodiment, S303 is executed to transmit the code stream 1 of the key frame and the code stream 2 of the non-key frame to the receiving end;

[0110] As a preferred embodiment, S304 is performed. After receiving the transmitted key frame code stream 1, the receiving end uses the VVenC decoder to decode the code stream 1 to obtain a reconstructed key frame.

[0111] As a preferred embodiment, S305 is executed. After the receiving end receives the transmitted non-key frame code stream 2, it performs corresponding lossless decoding on it to obtain the key point data of the Uint8 type of the non-key frame. The bidirectional frame prediction method can predict the image of the corresponding non-key frame based on the key point data of two adjacent key frames and a non-key frame in the middle. Therefore, in this embodiment, the two adjacent key frames reconstructed from the code stream 1 in S304 are combined with the key point data of a non-key frame in the middle decoded from the code stream 2, and the bidirectional frame prediction method is used to complete the reconstruction of a non-key frame in the middle. Repeat the above operation for each non-key frame image, and finally reconstruct all non-key frame images.

[0112] As a preferred embodiment, S306 is executed, and the key frames reconstructed from the code stream 1 and the non-key frames reconstructed from the code stream 2 need to be sorted in the time domain, so as to output the final video sequence.

[0113] Based on the same inventive concept, an embodiment of the present invention further provides an extremely low bit rate face video hybrid compression system, comprising:

[0114] The sending module samples the key frames of the complete video sequence, then encodes the sampled key frames using VVenC to obtain bitstream 1, and extracts the key point data of non-keyframes and performs lossless encoding to obtain bitstream 2;

[0115] The receiving module decodes received bitstream 1 using the VVenC decoder to reconstruct keyframes. It decodes received bitstream 2 to obtain keypoint data for non-keyframes. This data is then combined with the reconstructed keyframes and reconstructed using a bidirectional frame prediction method. Finally, the reconstructed keyframes and non-keyframes are arranged in time sequence to produce the final video sequence.

[0116] Figure 5 The following is a diagram of the components and architecture of the entire system. The system mainly consists of a sending module and a receiving module. The function of each module can be found in the implementation of the extremely low bit rate hybrid compression method for facial videos mentioned above, and will not be repeated here. In practical applications, the sampling interval N of the key frame and the encoding parameters of VVenC are both adjustable. When the encoding parameters of VVenC are fixed, the larger the sampling interval N of the key frame, the lower the final bit rate, and the final synthesis quality of the facial video also decreases accordingly; when the sampling interval N of the key frame is fixed, the encoding parameters of VVenC (such as encoding mode, QP value, etc.) can be changed according to needs. For scenarios requiring low latency, the LowDelay mode of VVenC can be used to encode key frames; for scenarios without latency requirements (such as offline video storage), the Random mode of VVenC can be used to encode key frames. When encoding key frames using VVenC, a QP value must be specified. A larger QP value results in a lower compression ratio and lower quality key frames reconstructed at the receiving end. Conversely, a smaller QP value results in a higher compression ratio and higher quality key frames reconstructed at the receiving end. Therefore, in this embodiment, the appropriate key frame sampling interval N and VVenC encoding parameters must be selected based on the encoding requirements of the current scenario and network conditions to achieve optimal results.

[0117] The following evaluation evaluates the results of the extremely low-bitrate hybrid facial video compression method described above. Fifteen video sequences from the VoxCeleb2 dataset test set, each containing approximately 300 frames, were selected as test sequences. Using VVenC, the most advanced conventional encoding method, quantitative and qualitative comparisons were performed with the hybrid facial video compression method described above.

[0118] Regarding the quantitative evaluation, the experimental results were evaluated both objectively and subjectively. In the objective evaluation, the objective indicator used was the peak signal-to-noise ratio (Y-PSNR) of the Y channel in the YUV format of the image. The larger the value, the less image distortion. In the subjective evaluation, 10 volunteers were invited to score the fidelity and aesthetics of the videos. The fidelity score was used to reflect the detail retention of the reconstructed image, and the aesthetic score was used to reflect the overall visual quality of the video. Specifically, a video encoded with VVenC and a video encoded using a hybrid compression method for face videos were played simultaneously, and the volunteers were told to choose the video with better fidelity and aesthetics, respectively. Therefore, the score is actually a percentage of user preference.

[0119] Table 1 shows the quantitative comparison results between VVenC and the above-described embodiment of the present invention, with bold numbers indicating optimal performance. Here, bpp (bits-per-pixel) represents the number of bits required per pixel, KB / s (KB / s) represents the number of bytes required per second, and "VVenC-QP62" represents the performance when all frames are encoded using VVenC with a QP value of 62. "Present embodiment method-QP34-N10" represents the performance when the present embodiment method has a sampling interval of 10 and a QP value of 34 when compressing key frames using VVenC. As can be seen from Table 1, when the bitrate is similar, the subjective and objective performance achieved by the present embodiment method is significantly better than that achieved by the method using only VVenC encoding. When the PSNR values ​​are similar, the present embodiment method requires a significantly lower bitrate than the method using only VVenC encoding, and the subjective quality is significantly better than that achieved by VVenC. All experimental results in Table 1 were measured on sequences with an image resolution of 256×256. Table 2 also shows the test results on a sequence with an image resolution of 512×512. From Table 2, it can be seen that the encoding performance of the method of this embodiment is significantly better than the method using only VVenC encoding at both 256×256 and 512×512 resolutions.

[0120] Table 1 Quantitative comparison results of VVenC and the method of this embodiment

[0121]

[0122]

[0123] Table 2 Test results of videos with different resolutions

[0124]

[0125] Figure 6The following figure compares the performance of the hybrid facial video compression method of an embodiment of the present invention with VVenC, showing the performance comparison in LowDelay (LDP) mode and RandomAccess (RA) mode. As can be seen, in LDP mode, the BR-rate reaches -43.12%, while in RA mode, the BR-rate reaches -58.48%. This means that the hybrid facial video compression method provided by this embodiment can significantly reduce the bitrate while maintaining the quality of the reconstructed video.

[0126] Figure 7 This is a schematic diagram of the effect of synthesizing a face image frame using the bidirectional frame prediction method according to an embodiment of the present invention. Figure 7 In the example, the face mask M t The weighted effect is to use the key frame image F k1 、F k2 The obtained non-key frame reconstructed image F t1 '、F t2 ', the final reconstructed non-key frame reconstructed image F is obtained t '. Among them E t1 F t With F t1 'Error diagram between, E t2 F t With F t2 '. It can be seen that the predicted mask represents the contribution of the non-key frame images reconstructed using the two key frame images to the final non-key frame reconstructed image. Figure 8 This figure compares the results of the ultra-low bitrate hybrid facial video compression method of an embodiment of the present invention and VVenC. As can be seen, at the same bitrate, the image quality reconstructed at the receiving end by the hybrid facial video compression method of this embodiment is significantly better than that of VVenC at the same bitrate.

[0127] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0131] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0132] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0133] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various modifications or variations within the scope of the claims without affecting the essence of the present invention. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A bidirectional frame prediction method for face videos, characterized in that: include: Sample three frames of face video in the time domain; Obtaining key point heat maps corresponding to each of the three frames of image; Reconstructing an intermediate frame image by combining the three frames of image and the key point heat map; The three frames of image of the face video sampled in the time domain include: The first and last frames are collected as key frame images, denoted as F k1 、F k2 ; Use F k1 and F k2 The prediction interval is the intermediate frame between two frames, that is, the non-key frame image F t ; The obtaining of the key point heat maps corresponding to the three frames of images includes: For the non-key frame image F between two key frames t , using the key point coordinates extracted by the key point extractor based on convolutional neural network to generate the key point heat map H t ; For the key frame image F k1 、F k2 , use the key point extractor based on convolutional neural network to obtain the key point heat map H k1 、H k2 ; Combining the three frames of images and the key point heat map to reconstruct an intermediate frame image includes: The key frame image F k1 , the key frame image F k1 Key point heat map H k1 And the non-key frame image F t Key point heat map H t Input into the dense motion network based on convolutional neural network to obtain dense optical flow T k1 and occlusion map O k1 ; The key frame image F k1 , the dense optical flow T k1 and the occlusion map O k1 Input into the generator based on convolutional neural network to obtain the key frame image F k1 Reconstructed non-keyframe image F t1 ’ ; The key frame image F k2 , the key frame image F k2 Key point heat map H k2 And the non-key frame image F t Key point heat map H t Input into the dense motion network based on convolutional neural network to obtain dense optical flow T k2 and occlusion map O k2 ; The key frame image F k2 , the dense optical flow T k2 and the occlusion map O k2 Input into the generator based on convolutional neural network to obtain the key frame image F k2 Reconstructed non-keyframe image F t2 ’ ; The non-key frame image F t Key point heat map H t , key frame image F k1 Key point heat map H k1 and key frame image F k2 Key point heat map H k2 Input into the mask predictor based on convolutional neural network to obtain the face mask M t ; Mask the face M t Weighted action is applied to the reconstructed non-keyframe image F t1 ’ 、F t2 ’ The final reconstructed non-key frame reconstructed image F is obtained t ’ .

2. A bidirectional frame prediction method for face videos according to claim 1, characterized in that: The key frame image F k1 , the dense optical flow T k1 and the occlusion map O k1 Input into the generator based on convolutional neural network to obtain the key frame image F k1 Reconstructed non-keyframe image F t1 ’ , or, The key frame image F k2 , the dense optical flow T k2 and the occlusion map O k2 Input into the generator based on convolutional neural network to obtain the key frame image F k2 Reconstructed non-keyframe image F t2 ’ ; include: The generator network extracts features of the keyframe image; Use dense optical flow to distort the features of key frame images; Multiply the distorted features with the occlusion map; The reconstructed non-keyframe image is obtained by decoding the features after distortion and occlusion processing.

3. The bidirectional frame prediction method for face video according to claim 1, characterized in that: The non-key frame reconstructed image F t ’ Acquisition, including: The mask predictor uses the key frame image F k1 、F k2 and non-keyframe image F t Key point heat map H k1 、H k2 and H t Predict a face mask M t ; M t Representation of the non-key frame image F reconstructed by two key frame images t1 ’ 、F t2 ’ Reconstruct the final non-key frame image F t ’ the extent of contribution; M t The size of is the same as the reconstructed image, and the weighting formula is as follows: 。 4. A very low bit rate face video hybrid compression method, characterized in that: include: Get stream 1 and stream 2 from the complete video sequence at the sender; Transmitting the code stream 1 and the code stream 2 to a receiving end; The code stream 1 is decoded to obtain a reconstructed key frame; Performing lossless decoding on the bitstream 2 to obtain key point data of non-key frames, combining the reconstructed key frames, and reconstructing the non-key frames using the bidirectional frame prediction method according to any one of claims 1 to 3; Arranging the reconstructed key frames and the reconstructed non-key frames in time sequence to obtain a final video sequence; The step of obtaining stream 1 and stream 2 from the complete video sequence at the transmitting end includes: For the complete video sequence at the transmitter, set the sampling interval n, that is, sample one frame every n frames as a key frame, regard the sampled key frame as a new video sequence and encode it using the VVenC encoder to obtain bitstream 1; Utilizing the convolutional neural network-based key point extractor, a number of Uint8-type key points of each non-key frame that has not been sampled are detected, and the detected Uint8-type key point data are losslessly encoded to obtain bitstream 2; The process of losslessly decoding the bitstream 2 to obtain key point data of non-key frames, combining the reconstructed key frames, and reconstructing the non-key frames using the bidirectional frame prediction method includes: After receiving the code stream 2, the receiving end decodes it to obtain the key point data of the Uint8 type of the non-key frame; The bidirectional frame prediction method predicts the image of the corresponding non-key frame based on the key point data of two adjacent reconstructed key frames and the decoded middle non-key frame, thereby completing the reconstruction of the non-key frame.

5. The method for hybrid compression of face videos at an extremely low bit rate according to claim 4, characterized in that: The lossless coding, i.e., using intra-frame prediction to eliminate spatial redundancy and using inter-frame prediction to eliminate temporal redundancy, includes: For the first non-key frame, intra-frame prediction is used to remove spatial redundancy, that is, the first key points to predict the Nth key point; Inter-frame prediction uses The key points in the frame are used to predict the corresponding key points in the Kth frame. The residual of the key point data values ​​of the frame and the K-th frame is encoded.

6. A very low bit rate face video hybrid compression system, characterized in that: include: The sending module samples key frames from the complete video sequence, encodes the key frames using VVenC to obtain bitstream 1, and extracts key point data of non-key frames and performs lossless encoding to obtain bitstream 2; A receiving module is configured to decode the received bitstream 1 using a VVenC decoder to obtain reconstructed key frames; decode the received bitstream 2 to obtain key point data of non-key frames, and reconstruct the non-key frames using the bidirectional frame prediction method according to any one of claims 1 to 3 in combination with the reconstructed key frames; and arrange the reconstructed key frames and the reconstructed non-key frames in time sequence to obtain a final video sequence; The sending module includes: A key frame sampling unit, wherein the key frame sampling unit samples the complete video sequence at a sampling interval n, that is, sampling one frame as a key frame every n frames; A video encoding unit, wherein the video encoding unit regards the sampled key frames as a new video sequence and encodes it using a VVenC encoder to obtain a bit stream 1; A key point extraction unit, wherein the key point extraction unit uses the key point extractor based on the convolutional neural network to extract a plurality of Uint8 type key points of each non-key frame; A key point encoding unit encodes the extracted key point data of Uint8 type using a lossless encoding scheme to obtain a code stream 2.

7. The extremely low bit rate face video hybrid compression system according to claim 6, characterized in that: The receiving module includes: A video decoding unit, wherein the video decoding unit decodes the bit stream 1 using a VVenC decoder to obtain a reconstructed key frame; A key point decoding unit, which decodes the code stream 2 to obtain key point data of the Uint8 type of non-key frame; a non-key frame reconstruction unit, wherein the non-key frame reconstruction unit utilizes the key point data of the decoded non-key frame, combines the reconstructed key frame, and utilizes the bidirectional frame prediction method to reconstruct the non-key frame; A sorting unit arranges the reconstructed key frames and the reconstructed non-key frames in time sequence to obtain a final video sequence.

Images