Facial video encoding method, decoding method and device

Abstract

The embodiments of the present application provide a facial video encoding method, decoding method, and apparatus. The facial video encoding method includes: obtaining a target facial video frame to be encoded and multiple initial reference facial video frames in a reference frame list; and encoding the multiple initial reference facial video frames and the target facial video frame, respectively, to obtain a facial video bitstream. In the embodiments of the present application, encoding and decoding operations are performed based on multiple reference facial video frames to obtain a fused facial video frame, the texture quality and motion information of which are simultaneously referenced by multiple different reference facial video frames. Therefore, the quality difference between the reconstructed fused facial video frame and the target facial video frame is small, thereby improving the reconstruction quality of the facial video frame.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a facial video encoding method, decoding method, and apparatus. Background Art

[0002] With the continuous development of video coding technology, a variety of video coding algorithms have emerged to improve video coding performance. For example, traditional video coding algorithms using block-based motion estimation and discrete cosine transform methods, as well as end-to-end video coding algorithms based on deep learning, have emerged.

[0003] Currently, when encoding and decoding facial videos, existing video coding algorithms typically select the first frame in a video as a reference frame and then encode and decode subsequent frames to obtain the corresponding reconstructed facial video frames. However, compared to the original facial video frames to be encoded, the texture quality of the reconstructed facial video frames obtained in this way is inferior, and the accuracy of motion description is also low. In other words, the quality of the reconstructed facial video frames obtained by these existing video coding algorithms is poor. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a facial video encoding method, decoding method, and apparatus to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present application, a facial video encoding method is provided, comprising:

[0006] Obtaining a target facial video frame to be encoded and a plurality of initial reference facial video frames in a reference frame list;

[0007] The multiple initial reference facial video frames and the target facial video frame are respectively encoded to obtain a facial video bitstream.

[0008] According to a second aspect of an embodiment of the present application, a facial video decoding method is provided, comprising:

[0009] Obtaining a facial video bitstream, the facial video bitstream comprising: a plurality of encoded reference facial video frames and encoded compact feature information; the encoded compact feature information representing key feature information of a target facial video frame to be reconstructed;

[0010] respectively decoding the plurality of encoded reference facial video frames to obtain a plurality of reference facial video frames;

[0011] Decoding the encoded compact feature information to obtain target compact features of the target facial video frame;

[0012] Based on the multiple reference facial video frames and the target compact features, facial video frames are reconstructed to obtain a fused facial video frame corresponding to the target facial video frame.

[0013] According to a third aspect of an embodiment of the present application, a method for generating a reference facial video frame is provided, comprising:

[0014] Obtain a target facial video frame and a plurality of initial reference facial video frames in a reference frame list;

[0015] Calculating an information difference value between the current facial video frame and each initial reference facial video frame, wherein the information difference value represents a degree of difference between information contained in the current facial video frame and information contained in each initial reference facial video frame;

[0016] If there is an information difference value greater than a preset threshold, the current facial video frame is added to the reference frame list as a new reference facial video frame.

[0017] According to a fourth aspect of an embodiment of the present application, a model training method is provided, comprising:

[0018] Encoding a plurality of initial reference facial video frame samples and a target facial video frame sample respectively to obtain facial video bitstream samples;

[0019] Decoding the facial video bitstream samples to obtain the multiple initial reference facial video frame samples and target compact feature samples of the target facial video frame samples;

[0020] Based on each reference facial video frame sample and the target compact feature sample, obtaining a driving information sample corresponding to each reference facial video frame sample;

[0021] Inputting each reference facial video frame sample and each corresponding driving information sample into the initial second generation model to obtain initial reconstructed facial video frame samples corresponding to each reference facial video frame sample;

[0022] Inputting each initial reconstructed facial video frame sample into the initial fusion model to obtain a fused facial video frame sample;

[0023] Constructing a loss function based on the fused facial video frame sample and the target facial video frame sample, and performing training based on the loss function to obtain a trained second generation model and a transition fusion model;

[0024] Based on the target facial video frame sample and the multiple initial reference facial video frame samples, and based on the trained second generation model, the transition fusion model is trained again to obtain a trained fusion model.

[0025] According to a fifth aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform an operation corresponding to the facial video encoding method as described in the first aspect, or an operation corresponding to the facial video decoding method as described in the second aspect, or an operation corresponding to the reference facial video frame generation method as described in the third aspect, or an operation corresponding to the model training method as described in the fourth aspect.

[0026] According to a sixth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the facial video encoding method as described in the first aspect, or the facial video decoding method as described in the second aspect, or the reference facial video frame generation method as described in the third aspect, or the model training method as described in the fourth aspect is implemented.

[0027] According to the seventh aspect of the embodiments of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the facial video encoding method as described in the first aspect, or operations corresponding to the facial video decoding method as described in the second aspect, or operations corresponding to the reference facial video frame generation method as described in the third aspect, or operations corresponding to the model training method as described in the fourth aspect.

[0028] According to the facial video encoding method, decoding method and device provided in the embodiments of the present application, encoding and decoding operations are performed on the target facial video frame to be encoded based on multiple reference facial video frames, thereby obtaining a fused reconstructed facial video frame corresponding to the target facial video frame. For the existing reconstructed facial video frame obtained based only on a single reference facial video frame, its texture quality and motion information mainly depend on the single reference facial video frame. That is, during the reconstruction process, the single reference facial video frame limits the reconstruction of texture information and motion information. In the embodiments of the present application, a fused facial video frame is obtained based on multiple reference facial video frames, and its texture quality and motion information refer to multiple different reference facial video frames at the same time. Therefore, the quality difference between the reconstructed fused facial video frame and the target facial video frame is small. In summary, the embodiments of the present application improve the reconstruction quality of facial video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0030] Figure 1 A schematic diagram of a framework for a coding and decoding method based on deep video generation;

[0031] Figure 2 This is a flowchart of a facial video encoding method according to the first embodiment of the present application;

[0032] Figure 3 for Figure 2 A schematic diagram of an example scenario in the illustrated embodiment;

[0033] Figure 4 This is a flowchart of a facial video decoding method according to the second embodiment of the present application;

[0034] Figure 5 for Figure 4 A schematic diagram of an example scenario in the illustrated embodiment;

[0035] Figure 6 A schematic diagram of a process for obtaining a fused facial video frame according to the second embodiment of the present application;

[0036] Figure 7 Schematic diagram of another process for obtaining a fused facial video frame according to the second embodiment of the present application;

[0037] Figure 8 1 is a flow chart of a method for generating a reference facial video frame according to the third embodiment of the present application;

[0038] Figure 9 This is a flowchart of the steps of a model training method according to the fourth embodiment of the present application;

[0039] Figure 10 4 is a structural block diagram of a facial video encoding device according to Embodiment 5 of the present application;

[0040] Figure 11 4 is a structural block diagram of a facial video decoding device according to Embodiment 6 of the present application;

[0041] Figure 12 This is a structural block diagram of a model training device according to Embodiment 7 of the present application;

[0042] Figure 13 This is a structural diagram of an electronic device according to Example 8 of the present application. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0044] See also Figure 1 , Figure 1 The main principle of this method is to deform the reference frame based on the motion of the frame to be encoded to obtain the reconstructed frame corresponding to the frame to be encoded. Figure 1 The basic framework of the encoding and decoding method based on deep video generation is explained:

[0045] In the first step, the encoding stage, the encoder uses a key point extractor to extract the target key point information of the target facial video frame to be encoded and encodes the target key point information; at the same time, the traditional image coding method (such as VVC, HEVC, etc.) is used to encode the reference facial video frame.

[0046] The second step is the decoding stage. The motion estimation module in the decoder extracts the reference key point information of the reference facial video frame through the key point extractor; and performs dense motion estimation based on the reference key point information and the target key point information to obtain a dense motion estimation map and an occlusion map. The dense motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in the feature domain represented by the key point information; the occlusion map represents the degree to which each pixel in the target facial video frame is occluded.

[0047] The third step is the decoding stage. The generation module in the decoder deforms the reference facial video frame based on the dense motion estimation map to obtain the deformation processing result, and then multiplies the deformation processing result with the occlusion map to output the reconstructed facial video frame.

[0048] Figure 1 In the illustrated method, subsequent video frames are encoded and decoded based on a single reference frame to obtain the corresponding reconstructed facial video frame. However, the texture quality and motion information of the reconstructed facial video frame obtained in this way are primarily dependent on the single reference facial video frame. In other words, during the reconstruction process, this single reference facial video frame limits the reconstruction of texture and motion information, resulting in lower texture quality and lower motion description accuracy compared to the original facial video frame to be encoded. In other words, the quality of the reconstructed facial video frame is poor.

[0049] In an embodiment of the present application, a fused facial video frame is obtained based on multiple reference facial video frames, and its texture quality and motion information simultaneously refer to multiple different reference facial video frames. Therefore, the quality difference between the reconstructed fused facial video frame and the target facial video frame is small, which can improve the reconstruction quality of the facial video frame.

[0050] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0051] Example 1

[0052] Reference Figure 2 , Figure 2 1 is a flowchart of a facial video encoding method according to Embodiment 1 of the present application. Specifically, the facial video encoding method provided in this embodiment includes the following steps:

[0053] Step 202: Obtain a target facial video frame to be encoded and a plurality of initial reference facial video frames in a reference frame list.

[0054] In the embodiment of the present application, there is no limitation on the specific setting method and data of the initial reference facial video frame, and it can be selected according to actual conditions.

[0055] For example, in low-latency scenarios such as video conferencing or live video streaming, the initial reference facial video frame can be any frame in the facial video with a timestamp earlier than the target facial video frame; it can also be a frame selected from the facial video frames according to a preset selection rule, etc. For example, it can be a preset number of frames in the facial video.

[0056] Step 204 : Encode the multiple initial reference facial video frames and the target facial video frame respectively to obtain a facial video bitstream.

[0057] Specifically, for each initial reference facial video frame, relatively small quantization distortion can be used for encoding, and the encoding process retains the complete data of the initial reference facial video frame. For example, the initial reference facial video frame can be encoded using the Versatile Video Coding (VVC) method.

[0058] To reduce the transmission bitrate of the target facial video frame, we can first extract its features to obtain a target compact feature. These features are then encoded. These features represent key feature information in the target facial video frame, such as facial feature location, posture, and expression. Furthermore, we can use a machine learning model to extract features from the target facial video frame to obtain the target compact feature.

[0059] See also Figure 3 , Figure 3 This is a schematic diagram of the scene corresponding to the first embodiment of the present application. Figure 3 The schematic diagram shown in the figure illustrates an embodiment of the present application using a specific scenario as an example:

[0060] Get the target facial video frame a and the multiple initial reference facial video frames in the reference frame list: A1, A2, A3, A4 (in the embodiment of the present application, the number of initial reference facial video frames is not limited, Figure 3 Only four examples are used for illustration and do not constitute a limitation on the embodiments of the present application); A1, A2, A3, and A4 are encoded respectively. In addition, a is also encoded (for example, compact features of a can be extracted, thereby encoding the compact features), thereby obtaining a facial video bitstream.

[0061] Furthermore, in some embodiments of the present application, after obtaining the facial video bitstream, the method further includes:

[0062] Calculating information difference values ​​between the target facial video frame and each initial reference facial video frame respectively; the information difference value represents the degree of difference between the information contained in the target facial video frame and the information contained in the initial reference facial video frame;

[0063] If there is an information difference value greater than a preset threshold, the target facial video frame is added to the reference frame list as a new reference facial video frame to update the reference frame list.

[0064] Specifically, if there is an information difference value greater than a preset threshold, the target facial video frame can be added to the reference frame list as a new reference facial video frame, and the initial reference facial video frame corresponding to the maximum information difference value is deleted, thereby obtaining an updated reference frame list, so as to perform encoding and decoding operations on the facial video frames after the current target facial video frame to obtain the corresponding reconstructed facial video frame.

[0065] In the embodiments of the present application, there is no limitation on the specific calculation method used to calculate the above-mentioned information difference value. For example, a mean square error calculation can be performed based on the pixel values ​​of each pixel point in the target facial video frame and the pixel values ​​of each pixel point in the initial reference facial video frame to obtain a pixel-level information difference value; or feature extraction can be performed on the target facial video frame and the initial reference facial video frame respectively to obtain features that can represent the key feature information of the current facial video frame and features that can represent the key feature information of the initial reference facial video frame, and then a mean square error calculation can be performed based on the above two features to obtain the information difference value of the feature domain, and so on.

[0066] In the embodiment of the present application, a facial video bitstream is obtained by encoding multiple reference facial video frames and a target facial video frame. This allows the decoding end to decode the target facial video frame based on the multiple reference facial video frames after obtaining the facial video bitstream, thereby obtaining a fused reconstructed facial video frame corresponding to the target facial video frame. For existing reconstructed facial video frames based only on a single reference facial video frame, their texture quality and motion information primarily depend on the single reference facial video frame. That is, during the reconstruction process, the single reference facial video frame limits the reconstruction of texture and motion information. However, in the embodiment of the present application, a fused facial video frame is obtained based on multiple reference facial video frames, and its texture quality and motion information simultaneously refer to multiple different reference facial video frames. Therefore, the quality difference between the reconstructed fused facial video frame and the target facial video frame is relatively small.

[0067] The facial video encoding method provided in the first embodiment of the present application can be executed by a video encoding terminal (encoder) to encode facial video files to achieve digital broadband compression of the facial video files. It can be applied to a variety of different scenarios, such as: the storage and streaming of conventional video games involving faces, specifically: the facial video encoding method provided in the embodiment of the present application can be used to encode game video frames to form a corresponding video stream for storage and transmission in video streaming services or other similar applications; another example: low-latency scenarios such as video conferencing and live video broadcasting, specifically: the facial video encoding method provided in the embodiment of the present application can be used to encode facial video data collected by a video capture device to form a corresponding video stream, and send it to a conference terminal, and the conference terminal decodes the video stream to obtain the corresponding facial video image; another example: virtual reality scenarios, the facial video encoding method provided in the embodiment of the present application can be used to encode facial video data collected by a video capture device to form a corresponding video stream, and send it to a virtual reality-related device (such as VR virtual glasses, etc.), and the VR device decodes the video stream to obtain the corresponding facial video image, and implement corresponding VR functions based on the facial video image, etc.

[0068] Example 2

[0069] Reference Figure 4 , Figure 4 Flowchart of a facial video decoding method according to Embodiment 2 of the present application. Specifically, the facial video decoding method provided in this embodiment includes the following steps:

[0070] Step 402: Obtain a facial video bitstream, where the facial video bitstream includes: a plurality of encoded reference facial video frames and encoded compact feature information.

[0071] The encoded compact feature information represents the key feature information of the target facial video frame to be reconstructed.

[0072] In an embodiment of the present application, the encoded compact feature information corresponds to the compact feature information obtained by extracting features from the target facial video frame for characterizing key feature information, and may also correspond to: the difference between the target compact features of adjacent target facial video frames.

[0073] Step 404 : Decode the plurality of encoded reference facial video frames respectively to obtain a plurality of reference facial video frames.

[0074] Step 406 : Decode the encoded compact feature information to obtain target compact features of the target facial video frame.

[0075] Step 408 : Reconstruct the facial video frame based on the multiple reference facial video frames and the target compact features to obtain a fused facial video frame corresponding to the target facial video frame.

[0076] Specifically, in this step, facial video frames can be reconstructed separately for each reference facial video frame in combination with the target compact features, thereby obtaining an initial reconstructed facial video frame corresponding to each reference facial video frame, and then the initial reconstructed facial video frames are fused to obtain a final fused facial video frame; alternatively, the final fused facial video frame can be obtained based on multiple reference facial video frames and the reference facial video frame at the same time.

[0077] In the embodiment of the present application, there is no limitation on the specific method used to obtain the fused facial video frame.

[0078] See Figure 5 , Figure 5 This is a schematic diagram of the scene corresponding to the second embodiment of the present application. Figure 5 The schematic diagram shown in the figure illustrates an embodiment of the present application using a specific scenario as an example:

[0079] Obtain a facial video bitstream consisting of multiple encoded reference facial video frames and encoded compact feature information; decode the encoded reference facial video frames to obtain multiple reference facial video frames: a1, a2, a3, and a4 (in the embodiment of the present application, the number of reference facial video frames is not limited, Figure 5 Only four examples are used for illustration and do not constitute a limitation to the embodiments of the present application), as well as target compact features; finally, based on multiple reference facial video frames a1, a2, a3, a4, and target compact features, facial video frames are reconstructed to obtain a fused facial video frame a0 corresponding to the target facial video frame.

[0080] Furthermore, in some embodiments of the present application, reconstructing a facial video frame based on multiple reference facial video frames and target compact features to obtain a fused facial video frame corresponding to the target facial video frame may include:

[0081] Facial video frames are reconstructed based on each reference facial video frame and the target compact feature to obtain an initial reconstructed facial video frame corresponding to each reference facial video frame; the initial reconstructed facial video frames are fused to obtain a fused facial video frame corresponding to the target facial video frame.

[0082] Specifically, in the embodiment of the present application, there is no limitation on the specific process of obtaining the initial reconstructed facial video frame.

[0083] For example, the following can be done: feature extraction is performed on each reference facial video frame to obtain reference compact features; sparse motion estimation is then performed based on the reference compact features and the target compact features to obtain a sparse motion estimation map; and based on the sparse motion estimation map, the reference facial video frame is deformed to obtain an initial reconstructed facial video frame corresponding to the target facial video frame. The sparse motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset sparse feature domain.

[0084] In addition, different fusion methods can be used when fusing the initial reconstructed facial video frames:

[0085] In some embodiments of the present application, weight values ​​corresponding to each initially reconstructed facial video frame may be first obtained; then, based on the weight values, linear weighting processing may be performed on each initially reconstructed facial video frame to obtain a fused facial video frame corresponding to the target facial video frame. Specifically, based on the weight values, linear weighting processing may be performed on the pixel values ​​corresponding to the pixels in each initially reconstructed facial video frame to obtain a fused facial video frame corresponding to the target facial video frame.

[0086] In some embodiments of the present application, fusion processing can also be performed with the help of a machine learning model, specifically:

[0087] Each initial reconstructed facial video frame is input into the fusion model so that the fusion model outputs a fused facial video frame corresponding to the target facial video frame.

[0088] In the embodiments of this application, the specific structure and reference of the fusion model are not limited and can be set according to actual needs. For example, the fusion model can be a U-Net network composed of a convolutional layer and a generalized division normalization layer; it can also be an hourglass model composed of multiple downsampling modules and multiple corresponding upsampling modules, etc.

[0089] Furthermore, in some embodiments of the present application, reconstructing a facial video frame based on multiple reference facial video frames and target compact features to obtain a fused facial video frame corresponding to the target facial video frame may include:

[0090] For each reference facial video frame, driving information corresponding to the reference facial video frame is obtained based on the reference facial video frame and the target compact feature, where the driving information includes a motion estimation map between the reference facial video frame and the target facial video frame; each reference facial video frame and the driving information corresponding to each reference facial video frame are input into a first generation model to obtain a fused facial video frame corresponding to the target facial video frame.

[0091] The motion estimation map represents the relative motion relationship between the reference facial video frame and the target facial video frame. The motion estimation map may include a sparse motion map and a dense motion map. The sparse motion map represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset sparse feature domain; the dense motion map represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset dense feature domain.

[0092] Furthermore, in order to improve the quality of facial video frame reconstruction, the driving information may further include an occlusion map showing the occlusion degree of each pixel in the target facial video frame, and so on.

[0093] See also Figure 6 , Figure 6 The following is a flow chart illustrating a process for obtaining a fused facial video frame. This diagram uses two reference facial video frames as an example and does not limit the embodiments of this application. Specifically, based on reference facial video frame 1 and the target compact feature, drive information 1 corresponding to reference facial video frame 1 is obtained; based on reference facial video frame 2 and the target compact feature, drive information 2 corresponding to reference facial video frame 2 is obtained; and reference facial video frames 1 and 2, along with drive information 1 and 2, are then input into a first generation model to obtain a fused facial video frame.

[0094] Furthermore, in some other embodiments of the present application, reconstructing a facial video frame based on multiple reference facial video frames and target compact features to obtain a fused facial video frame corresponding to the target facial video frame may further include:

[0095] For each reference facial video frame, driving information corresponding to the reference facial video frame is obtained based on the reference facial video frame and the target compact feature; each reference facial video frame and the driving information corresponding to the reference facial video frame are input into a second generation model to obtain initial reconstructed facial video frames corresponding to each reference facial video frame; each initial reconstructed facial video frame is input into a fusion model so that the fusion model outputs a fused facial video frame corresponding to the target facial video frame.

[0096] See also Figure 7 , Figure 7 To obtain another flow diagram of facial video frame fusion, the figure also uses two reference facial video frames as an example. Specifically, based on reference facial video frame 1 and the target compact feature, driving information 1 corresponding to reference facial video frame 1 is obtained; based on reference facial video frame 2 and the target compact feature, driving information 2 corresponding to reference facial video frame 2 is obtained; reference facial video frame 1 and driving information 1 are input into the second generative model to obtain initial reconstructed facial video frame 1; reference facial video frame 2 and driving information 2 are input into the second generative model to obtain initial reconstructed facial video frame 2; initial reconstructed facial video frame 1 and initial reconstructed facial video frame 2 are then input into the fusion model to ultimately obtain a fused facial video frame.

[0097] In the embodiment of the present application, encoding and decoding operations are performed on the target facial video frame to be encoded based on multiple reference facial video frames, thereby obtaining a fused reconstructed facial video frame corresponding to the target facial video frame. For the existing reconstructed facial video frame obtained based only on a single reference facial video frame, its texture quality and motion information mainly depend on the single reference facial video frame. That is to say, during the reconstruction process, the single reference facial video frame limits the reconstruction of texture information and motion information. In the embodiment of the present application, a fused facial video frame is obtained based on multiple reference facial video frames, and its texture quality and motion information refer to multiple different reference facial video frames at the same time. Therefore, the quality difference between the reconstructed fused facial video frame and the target facial video frame is small. In summary, the embodiment of the present application improves the reconstruction quality of facial video frames.

[0098] The facial video decoding method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: a server, a PC, etc.

[0099] Example 3

[0100] Reference Figure 8 , Figure 8 Flowchart of the steps of a method for generating a reference facial video frame according to the third embodiment of the present application. Specifically, the method for generating a reference facial video frame provided in this embodiment includes the following steps:

[0101] Step 802: Obtain a target facial video frame and a plurality of initial reference facial video frames in a reference frame list.

[0102] In the embodiment of the present application, there is no limitation on the specific setting method and data of the initial reference facial video frame, and it can be selected according to actual conditions.

[0103] For example, in low-latency scenarios such as video conferencing or live video streaming, the initial reference facial video frame can be any frame in the facial video with a timestamp earlier than the target facial video frame; it can also be a frame selected from the facial video frames according to a preset selection rule, etc. For example, it can be a preset number of frames in the facial video.

[0104] Step 804 : Calculate information difference values ​​between the current facial video frame and each initial reference facial video frame.

[0105] The facial information difference value represents the degree of difference between the information included in the current facial video frame and the information included in each initial reference facial video frame.

[0106] In the embodiments of the present application, there is no limitation on the specific calculation method used to calculate the above-mentioned information difference value. For example, a mean square error calculation can be performed based on the pixel values ​​of each pixel point in the target facial video frame and the pixel values ​​of each pixel point in the initial reference facial video frame to obtain a pixel-level information difference value; or feature extraction can be performed on the target facial video frame and the initial reference facial video frame respectively to obtain features that can represent the key feature information of the current facial video frame and features that can represent the key feature information of the initial reference facial video frame, and then a mean square error calculation can be performed based on the above two features to obtain the information difference value of the feature domain, and so on.

[0107] Step 806 : If there is an information difference value greater than a preset threshold, the current facial video frame is added as a new reference facial video frame to the facial reference frame list.

[0108] Specifically, the number of reference facial video frames included in the facial reference frame list is usually constant (can be set according to actual conditions). Therefore, when the current facial video frame is determined as a newly added reference facial video frame, the reference frame list needs to be updated to obtain an updated reference frame list, so as to perform subsequent facial video frame encoding and decoding operations.

[0109] Specifically, the reference frame list can be updated by adding a new reference facial video frame to the reference frame list and deleting an initial reference facial video frame from all initial reference facial video frames to obtain an updated reference frame list. Specifically, the initial reference facial video frames can be deleted based on the information difference between each initial reference facial video and the new reference facial video frame. For example, the initial reference facial video with the largest information difference value can be deleted. Alternatively, the initial reference facial video frame with the earliest timestamp can be deleted based on the timestamp of each initial reference facial video.

[0110] The reference facial video frame generation provided by the embodiment of the present application is based on the information difference value between the current facial video frame and each initial reference facial video frame. When there is an initial reference facial video frame with a large information difference value (greater than a preset threshold), it indicates that the information difference between the current facial video frame and one or several initial reference facial video frames is large. In other words, as time changes, the information in the facial video frame has undergone a large migration (change). Then, based on the initial reference facial video frame with a large information difference value, the subsequent facial video frame is encoded and decoded. The information difference between the reconstructed facial video frame and the current facial video frame may be large. Therefore, in the present application, the current facial video frame is added as a new reference facial video frame in the reference frame list, and an updated reference frame list is obtained, so that when subsequent facial video frames are reconstructed based on the updated reference frame list, the quality of the reconstructed video frame can be improved.

[0111] Example 4

[0112] Reference Figure 9 , Figure 9 The following is a flowchart of a model training method according to the fourth embodiment of the present application. Specifically, the model training method provided in this embodiment includes the following steps:

[0113] Step 902 : Encode a plurality of initial reference facial video frame samples and a target facial video frame sample respectively to obtain facial video bitstream samples.

[0114] Step 904 : Decode facial video bitstream samples to obtain a plurality of initial reference facial video frame samples and target compact feature samples of target facial video frame samples.

[0115] Step 906 : Obtain a driving information sample corresponding to each reference facial video frame sample based on each reference facial video frame sample and the target compact feature sample.

[0116] Step 908 : Input each reference facial video frame sample and each corresponding driving information sample into the initial second generation model to obtain initial reconstructed facial video frame samples corresponding to each reference facial video frame sample.

[0117] Step 910: Input each initial reconstructed facial video frame sample into the initial fusion model to obtain a fused facial video frame sample.

[0118] Step 912: construct a loss function based on the fused facial video frame samples and the target facial video frame samples, and perform training based on the loss function to obtain a trained second generation model and a transition fusion model.

[0119] Step 914 : Based on the target facial video frame sample and the multiple initial reference facial video frame samples, and based on the trained second generation model, the transition fusion model is trained again to obtain a trained fusion model.

[0120] In an embodiment of the present application, model training is divided into two stages. In the first stage, the second generation model and the fusion model are preliminarily trained based on multiple initial reference facial video frame samples and target facial video frame samples to obtain the trained second generation model and transition fusion model. In the second stage, the transition fusion model is fine-tuned based on multiple initial reference facial video frame samples, target facial video frame samples, and the trained second generation model to obtain the final trained fusion model. The advantage of the above model training is that the entire training task is divided into two different subtasks. Compared with the method of training the second generation model and the fusion model at the same time, it can reduce the difficulty of machine learning and improve the efficiency of model training.

[0121] The model training method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: servers, PCs, etc.

[0122] Example 5

[0123] See also Figure 10 , Figure 10 : This is a structural block diagram of a facial video encoding device according to Embodiment 5 of the present application. The facial video encoding device provided in this embodiment of the present application includes:

[0124] A facial video frame acquisition module 1002 is configured to acquire a target facial video frame to be encoded and a plurality of initial reference facial video frames in a reference frame list;

[0125] The video stream obtaining module 1004 is configured to encode the multiple initial reference facial video frames and the target facial video frames respectively to obtain a facial video bitstream.

[0126] Optionally, in some of the embodiments, the facial video encoding apparatus further comprises:

[0127] a reference frame list updating module for calculating, after obtaining the facial video bitstream, information difference values ​​between the target facial video frame and each initial reference facial video frame; the information difference values ​​representing the degree of difference between information contained in the target facial video frame and information contained in the initial reference facial video frame;

[0128] If there is an information difference value greater than a preset threshold, the target facial video frame is added to the reference frame list as a new reference facial video frame to update the reference frame list.

[0129] Optionally, in some embodiments, the video stream obtaining module 1004 is specifically configured to:

[0130] Extracting features of the target facial video frame to obtain target compact features of the target facial video frame;

[0131] The target compact features and each initial reference facial video frame are encoded respectively to obtain a facial video bitstream.

[0132] The facial video encoding device of this embodiment is used to implement the corresponding facial video encoding methods of the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which are not further described here. Furthermore, the functional implementation of each module in the facial video encoding device of this embodiment can refer to the corresponding descriptions of the aforementioned method embodiments and are not further described here.

[0133] Example 6

[0134] See also Figure 11 , Figure 11 : This is a structural block diagram of a facial video decoding device according to Embodiment 6 of the present application. The facial video decoding device provided in this embodiment of the present application includes:

[0135] The video stream acquisition module 1102 is configured to acquire a facial video bitstream, wherein the facial video bitstream includes: a plurality of encoded reference facial video frames and encoded compact feature information; the encoded compact feature information represents key feature information of a target facial video frame to be reconstructed;

[0136] A first decoding module 1104 is configured to decode the plurality of encoded reference facial video frames respectively to obtain a plurality of reference facial video frames;

[0137] A second decoding module 1106 is configured to decode the encoded compact feature information to obtain target compact features of the target facial video frame;

[0138] The fused facial video frame obtaining module 1108 is configured to reconstruct a facial video frame based on multiple reference facial video frames and target compact features to obtain a fused facial video frame corresponding to the target facial video frame.

[0139] Optionally, in some embodiments, the facial video frame fusion module 1108 is specifically configured to:

[0140] Reconstructing facial video frames based on each reference facial video frame and the target compact feature to obtain an initial reconstructed facial video frame corresponding to each reference facial video frame;

[0141] The initial reconstructed facial video frames are fused to obtain a fused facial video frame corresponding to the target facial video frame.

[0142] Optionally, in some embodiments, the fused facial video frame obtaining module 1108, when performing the step of fusing the initial reconstructed facial video frames to obtain a fused facial video frame corresponding to the target facial video frame, is specifically configured to:

[0143] Obtaining weight values ​​corresponding to each initial reconstructed facial video frame;

[0144] Based on the weight values, linear weighted processing is performed on the initial reconstructed facial video frames to obtain a fused facial video frame corresponding to the target facial video frame.

[0145] Optionally, in some embodiments, the fused facial video frame obtaining module 1108, when performing the step of fusing the initial reconstructed facial video frames to obtain a fused facial video frame corresponding to the target facial video frame, is specifically configured to:

[0146] Each initial reconstructed facial video frame is input into the fusion model so that the fusion model outputs a fused facial video frame corresponding to the target facial video frame.

[0147] Optionally, in some embodiments, the facial video frame fusion module 1108 is specifically configured to:

[0148] For each reference facial video frame, based on the reference facial video frame and the target compact feature, obtaining driving information corresponding to the reference facial video frame, the driving information including: a motion estimation map between the reference facial video frame and the target facial video frame;

[0149] Each reference facial video frame and the driving information corresponding to each reference facial video frame are input into a first generation model to obtain a fused facial video frame corresponding to the target facial video frame.

[0150] Optionally, in some embodiments, the facial video frame fusion module 1108 is specifically configured to:

[0151] For each reference facial video frame, obtaining driving information corresponding to the reference facial video frame based on the reference facial video frame and the target compact feature;

[0152] Inputting each reference facial video frame and driving information corresponding to each reference facial video frame into a second generation model to obtain initial reconstructed facial video frames corresponding to each reference facial video frame;

[0153] Each initial reconstructed facial video frame is input into the fusion model so that the fusion model outputs a fused facial video frame corresponding to the target facial video frame.

[0154] The facial video decoding device of this embodiment is used to implement the corresponding facial video decoding methods of the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. Furthermore, the functional implementation of each module in the facial video decoding device of this embodiment can be referenced to the corresponding descriptions of the aforementioned method embodiments and will not be described in detail here.

[0155] Example 7

[0156] See also Figure 12 , Figure 12 This is a structural block diagram of a model training device according to embodiment 7 of the present application. The model training device provided in this embodiment of the present application includes:

[0157] The video stream sample obtaining module 1202 is used to encode the multiple initial reference facial video frame samples and the target facial video frame samples respectively to obtain facial video bitstream samples;

[0158] The video stream sample decoding module 1204 is used to decode the encoded facial video stream samples to obtain a plurality of initial reference facial video frame samples and target compact feature samples of the target facial video frame samples;

[0159] A driving information sample obtaining module 1206 is configured to obtain a driving information sample corresponding to each reference facial video frame sample based on each reference facial video frame sample and the target compact feature sample;

[0160] An initial reconstructed facial video frame sample obtaining module 1208 is configured to input each reference facial video frame sample and each corresponding driving information sample into an initial second generation model to obtain initial reconstructed facial video frame samples corresponding to each reference facial video frame sample;

[0161] A fused facial video frame sample obtaining module 1210 is configured to input each initial reconstructed facial video frame sample into an initial fusion model to obtain a fused facial video frame sample;

[0162] A first training module 1212 is configured to construct a loss function based on the fused facial video frame samples and the target facial video frame samples, and perform training based on the loss function to obtain a trained second generation model and a transition fusion model;

[0163] The second training module 1214 is configured to train the transition fusion model again based on the target facial video frame sample and a plurality of initial reference facial video frame samples and based on the trained second generation model to obtain a trained fusion model.

[0164] For the specific execution process of each of the above steps, please refer to the corresponding steps in the above embodiment 1 or embodiment 2, and will not be repeated here.

[0165] The model training device of this embodiment is used to implement the corresponding model training methods of the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the model training device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0166] Example 8

[0167] Reference Figure 13 , shows a structural diagram of an electronic device according to embodiment eight of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0168] like Figure 13 As shown, the conference terminal may include: a processor (processor) 1302 , a communications interface (Communications Interface) 1304 , a memory (memory) 1306 , and a communication bus 1308 .

[0169] in:

[0170] The processor 1302 , the communication interface 1304 , and the memory 1306 communicate with each other via a communication bus 1308 .

[0171] The communication interface 1304 is used to communicate with other electronic devices or servers.

[0172] Processor 1302 is used to execute program 1310, and specifically can execute the above-mentioned facial video encoding method, or facial video decoding method, or reference facial video frame generation method, or relevant steps in the model training method embodiment.

[0173] Specifically, the program 1310 may include program codes, which include computer operation instructions.

[0174] The processor 1302 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0175] The memory 1306 is used to store the program 1110. The memory 1106 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0176] Program 1310 can specifically be used to enable processor 1302 to perform the following operations: obtain a target facial video frame to be encoded and multiple initial reference facial video frames in a reference frame list; encode multiple initial reference facial video frames and target facial video frames respectively to obtain a facial video bitstream.

[0177] or,

[0178] Program 1310 can be specifically used to enable processor 1302 to perform the following operations: obtain a facial video bitstream, the facial video bitstream including: multiple encoded reference facial video frames and encoded compact feature information; the facial encoded compact feature information represents key feature information of a target facial video frame to be reconstructed; decode the multiple encoded reference facial video frames of the face respectively to obtain multiple reference facial video frames; decode the facial encoded compact feature information to obtain target compact features of the target facial video frame of the face; reconstruct the facial video frame based on the multiple reference facial video frames and the facial target compact features to obtain a fused facial video frame corresponding to the target facial video frame of the face.

[0179] or,

[0180] Program 1310 can specifically be used to enable processor 1302 to perform the following operations: obtain a target facial video frame and multiple initial reference facial video frames in a reference frame list; calculate an information difference value between a current facial video frame and each initial reference facial video frame, where the facial information difference value represents the degree of difference between the information contained in the current facial video frame and the information contained in each initial reference facial video frame; if there is an information difference value greater than a preset threshold, delete the initial reference facial video frame corresponding to the maximum information difference value from the facial reference frame list, and add the current facial video frame as a new reference facial video frame to the facial reference frame list.

[0181] or,

[0182] Program 1310 can be specifically used to enable processor 1302 to perform the following operations: encode multiple initial reference facial video frame samples and target facial video frame samples respectively to obtain facial video bitstream samples; decode the facial video bitstream samples to obtain multiple initial reference facial video frame samples and target compact feature samples of the face target facial video frame samples; obtain a driving information sample corresponding to each reference facial video frame sample based on each reference facial video frame sample and the facial target compact feature sample; input each reference facial video frame sample and the corresponding driving information sample into an initial second generation model to obtain an initial reconstructed facial video frame sample corresponding to each reference facial video frame sample; input each initial reconstructed facial video frame sample into an initial fusion model to obtain a fused facial video frame sample; construct a loss function based on the facial fused facial video frame samples and the facial target facial video frame samples, and train based on the facial loss function to obtain a trained second generation model and a transition fusion model; based on the facial target facial video frame samples and the multiple initial reference facial video frame samples of the face and based on the trained second generation model, the facial transition fusion model is retrained to obtain a trained fusion model.

[0183] The specific implementation of each step in program 1310 can be found in the corresponding descriptions of the corresponding steps and units in the above-mentioned facial video encoding method, facial video decoding method, facial video frame generation method, or model training method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0184] The electronic device of this embodiment performs encoding and decoding operations on the target facial video frame to be encoded based on multiple reference facial video frames, thereby obtaining a fused reconstructed facial video frame corresponding to the target facial video frame. For existing reconstructed facial video frames obtained based only on a single reference facial video frame, their texture quality and motion information mainly depend on the single reference facial video frame. That is, during the reconstruction process, the single reference facial video frame limits the reconstruction of texture information and motion information. In the embodiment of the present application, a fused facial video frame is obtained based on multiple reference facial video frames, and its texture quality and motion information refer to multiple different reference facial video frames at the same time. Therefore, the quality difference between the reconstructed fused facial video frame and the target facial video frame is small. In summary, the embodiment of the present application improves the reconstruction quality of facial video frames.

[0185] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0186] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0187] The above-described method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, processor or hardware, the facial video encoding method, or the facial video decoding method, or the reference facial video frame generation method, or the model training method described herein is implemented. In addition, when a general-purpose computer accesses code for implementing the facial video encoding method, or the facial video decoding method, or the reference facial video frame generation method, or the model training method shown herein, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the facial video encoding method, or the facial video decoding method, or the reference facial video frame generation method, or the model training method shown herein.

[0188] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0189] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A facial video encoding method, comprising: Obtaining a target facial video frame to be encoded and a plurality of initial reference facial video frames in a reference frame list; Encoding the multiple initial reference facial video frames and the target facial video frame respectively to obtain a facial video bitstream; Calculating information difference values ​​between the target facial video frame and each initial reference facial video frame respectively; the information difference values ​​represent the degree of difference between information contained in the target facial video frame and information contained in the initial reference facial video frame; If there is an information difference value greater than a preset threshold, the target facial video frame is added as a new reference facial video frame to the reference frame list to update the reference frame list.

2. The method according to claim 1, wherein The encoding of the multiple initial reference facial video frames and the target facial video frame to obtain a facial video bitstream includes: Performing feature extraction on the target facial video frame to obtain target compact features of the target facial video frame; The target compact features and each initial reference facial video frame are encoded respectively to obtain a facial video bitstream.

3. A facial video decoding method, comprising: Obtaining a facial video bitstream, the facial video bitstream comprising: a plurality of encoded reference facial video frames and encoded compact feature information; the encoded compact feature information representing key feature information of a target facial video frame to be reconstructed; respectively decoding the plurality of encoded reference facial video frames to obtain a plurality of reference facial video frames; Decoding the encoded compact feature information to obtain target compact features of the target facial video frame; For each reference facial video frame, based on the reference facial video frame and the target compact feature, obtaining driving information corresponding to the reference facial video frame, the driving information comprising: a motion estimation map between the reference facial video frame and the target facial video frame; Each reference facial video frame and the driving information corresponding to each reference facial video frame are input into a first generation model to obtain a fused facial video frame corresponding to the target facial video frame.

4. The method according to claim 3, wherein: The method further comprises: For each reference facial video frame, obtaining driving information corresponding to the reference facial video frame based on the reference facial video frame and the target compact feature; Inputting each reference facial video frame and driving information corresponding to each reference facial video frame into a second generation model to obtain initial reconstructed facial video frames corresponding to each reference facial video frame; Each initial reconstructed facial video frame is input into a fusion model so that the fusion model outputs a fused facial video frame corresponding to the target facial video frame.

5. A method for generating a reference facial video frame, comprising: Obtain a target facial video frame and a plurality of initial reference facial video frames in a reference frame list; Calculating an information difference value between the current facial video frame and each initial reference facial video frame, wherein the information difference value represents a degree of difference between information contained in the current facial video frame and information contained in each initial reference facial video frame; If there is an information difference value greater than a preset threshold, the current facial video frame is added to the reference frame list as a new reference facial video frame.

6. A model training method comprising: Encoding a plurality of initial reference facial video frame samples and a target facial video frame sample respectively to obtain facial video bitstream samples; Decoding the facial video bitstream samples to obtain the multiple initial reference facial video frame samples and target compact feature samples of the target facial video frame samples; Based on each reference facial video frame sample and the target compact feature sample, obtaining a driving information sample corresponding to each reference facial video frame sample; Inputting each reference facial video frame sample and each corresponding driving information sample into the initial second generation model to obtain initial reconstructed facial video frame samples corresponding to each reference facial video frame sample; Inputting each initial reconstructed facial video frame sample into the initial fusion model to obtain a fused facial video frame sample; Constructing a loss function based on the fused facial video frame sample and the target facial video frame sample, and performing training based on the loss function to obtain a trained second generation model and a transition fusion model; Based on the target facial video frame sample and the multiple initial reference facial video frame samples, and based on the trained second generation model, the transition fusion model is trained again to obtain a trained fusion model.

7. A facial video encoding apparatus, comprising: A facial video frame acquisition module is used to acquire a target facial video frame to be encoded and a plurality of initial reference facial video frames in a reference frame list; A facial video frame encoding module is configured to encode the multiple initial reference facial video frames and the target facial video frame to obtain a facial video bitstream; The reference frame list updating module is configured to calculate information difference values ​​between the target facial video frame and each initial reference facial video frame, respectively; the information difference values ​​represent the degree of difference between the information contained in the target facial video frame and the information contained in the initial reference facial video frames; if there is an information difference value greater than a preset threshold, the target facial video frame is added to the reference frame list as a new reference facial video frame to update the reference frame list.

8. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, which enables the processor to perform an operation corresponding to the facial video encoding method according to claim 1 or 2, or an operation corresponding to the facial video decoding method according to claim 3 or 4, or an operation corresponding to the reference facial video frame generation method according to claim 5, or an operation corresponding to the model training method according to claim 6.

Images