Facial video encoding method, decoding method and device

Abstract

The embodiments of the present application provide a facial video encoding method, decoding method, and apparatus. The facial video encoding method includes: obtaining a target facial video frame to be encoded and a reference facial video frame; performing feature extraction on the target facial video frame to obtain a target compact feature, wherein the target compact feature represents key feature information in the target facial video frame; and encoding the target compact feature and the reference facial video frame, respectively, to obtain a facial video bitstream. The embodiments of the present application can reduce the encoding bitrate while ensuring the quality of facial video encoding, thereby better meeting the requirements of low-bitrate facial video encoding.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a facial video encoding method, decoding method, and apparatus. Background Art

[0002] With the continuous development of video coding and decoding technology, video coding and decoding equipment has been widely used in various scenarios, such as video conferencing, video live broadcasting, etc.

[0003] Currently, more traditional video encoding and decoding methods are usually targeted at common natural scenes, and use block-based motion estimation, discrete cosine transform and other methods to encode and decode video frames.

[0004] When using traditional video coding methods to encode facial videos, in order to ensure the video encoding quality, the video encoding rate is usually low and cannot meet the requirements of low-bitrate facial video encoding. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a facial video encoding method, decoding method, and apparatus to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, a facial video encoding method is provided, comprising:

[0007] Obtaining a target facial video frame to be encoded and a reference facial video frame;

[0008] Performing feature extraction on the target facial video frame to obtain a target compact feature, wherein the target compact feature represents key feature information in the target facial video frame;

[0009] The target compact feature and the reference facial video frame are encoded respectively to obtain a facial video bitstream.

[0010] According to a second aspect of an embodiment of the present application, a facial video decoding method is provided, comprising:

[0011] Obtaining a facial video bitstream, the facial video bitstream comprising: an encoded reference facial video frame and encoded compact feature information; the encoded compact feature information representing key feature information of a target facial video frame to be reconstructed;

[0012] decoding the encoded reference facial video frame, and performing feature extraction on the decoded reference facial video frame to obtain a reference compact feature;

[0013] Decoding the encoded compact feature information to obtain target compact features of the target facial video frame;

[0014] performing sparse motion estimation based on the reference compact features and the target compact features to obtain a sparse motion estimation map, wherein the sparse motion estimation map represents a relative motion relationship between the target facial video frame and the reference facial video frame in a preset sparse feature domain;

[0015] A reconstructed facial video frame corresponding to the target facial video frame is obtained according to the sparse motion estimation map and the reference facial video frame.

[0016] According to a third aspect of an embodiment of the present application, a model training method is provided, comprising:

[0017] Inputting a target facial video frame sample into a feature extraction model to obtain a target compact feature sample; encoding the target compact feature sample and a reference facial video frame sample respectively to obtain a facial video bitstream sample;

[0018] Decoding the facial video bitstream sample to obtain the reference facial video frame sample and the target compact feature sample; inputting the reference facial video frame sample into the feature extraction model to obtain a reference compact feature sample;

[0019] Performing sparse motion estimation based on the reference compact feature samples and the target compact feature samples to obtain a sparse motion estimation sample graph; inputting the sparse motion estimation sample graph and the reference facial video frame samples into a deformable image prediction model to obtain an initial reconstructed facial video frame sample;

[0020] Constructing a perceptual loss function and an adversarial loss function based on the initial reconstructed facial video frame sample and the target facial video frame sample, respectively; and obtaining a rate-distortion loss function based on the transmission bit rates corresponding to the initial reconstructed facial video frame, the target facial video frame sample, and the target compact feature sample;

[0021] The perceptual loss function, the adversarial loss function, and the rate-distortion loss function are fused to obtain a training loss function; and the feature extraction model and the deformable image prediction model are trained according to the training loss function.

[0022] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the facial video encoding method as described in the first aspect, or an operation corresponding to the facial video decoding method as described in the second aspect, or an operation corresponding to the model training method as described in the third aspect.

[0023] According to a fifth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the facial video encoding method as described in the first aspect, or the facial video decoding method as described in the second aspect, or the model training method as described in the third aspect is implemented.

[0024] According to a sixth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the facial video encoding method as described in the first aspect, or operations corresponding to the facial video decoding method as described in the second aspect, or operations corresponding to the model training method as described in the third aspect.

[0025] According to the facial video encoding method and decoding method provided by the embodiments of the present application, in the encoding stage, the target compact feature is extracted from the target facial video frame, and a facial video bitstream is obtained by encoding the above-mentioned target compact feature. Since the target compact feature is a feature that characterizes the key feature information in the target facial video frame, it characterizes the key information in the entire facial video frame with a smaller amount of data. Therefore, the facial video bitstream obtained by encoding the target compact feature has a smaller amount of data, and the corresponding bitstream is also smaller (lower bit rate) when the video stream is transmitted. In addition, in the decoding stage, the facial video bitstream obtained above is decoded, and then the facial video frame is reconstructed based on the decoded target compact feature that characterizes the key feature information in the target facial video frame. The quality difference between the reconstructed video frame and the target facial video frame is also smaller. In summary, the embodiments of the present application can reduce the encoding bit rate while ensuring the quality of facial video reconstruction, thereby better meeting the needs of low-bitrate facial video encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0027] Figure 1 A schematic diagram of a framework for a coding and decoding method based on deep video generation;

[0028] Figure 2 This is a flowchart of a facial video encoding method according to the first embodiment of the present application;

[0029] Figure 3 for Figure 2 A schematic diagram of an example scenario in the illustrated embodiment;

[0030] Figure 4 This is a flowchart of a facial video decoding method according to the second embodiment of the present application;

[0031] Figure 5 for Figure 4 A schematic diagram of an example scenario in the illustrated embodiment;

[0032] Figure 6 for Figure 4 A schematic diagram of another example scenario in the illustrated embodiment;

[0033] Figure 7 This is a flowchart of the steps of a model training method according to Example 3 of the present application;

[0034] Figure 8 for Figure 7 A schematic diagram of an example scenario in the illustrated embodiment;

[0035] Figure 9 4 is a structural block diagram of a facial video encoding device according to the fourth embodiment of the present application;

[0036] Figure 10 4 is a structural block diagram of a facial video decoding device according to Embodiment 5 of the present application;

[0037] Figure 11 This is a structural block diagram of a model training device according to Example 6 of the present application;

[0038] Figure 12 This is a structural diagram of an electronic device according to embodiment 7 of the present application. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0040] See also Figure 1 , Figure 1 The main principle of this method is to deform the reference frame based on the motion of the frame to be encoded to obtain the reconstructed frame corresponding to the frame to be encoded. Figure 1 The basic framework of the encoding and decoding method based on deep video generation is explained:

[0041] In the first step, the encoding stage, the encoder uses a key point extractor to extract the target key point information of the target facial video frame to be encoded and encodes the target key point information; at the same time, the traditional image coding method (such as VVC, HEVC, etc.) is used to encode the reference facial video frame.

[0042] The second step is the decoding stage. The motion estimation module in the decoder extracts the reference key point information of the reference facial video frame through the key point extractor; and performs dense motion estimation based on the reference key point information and the target key point information to obtain a dense motion estimation map and an occlusion map. The dense motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in the feature domain represented by the key point information; the occlusion map represents the degree to which each pixel in the target facial video frame is occluded.

[0043] The third step is the decoding stage. The generation module in the decoder deforms the reference facial video frame based on the dense motion estimation map to obtain the deformation processing result, and then multiplies the deformation processing result with the occlusion map to output the reconstructed facial video frame.

[0044] Figure 1 In the method shown, facial video frames are reconstructed based on key point information extracted from the facial video frames. The key point information is information for display representation. During the encoding process, the data volume of the key point information cannot be further reduced based on the specific requirements for encoding bit consumption. Therefore, the above method cannot meet the encoding requirements of low-bitrate facial video frames.

[0045] In addition, compared with the original target facial video frame, the facial posture information and expression information of the reconstructed facial video frame based on the key point information is usually not reconstructed more accurately, that is, the reconstruction quality of the video frame is low.

[0046] In an embodiment of the present application, facial video frames are reconstructed based on compact features extracted from facial video frames to represent their key feature information. Compared with key point information, compact features, as implicit features, can not only represent the key feature information in the video frame, but also the size of the compact feature matrix can be further reduced according to the specific requirements of bit consumption. In other words, compact features can represent the key information in the entire facial video frame with a smaller amount of data. Therefore, the facial video bitstream obtained by encoding the compact features also has a smaller data volume, and the corresponding bitstream is also smaller (lower bit rate) when transmitting the video stream.

[0047] Furthermore, for facial video frames, key feature information can include facial features, position, posture, and expression. Therefore, compared to keypoint information, compact feature representations provide richer information, resulting in a reconstructed video frame with a closer image quality to the original target facial video frame.

[0048] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0049] Example 1

[0050] Reference Figure 2 , Figure 2 1 is a flowchart of a facial video encoding method according to Embodiment 1 of the present application. Specifically, the facial video encoding method provided in this embodiment includes the following steps:

[0051] Step 202: Obtain a target facial video frame to be encoded and a reference facial video frame.

[0052] Step 204 : extracting features from the target facial video frame to obtain target compact features, where the target compact features represent key feature information in the target facial video frame.

[0053] In an embodiment of the present application, a machine learning model can be used to extract features from target facial video frames to obtain target compact features. Specifically, the target facial video frames can be input into a pre-trained feature extraction model so that the feature extraction model outputs target compact features for each target facial video frame.

[0054] For facial video frames, key feature information may specifically include: facial feature position information, posture information, expression information, etc.

[0055] In the embodiment of the present application, there is no limitation on the structure and parameters of the feature extraction model, which can be set according to actual needs. For example, the feature extraction model can be a U-Net network composed of a convolutional layer and a generalized division normalization layer, etc.

[0056] Step 206 : Encode the target compact feature and the reference facial video frame respectively to obtain a facial video bitstream.

[0057] Specifically, the reference facial video frame can be encoded with relatively small quantization distortion, preserving the complete data of the reference facial video frame. For example, the reference facial video frame can be encoded using Versatile Video Coding (VVC). Target compact features can be encoded using quantization and entropy coding.

[0058] See also Figure 3 , Figure 3 This is a schematic diagram of the scene corresponding to the first embodiment of the present application. Figure 3 The schematic diagram shown in the figure illustrates an embodiment of the present application using a specific scenario as an example:

[0059] A target facial video frame a and a reference facial video frame a0 are obtained respectively; features are extracted from the target facial video frame a to obtain target compact features; the target compact features and the reference facial video frame a0 are encoded respectively to obtain a facial video bitstream, which can then be transmitted to a decoding end, so that the facial video stream based on the facial video bitstream is decoded by the decoding end to obtain a reconstructed facial video frame corresponding to the target facial video frame a.

[0060] Furthermore, in some embodiments of the present application, in order to further reduce the bit rate of facial video encoding, a differential operation can be performed based on the target compact features of adjacent target facial video frames, and the difference obtained by the differential operation is encoded to form a facial video bitstream.

[0061] The specific process is as follows:

[0062] Feature extraction is performed on each target facial video frame to obtain a target compact feature of each target facial video frame; a differential operation is performed on the target compact features of two adjacent target facial video frames to obtain a target compact feature residual; the target compact feature residual and the reference facial video frame are encoded respectively to obtain a facial video bitstream.

[0063] Compared with the method of directly encoding based on target compact features, the above method performs encoding based on the difference between target compact features to obtain a facial video bit stream. Obviously, the data amount of the difference between target compact features is smaller than the data amount of the target compact features themselves. Therefore, encoding based on the difference between target compact features can effectively reduce the bit rate of facial video encoding.

[0064] In the embodiment of the present application, during the encoding stage, a target compact feature is extracted from the target facial video frame, and a facial video bitstream is obtained by encoding the target compact feature. Since the target compact feature is a feature that characterizes the key feature information in the target facial video frame, it characterizes the key information in the entire facial video frame with a smaller amount of data. Therefore, the facial video bitstream obtained by encoding the target compact feature also has a smaller amount of data, and the corresponding bitstream is also smaller (lower bit rate) when the video stream is transmitted. The embodiment of the present application can reduce the encoding bit rate, better meeting the needs of low-bitrate facial video encoding.

[0065] The facial video encoding method provided in the first embodiment of the present application can be executed by a video encoding terminal (encoder) to encode facial video files to achieve digital broadband compression of the facial video files. It can be applied to a variety of different scenarios, such as: the storage and streaming of conventional video games involving faces, specifically: the facial video encoding method provided in the embodiment of the present application can be used to encode game video frames to form a corresponding video stream for storage and transmission in video streaming services or other similar applications; another example: low-latency scenarios such as video conferencing and live video broadcasting, specifically: the facial video encoding method provided in the embodiment of the present application can be used to encode facial video data collected by a video capture device to form a corresponding video stream, and send it to a conference terminal, and the conference terminal decodes the video stream to obtain the corresponding facial video image; another example: virtual reality scenarios, the facial video encoding method provided in the embodiment of the present application can be used to encode facial video data collected by a video capture device to form a corresponding video stream, and send it to a virtual reality-related device (such as VR virtual glasses, etc.), and the VR device decodes the video stream to obtain the corresponding facial video image, and implement corresponding VR functions based on the facial video image, etc.

[0066] Example 2

[0067] Reference Figure 4 , Figure 4 Flowchart of a facial video decoding method according to Embodiment 2 of the present application. Specifically, the facial video decoding method provided in this embodiment includes the following steps:

[0068] Step 402: Obtain a facial video bitstream, where the facial video bitstream includes: an encoded reference facial video frame and encoded compact feature information.

[0069] The encoded compact feature information represents the key feature information of the target facial video frame to be reconstructed.

[0070] In an embodiment of the present application, the encoded compact feature information corresponds to the compact feature information obtained by extracting features from each target facial video frame for characterizing key feature information, and may also correspond to: the difference between the target compact features of adjacent target facial video frames.

[0071] Step 404 : Decode the encoded reference facial video frame, and perform feature extraction on the decoded reference facial video frame to obtain a reference compact feature.

[0072] The machine learning model can be used to extract features from the reference facial video frames to obtain reference compact features. Specifically, the reference facial video frames can be input into a pre-trained feature extraction model so that the feature extraction model outputs reference compact features for each reference facial video frame.

[0073] Step 406 : Decode the encoded compact feature information to obtain target compact features of the target facial video frame.

[0074] When the encoded compact feature information corresponds to the compact feature information obtained by feature extraction of each target facial video frame for characterizing key feature information, the encoded compact feature information can be decoded to obtain the target compact feature of the target facial video frame; when the encoded compact feature information corresponds to: the difference between the target compact features of adjacent target facial video frames, then after obtaining the target compact feature of the previous target facial video frame, the target compact feature of the subsequent target facial video frame can be calculated based on the difference between the decoded target compact features.

[0075] Step 408 : Perform sparse motion estimation based on the reference compact features and the target compact features to obtain a sparse motion estimation map.

[0076] The sparse motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset sparse feature domain.

[0077] The relative motion relationship between two different facial video frames can be represented in several ways. For example, the relative motion relationship between each pixel in the two facial video frames can be calculated separately at the pixel level; or, feature extraction can be performed on the two facial video frames to obtain corresponding relatively sparse feature maps, and then the relative motion relationship between the two facial video frames can be calculated separately at the above-mentioned feature map (feature domain) level.

[0078] In this step, the latter method can be used to obtain the relative motion relationship between the reference facial video frame and the target facial video frame in the compact feature domain based on the reference compact feature and the target compact feature at the compact feature level.

[0079] Step 410 : Obtain a reconstructed facial video frame corresponding to the target facial video frame according to the sparse motion estimation map and the reference facial video frame.

[0080] See also Figure 5 , Figure 5 The schematic diagram of the scene corresponding to the second embodiment of the present application is shown below. Figure 5 The schematic diagram shown in the figure illustrates an embodiment of the present application using a specific scenario as an example:

[0081] A facial video bitstream consisting of an encoded reference facial video frame and encoded compact feature information is obtained; the encoded reference facial video frame is decoded to obtain a reference facial video frame, and features are extracted from the reference facial video frame to obtain reference compact features; in addition, the encoded compact feature information is decoded to obtain target compact features of a target facial video frame; then, sparse motion estimation is performed based on the obtained reference compact features and target compact features to obtain a sparse motion estimation map; finally, the facial video frame can be reconstructed based on the sparse motion estimation map and the reference facial video frame to obtain a reconstructed facial video frame corresponding to the target facial video frame.

[0082] Since the sparse motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in the sparse feature domain, that is, the sparse motion estimation map represents a relatively rough relative motion relationship, therefore, directly generating a reconstructed facial video frame corresponding to the target facial video frame based on the sparse motion estimation map and the reference facial video frame may result in a large difference in quality between the reconstructed facial video frame and the target facial video frame.

[0083] Therefore, in order to further improve the quality of the reconstructed facial video frame, in some embodiments, obtaining the reconstructed facial video frame corresponding to the target facial video frame based on the sparse motion estimation map and the reference facial video frame may include:

[0084] Based on the sparse motion estimation graph, the reference facial video frame is deformed to obtain the initial reconstructed facial video frame corresponding to the target facial video frame;

[0085] Performing a differential operation on the reference compact feature and the target compact feature to obtain a compact feature difference;

[0086] Dense motion estimation is performed based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map, which represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset dense feature domain;

[0087] According to the dense motion estimation map and the reference facial video frame, a reconstructed facial video frame corresponding to the target facial video frame is obtained.

[0088] In the above method, a dense motion estimation map is obtained again based on the difference between compact features and the initial reconstructed facial video frame generated based on the sparse motion estimation map and the reference facial video frame. That is, in a denser feature domain, the relative motion relationship between the target facial video frame and the reference facial video frame is more accurate than the relative motion relationship represented by the sparse motion estimation map. Therefore, a reconstructed facial video frame is generated based on the dense motion estimation map and the reference facial video frame, which can improve the quality of the reconstructed facial video frame.

[0089] Furthermore, in some embodiments, performing dense motion estimation based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map may further include:

[0090] Dense motion estimation is performed based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map and an occlusion map. The occlusion map represents the degree to which each pixel in the target facial video frame is occluded.

[0091] According to the dense motion estimation map and the reference facial video frame, a reconstructed facial video frame corresponding to the target facial video frame is obtained, including:

[0092] According to the dense motion estimation map, the reference facial video frame and the occlusion map, a reconstructed facial video frame corresponding to the target facial video frame is obtained.

[0093] Specifically, the reference facial video frame can be deformed according to the dense motion estimation map to obtain a deformed facial video frame, and then the deformed facial video frame can be further deformed based on the occlusion map to obtain the final reconstructed facial video frame.

[0094] In the reference and target facial video frames, the faces may be twisted at a certain angle, resulting in occlusion of some pixels. For example, the reference facial video frame may show a frontal face, while the target facial video frame may be slightly rotated to the left or right. In this case, some pixels may be occluded.

[0095] Therefore, in order to further improve the quality of reconstructed facial video frames, in the process of generating reconstructed facial video frames, we can consider the probability of each pixel in the video frame being occluded on the basis of the dense motion estimation map, and perform deformation processing on the reference facial video frame based on the dense motion estimation map and the occlusion map, so as to obtain a more accurate reconstructed facial video frame.

[0096] Furthermore, to improve the overall efficiency of facial video decoding, some of the above steps can be processed with the help of machine learning models. Specifically:

[0097] Feature extraction is performed on the decoded reference facial video frame to obtain reference compact features, which may include:

[0098] The decoded reference facial video frame is input into the feature extraction model so that the feature extraction model outputs a reference compact feature.

[0099] Based on the sparse motion estimation graph, the reference facial video frame is deformed to obtain an initial reconstructed facial video frame corresponding to the target facial video frame, which may include:

[0100] The sparse motion estimation map and the reference facial video frame are input into the deformable image prediction model, so that the deformable image prediction model outputs an initial reconstructed facial video frame corresponding to the target facial video frame.

[0101] Dense motion estimation is performed based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map, which may include:

[0102] The compact feature difference and the initial reconstructed facial video frame are input into a dense motion estimation model, so that the dense motion estimation model outputs a dense motion estimation map.

[0103] Obtaining a reconstructed facial video frame corresponding to a target facial video frame according to the dense motion estimation map, the reference facial video frame, and the occlusion map may include:

[0104] The dense motion estimation map, the reference facial video frame, and the occlusion map are input into the generative model so that the generative model outputs a reconstructed facial video frame corresponding to the target facial video frame.

[0105] See also Figure 6 , Figure 6 This is another scene diagram corresponding to the second embodiment of the present application. Figure 5 Based on the scene shown, the reference facial video frame is deformed based on the dense motion estimation map and the occlusion map to obtain the final reconstructed facial video frame. Specifically:

[0106] exist Figure 5 On the basis of the sparse motion estimation map and the reference facial video frame, the facial video frame is reconstructed to obtain an initial reconstructed facial video frame corresponding to the target facial video frame; at the same time, a differential operation is performed on the reference facial video frame and the target facial video frame to obtain a differential operation result; based on the initial reconstructed facial video frame and the above differential operation result, dense motion estimation is performed to obtain a dense motion estimation map and an occlusion map; finally, based on the dense motion estimation map, the occlusion map and the reference facial video frame, the final reconstructed facial video frame corresponding to the target facial video frame is obtained.

[0107] In this embodiment of the present application, during the decoding phase, the facial video bitstream obtained during the encoding phase is decoded, and then facial video frames are reconstructed based on the target compact features obtained during decoding. Because the target compact features can represent key feature information in the target facial video frames, the quality difference between the reconstructed video frames obtained based on the target compact features and the target facial video frames is relatively small. This embodiment of the present application can reduce the encoding bit rate while obtaining higher-quality reconstructed facial video frames.

[0108] The facial video decoding method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: a server, a PC, etc.

[0109] Example 3

[0110] Reference Figure 7 , Figure 7 The following is a flowchart of a model training method according to the third embodiment of the present application. Specifically, the model training method provided in this embodiment includes the following steps:

[0111] Step 702: Input the target facial video frame sample into the feature extraction model to obtain a target compact feature sample; encode the target compact feature sample and the reference facial video frame sample respectively to obtain a facial video bitstream sample.

[0112] In this application, there is no limitation on the structure and parameters of the feature extraction model, which can be set according to actual needs. For example, the feature extraction model can be a U-Net network composed of a convolutional layer and a generalized division normalization layer, etc.

[0113] Step 704 : Decode the facial video bitstream sample to obtain a reference facial video frame sample and a target compact feature sample; input the reference facial video frame sample into a feature extraction model to obtain a reference compact feature sample.

[0114] The feature extraction model in this step may be a model that is completely identical to the feature extraction model in step 602 , so as to obtain a reference compact feature sample corresponding to the target compact feature sample.

[0115] Step 706 , performing sparse motion estimation based on the reference compact feature samples and the target compact feature samples to obtain a sparse motion estimation sample map; inputting the sparse motion estimation sample map and the reference facial video frame samples into a deformable image prediction model to obtain initial reconstructed facial video frame samples.

[0116] In this application, there is no limitation on the structure and parameters of the deformable image prediction model, which can be set according to actual needs. For example, it can also be a U-Net network composed of a convolutional layer and a generalized division normalization layer, etc.

[0117] The specific execution process of each step in the above steps 702 to 706 can refer to the corresponding steps in the above embodiment 1 or embodiment 2, and will not be repeated here.

[0118] Step 708: construct a perceptual loss function and an adversarial loss function based on the initial reconstructed facial video frame sample and the target facial video frame sample, respectively; and obtain a rate-distortion loss function based on the transmission bit rates corresponding to the initial reconstructed facial video frame, the target facial video frame sample, and the target compact feature sample.

[0119] Specifically, the perceptual loss function can be constructed as follows:

[0120] The initial reconstructed facial video frame samples and the target facial video frame samples are respectively input into a preset trained image classification model, such as the VGG-19 network model, to obtain the initial feature map corresponding to the initial reconstructed facial video frame samples and the target feature map corresponding to the target facial video frame samples; then, based on the initial feature map and the target feature map, the mean square error is calculated to obtain the perceptual loss function.

[0121] The adversarial loss function can be constructed as follows: the initial reconstructed facial video frame sample and the target facial video frame sample are simultaneously input into a pre-trained classifier, and the corresponding adversarial loss function is constructed based on the classification results (whether they are video frames of the same type).

[0122] The process of constructing the rate-distortion loss function may include: first obtaining the transmission bit rate corresponding to the target compact feature sample, and then constructing a distortion function based on the initial reconstructed facial video frame sample and the target facial video frame sample (in the embodiment of the present application, the specific method used to construct the distortion function is not limited, for example: the distortion function can be constructed using the deep image structure and texture similarity algorithm, etc.), and then fusing the above-mentioned transmission bit rate and the constructed distortion function (such as addition, etc.) to obtain the rate-distortion loss function.

[0123] In step 710 , the perceptual loss function, the adversarial loss function, and the rate-distortion loss function are integrated to obtain a training loss function; and a feature extraction model and a deformable image prediction model are trained based on the training loss function.

[0124] Specifically, corresponding weight values ​​can be set for the perceptual loss function, the adversarial loss function, and the rate-distortion loss function, respectively. Then, based on the set weight values, the perceptual loss function, the adversarial loss function, and the rate-distortion loss function are summed to obtain the final training loss function. Specifically, see the following formula:

[0125] L=λ1L per +λ2L GD+λ3L RD

[0126] Among them, L is the final training loss function; L per is the perceptual loss function; L GD is the adversarial loss function; L RD is the rate-distortion loss function; λ1, λ2, and λ3 are the weight values ​​of the perceptual loss function, the adversarial loss function, and the rate-distortion loss function, respectively.

[0127] See also Figure 8 , Figure 8 This is a schematic diagram of the scene corresponding to the third embodiment of this application. Figure 8 The schematic diagram shown in the figure illustrates an embodiment of the present application using a specific scenario as an example:

[0128] Input the target facial video frame sample into the feature extraction model to be trained to obtain the target compact feature sample; encode the target compact feature sample and the reference facial video frame sample respectively to obtain the facial video bitstream sample; decode the encoded reference facial video frame sample in the facial video bitstream sample to obtain the reference facial video frame sample; decode the encoded compact feature sample in the facial video bitstream sample to obtain the target compact feature sample; then input the reference facial video frame sample into the above feature extraction model to obtain the reference compact feature sample; perform sparse motion estimation based on the reference compact feature sample and the target compact feature sample to obtain the sparse motion estimation sample graph; input the sparse motion estimation sample graph and the reference facial video frame sample into the deformable image prediction model to be trained to obtain the initial reconstructed facial video frame sample; construct the perceptual loss function L according to the initial reconstructed facial video frame sample and the target facial video frame sample. per And the adversarial loss function L GD Based on the transmission bit rate corresponding to the initial reconstructed facial video frame, the target facial video frame sample and the target compact feature sample, the rate-distortion loss function L is obtained RD ; for L per , L GD and L RD The training loss function L is obtained by fusion. According to L, the feature extraction model and deformable image prediction model to be trained are trained to obtain the trained feature extraction model and deformable image prediction model.

[0129] Further, with Figure 6 Correspondingly, in some of the embodiments, a dense motion estimation model and a generation model may be further introduced. Figure 8Based on the training procedure shown, the following improvements are made: the sparse motion estimation sample map and the reference facial video frame samples are input into the deformable image prediction model to obtain the initial reconstructed facial video frame samples; the reference compact feature samples and the target compact feature samples are differentially operated to obtain the compact feature sample difference; the compact feature sample difference and the initial reconstructed facial video frame samples are input into the dense motion estimation model to be trained to obtain the dense motion estimation sample map and the occlusion sample map; the dense motion estimation sample map, the reference facial video frame samples and the occlusion sample map are input into the generative model to be trained to obtain the reconstructed facial video frame.

[0130] Afterwards, a first perceptual loss function is constructed based on the initial reconstructed facial video frame samples and the target facial video frame samples; a second perceptual loss function is constructed based on the reconstructed facial video frame and the target facial video frame samples; an adversarial loss function is constructed based on the reconstructed facial video frame and the target facial video frame samples; a rate-distortion loss function is obtained based on the distortion loss function constructed based on the reconstructed facial video frame and the target facial video frame samples, and the transmission bit rate corresponding to the target compact feature sample; the first perceptual loss function, the second perceptual loss function, the adversarial loss function and the rate-distortion loss function are fused to obtain a training loss function; according to the training loss function, the above-mentioned feature extraction model, deformable image prediction model, dense motion estimation model and generation model to be trained are trained to obtain the trained feature extraction model, deformable image prediction model, dense motion estimation model and generation model.

[0131] The model training method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: servers, PCs, etc.

[0132] Example 4

[0133] See also Figure 9 , Figure 9 : This is a structural block diagram of a facial video encoding device according to the fourth embodiment of the present application. The facial video encoding device provided in this embodiment of the present application includes:

[0134] The facial video frame acquisition module 902 is configured to acquire a target facial video frame to be encoded and a reference facial video frame.

[0135] The feature extraction module 904 is configured to extract features from the target facial video frame to obtain target compact features, where the target compact features represent key feature information in the target facial video frame.

[0136] The encoding module 906 is configured to encode the target compact feature and the reference facial video frame respectively to obtain a facial video bitstream.

[0137] Optionally, in some embodiments, the target facial video frame is a plurality of continuous facial video frames; the feature extraction module 904 is specifically configured to: perform feature extraction on each target facial video frame to obtain a target compact feature of each target facial video frame;

[0138] The encoding module 906 is specifically configured to perform a differential operation on the target compact features of two adjacent target facial video frames to obtain a target compact feature residual;

[0139] The target compact feature residual and the reference facial video frame are encoded separately to obtain the facial video bitstream.

[0140] Optionally, in some embodiments, the feature extraction module 904 is specifically configured to input each target facial video frame into a feature extraction model, so that the feature extraction model outputs a target compact feature of each target facial video frame.

[0141] The facial video encoding device of this embodiment is used to implement the corresponding facial video encoding methods of the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which are not further described here. Furthermore, the functional implementation of each module in the facial video encoding device of this embodiment can refer to the corresponding descriptions of the aforementioned method embodiments and are not further described here.

[0142] Example 5

[0143] See also Figure 10 , Figure 10 FIG1 is a structural block diagram of a facial video decoding device according to Embodiment 5 of the present application. The facial video decoding device provided in the embodiment of the present application includes:

[0144] The video bitstream acquisition module 1002 is configured to acquire a facial video bitstream, wherein the facial video bitstream includes: an encoded reference facial video frame and encoded compact feature information; the encoded compact feature information represents key feature information of a target facial video frame to be reconstructed;

[0145] A first decoding module 1004 is configured to decode the encoded reference facial video frame and perform feature extraction on the decoded reference facial video frame to obtain a reference compact feature;

[0146] A second decoding module 1006 is configured to decode the encoded compact feature information to obtain target compact features of the target facial video frame;

[0147] A sparse motion estimation module 1008 is configured to perform sparse motion estimation based on the reference compact features and the target compact features to obtain a sparse motion estimation map, wherein the sparse motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset sparse feature domain;

[0148] The reconstructed facial video frame obtaining module 1010 is configured to obtain a reconstructed facial video frame corresponding to a target facial video frame according to the sparse motion estimation map and the reference facial video frame.

[0149] Optionally, in some embodiments, the facial video frame reconstruction module 1010 is specifically configured to:

[0150] Based on the sparse motion estimation graph, the reference facial video frame is deformed to obtain the initial reconstructed facial video frame corresponding to the target facial video frame;

[0151] Performing a differential operation on the reference compact feature and the target compact feature to obtain a compact feature difference;

[0152] Dense motion estimation is performed based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map, which represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset dense feature domain;

[0153] According to the dense motion estimation map and the reference facial video frame, a reconstructed facial video frame corresponding to the target facial video frame is obtained.

[0154] Optionally, in some embodiments, when executing the step of performing dense motion estimation based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map, the reconstructed facial video frame obtaining module 1010 is specifically used to: perform dense motion estimation based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map and an occlusion map, wherein the occlusion map represents the degree to which each pixel in the target facial video frame is occluded; when executing the step of obtaining a reconstructed facial video frame corresponding to the target facial video frame based on the dense motion estimation map and the reference facial video frame, the reconstructed facial video frame corresponding to the target facial video frame is specifically used to: obtain a reconstructed facial video frame corresponding to the target facial video frame based on the dense motion estimation map, the reference facial video frame and the occlusion map.

[0155] Optionally, in some embodiments, when performing the step of extracting features from the decoded reference facial video frame to obtain reference compact features, the first decompression module 1004 is specifically configured to:

[0156] The decoded reference facial video frame is input into the feature extraction model so that the feature extraction model outputs a reference compact feature.

[0157] Optionally, in some embodiments, the reconstructed facial video frame obtaining module 1010, when executing the step of deforming the reference facial video frame based on the sparse motion estimation map to obtain the initial reconstructed facial video frame corresponding to the target facial video frame, is specifically used to: input the sparse motion estimation map and the reference facial video frame into the deformable image prediction model, so that the deformable image prediction model outputs the initial reconstructed facial video frame corresponding to the target facial video frame.

[0158] Optionally, in some embodiments, the reconstructed facial video frame obtaining module 1010, when performing the step of performing dense motion estimation based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map, is specifically used to: input the compact feature difference and the initial reconstructed facial video frame into a dense motion estimation model, so that the dense motion estimation model outputs a dense motion estimation map.

[0159] Optionally, in some embodiments, the reconstructed facial video frame obtaining module 1010, when executing the step of obtaining a reconstructed facial video frame corresponding to the target facial video frame according to the dense motion estimation map, the reference facial video frame, and the occlusion map, is specifically configured to:

[0160] The dense motion estimation map, the reference facial video frame, and the occlusion map are input into the generative model so that the generative model outputs a reconstructed facial video frame corresponding to the target facial video frame.

[0161] The facial video decoding device of this embodiment is used to implement the corresponding facial video decoding methods of the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. Furthermore, the functional implementation of each module in the facial video decoding device of this embodiment can be referenced to the corresponding descriptions of the aforementioned method embodiments and will not be described in detail here.

[0162] Example 6

[0163] See also Figure 11 , Figure 11 This is a structural block diagram of a model training device according to Example 6 of the present application. The model training device provided in this embodiment of the application includes:

[0164] The facial video bitstream sample obtaining module 1102 is configured to input the target facial video frame sample into a feature extraction model to obtain a target compact feature sample; and to encode the target compact feature sample and the reference facial video frame sample respectively to obtain a facial video bitstream sample.

[0165] The compact feature sample obtaining module 1104 is configured to decode the encoded facial video stream sample to obtain a reference facial video frame sample and a target compact feature sample; and input the reference facial video frame sample into a feature extraction model to obtain a reference compact feature sample.

[0166] Initial reconstructed facial video frame sample obtaining module 1106 is configured to perform sparse motion estimation based on the reference compact feature samples and the target compact feature samples to obtain a sparse motion estimation sample map; the sparse motion estimation sample map and the reference facial video frame samples are input into a deformable image prediction model to obtain initial reconstructed facial video frame samples;

[0167] A rate-distortion loss function obtaining module 1108 is configured to construct a perceptual loss function and an adversarial loss function based on the initial reconstructed facial video frame sample and the target facial video frame sample, respectively; and obtain a rate-distortion loss function based on the transmission bit rates corresponding to the initial reconstructed facial video frame, the target facial video frame sample, and the target compact feature sample;

[0168] The model training module 1110 is used to fuse the perceptual loss function, the adversarial loss function and the rate-distortion loss function to obtain a training loss function; and train the feature extraction model and the deformable image prediction model according to the training loss function.

[0169] The model training device of this embodiment is used to implement the corresponding model training methods of the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the model training device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0170] Example 7

[0171] Reference Figure 12 , shows a structural diagram of an electronic device according to embodiment seven of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0172] like Figure 12 As shown, the conference terminal may include: a processor (processor) 1202 , a communications interface (Communications Interface) 1204 , a memory (memory) 1206 , and a communication bus 1208 .

[0173] in:

[0174] The processor 1202 , the communication interface 1204 , and the memory 1206 communicate with each other via a communication bus 1208 .

[0175] The communication interface 1204 is used to communicate with other electronic devices or servers.

[0176] The processor 1202 is used to execute the program 1210, and specifically can execute the relevant steps in the above-mentioned facial video encoding method, or the facial video decoding method, or the model training method embodiment.

[0177] Specifically, the program 1210 may include program codes, which include computer operation instructions.

[0178] The processor 1202 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0179] The memory 1206 is used to store the program 1210. The memory 1206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0180] Program 1210 can specifically be used to enable processor 1202 to perform the following operations: obtain a target facial video frame to be encoded and a reference facial video frame; perform feature extraction on the target facial video frame to obtain a target compact feature, where the target facial compact feature represents key feature information in the target facial video frame; and encode the target facial compact feature and the reference facial video frame respectively to obtain a facial video bitstream.

[0181] or,

[0182] Program 1210 can be specifically used to enable processor 1202 to perform the following operations: obtain a facial video bitstream, the facial video bitstream including: an encoded reference facial video frame and encoded compact feature information; the facial encoded compact feature information represents key feature information of a target facial video frame to be reconstructed; decode the facial encoded reference facial video frame, and perform feature extraction on the decoded reference facial video frame to obtain a reference compact feature; decode the facial encoded compact feature information to obtain a target compact feature of a target facial video frame; perform sparse motion estimation based on the facial reference compact feature and the facial target compact feature to obtain a sparse motion estimation map, the facial sparse motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset sparse feature domain; obtain a reconstructed facial video frame corresponding to the target facial video frame based on the facial sparse motion estimation map and the facial reference facial video frame.

[0183] or,

[0184] The program 1210 can be specifically used to enable the processor 1202 to perform the following operations: input the target facial video frame sample into the feature extraction model to obtain the target compact feature sample; encode the facial target compact feature sample and the reference facial video frame sample respectively to obtain the facial video bitstream sample; decode the facial video bitstream sample to obtain the facial reference facial video frame sample and the facial target compact feature sample; input the facial reference facial video frame sample into the facial feature extraction model to obtain the reference compact feature sample; perform sparse motion estimation based on the facial reference compact feature sample and the facial target compact feature sample to obtain the sparse motion estimation sample graph; and decode the facial sparse motion estimation sample into the facial sparse motion estimation sample graph. The motion estimation sample map and the facial reference facial video frame samples are input into the deformable image prediction model to obtain the initial reconstructed facial video frame samples; according to the facial initial reconstructed facial video frame samples and the facial target facial video frame samples, the perceptual loss function and the adversarial loss function are constructed respectively; based on the transmission bit rate corresponding to the facial initial reconstructed facial video frame, the facial target facial video frame samples and the facial target compact feature samples, the rate-distortion loss function is obtained; the facial perceptual loss function, the adversarial loss function and the rate-distortion loss function are fused to obtain the training loss function; according to the facial training loss function, the facial feature extraction model and the facial deformable image prediction model are trained.

[0185] The specific implementation of each step in program 1210 can be found in the corresponding descriptions of the corresponding steps and units in the above-mentioned facial video encoding method, facial video decoding method, or model training method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiments, and will not be repeated here.

[0186] Through the electronic device of this embodiment, in the encoding stage, the target compact feature is extracted from the target facial video frame, and the facial video bitstream is obtained by encoding the above-mentioned target compact feature. Since the target compact feature is a feature that characterizes the key feature information in the target facial video frame, it characterizes the key information in the entire facial video frame with a smaller amount of data. Therefore, the facial video bitstream obtained by encoding the target compact feature has a smaller amount of data, and the corresponding bitstream is also smaller (lower bit rate) when the video stream is transmitted. In addition, in the decoding stage, the facial video bitstream obtained above is decoded, and then the facial video frame is reconstructed based on the decoded target compact feature that characterizes the key feature information in the target facial video frame. The quality difference between the reconstructed video frame and the target facial video frame is also smaller. In summary, the embodiment of the present application can reduce the encoding bit rate while ensuring the quality of facial video reconstruction, thereby better meeting the needs of low-bitrate facial video encoding.

[0187] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0188] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0189] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the facial video encoding method, or the facial video decoding method, or the model training method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the facial video encoding method, or the facial video decoding method, or the model training method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the facial video encoding method, or the facial video decoding method, or the model training method shown herein.

[0190] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0191] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A facial video encoding method, comprising: Obtaining a target facial video frame to be encoded and a reference facial video frame; Performing feature extraction on the target facial video frame to obtain a target compact feature, wherein the target compact feature represents key feature information in the target facial video frame; The target compact features and the reference facial video frame are respectively encoded to obtain a facial video bitstream, wherein the facial video bitstream is used to reconstruct the target facial video frame at a decoding end, wherein the reconstructed target facial video frame is obtained based on the reference facial video frame and a dense motion estimation map, wherein the dense motion estimation map is obtained by performing dense motion estimation on an initial reconstructed facial video frame corresponding to the target facial video frame and a compact feature difference; the initial reconstructed facial video frame is obtained by deforming the reference facial video frame based on the sparse motion estimation map; The sparse motion estimation map is obtained by performing sparse motion estimation on the reference compact features corresponding to the reference facial video frame and the target compact features, and the compact feature difference is obtained by performing a differential operation on the reference compact features corresponding to the reference facial video frame and the target compact features.

2. The method according to claim 1, wherein The target facial video frames are a plurality of continuous facial video frames; and extracting features from the target facial video frames to obtain compact features of the target facial video frames includes: Extract features of each target facial video frame respectively to obtain target compact features of each target facial video frame; The encoding of the target compact feature and the reference facial video frame to obtain a facial video bitstream includes: Perform differential operation on the target compact features of two adjacent target facial video frames to obtain the target compact feature residual; The target compact feature residual and the reference facial video frame are encoded respectively to obtain a facial video bitstream.

3. The method according to claim 2, wherein: The feature extraction of each target facial video frame is performed to obtain target compact features of each target facial video frame, including: Each target facial video frame is input into a feature extraction model respectively, so that the feature extraction model outputs a target compact feature of each target facial video frame.

4. A facial video decoding method, comprising: Obtaining a facial video bitstream, the facial video bitstream comprising: an encoded reference facial video frame and encoded compact feature information; the encoded compact feature information representing key feature information of a target facial video frame to be reconstructed; decoding the encoded reference facial video frame, and performing feature extraction on the decoded reference facial video frame to obtain a reference compact feature; Decoding the encoded compact feature information to obtain target compact features of the target facial video frame; performing sparse motion estimation based on the reference compact features and the target compact features to obtain a sparse motion estimation map, wherein the sparse motion estimation map represents a relative motion relationship between the target facial video frame and the reference facial video frame in a preset sparse feature domain; A reconstructed facial video frame corresponding to the target facial video frame is obtained according to the sparse motion estimation map and the reference facial video frame, including: performing deformation processing on the reference facial video frame based on the sparse motion estimation map to obtain an initial reconstructed facial video frame corresponding to the target facial video frame; performing a differential operation on the reference compact feature and the target compact feature to obtain a compact feature difference; performing dense motion estimation according to the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map; and obtaining a reconstructed facial video frame corresponding to the target facial video frame according to the dense motion estimation map and the reference facial video frame.

5. The method according to claim 4, wherein The dense motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in a preset dense feature domain.

6. The method according to claim 5, wherein: The performing dense motion estimation based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map includes: performing dense motion estimation based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map and an occlusion map, wherein the occlusion map represents the degree to which each pixel in the target facial video frame is occluded; The step of obtaining a reconstructed facial video frame corresponding to the target facial video frame according to the dense motion estimation map and the reference facial video frame includes: A reconstructed facial video frame corresponding to the target facial video frame is obtained according to the dense motion estimation map, the reference facial video frame, and the occlusion map.

7. The method according to claim 6, wherein: The feature extraction of the decoded reference facial video frame to obtain a reference compact feature includes: The decoded reference facial video frame is input into a feature extraction model so that the feature extraction model outputs a reference compact feature.

8. The method according to claim 6, wherein: The step of performing deformation processing on the reference facial video frame based on the sparse motion estimation map to obtain an initial reconstructed facial video frame corresponding to the target facial video frame includes: The sparse motion estimation map and the reference facial video frame are input into a deformable image prediction model, so that the deformable image prediction model outputs an initial reconstructed facial video frame corresponding to the target facial video frame.

9. The method according to claim 6, wherein: The performing dense motion estimation based on the compact feature difference and the initial reconstructed facial video frame to obtain a dense motion estimation map includes: The compact feature difference and the initial reconstructed facial video frame are input into a dense motion estimation model, so that the dense motion estimation model outputs a dense motion estimation map.

10. The method according to claim 6, wherein: Obtaining a reconstructed facial video frame corresponding to the target facial video frame according to the dense motion estimation map, the reference facial video frame, and the occlusion map includes: The dense motion estimation map, the reference facial video frame, and the occlusion map are input into a generative model so that the generative model outputs a reconstructed facial video frame corresponding to the target facial video frame.

11. A model training method comprising: Input the target facial video frame sample into the feature extraction model to obtain the target compact feature sample; Encoding the target compact feature sample and the reference facial video frame sample respectively to obtain a facial video bitstream sample; Decoding the facial video bitstream sample to obtain the reference facial video frame sample and the target compact feature sample; Inputting the reference facial video frame sample into the feature extraction model to obtain a reference compact feature sample; Performing sparse motion estimation based on the reference compact feature samples and the target compact feature samples to obtain a sparse motion estimation sample graph; inputting the sparse motion estimation sample graph and the reference facial video frame samples into a deformable image prediction model to obtain an initial reconstructed facial video frame sample; Constructing a perceptual loss function and an adversarial loss function based on the initial reconstructed facial video frame sample and the target facial video frame sample, respectively; and obtaining a rate-distortion loss function based on the transmission bit rates corresponding to the initial reconstructed facial video frame, the target facial video frame sample, and the target compact feature sample; The perceptual loss function, the adversarial loss function, and the rate-distortion loss function are integrated to obtain a training loss function; and the feature extraction model and the deformable image prediction model are trained according to the training loss function; The training of the feature extraction model and the deformable image prediction model according to the training loss function includes: training the feature extraction model, the deformable image prediction model, the dense motion estimation model, and the generative model according to the training loss function; The dense motion estimation model is used to obtain a dense motion estimation sample graph based on the compact feature sample difference obtained by performing a differential operation on the reference compact feature sample and the target compact feature sample, and the initial reconstructed facial video frame sample; the generation model is used to obtain a reconstructed facial video frame based on the dense motion estimation sample graph and the reference facial video frame sample, and the reconstructed facial video frame is used to construct the training loss function.

12. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, which enables the processor to perform an operation corresponding to the facial video encoding method according to any one of claims 1 to 3, or an operation corresponding to the facial video decoding method according to any one of claims 4 to 9, or an operation corresponding to the model training method according to claim 11.

Images