A face video decoding method and apparatus, and a communication method and system

Abstract

The application discloses a face video decoding method and device, and a communication method and system, relates to the field of image coding and decoding, and aims to solve the problems of face distortion and poor decoupling effect of foreground and background in face video communication. The application performs feature alignment on a data stream obtained by encoding an intra-frame image and features extracted from an inter-frame image, obtains a decoded intra-frame image, and obtains aligned intra-frame features and inter-frame features; extracts intra-frame appearance features from the decoded intra-frame image; estimates affine parameters according to the aligned intra-frame features and the inter-frame features; performs affine transformation on the intra-frame appearance features based on the affine parameters, and performs image reconstruction on the result after the affine transformation based on the affine parameters. The application introduces affine transformation to model large motion, effectively restores the face, and retains detailed motion details, so that the modeling precision of the face and the decoupling effect of the face and the background are excellent.

Description

Technical Field

[0001] This invention relates to the field of image encoding and decoding, and in particular to a face video decoding method and apparatus based on affine transformation, as well as a face video communication method and system. Background Technology

[0002] In recent years, facial video communication has been widely used in scenarios such as video conferencing, online education, live streaming, healthcare, animation, and virtual avatars. This type of communication focuses on processing and transmitting facial images to achieve more natural and efficient communication. However, providing high-quality facial video requires significant bandwidth and storage space, posing a major challenge in resource-constrained environments. Existing video encoding methods often lead to severe quality distortion under low bandwidth conditions. Therefore, designing an efficient facial video compression framework to achieve high-quality video reconstruction under limited bandwidth is crucial.

[0003] Traditional video compression standards, such as H.264 / AVC, H.265 / HEVC, and the latest H.266 / VVC, have continuously improved their redundancy reduction capabilities through well-designed hybrid coding frameworks. These improvements effectively reduce bandwidth requirements without significantly degrading visual quality. However, the emergence of neural networks with powerful representational capabilities has led to increased interest in end-to-end video compression. Unfortunately, existing methods are primarily designed for general videos, such as natural scenes or screen content, using conventional bandwidth. In scenarios like face video communication under low bandwidth conditions, these methods remain insufficient, severely impacting image quality and smoothness.

[0004] Considering the strong face priors in videos, model-based face video compression methods have been proposed, but they are limited by application and synthesis techniques. The emergence of deep generative models enables the generation of high-quality images from limited information. In recent years, large-scale generative face video compression (GFVC) has been proposed, most of which divide the input face video into intra-frame and post-frame representations. The intra-frame representations represent appearance information and are typically compressed by existing codecs such as H.266 / VVC. Existing works mainly employ various compact representations based on optical flow as motion, and although they have achieved good results, we still find that optical flow has certain limitations and drawbacks. First, while optical flow is powerful and flexible enough to represent any motion on a plane, it also poses a challenge for accurate estimation of 256×256 images. This is because for 256×256 images, the output dimension of the internal representation can be reduced to 4×4, and estimating such dense optical flow from low-dimensional data is an ill-posed problem. Secondly, optical flow has limitations in representing large motions because in videos of speaking faces, most areas involve slow motion, while large motions are rare, making it difficult for deep models to learn good weights. Thirdly, during decoding, it was found that the background moves with the head when it rotates, meaning that the learned optical flow cannot decouple the foreground from the background. Furthermore, there are also noticeable deformations within the head itself. Summary of the Invention

[0005] The purpose of this invention is to provide a face video decoding method, apparatus, communication method, and system to address all or part of the problems mentioned above, thereby solving the problems of face distortion and poor decoupling between foreground and background in face video communication.

[0006] The technical solution adopted in this invention is as follows:

[0007] A face video decoding method based on affine transformation, comprising:

[0008] The received data stream is feature aligned to obtain the decoded intra-frame image, as well as the aligned intra-frame features and inter-frame features; the data stream contains the encoded intra-frame image and features extracted from the inter-frame image.

[0009] Extract intra-frame appearance features from the decoded intra-frame image;

[0010] The affine parameters are estimated based on the aligned intra-frame and inter-frame features;

[0011] An affine transformation is performed on the intra-frame appearance features based on the affine parameters, and image reconstruction is performed on the result of the affine transformation based on the affine parameters.

[0012] Furthermore, the affine parameters include coarse affine parameters and fine affine parameters;

[0013] Performing an affine transformation on the intra-frame appearance features includes: a coarse affine transformation on the intra-frame appearance features based on the coarse affine parameters, and a fine affine transformation on the result of the coarse affine transformation based on the fine affine parameters.

[0014] Furthermore, image reconstruction is performed on the image features after affine transformation based on the affine parameters, including:

[0015] Image reconstruction is performed on the result of the fine affine transformation based on the fine affine parameters.

[0016] Furthermore, the method for obtaining the coarse affine parameters includes:

[0017] After concatenating the aligned intra-frame and inter-frame features, the coarse affine parameters are estimated using a parameter estimation network.

[0018] The method for obtaining the fine affine parameters includes:

[0019] The fine affine parameters are estimated using the parameter estimation network by combining the aligned intra-frame and inter-frame features, as well as the residual features between the intra-frame and inter-frame features.

[0020] Furthermore, the parameter estimation network is composed of multiple downsampling modules connected sequentially.

[0021] Furthermore, the feature alignment of the received data stream includes:

[0022] Encoded intra-frame images and features extracted from inter-frame images are extracted from the data stream, respectively.

[0023] Decoding the encoded intra-frame image yields the decoded intra-frame image, and decoding the encoded features extracted from the inter-frame image yields the decoded inter-frame features.

[0024] Extract intra-frame features from the decoded intra-frame image;

[0025] Align the extracted intra-frame features with the decoded inter-frame features.

[0026] Furthermore, aligning the extracted intra-frame features with the decoded inter-frame features includes:

[0027] The extracted intra-frame features are concatenated with the decoded inter-frame features.

[0028] The present invention also provides a face video decoding device based on affine transformation. The decoding device includes a processor and a storage medium. The processor runs a computer program in the storage medium to execute the above-described decoding method.

[0029] The present invention also provides a face video communication method, which includes:

[0030] The data stream is obtained by encoding the intra-frame image and the features extracted from the inter-frame image;

[0031] The data stream is received, and the data stream is decoded using the decoding method described above.

[0032] The present invention also provides a face video communication system, which includes an encoding device and the above-mentioned decoding device; the encoding device encodes intra-frame images and features extracted from inter-frame images to obtain a data stream.

[0033] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0034] This invention proposes a generative face video encoding / decoding and communication scheme that differs from previous schemes that use optical flow to model motion information. Instead, it introduces affine transformation to model large motions, alleviating motion estimation problems under low bandwidth. Since the affine transformation is designed on the decoding device, this framework is parameterless and does not introduce any overhead. Furthermore, a feature alignment step is introduced to map the decoded intra-frame image and the compact inter-frame representation to the same domain, obtaining better affine parameter estimation and improving the face reconstruction effect. In addition, this invention proposes a coarse-to-fine affine transformation to effectively capture large head movements while preserving detailed motion details, resulting in excellent face modeling accuracy and decoupling from the background. Attached Figure Description

[0035] The present invention will be described by way of example and with reference to the accompanying drawings, wherein:

[0036] Figure 1 This is a data flow diagram of a face video communication method in one embodiment.

[0037] Figure 2 This is a data flow diagram of an embodiment of coarse and fine affine parameter estimation.

[0038] Figure 3 This is a data flow diagram of one embodiment of image reconstruction. Detailed Implementation

[0039] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0040] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0041] Current deep generative model-based face video compression frameworks all use optical flow as an intermediate representation. However, due to the high flexibility of optical flow, distortion can easily occur when modeling faces, such as head deformation. Secondly, for large movements, it is difficult to learn complex representations using dense optical flow, thus failing to effectively decouple the foreground and background.

[0042] Embodiments of this application provide a face video communication system, such as... Figure 1 As shown, it mainly consists of an encoding unit and a decoding unit. In the encoding unit, the input face is first divided into intra-frame and inter-frame frames. Intra-frame compression is then performed by the image encoding unit. For subsequent inter-frame frames, a compact extractor and compression module are used to represent and compress the inter-frames. In the decoding unit, once the data stream is received, feature alignment is first established to obtain the decoded intra-frame features and aligned intra-frame and inter-frame features. Then, intra-frame appearance features are extracted by the appearance encoding unit. The aligned intra-frame and inter-frame features, along with the appearance features, are then input into the proposed coarse-to-fine affine transformation module to estimate the affine parameters, obtaining finer, aligned output features. Finally, the image generation module generates feature reconstruction from these features.

[0043] Based on the inventive concept of this application, the implementation methods of the relevant design details are described below.

[0044] I. Encoding Methods.

[0045] The encoding method is responsible for encoding the intra-frame image and the features extracted from the inter-frame image to obtain the data stream. This specifically includes encoding the intra-frame image and encoding the features extracted from the inter-frame image.

[0046] 1) Intraframe image encoding.

[0047] First, intra-frames are compressed using an image encoder. In some embodiments, the H.266 / VVC Reference Encoding Apparatus (JEM) is used, which boasts state-of-the-art coding capabilities among conventional encoding devices. In the JEM, quantization coefficients control both quality and bit consumption. This encoding apparatus is then used to perform image compression encoding on the intra-frames.

[0048] 2) Encoding of features extracted from inter-frame images.

[0049] The feature extractor compresses subsequent inter-frame images into compact features. In some embodiments, given a frame, the image is first downsampled by a scaling factor s, and then a compact feature representation is extracted using a U-Net network, resulting in features extracted from the inter-frame image. These features are then compressed into a final 4×4 feature set by a network consisting of convolutions and GDN operations. These features are then compressed into a bitstream and subsequently subjected to prediction, quantization, and entropy coding.

[0050] The bitstream of intra-frame image encoding and the bitstream of feature encoding extracted from inter-frame images are combined to obtain the data stream of the encoding stage.

[0051] Based on this decoding method, this application embodiment also provides an encoding device that encodes intra-frame images and features extracted from inter-frame images to obtain a data stream.

[0052] II. Decoding Method.

[0053] The decoding method is based on affine transformation. It includes:

[0054] The received data stream is feature aligned to obtain the decoded intra-frame image, as well as the aligned intra-frame and inter-frame features. The data stream is the encoded data stream, which contains the encoded intra-frame image and features extracted from the inter-frame image.

[0055] Extract intra-frame appearance features from the decoded intra-frame image.

[0056] Affine parameters are estimated based on aligned intra-frame and inter-frame features.

[0057] Affine transformation is performed on the intra-frame appearance features based on affine parameters, and image reconstruction is performed on the result of the affine transformation based on the affine parameters.

[0058] The decoding method involves feature alignment, affine transformation, appearance feature extraction, and image reconstruction. Feature alignment is initially used to decode intra-frames and align intra-frame and inter-frame features. Since an affine transformation is established in the feature domain, an appearance encoder is defined to extract intra-frame appearance features. Using these features (i.e., intra-frame appearance features, as well as aligned intra-frame and inter-frame features), a coarse-to-fine affine transformation is established. Then, the final reconstructed image is recovered from the finer transformed features by the image generation module.

[0059] 1) Feature alignment.

[0060] For the data stream received at the decoding end, decompression is first performed to obtain the decoded intra-frame and inter-frame features. To simplify affine parameter estimation, in this embodiment, the intra-frame image in the pixel domain is aligned with the feature domain, which means that intra-frame features are extracted from the decoded intra-frame image.

[0061] In some embodiments, feature alignment of the received data stream includes the following steps:

[0062] The encoded intra-frame images and features extracted from the inter-frame images are extracted from the data stream, respectively.

[0063] Decoding the encoded intra-frame image yields the decoded intra-frame image, and decoding the encoded features extracted from the inter-frame image yields the decoded inter-frame features (i.e., inter-frame representation).

[0064] Extract intra-frame features from the decoded intra-frame image;

[0065] Align the extracted intra-frame features with the decoded inter-frame features.

[0066] like Figure 1 As shown, in some specific implementations, aligning the extracted intra-frame features with the decoded inter-frame features includes:

[0067] The extracted intra-frame features are concatenated with the decoded inter-frame features. In this way, the decoded intra-frame image and the decoded inter-frame features are aligned to the same domain (feature domain).

[0068] 2) Affine transformation.

[0069] To improve the accuracy of facial organ modeling and accurately reconstruct face videos, affine transformations were performed on the intra-frame appearance features.

[0070] In some embodiments, affine transformation involves estimating affine parameters, namely rotation parameters, translation parameters, and scaling parameters.

[0071] Affine parameters include coarse affine parameters and fine affine parameters. Correspondingly, performing an affine transformation on intra-frame appearance features includes: a coarse affine transformation based on the coarse affine parameters, and a fine affine transformation based on the result of the coarse affine transformation using the fine affine parameters. Coarse affine transformations can very effectively recover large-scale motion in face videos. However, with only a coarse affine transformation, almost all lip movement is lost during image reconstruction, resulting in poor visual quality and user experience. Analysis shows that this is because global affine transformations are limited in processing local regions. To address this issue, embodiments of this application introduce a fine affine transformation to capture more detailed motion.

[0072] Specifically, for affine parameters, in some embodiments, the method for obtaining them includes:

[0073] The methods for obtaining coarse affine parameters include:

[0074] After concatenating the aligned intra-frame and inter-frame features, the coarse affine parameters are estimated using a parameter estimation network.

[0075] Since only implicit intra-frame appearance features are available during the decoding stage, it is difficult to find their correlation with lip movements. In some embodiments of this application, a simple residual is introduced into the affine parameter estimation network to address the difference between intra-frame and inter-frame features. Therefore, for fine affine parameters, the method for obtaining them includes:

[0076] The fine affine parameters are estimated using a parameter estimation network by aligning intra-frame and inter-frame features, as well as the residual features between the intra-frame and inter-frame features.

[0077] like Figure 2 As shown, for coarse affine parameters, intra-frame and inter-frame features are concatenated, and then a parameter estimation network consisting of multiple (four in the illustrated example, but other numbers are possible) downsampling modules is used to extract coarse-grained parameters. For fine affine parameters, in addition to intra-frame and inter-frame features, the residual features between intra-frame and inter-frame features are also used as input. The concatenated intra-frame features, inter-frame features, and residual features are passed through the parameter estimation network to obtain fine affine parameters. The coarse-grained affine parameter estimation network has the same structure as the fine-grained affine parameter estimation network.

[0078] 3) Extraction of appearance features.

[0079] Affine transformation is applicable to the feature domain. The feature alignment step described earlier has already aligned the decoded intra-frame image and inter-frame representation to the feature domain. Here, tools such as appearance encoders are used to extract the intra-frame appearance features of the intra-frame image, so that affine transformation can be performed using affine parameters.

[0080] 4) Image reconstruction.

[0081] After performing an affine transformation on the intra-frame appearance features, image reconstruction is performed on the result of the fine affine transformation based on the fine affine parameters.

[0082] like Figure 3 The diagram shows the image reconstruction data flow. Fine affine parameters and fine-grained features after fine affine transformation are input into an AdaIn module, which transforms the affine motion model onto the features to enrich the detailed features. Then, an appearance decoder (corresponding to the appearance encoder that extracts intra-frame appearance features) is used to generate the final reconstructed image, completing the decoding process.

[0083] It should be noted that the networks or tools involved in each step of the encoding or decoding methods listed in the embodiments of this application, such as image compression and feature extraction involved in the encoding method, and appearance encoders, appearance decoders, parameter estimation networks, etc. involved in the decoding method, can be replaced with other designs without violating the basic design ideas and purposes of this application.

[0084] Based on the decoding method proposed in this application, embodiments of this application also provide a decoding device, which is configured as follows:

[0085] The received data stream is feature aligned to obtain the decoded intra-frame image, as well as the aligned intra-frame and inter-frame features; the data stream contains the encoded intra-frame image and features extracted from the inter-frame image.

[0086] Extract intra-frame appearance features from the decoded intra-frame image;

[0087] The affine parameters are estimated based on the aligned intra-frame and inter-frame features;

[0088] Affine transformation is performed on the intra-frame appearance features based on affine parameters, and image reconstruction is performed on the result of the affine transformation based on the affine parameters.

[0089] Furthermore, according to the face video communication system described above, the encoding device encodes intra-frame images and features extracted from inter-frame images to obtain a data stream; the decoding device decodes the data stream. This application also provides a face video communication method, which includes:

[0090] The data stream is obtained by encoding the intra-frame image and the features extracted from the inter-frame image;

[0091] Receive the data stream and decode it using the decoding method described above.

[0092] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A face video decoding method based on affine transformation, characterized in that, include: The received data stream is feature aligned to obtain the decoded intra-frame image, as well as the aligned intra-frame features and inter-frame features; the data stream contains the encoded intra-frame image and features extracted from the inter-frame image. Extract intra-frame appearance features from the decoded intra-frame image; Affine parameters are estimated based on aligned intra-frame and inter-frame features; the affine parameters include coarse affine parameters and fine affine parameters; the method for obtaining the coarse affine parameters includes: concatenating aligned intra-frame and inter-frame features, and then estimating the coarse affine parameters using a parameter estimation network; the method for obtaining the fine affine parameters includes: concatenating aligned intra-frame and inter-frame features, and the residual features between the intra-frame and inter-frame features, and then estimating the fine affine parameters using the parameter estimation network; the parameter estimation network is composed of multiple downsampling modules connected sequentially; A coarse affine transformation is performed on the intra-frame appearance features based on the coarse affine parameters, a fine affine transformation is performed on the result of the coarse affine transformation based on the fine affine parameters, and image reconstruction is performed on the result of the fine affine transformation based on the fine affine parameters.

2. The face video decoding method based on affine transformation as described in claim 1, characterized in that, The feature alignment of the received data stream includes: Encoded intra-frame images and features extracted from inter-frame images are extracted from the data stream, respectively. Decoding the encoded intra-frame image yields the decoded intra-frame image, and decoding the encoded features extracted from the inter-frame image yields the decoded inter-frame features. Extract intra-frame features from the decoded intra-frame image; Align the extracted intra-frame features with the decoded inter-frame features.

3. The face video decoding method based on affine transformation as described in claim 2, characterized in that, Aligning the extracted intra-frame features with the decoded inter-frame features includes: The extracted intra-frame features are concatenated with the decoded inter-frame features.

4. A face video decoding device based on affine transformation, characterized in that, The decoding device includes a processor and a storage medium, wherein the processor runs a computer program in the storage medium to perform the decoding method as described in any one of claims 1-3.

5. A face video communication method, characterized in that, include: The data stream is obtained by encoding the intra-frame image and the features extracted from the inter-frame image; Receive the data stream and decode the data stream using the decoding method as described in any one of claims 1-3.

6. A facial video communication system, characterized in that, It includes an encoding device and a decoding device as described in claim 4; the encoding device encodes intra-frame images and features extracted from inter-frame images to obtain a data stream.

Images