Facial animation synthesis method, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a facial animation synthesis method and electronic equipment, and relates to the technical field of virtual digital images, and the method comprises the steps: obtaining an input voice to be subjected to animation synthesis and a face source image; segmenting the input voice into corresponding voice unit sequences; extracting a facial action feature sequence corresponding to the voice unit sequence; based on a vector quantizer, compressing and converting the facial action feature sequence into a corresponding action discrete token set; and processing the set of action discrete tokens and the face source image based on an image renderer to generate a corresponding facial animation. Therefore, the facial action feature sequence is compressed into the discrete token set through the vector quantizer, a complex continuous representation model is avoided, more efficient facial animation synthesis is achieved, the requirement for computing resources is reduced, and meanwhile the problem of identity leakage is also avoided.

Description

Technical Field

[0001] The present application relates to the technical field of virtual digital images, and in particular to a facial animation synthesis method, a storage medium and an electronic device. Background Art

[0002] Virtual digital image technology has broad application prospects in virtual reality, computer games, human-computer interaction and other fields, and the generation technology of virtual digital faces is one of the key links. In speech-driven facial animation generation technology, the main focus is on the synchronization of speech and lip movement, so as to generate more realistic lip animation.

[0003] At present, facial animation is usually generated by using techniques based on explicit structural representation, specifically by using Blendshapes, 3D Morphable Model (3DMM) or facial landmarks to separate facial movements and texture information. Specifically, Blend shapes is a deformation technology commonly used in 3D modeling software. It can generate intermediate transition states by mixing multiple target shapes and is often used to make facial expression animations. 3D morphable model (3DMM) is a parameterized statistical model of the face. The 3D face model corresponding to the input face image can be reconstructed through shape and texture parameters. Facial landmarks are a set of key points marked on the face image, usually including the contour points of facial organs such as eyebrows, eyes, nose, mouth, etc., which can be used to describe the shape and expression characteristics of the face.

[0004] However, these solutions all use explicit action representation, which lacks the ability to capture subtle facial dynamics and is difficult to generate non-speech related facial movements such as blinking and expression changes. In addition, the dimensions of the action representation of the explicit structure are limited and cannot fully describe the complexity of facial dynamics, making it difficult to cope with the differences between different identities, resulting in generally single generation effects. The facial motion representation used by these solutions is too limited and simplistic. It generally only focuses on lip synchronization or uses parametric models to approximate facial motion, which cannot fully depict the rich diversity of facial dynamics.

[0005] In addition, some experts and scholars have also envisioned that the refined structure based on self-supervision attempts to build a more refined facial representation through self-supervised learning, which is usually learned by transferring representations between two facial images in the video to capture more subtle facial movements. However, this method often implicitly encodes the target identity information, leading to identity leakage problems. Of course, there are also related optimization solutions to deal with identity leakage problems, but they will still increase model complexity and computational overhead. Secondly, these methods usually require a large amount of training data to obtain satisfactory results, which may be a challenge for some languages ​​with fewer resources. In addition, these continuous representation methods have limited generalization capabilities on unknown languages ​​and are difficult to meet the needs of cross-language scenarios. For example, the current main training data sets are concentrated on data sets based on Indo-European languages ​​(such as VoxCeleb and HDTF), which leads to limited generalization capabilities on non-Indo-European languages. The nature of continuous representation also makes it difficult for the model to effectively capture the limited but unique speech-visual associations in different language families, limiting its adaptability in cross-language scenarios.

[0006] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention

[0007] The present application provides a facial animation synthesis method storage medium and an electronic device, which are used to at least solve the problem of identity leakage or inability to support cross-language scenarios during the synthesis of facial animation in the current related technology.

[0008] In a first aspect, an embodiment of the present application provides a facial animation synthesis method, comprising: obtaining input speech and a face source image to be animated; segmenting the input speech into a corresponding speech unit sequence; extracting a facial action feature sequence corresponding to the speech unit sequence; based on a vector quantizer, compressing and converting the facial action feature sequence into a corresponding action discrete token set; and processing the action discrete token set and the face source image based on an image renderer to generate a corresponding facial animation.

[0009] In a second aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the facial animation synthesis method of any embodiment of the present application.

[0010] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the facial animation synthesis method of any embodiment of the present application are implemented.

[0011] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the facial animation synthesis method of any embodiment of the present application.

[0012] The beneficial effects of the embodiments of the present application are:

[0013] Through the embodiments of the present application, based on the linguistic principle that human speech is composed of finite phonemes and corresponding visemes, the discretized speech units are mapped to discretized facial action tokens. The discretized action representation can effectively capture the unique speech-visual associations in different language families, effectively improve the generalization ability in multilingual scenarios, and enhance cross-language adaptability. In addition, through the discretization of facial action features, the generated facial animation no longer depends on an explicit parameterized model, thereby reducing the impact of differences between different identities on the animation generation effect. The discretized action representation can better adapt to different facial features, thereby improving the generalization ability across identities. In addition, unlike some refined structures based on self-supervised learning, in the embodiments of the present application, the facial action feature sequence is compressed into a discrete token set through a vector quantizer, avoiding a complex continuous representation model, achieving more efficient facial animation synthesis, reducing the demand for computing resources, and also avoiding the problem of identity leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 A schematic diagram showing an effect of converting audio tags into facial action tags;

[0016] Figure 2 A flowchart showing an example of a facial animation synthesis method according to an embodiment of the present application is shown;

[0017] Figure 3 An operation flow chart of an example of compressing and converting a facial action feature sequence into a set of action discrete tokens according to an embodiment of the present application is shown;

[0018] Figure 4 A schematic diagram of the architecture of an example of a VQTalker system according to an embodiment of the present application is shown;

[0019] Figure 5 A schematic structural diagram of an example of a codebook interleaving mode according to an embodiment of the present application is shown;

[0020] Figure 6 An operation flow chart showing an example of extracting a facial action feature sequence of a speech unit sequence according to an embodiment of the present application is shown;

[0021] Figure 7 An operation flow chart of an example of a facial image reconstruction method according to an embodiment of the present application is shown;

[0022] Figure 8 A schematic diagram showing the effect of an example of a method for generating a talking avatar based on speech drive in a non-Indo-European language according to an embodiment of the present application is shown;

[0023] Fig. 9 A schematic diagram showing the effect of an example of an ablation study on eye gaze and mouth shape modeling according to an embodiment of the present application is shown;

[0024] Fig.10 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be noted that audio-driven talking head generation aims to create realistic facial animations synchronized with the input audio, and is widely used in film dubbing, animation production, and virtual assistant development. The core challenge is to achieve accurate lip synchronization, which is particularly critical for the McGurk effect. This effect demonstrates the complex interaction between auditory and visual cues in human speech perception. Although current audio-driven talking head generation methods have made significant progress, existing algorithms still face synchronization problems, such as misalignment between audio and visual cues, or the lack of specific lip shapes corresponding to certain phonemes. These problems are particularly prominent when dealing with languages ​​outside the Indo-European language family.

[0027] This limitation mainly stems from two key factors. Commonly used training datasets, such as VoxCeleb and HDTF (High-definition Talking Face), mainly consist of Indo-European languages. Therefore, models trained on these datasets perform well within Indo-European languages, but often fail to accurately capture lip movements and facial expressions associated with other language groups. Second, existing methods rely on continuous representations, which further exacerbates the problem. Current methods typically operate in continuous spaces, such as the acoustic feature input space and the latent space of facial action output. However, different language families have a limited number of unique phonetic structures and visual articulation movements. Continuous representations allow for unlimited variations, which can increase the difficulty of training and lead to overfitting to specific languages.

[0028] Although some approaches focus on increasing the amount of training data for other languages, this strategy is resource-intensive and particularly challenging for a small number of languages. In addition, simply increasing the amount of data may not fully address the need for models to generalize across diverse language structures and speech systems.

[0029] It should be noted that, based on the principles of linguistics, human speech is composed of a limited number of phonemes (unique sound units) and their corresponding pronunciation actions (the mouth shapes of these sounds). In addition, the mapping of these phonemes-pronunciation actions often has commonalities between different languages. Based on this insight, VQTalker is proposed in an embodiment of the present application, which maps discrete speech units to discrete facial action markers (or tokens). Each speech unit logically represents a phoneme, and each facial action marker corresponds to a pronunciation action. This discrete representation enables the model to capture the universal pattern of the correlation between speech and facial actions, providing a solid foundation for multilingual or cross-lingual speaking avatar generation. Figure 1 FIG. 4 is a schematic diagram showing an example of converting audio tags into facial action tags. Figure 1 As shown, VQTalker converts audio markers into facial action markers, enabling the source portrait to display diverse discrete mouth shapes. The red text in the figure represents speech sounds, and the corresponding pronunciation action categories are displayed in grayscale text.

[0030] In view of this, Figure 2 A flowchart of an example of a facial animation synthesis method according to an embodiment of the present application is shown.

[0031] Regarding the executor of the method of the embodiment of the present application, it can be any controller or processor with computing or processing capabilities. By introducing a vector quantizer, the facial action feature sequence is discretized, breaking through the limitations of the traditional explicit structure representation method, significantly improving the dynamic expressiveness of facial animation, and the adaptability across identities and languages, reducing the model complexity and computing overhead, while improving the flexibility and diversity of animation generation.

[0032] In some examples, it may be integrated and configured in an electronic device or terminal by means of software, hardware, or a combination of software and hardware, and the type of the terminal or electronic device may be diverse, such as a mobile phone, a tablet computer, or a desktop computer, etc.

[0033] like Figure 2 As shown, in step S210, the input speech and face source image to be animated are obtained.

[0034] In some embodiments, the voice signal can be collected by a microphone device or read from an existing audio file, which contains the speech content to be generated. The face source image is the initial face image used for animation synthesis, which can be a photo uploaded by the user or a frame extracted from a video. The source image provides the appearance information of the target person. More preferably, the image can also be pre-processed, such as face detection and alignment, to ensure that the position and direction of the face in the image are consistent, and filtering, illumination correction and other technologies can be applied to optimize the image quality, thereby improving the accuracy of subsequent processing.

[0035] In step S220, the input speech is segmented into corresponding speech unit sequences.

[0036] Here, the speech unit can be a phoneme, a syllable, or a higher-level language unit, such as a word or a phrase. Exemplarily, the segmentation process is usually completed by a speech recognition algorithm, using an existing speech model for segmentation, and assigning a corresponding phoneme label to each speech segment in combination with the timing information of the speech.

[0037] In some implementations, a pre-trained CosyVoice speech segmenter is used to convert input speech into discrete speech units, and a continuous speech signal is converted into a series of discrete phonemes or speech features.

[0038] In step S230, a facial action feature sequence corresponding to the speech unit sequence is extracted.

[0039] In some embodiments, various non-limiting pre-trained speech-to-facial action feature models can be used to input each speech unit and convert it into corresponding facial action features. Here, the model can be trained by a large-scale speech and facial expression pairing data set to capture the complex association between phonemes and facial movements. In addition, facial action features should be used to describe various facial movement features related to pronunciation, such as the opening and closing of the lips, the movement of the corners of the mouth, the lifting of the cheekbones, etc., as well as other facial muscle movements related to pronunciation.

[0040] In step S240, based on a vector quantizer, the facial action feature sequence is compressed and converted into a corresponding action discrete token set.

[0041] It should be noted that the role of the vector quantizer is to map continuous high-dimensional facial action feature vectors to limited discrete labels through code table quantization query, thereby simplifying and encoding complex facial actions and avoiding the problem of identity leakage. Here, the type of vector quantizer can be diverse, for example, the vector quantizer is selected from any one of the following: single codebook vector quantizer, group vector quantizer, residual vector quantizer, group residual vector quantizer or group residual finite scalar quantizer. The selection and comparison of quantizers will be expanded in conjunction with other examples below, so I will not go into details here.

[0042] In step S250, the image-based renderer processes the action discrete token set and the face source image to generate corresponding facial animation.

[0043] In some embodiments, the image renderer receives the action discrete token set and the source face image, and generates corresponding facial animation based on the discrete token set, and generates continuous facial expression changes by simulating and driving facial muscles on the face image. Here, the image renderer can use various non-limiting renderers, such as a neural network-based renderer or a generative adversarial network, etc., to map the action token set to a high-quality facial animation image, which can process subtle expression changes and generate natural and realistic animation effects.

[0044] In some embodiments, the image renderer uses an encoder-decoder structure to restore detail information through skip connections, ensuring that the generated image not only maintains the identity characteristics of the source person, but also accurately shows the corresponding lip shape and expression. The final output is a talking head video synchronized with the input speech, which maintains the identity characteristics of the source image and accurately shows the lip shape and expression changes that match the speech content.

[0045] Through the embodiments of the present application, based on the linguistic principle that human speech is composed of finite phonemes and corresponding visemes, the discretized speech units are mapped to discretized facial action tokens. The discretized action representation can effectively capture the unique speech-visual associations in different language families, effectively improve the generalization ability in multilingual scenarios, and enhance cross-language adaptability. In addition, through the discretization of facial action features, the generated facial animation no longer depends on an explicit parameterized model, thereby reducing the impact of differences between different identities on the animation generation effect. The discretized action representation can better adapt to different facial features, thereby improving the generalization ability across identities. In addition, unlike some refined structures based on self-supervised learning, in the embodiments of the present application, the facial action feature sequence is compressed into a discrete token set through a vector quantizer, avoiding a complex continuous representation model, achieving more efficient facial animation synthesis, reducing the demand for computing resources, and also avoiding the problem of identity leakage.

[0046] Through the embodiments of the present application, linguistic principles (a limited set of phonemes and visemes) are cleverly combined with computer vision and deep learning technology, and a new discrete representation method is proposed to realize business applications based on interdisciplinary thinking.

[0047] For the vector quantizer in step S240, it captures a limited set of mouth shapes of facial movements related to speech sounds. However, directly modeling facial movements at high resolution faces significant challenges. In order to solve this problem, the embodiment of the present application adopts a multi-strategy quantization method based on group residual finite scalar quantization (GRFSQ). This combined method takes advantage of different strategies: group quantization reduces the codebook size, residual quantization allows modeling of movements in a coarse to fine manner at different granularities, and finite scalar quantization (FSQ) does not require a codebook and minimizes identity leakage. Ultimately, facial movements with a resolution of 512×512 can be modeled at a low bit rate of about 11kbps.

[0048] In the process of practicing this application, the inventors conducted ablation experiments to compare various types of vector quantizers. Vector quantization (VQ) maps continuous input vectors to a limited set of discrete vectors for data compression or improving the efficiency of neural network models. Traditional VQ uses only one codebook, which is prone to mode collapse. Subsequently, group vector quantization (GVQ) uses multiple groups to model different modes, reducing the occurrence of mode collapse in a single codebook. Residual vector quantization (RVQ) is another extension of VQ, which iteratively quantizes vectors and encodes residual errors in each step to more accurately and efficiently represent high-dimensional data. Compared with the original VQ, RVQ can avoid exponential complexity growth. In addition, RVQ is particularly suitable for features with different granularity representations. For example, in audio processing, shallow semantic features and deep acoustic features can be modeled at different granularities in different residual layers. In terms of codebook learning, compared with the original codebook optimization, limited scalar quantization (FSQ) optimizes the table lookup process and integrates the straight-through estimator (STE) technology into the network. It projects the continuous representation of the neural network into a few dimensions, and then quantizes each dimension into a finite set of values. Thus, good performance and better codebook utilization are achieved without complex auxiliary losses. In the embodiment of the present application, the advantages of the group, residual and FSQ methods are combined to achieve the discretization of facial actions.

[0049] For the single codebook vector quantizer, this version only uses a single vector quantization (VQ) strategy, which has the advantages of simple implementation, fast training speed, and can still capture basic facial movements at low bit rates. However, its disadvantage is that the motion stability index (MSI) is low, at 0.786, indicating that the generated facial movements have obvious jitters and it is difficult to capture subtle changes in facial expressions, especially at high resolution (512x512).

[0050] For the group vector quantizer, it is a version that only uses Group VQ. This version introduces a group quantization strategy, but does not use residual quantization and FSQ. The ablation experiment results show that it has better expressiveness than a single VQ and can reduce the mode collapse problem to a certain extent. However, its drawback is that the MSI index is very low, only 0.062, indicating that the generated actions are extremely unstable, and the CSIM (cosine similarity) is low, at 0.548, indicating that the identity preservation ability is insufficient.

[0051] For the residual vector quantizer, it belongs to the version that only uses Residual VQ. This version introduces residual quantization, but does not use grouping strategy and FSQ. Its advantage is that it has better reconstruction quality than single VQ and Group VQ, and can capture facial action information of different scales. But its disadvantage is that the MSI index is 0.566, which is better than Group VQ, but still not stable enough. LPIPS (learned perceptual image block similarity) is 0.333, indicating that the generation quality needs to be improved.

[0052] For the group residual vector quantizer, this version combines Group VQ and Residual VQ, but has not yet introduced FSQ. Its advantages are that it improves expressiveness while maintaining a low bit rate. In addition, the MSI index is increased to 0.552, and the motion stability is improved. However, its disadvantages are that the LPIPS is 0.342, and there is still room for improvement in image quality. In addition, the codebook utilization rate is not high, and there are underutilized codewords.

[0053] The final version, GRFSQ, combines the advantages of Group VQ, Residual VQ and FSQ. Its advantages are: the MSI index is greatly improved to 2.015, which significantly improves motion stability; LPIPS is reduced to 0.340, which improves image quality; the low bit rate of about 11kbps is still maintained at a resolution of 512x512; CSIM is increased to 0.577, which improves identity preservation.

[0054] Through ablation experiments on different beta versions, the design of VQTalker was gradually optimized. Each step of improvement targets specific issues such as motion stability, image quality, identity preservation, etc. The final version of GRFSQ successfully balances these factors, achieving high-quality and stable facial animation generation while maintaining a low bitrate, and deeply understanding the impact of different quantization strategies on facial animation generation. The specific details of the ablation study will be expanded in other sections below.

[0055] Figure 3 An operational flow chart showing an example of compressing and converting a facial action feature sequence into a discrete action token set according to an embodiment of the present application is shown. Figure 4 A schematic diagram of the architecture of an example of a VQTalker system according to an embodiment of the present application is shown.

[0056] like Figure 3As shown, in step S310, the facial action feature sequence is divided into multiple feature sequence groups, and quantization processing is performed in parallel on each feature sequence group to determine the corresponding action codebook index, and the quantization processing includes multiple iterations of residual vector quantization, wherein finite scalar quantization is applied to the residual in each residual vector quantization.

[0057] In step S320, the action discrete token set is determined according to the action codebook index corresponding to each feature sequence group.

[0058] In the embodiment of the present application, the quantizer is designed as group residual finite scalar quantization (GRFSQ), which combines multiple strategies, including group vector quantization (Group VQ), residual vector quantization (Residual VQ) and finite scalar quantization (FSQ). This method combines the advantages of three quantization techniques: group vector quantization divides the input into multiple groups for parallel processing, residual vector quantization improves accuracy by iteratively quantizing residual errors, and finite scalar quantization efficiently quantizes scalar values ​​at a finite level. Figure 4 As shown in (a) of , in GRFSQ, the input is first divided into G groups. Each group undergoes R residual quantization, where FSQ is applied to the residual at each step. This process generates G×R groups of codebook indices, and the quantized outputs of all groups are concatenated to form the final quantized vector. GRFSQ aims to strike a balance between quantization accuracy and computational efficiency.

[0059] Here, GRFSQ combines the advantages of Group VQ, Residual VQ and FSQ, and can effectively represent high-resolution (e.g., 512×512) facial actions at low bit rate, solving the problems of coarse representation granularity and identity information leakage.

[0060] In some examples of the embodiments of the present application, the codebook adopts a codebook interleaving mode. Figure 5 A structural schematic diagram of an example of a codebook interleaving mode according to an embodiment of the present application is shown.

[0061] Specifically, the codebook uses a codebook that interleaves the time dimension and the action granularity dimension, so that the feature data can be quantized alternately in the time dimension and the action granularity dimension. Figure 5 As shown, in the horizontal direction (horizontal), the generated sequence of each time point is displayed, indicating the facial action data generated at each time point. In the vertical direction (longitudinal), the residual codebook corresponding to the action granularity is displayed. Thus, the codebook quantization is performed alternately in the time dimension and the action granularity dimension to ensure that the generated facial action is both consistent in time and gradually refined in details.

[0062] In some examples of the embodiments of the present application, the codebook quantization for the time dimension adopts a non-autoregressive method, indicating that the action sequence at each time point is generated simultaneously and has no dependencies between each other, allowing the model to simultaneously consider the bidirectional time context (i.e., the previous and subsequent time points), thereby speeding up the generation speed and ensuring the temporal consistency between actions. In addition, the codebook quantization for the action granularity dimension adopts a multi-level progressive iterative autoregressive method, from r 1 to r 4 Each layer (r 1 ,r 2 ,r 3 ,r 4 ) represents a level of refinement in the generation process, that is, gradually from a coarse representation (r 1 ) is refined to a finer representation (r 4 ). In this vertical direction, the generation process is autoregressive, which means that the generation of each layer depends on the result of the previous layer, so that the model can gradually add details and ultimately generate more accurate and detailed facial movements. Through the embodiment of the present application, a hybrid facial action matrix is ​​generated by combining non-autoregressive and autoregressive techniques, using non-autoregressive generation in time and autoregressive generation in fineness, balancing temporal consistency and multi-granularity feature capture.

[0063] Through the embodiments of the present application, the facial action quantizer based on GRFSQ converts continuous facial features into discrete tokens, effectively creating an information bottleneck, which can better capture subtle lip movements while avoiding encoding too much identity information. Since GRFSQ combines the advantages of GroupVQ, Residual VQ and FSQ, efficient facial action coding is achieved, which can effectively represent high-resolution (e.g., 512×512) facial actions while maintaining a low bit rate. In addition, by adopting a hybrid facial action matrix generated by combining non-autoregressive and autoregressive techniques, non-autoregressive generation is used in time, and autoregressive generation is used in fineness, which balances temporal consistency and multi-granularity feature capture.

[0064] Figure 6 An operational flowchart of an example of extracting a facial action feature sequence from a speech unit sequence according to an embodiment of the present application is shown.

[0065] like Figure 6 As shown, in step S610, corresponding pronunciation action control information is added to each speech unit in the speech unit sequence to generate a corresponding speech token sequence.

[0066] Here, the pronunciation action control information includes at least one of the following: lip control information, blink control information, head posture control information, and sight direction control information.

[0067] In some embodiments, the control information is frame-level features extracted by a specialized pre-trained feature extractor, and these features are then connected along the time axis to form a composite input vector aligned with the speech unit sequence.

[0068] In step S620, each speech token in the speech token sequence is converted into a corresponding facial action feature through multiple layers of residual iteration based on the BERT model to generate a facial action feature sequence.

[0069] Here, the BERT model is used to process the generated speech token sequence, and the input speech token sequence is encoded and converted through a multi-layer residual network structure. As a result, the multi-layer residual iteration of the BERT model can effectively capture the timing information and contextual associations contained in the speech tokens, and gradually convert the speech tokens into facial action features. In this way, each speech token is processed through the multi-layer iteration of the BERT model to generate corresponding facial action features, which can describe subtle facial movements including mouth shape changes, blinking, head posture, etc.

[0070] In one embodiment, an 8-layer BERT network is used to process discretized speech units, map speech features to facial action features, and adopt a coarse-to-fine generation process to iteratively generate facial action tokens through multiple residual layers, allowing the model to capture facial action information of different granularities.

[0071] By attaching pronunciation action control information to speech units and performing multi-layer residual iterative processing based on the BERT model, efficient conversion from speech to facial action features is achieved, and the realism of the animation is ensured through rich facial control information.

[0072] In some extended business application scenarios of the embodiments of the present application, VQTalker not only has good performance in voice-driven scenarios, but is also competitive in video reconstruction tasks.

[0073] Figure 7 An operational flowchart of an example of a facial image reconstruction method according to an embodiment of the present application is shown.

[0074] like Figure 7 As shown, in step S710, a face target image is acquired, and first facial action features and second facial action features corresponding to the face source image and the face target image are extracted respectively.

[0075] In step S720, facial action feature difference information between the first facial action feature and the second facial action feature is determined, and the facial action feature difference information is converted into target action discrete tokens based on a vector quantizer.

[0076] In step S730, the target action discrete token and the face source image are input into an image renderer to reconstruct the face target image.

[0077] By obtaining the facial action features of the source image and the target image, calculating the difference information, and then using the vector quantizer to discretize the difference information, the image renderer is finally used to reconstruct the target face image. In this way, the dynamic changes of the face can be accurately captured, and the discretized action tokens can be used to achieve efficient generation of target face images, avoiding the problem of identity leakage, and ensuring that the generated image can reach a high level in terms of visual effects and action synchronization.

[0078] Through the embodiment of the present application, VQTalker can generate smooth and natural talking head videos at a high resolution of 512x512, while maintaining a low bit rate (about 11kbps), supporting high-quality multilingual speaking face generation. In addition, through fine facial action quantization, the synchronization of lip shape and speech is significantly improved, especially in non-Indo-European languages, lip synchronization is effectively improved. In addition, identity information leakage is effectively reduced, and the identity characteristics of the source image are maintained. Further, compared with other methods (such as methods based on diffusion models), VQTalker has higher computational efficiency.

[0079] Through the application of the embodiment of the present application, cross-language communication innovation can be effectively realized, the localization of global video content can be promoted, the dubbing and localization costs can be greatly reduced, more high-quality audio-visual content can be created for small languages, cultural diversity can be promoted, and the accessibility of international education resources can be improved, especially in the field of online education. In addition, a more natural lip reading aid is provided for the hearing-impaired, the ability to obtain information is improved, the development of a sign language translation system is supported, and sign language movements are converted into oral expressions. In addition, it helps the innovation of human-computer interaction, improves the expressiveness of virtual assistants and digital people, makes human-computer interaction more natural and attractive, and gives more realistic expressions and lip shapes to the characters in the metaverse and virtual reality environment. In addition, it can also help the transformation of the film and television industry, simplify the cross-language production process of movies and TV programs, reduce internationalization costs, provide new tools for the restoration of historical figures and the creation of fictional characters, and enrich visual narrative means. In addition, it can also be effectively used in medical and mental health, realize auxiliary language therapy, provide visual feedback for patients, such as developing facial expression training tools, and help autistic patients improve their emotional expression ability, which has great promotion significance.

[0080] In an embodiment of the present application, VQTalker is proposed, which is a new framework for multilingual conversational avatar generation, designed to address the challenges of lip synchronization and natural movements in different languages. Specifically, a facial action tokenizer based on group residual finite scalar quantization (GRFSQ) is introduced, which creates a discretized representation of facial features. This method can fully capture facial movements while improving the generalization ability to multiple languages, even when training data is limited. Based on this quantized representation, a coarse-to-fine action generation process is implemented using a bidirectional attention mechanism and an autoregressive training strategy. This method not only improves the generation quality, but also improves temporal consistency and lip movement transitions.

[0081] Extensive experiments show that VQTalker achieves state-of-the-art results in both video-driven and speech-driven scenarios, especially in multilingual settings. Notably, our approach achieves high-quality results at a resolution of 512×512 pixels while maintaining a significantly lower bitrate of approximately 11 kbps.

[0082] In addition, VQTalker shows substantial improvements in structural similarity, identity preservation, and motion stability in a variety of languages, including non-Indo-European languages. Through the embodiments of the present application, new possibilities are opened up for cross-language applications such as movie dubbing, virtual assistants, and educational tools.

[0083] Based on this structure, an interleaved mode of codebook generation is introduced, combining autoregressive and non-autoregressive elements. Within each action granularity, a non-autoregressive method is used to maintain bidirectional attention and make full use of temporal correlation. Between different granularities, an autoregressive strategy is used to achieve a coarse-to-fine action generation process, achieving a balance between generation speed and complexity, ensuring the efficiency and quality of generated animations.

[0084] Through the embodiments of the present application, a novel method for discrete representation of facial movements is proposed, which utilizes the finiteness of speech pronunciation movements to enhance the model's ability to capture subtle lip movements and improves its generalization ability in different languages. In addition, a multi-strategy quantization method based on combined residual finite scalar quantization (GRSQ) is proposed, which effectively reduces the codebook size while maintaining high-quality facial movement representation. In addition, a coarse-to-fine action generation process is adopted, combined with non-autoregressive and autoregressive training, which improves temporal consistency and simplifies generation complexity.

[0085] It should be noted that efficient representation of talking head videos has always been a research focus. Unlike general video coding methods, facial features are highly structured. Therefore, methods based on disentangled structures have become mainstream, providing better interpretability and controllability. At present, the latent representation of motion mainly relies on predefined structured features, such as landmarks or 3DMM expression parameters. However, these features are relatively coarse and difficult to capture subtle facial movements. Recent studies have attempted to learn more refined representations through self-supervised learning, usually by transferring representations between two facial images in a video. However, this approach often implicitly encodes target identity information, leading to identity leakage problems. Although various strategies have been proposed to address this issue, they usually introduce additional model complexity and computational overhead. To address these challenges and inspired by the concepts of finite phonemes and articulatory actions, it is proposed to explore discrete coding techniques in latent facial representations. Discrete coding naturally forms an information bottleneck by quantizing the continuous feature space into a finite discrete codebook. This approach forces the model to learn more compact representations more efficiently without the need for complex additional loss functions or model components.

[0086] Image VQ methods in the current related art usually rely on VQ-VAE, patch-based tokenization, which limits quantization to local areas and fails to capture global semantics, thus limiting their application at low resolutions (e.g., 256×256). Extending these methods to higher resolutions (e.g., 512×512) usually requires complex networks with billions of parameters, similar to stable diffusion models. The disadvantage of this approach is that texture and motion information are coupled together. Since the texture part contains detailed information, the network will increase network parameters when reconstructing these details, making it difficult to expand to higher resolutions or requiring a larger network. In contrast, in the present application, by independently quantizing the motion and using a separate rendering network to restore the detailed parts, these elements are decoupled, which can go beyond local patch semantics to global motion semantics and achieve holistic modeling of facial dynamics without an overly large network.

[0087] The VQTalker framework proposed in the embodiment of the present application solves the challenge of generating multilingual talking heads from voice tokens to visual motion tokens. Figure 4 As shown in Figure 1, the VQTalker architecture consists of two main components: (1) a facial action tokenizer, a quantized facial action codec that learns a universal motion representation through self-supervised learning and group residual finite scalar quantization (GRFSQ), and (2) a coarse-to-fine motion generation process. The coarse-to-fine motion generation process uses the BERT model, which converts speech tokens into facial action tokens in multiple layers of residuals. Then, the image renderer synthesizes the final talking head video from these representations.

[0088] In some examples of the embodiments of the present application, a facial action tokenizer is designed specifically for the talking head generation task, which uses self-supervised learning and multi-strategy quantization methods to create discrete representations of facial actions, including lip movements, blinks, head postures, and gaze directions.

[0089] The backbone architecture of the tokenizer is encoder-quantizer-decoder. The encoder extracts three types of features: fine-grained facial features (especially high-frequency details), global facial features as facial cues (f g ) and facial action features from the source image (f s ) and the facial action features of the target image (f t ). The quantizer converts the action difference (f t -f s ) is converted into discrete tokens, acting as a strong bottleneck to retain only facial action information. The decoder (also called image renderer) reconstructs facial features from the quantized representation and compares it with the global facial features (f g ) to guide the final rendering. To recover details, each decoder layer is connected to its corresponding encoder layer via skip connections, generating deformations at each stage. This process maximizes the recovery of details in the output image and makes up for the limitations of the global motion representation alone.

[0090] In the embodiment of the present application, the quantizer is designed as group residual finite scalar quantization (GRFSQ), which combines multiple strategies, including group vector quantization (Group VQ), residual vector quantization (Residual VQ) and finite scalar quantization (FSQ). This method combines the advantages of three quantization techniques: group vector quantization divides the input into multiple groups for parallel processing, residual vector quantization improves accuracy by iteratively quantizing the residual error, and finite scalar quantization efficiently quantizes scalar values ​​at a finite level. The whole process is as follows Figure 4 As shown in (a) in . In GRFSQ, the input is first divided into G groups. Each group undergoes R residual quantization, where FSQ is applied to the residual at each step. This process generates G×R groups of codebook indices, and the quantized outputs of all groups are concatenated to form the final quantized vector. GRFSQ aims to strike a balance between quantization accuracy and computational efficiency.

[0091] The self-supervised structure design and the introduction of the quantizer enable this model to generate robust representations that can finely model facial movements. This architecture ensures that the representations generated by the quantizer can effectively capture facial movements while discarding irrelevant information. Once the above model training is completed, all model weights in subsequent stages will be fixed. The total loss is the same as LIA, and the formula is as follows:

[0092] L tokenizer =L recon +λ 1 Lpercep +λ 2 L adv , formula (1)

[0093] In the coarse-to-fine motion generation process, a quantized facial action codec is utilized and the BERT model architecture is adopted to convert discrete audio features into facial action tokens. This coarse-to-fine motion generation process bridges the gap between audio input and facial animation output.

[0094] Furthermore, the process starts with a pre-trained speech tokenizer input to generate discrete speech units. Let A = {a 1 ,a 2 ,...,a T}∈{1,2,...,M} T represents a sequence of discrete speech tokens of length T, where each token belongs to a vocabulary of M tokens. These speech tokens are augmented with additional control information: head pose Sight direction and blink data The head pose h is a 3D vector representing yaw, pitch, and roll. The gaze direction g is a 2D vector encoding yaw and pitch. The blink data b is a scalar value representing the degree of eye openness. These control features are extracted as frame-level features by a specialized pre-trained feature extractor. These features are then concatenated along the time dimension to construct a composite input vector C r =[A,h,g,b,R r-1 ], where r>0. When r=0, R r-1 Replace it with a zero matrix. The vector C r As the input of the BERT model, R r Represents the output, where r represents the result of the rth residual codebook.

[0095] like Figure 5 As shown in Figure 1, non-autoregressive and autoregressive techniques are combined in the GRFSQ generation process. The matrix shows the residual codebook r in the vertical direction. 1 to r 4 and the sequence s generated in the horizontal direction 1 to n Each cell represents the sequence position s j Upper code book r i The group quantization value of 1 to r 4 ) represents the residual codebook with gradually increasing details, column (s 1 to n ) shows the generated sequence, while the color gradient indicates the iterative refinement process across the residual layers.

[0096] In time (horizontally), the process is non-autoregressive: the sequence s 1 ,s 2 ,...,s n Generated simultaneously, allowing bidirectional context to be considered and speeding up processing. This is evident in the consistent colors within each row.

[0097] At the action granularity (vertical direction), the generation process is done through the residual layer r 1 to r 4 Each layer refines the facial action representation from the coarse r 1 To fine r 4 Details. This vertical progression involves 4 iterations, which is significantly less than the sequence length n, maintaining efficiency.

[0098] The color gradient from light blue to dark blue visualizes this iterative refinement process. Elements of the same color are generated at the same time, and the deepening of the blue indicates the evolution from r 1 to r 4 This hybrid approach balances temporal smoothness and multi-granularity feature capture, producing a model that can effectively generate temporally consistent and detailed facial animations.

[0099] The main goal of the learning process is to reconstruct the categories of the (r+1)th layer codebook, conditioned on the input from the rth layer C r The training objective is to minimize the negative log-likelihood, which can be expressed mathematically as:

[0100]

[0101] In the formula, y g,t,r+ 1 Indicates the target category of the g-th group in the (r+1)-th layer codebook at time step t. In the training phase, softmax is used for classification. In the inference phase, argmax is used to select the category with the highest probability for sampling.

[0102] The following will describe the relevant details of the experimental part of the VQTalker framework of the embodiment of the present application.

[0103] In terms of training datasets, three publicly available datasets were utilized: the VoxCeleb dataset, the HDTF dataset, and the VFHQ dataset. To ensure consistency, the original videos were re-downloaded and processed using a unified method. The method includes filtering out faces smaller than 512 pixels and applying standardized facial processing methods. This comprehensive effort produced a dataset containing approximately 160,000 video clips with a total length of 210 hours. After statistical analysis, it can be seen that Indo-European languages ​​​​(such as English, German, and French, of which English accounts for the largest proportion, about 70%) account for more than 98% in the constructed dataset.

[0104] In terms of evaluation datasets, in order to evaluate the effectiveness of non-talking head algorithms on non-Indo-European languages, a Multilingual Non-Indo-European Talking Head Evaluation Corpus (MNTE) was compiled. This dataset includes Mandarin, Korean, Japanese, Arabic, Swahili, and Turkish, and aims to cover commonly used non-Indo-European languages. For each language, 5 talking head video clips were collected, each clip was no less than 3 seconds long, for a total of 30 videos. The shortest clip length was 3.25 seconds, the average was 10.4 seconds, and the longest was 21.73 seconds.

[0105] MNTE is used to verify the results of voice-driven related algorithms. In addition, for tasks that do not involve audio input (such as video reconstruction and face reproduction, the former involves self-driving and the latter is driven by people of different identities), a part of the widely adopted HDTF dataset is used as a test set, which is a reserved part.

[0106] In terms of model configuration, the self-supervised training paradigm is mainly based on LIA when training the facial action tokenizer. In terms of infrastructure, LIA is extended to support a resolution of 512×512 pixels. In addition, the linear motion decomposition (LMD) component is removed from the original implementation.

[0107] In the second stage, the non-autoregressive (NAR) phase, an 8-layer BERT network is used to iteratively generate a four-layer residual codebook for the face tokenizer. The maximum length is 4096. CosyVoice’s pre-trained voice tokenizer serves as an audio feature encoder and contains a downsampling layer that adjusts the audio sampling rate from 50Hz to 25Hz to synchronize with the video frame rate.

[0108] In terms of training and inference hardware, in training, four A100 (40G) GPUs were used to train each stage until the loss converged. Except for calculating the perceptual loss, no pre-training parameters were introduced. The first stage first trained a model with an input size of 256×256, and then used these parameters as initialization to train a 512×512 model. The 256 stage took 72 hours, and the subsequent 512 stage took an additional 48 hours. The second stage requires longer training time, about 120 hours. Insufficient training time will result in noticeable jitter in the rendered images. In inference, a GeForce RTX 3090 (24G) GPU was used, and the process started with the speech tokenizer generating a motion sequence, which was then rendered frame by frame.

[0109] Furthermore, the VQTalker framework is evaluated in two different scenarios: speech-driven scenarios and video reconstruction scenarios. The speech-driven scenarios are designed to test the performance of the algorithm under various audio-driven conditions, evaluating its ability to generate appropriate facial expressions and movements based on different speech inputs. In these tasks, when pose information is required, it is provided from the ground truth to isolate the impact of audio on facial animation. In contrast, the video reconstruction scenario is used to evaluate the effectiveness of the face tokenizer, focusing on its ability to accurately encode and reconstruct facial features and expressions in video data.

[0110] In some embodiments, multiple metrics are used to evaluate the quality and similarity of generated images and videos. In order to quantify the structural and perceptual similarities between generated images and real images, the structural similarity index (SSIM) and the learned perceptual image patch similarity (LPIPS) are used. Facial similarity is evaluated by the cosine similarity (CSIM) of the pre-trained facial recognition model. For video-specific indicators, the motion stability index (MSI) is used, which measures the motion stability in the speaking face video by the inverse of the variance of the facial key point acceleration. Image quality is evaluated by the cumulative probability of blur detection (CPBD), which is a reference-free indicator that evaluates image clarity by analyzing the probability of blur detection for different edge widths. In addition, the landmark distance (LMD) is used as an indicator to measure the accuracy of facial feature positioning, which is defined as the normalized L2 distance between the corresponding facial landmarks in the two images. To ensure fair comparison, the output resolution of all algorithms is normalized to 256×256.

[0111] A comprehensive evaluation of the Facial Action Tokenizer against several state-of-the-art face reenactment methods is performed. This scenario tests the ability of the algorithm to accurately reconstruct the original video using a video-driven portrait of the same identity (usually the first frame of the video). All compared methods use a variant of self-supervised learning. Table 1 shows the results of this evaluation.

[0112] Table 1: Video reconstruction results on HDTF. Reso. represents the image resolution (width and height), and the bitrate is in kbps. The bold and underlined values ​​represent the best and second-best results, respectively.

[0113]

[0114] The facial action tokenizer of VQTalker provided by the embodiment of the present application achieves competitive results with significantly lower bit rates while supporting higher resolutions (512×512). Specifically, in the VQTalker framework, 12 group layers, 4 residual layers, 625 codebook entries per group, and a total bit rate of about 11kbps (calculated as 4×12×log2(625)). For other methods using continuous variables, they are converted to bits by multiplying by 32-bit floating-point numbers, and all bit rates are calculated at 25 frames per second. Compared with other methods, this low bit rate is particularly significant. For example, AniTalker uses a 20-dimensional intermediate representation, and its bit rate is 20×32×25=16000, i.e. 16kbps.

[0115] Despite the lower bitrate, VQTalker performs well on key metrics. VQTalker's facial action tokenizer achieves the second highest scores on image structure metrics such as SSIM (0.874) and LPIPS (0.083). VQTalker's CSIM score (0.919) is very close to the state-of-the-art methods, indicating strong facial identity preservation during reconstruction. Notably, VQTalker achieves the highest CPBD score (0.421), indicating that VQTalker's facial action tokenizer performs better in terms of image sharpness and clarity compared to other methods. While not achieving the highest scores on all metrics, the slight differences on some metrics can be attributed to the information loss caused by quantization and slightly less smooth representation compared to continuous methods.

[0116] In addition, quantization-based methods show strong identity and motion disentanglement capabilities without relying on explicit disentanglement modules. Unlike AniTalker’s use of mutual information or MegaPortraits’ use of cycle consistency loss, VQTalker’s facial action tokenizer achieves this through an efficient bottleneck structure. This structure features low bitrate and codebook-free design, effectively preserving important facial action information.

[0117] In summary, VQTalker’s face tokenizer strikes a balance between low bitrate, high resolution, and various image quality, sharpness, and identity preservation metrics, highlighting the effectiveness of quantized facial action codecs for the challenging task of video reconstruction.

[0118] In the coarse-to-fine generation process, VQTalker is compared with existing state-of-the-art speech-driven methods, including SadTalker, EAT, PD-FGC, AniTalker, EDTalker, and EchoMimic. The quantitative results are shown in Table 2.

[0119] Table 2: Quantitative comparison with existing speech-driven methods.

[0120]

[0121] In subjective evaluation, the method provided by the embodiment of the present application consistently shows improvements in fidelity, lip synchronization accuracy, naturalness, and reduction of motion jitter, especially in enhancing the naturalness of motion. These advances can be attributed to complex universal motion representation. Objective evaluation involves using audio-driven images of images. Compared with these methods, VQTalker shows significant improvements in SSIM, CSIM, and MSI in self-driven audio scenarios. Although our performance may not be the best in the reproduction scene, the VQTalker method shows significant advantages in audio-driven tasks. In the design of VQTalker, texture and motion can be better decoupled, enabling the model to generate motion more efficiently and reduce the interference of texture information on the final result. In addition, the lip motion stability index was measured, and it was found that the lip motion generated by the audio-driven method was more stable, further highlighting the robustness and quality of the VQTalker framework in generating natural and consistent audio-driven facial animations.

[0122] Next, the details of the ablation study will be expanded.

[0123] In terms of codebook design, the importance of different codebook designs on the results was explored and various codebooks were tested, as shown in Table 3. “#Group” refers to the number of groups in group quantization, and “#Residual” indicates the number of residual quantizations. “FSQ” stands for finite scalar quantization, while “#Codewords” indicates the number of codewords in a single codebook. For FSQ, this column shows the dimension of the level. “-” indicates that this particular case is not applicable.

[0124] Table 3: Different intermediate representations under the Cross-Reenactment setting.

[0125]

[0126] As shown in Table 3, the first four rows tested the classic codebooks: VQ, GRVQ, RVQ, and GRVQ. VQ represents the basic codebook, 'G' represents the group quantization mechanism, and 'R' represents the residual quantization mechanism. It is found that the MSI index is poor, which indicates that these methods have weak modeling capabilities and insufficient quantization, resulting in considerable jitter.

[0127] In the last four rows in Table 3, GRFSQ was tested to verify the impact of the number of groups, the number of residuals, and the FSQ level. Compared with the final method (last row), the number of groups (fifth row), the number of residuals (sixth row), and the number in a single codebook (seventh row) were reduced, respectively. Thus, it is found that an optimal solution is formed by the method provided by the embodiment of the present application. As to why increasing the codebook (level parameter) in the sixth row actually leads to a decrease in performance, it can be attributed to the diminishing returns of FSQ when the codebook is expanded. It should be noted that although increasing the codebook size will initially improve performance, after a certain point, further increases will lead to more complex representations that are difficult for the model to learn effectively.

[0128] Figure 8 A schematic diagram showing the effect of an example of a speech-driven talking avatar generation method in a non-Indo-European language according to an embodiment of the present application is shown.

[0129] like Figure 8 As shown, each row shows a source portrait (far left) animated by a different algorithm for pronouncing a specific syllable (in red). The upper and lower images in each example show the transition of lip movements. This visualization highlights the ability of the VQTalker framework to model comprehensive pronunciation features and support multiple languages.

[0130] In terms of discrete and continuous representation, in order to verify the advantages of discrete values ​​over continuous values, three scenarios were tested in this application: completely continuous input, discrete input, and discrete output. For fair comparison, the Whisper Large v2 model was used to extract audio features in the continuous input scenario. Whisper supports multi-language input, which enables a fairer comparison. For the continuous output scenario, we used non-quantized features and used continuous values ​​as output.

[0131] Table 4: Discrete and continuous representation

[0132]

[0133] Fig. 9 A schematic diagram showing the effect of an example of an ablation study on eye gaze and mouth shape modeling according to an embodiment of the present application is shown.

[0134] like Fig. 9 As shown, in an ablation experiment for eye and mouth motion modeling, the results with and without specific losses for the eye and mouth regions are compared, demonstrating the improvement in facial feature control without the need for additional specialized models. Specifically, the top two rows show examples of gaze modeling. When the target's motion is used to drive the source portrait, the effect of eye loss is tested with and without the additional weight applied to the eye region.

[0135] The bottom two rows show experiments on mouth shape modeling. Similar to the eye experiments, the results with and without additional weights on the mouth region (mouth loss) are compared.

[0136] These experiments show that there is no need to introduce additional pre-trained models for gaze modeling or mouth shape prediction. Instead, the modeling of specific facial features can be improved simply by applying additional weights to specific regions. This approach enhances the overall performance while maintaining efficiency.

[0137] The results show that eye movements and mouth shapes are significantly improved when the corresponding losses are applied. In the example of gaze direction, the eyes in column (b) follow the target gaze more accurately than those in column (a). Similarly, for mouth shape, the results in column (b) more reliably reproduce the target mouth expression.

[0138] This ablation study highlights the effectiveness of region-specific weighted methods in capturing subtle facial motions without relying on complex specialized models, demonstrating that high-quality results can be achieved through targeted optimization within a unified framework.

[0139] Through the VQTalker provided in the embodiments of the present application, a new framework for multilingual talking head generation using discretized facial action representation is implemented. Based on experiments, the group residual finite scalar quantization (GRFSQ) method achieves an effective balance between low bit rate and high-quality facial action representation. Although competitive in video reconstruction tasks, VQTalker performs particularly well in speech-driven scenarios, especially in non-Indo-European languages ​​where previous methods often encounter difficulties. The coarse-to-fine generation process significantly enhances temporal consistency and natural lip movements in a multilingual context. These findings show that well-designed discretized representations can effectively capture and reproduce complex facial dynamics while maintaining efficiency. VQTalker represents an important advance in creating more inclusive and diverse talking head technology, and has good prospects for cross-cultural communication, accessibility and entertainment applications.

[0140] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0141] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of any of the above-mentioned facial animation synthesis methods of the present application.

[0142] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of any one of the above-mentioned facial animation synthesis methods.

[0143] In some embodiments, the embodiments of the present application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the facial animation synthesis method.

[0144] Fig.10 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for executing a facial animation synthesis method provided by another embodiment of the present application. Fig.10 As shown, the device includes:

[0145] One or more processors 1010 and memory 1020, Fig.10 A processor 1010 is taken as an example.

[0146] The apparatus for executing the facial animation synthesis method may further include: an input device 1030 and an output device 1040 .

[0147] The processor 1010, the memory 1020, the input device 1030 and the output device 1040 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0148] The memory 1020 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the facial animation synthesis method in the embodiment of the present application. The processor 1010 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1020, that is, the facial animation synthesis method in the above method embodiment is implemented.

[0149] The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1020 may optionally include a memory remotely arranged relative to the processor 1010, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The input device 1030 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 1040 may include a display device such as a display screen.

[0151] The one or more modules are stored in the memory 1020, and when executed by the one or more processors 1010, the facial animation synthesis method in any of the above method embodiments is executed.

[0152] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0153] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0154] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0155] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0156] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0157] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A facial animation synthesis method, comprising: Obtaining input speech and face source images to be synthesized for animation; Segmenting the input speech into corresponding speech unit sequences; Extracting a facial action feature sequence corresponding to the speech unit sequence; Based on a vector quantizer, compressing the facial action feature sequence into a corresponding action discrete token set; The image-based renderer processes the action discrete token set and the face source image to generate corresponding facial animation.

2. The method according to claim 1, wherein: The vector quantizer is selected from any one of the following: a single codebook vector quantizer, a group vector quantizer, a residual vector quantizer, a group residual vector quantizer or a group residual finite scalar quantizer.

3. The method according to claim 2, wherein: The vector quantizer is a group residual finite scalar quantizer; the facial action feature sequence is compressed and converted into a corresponding action discrete token set, comprising: Dividing the facial action feature sequence into a plurality of feature sequence groups, and performing quantization processing in parallel on each of the feature sequence groups to determine a corresponding action codebook index; the quantization processing comprises iterating a plurality of times of residual vector quantization, wherein a finite scalar quantization is applied to the residual in each residual vector quantization; The action discrete token set is determined according to the action codebook index corresponding to each of the feature sequence groups.

4. The method according to claim 3, wherein: The codebook uses a codebook that interleaves the time dimension and the action granularity dimension, so that the feature data can be codebook quantized alternately in the time dimension and the action granularity dimension.

5. The method according to claim 4, wherein: A non-autoregressive method is used for codebook quantization in the time dimension, and a multi-level progressive iterative autoregressive method is used for codebook quantization in the action granularity dimension.

6. The method according to claim 1, wherein: The step of extracting the facial action feature sequence corresponding to the speech unit sequence comprises: Adding corresponding pronunciation action control information to each speech unit in the speech unit sequence to generate a corresponding speech token sequence; the pronunciation action control information includes at least one of the following: lip control information, blink control information, head posture control information and sight direction control information; Based on the BERT model, each speech token in the speech token sequence is converted into a corresponding facial action feature through multiple layers of residual iteration to generate a facial action feature sequence.

7. The method according to claim 1, wherein: After obtaining the input speech and the source face image to be animated, the method further includes: Acquire a target face image, and extract a first facial action feature and a second facial action feature corresponding to the source face image and the target face image respectively; determining facial action feature difference information between the first facial action feature and the second facial action feature, and converting the facial action feature difference information into target action discrete tokens based on the vector quantizer; The target action discrete token and the human face source image are input into the image renderer to reconstruct the human face target image.

8. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images