Method for generating digital human through image audio

Abstract

The invention relates to a method for generating a digital human from an image audio, and the method comprises the following steps: S1, carrying out the feature coding of an input audio signal through employing a Whisper model, extracting the time sequence features in a voice, obtaining the time sequence change of pronunciation based on the time sequence features, and carrying out the feature coding of the input audio signal, the time sequence features are input into an action generation network based on a Transform architecture, in the Transform architecture, the time sequence features are processed frame by frame, and relevance of different parts in audio is identified through a self-attention mechanism, so that the time sequence features of audio signals are mapped to facial actions; s2, setting an efficient control module, wherein the efficient control module comprises a Stilling module and an eye and lip redirection module; and S3, generating a digital human based on audio-action mapping and an efficient control module. According to the invention, the effect of generating the facial animation of the digital human is improved.

Description

Technical Field

[0001] This invention relates to the field of digital human generation technology, and in particular to a method for generating digital humans from images and audio. Background Technology

[0002] Existing technologies for generating digital humans have the following shortcomings:

[0003] (1) Insufficient synchronicity and controllability

[0004] Current audio-driven portrait animation technology suffers from significant synchronization issues, particularly in matching lip movements with audio rhythm and controlling eye movements. This often results in delayed lip movements and incomplete eye closure, leading to a lack of realism in the animation.

[0005] (2) Poor style compatibility

[0006] Most existing portrait animation models are typically trained only for realistic styles, which often makes it difficult to maintain stylistic consistency when dealing with non-realistic styles (such as anime, illustration, and cartoons). Due to the limited variety of training datasets, these models are prone to texture distortion, facial feature misalignment, and style fragmentation when faced with diverse styles, resulting in animation effects that do not match the creator's expectations. With the diversification of digital creation, creators require more stylistic freedom and personalized expression when producing content. Therefore, the lack of style adaptability in existing technologies severely limits their practical value in a wide range of application scenarios.

[0007] (3) The contradiction between efficiency and cost

[0008] Although the diffusion class methods can generate high-quality animations, they require multiple denoising iterations and have a slow inference speed (typically tens of milliseconds), posing a significant challenge in real-time applications, especially in latency-sensitive scenarios such as video conferencing and virtual interaction.

[0009] (4) Lack of support for multi-character scenes

[0010] Most existing technologies are only designed for portrait animation of a single character. In scenes with multiple characters interacting, the characters' movements are prone to conflict, and the splicing between the background and the characters is unnatural, making it difficult to achieve audio-driven synchronized animation for multiple characters.

[0011] Therefore, it is necessary to provide an image and audio generation method for digital humans to improve the effect of digital human facial animation generation. Summary of the Invention

[0012] The purpose of this invention is to provide a method for generating digital humans from images and audio, so as to improve the effect of digital human facial animation generation.

[0013] To address the problems existing in the prior art, this invention provides a method for generating digital humans from images and audio, comprising the following steps:

[0014] S1: The Whisper model is used to encode the features of the input audio signal to extract the temporal features in the speech. The temporal features include rhythm features, intonation change features and emotional fluctuation features. Based on the temporal features, the temporal changes of pronunciation are obtained. The temporal features are input into the action generation network based on the Transformer architecture. In the Transformer architecture, the temporal features are processed frame by frame and the correlation between different parts of the audio is identified through the self-attention mechanism so as to map the temporal features of the audio signal onto facial movements.

[0015] S2: Set up the high-efficiency control module, which includes the Stitching module and the eye and lip repositioning module;

[0016] S3: Generates digital humans based on audio-motion mapping and a high-efficiency control module.

[0017] Optionally, in the image-audio digital human generation method, the Stitching module works as follows:

[0018] Keypoint offset calculation: The Stitching module first takes audio features and the pose of the character's position as input, and uses an MLP network to accurately calculate the offset of each driving keypoint. Each offset is used to adjust and correct the splicing effect of multiple characters in the same scene.

[0019] Natural Blending: Through precise key point adjustments, the Stitching module ensures natural transitions between different characters and between characters and the background, resulting in a more coherent and natural visual effect.

[0020] Optionally, in the image-audio digital human generation method, the eye and lip retargeting module is implemented as follows:

[0021] Set up an independent MLP network: The MLP network for the eye and lip repositioning module is trained independently and receives eye opening and lip opening / closing coefficients from audio-driven signals;

[0022] Fine-grained control: The eye and lip repositioning module allows for fine-grained control as needed;

[0023] Detailed adjustments: Through the eye and lip repositioning module, users can control every detail of facial expressions, ensuring that the movements of the eyes and lips are perfectly matched with the audio rhythm.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] This invention significantly improves the quality of facial animation generation through a series of innovative technologies, demonstrating remarkable progress in several aspects. First, synchronization and realism are greatly enhanced. By optimizing the synchronization between audio and facial movements, especially in matching lip movements with audio rhythm, this invention controls the synchronization error to within 2 milliseconds, greatly reducing the motion lag and misalignment problems common in traditional methods. Eye movements can also be automatically adjusted according to pauses, emphasis, and other contextual elements in the audio, making the animation more realistic and natural. Subjective evaluation results show that compared with existing technologies, the realism of the animation is improved by more than 35%, and the audience's immersion and interactivity are significantly enhanced. Second, this invention achieves a breakthrough in multi-style adaptability, efficiently handling portraits of different styles such as realistic, anime, and illustration. In relevant tests, the model performs excellently on the LPIPS (Perceptual Similarity Index), with a mean value below 0.07, and the texture distortion rate is reduced by 40% compared to traditional methods. This performance ensures that the generated animation stably conforms to the characteristics of the target style, greatly expanding the application scenarios of this technology in creation and providing creators with greater freedom and flexibility. Furthermore, the balance between efficiency and quality has been optimized to the extreme, with an inference speed of 12.8 milliseconds / frame, more than 3 times faster than the traditional diffusion method (50 milliseconds / frame), while also achieving better performance in objective quality indicators. The PSNR (Peak Signal-to-Noise Ratio) reached 31.56dB, and the SSIM (Structural Similarity) was 0.765, indicating that the generated animation has a visual quality far exceeding that of traditional non-diffusion methods, with higher detail and clarity. In addition, the technology performs excellently when handling multi-character scenes, stably generating coordinated and independent facial movements between multiple characters. For portrait scenes of 2-5 people, the character movement conflict rate is controlled below 5%, and the misalignment error in splicing areas such as the shoulders is less than 1 pixel, ensuring a high degree of consistency between each character's facial movements and the interaction with the background and other characters. Overall, this invention significantly improves the generation quality and practicality of portrait animation through optimizations in synchronization, style adaptation, efficiency, quality, and multi-character processing, broadens its applicability in various application scenarios, and greatly promotes technological progress in fields such as virtual interaction, creation, and entertainment. Attached Figure Description

[0026] Figure 1 A flowchart of a method for generating a digital human provided in an embodiment of the present invention. Detailed Implementation

[0027] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0028] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0029] Existing technologies have many shortcomings in generating digital humans.

[0030] To address the problems existing in the prior art, this invention provides a method for generating digital humans from images and audio, such as... Figure 1 As shown, it includes the following steps:

[0031] S1: This invention employs the Whisper model to encode features in the input audio signal, thereby extracting temporal features such as rhythm, intonation changes, and emotional fluctuations in speech. Whisper is an efficient and powerful speech recognition model that processes audio through deep neural networks. It can not only accurately identify speech content but also extract emotional, intonation, and rhythmic information from the speech. Through these audio features, we can obtain the temporal changes in pronunciation, such as the duration, pitch, tone changes, and emotional fluctuations of each syllable.

[0032] Next, these audio features are fed into an action generation network based on the Transformer architecture. The Transformer architecture has powerful sequence modeling capabilities, enabling it to capture long-term dependencies in audio features. In this architecture, audio features are processed frame by frame, and a self-attention mechanism is used to identify the correlations between different parts of the audio. For example, some syllables may be closely related to specific facial movements (such as the "b" sound and the opening and closing of the lips), while other syllables may be related to changes in the eyes or eyebrows. In this way, the temporal features of the audio signal can be accurately mapped to facial movements.

[0033] In the action generation network, audio features are transformed into a series of key facial action parameters, including lip opening / closing coefficients, eye opening coefficients, and eyebrow movement. These facial parameters are generated through the model's regression or classification layers, ensuring that each facial movement is highly synchronized with the rhythm and intonation changes of the audio. For example, when rapid speech changes occur in the audio, the model can adjust the speed and amplitude of lip opening / closing in a timely manner; when the pitch fluctuates, facial features such as the eyes and eyebrows will also adjust accordingly based on the audio changes.

[0034] This precise synchronization between audio and facial movements not only effectively avoids the problems of lagging and misaligned facial movements in traditional models, but also enables more natural facial expression generation. By combining the powerful speech feature extraction capabilities of the Whisper model with the temporal modeling capabilities of the Transformer architecture, this invention can generate high-quality, real-time facial animations, ensuring a realistic and accurate user experience in various application scenarios such as video conferencing, virtual character creation, and entertainment creation.

[0035] S2: In order to further optimize the accuracy and naturalness of facial portrait animation, this invention designs several efficient control modules, including a Stitching module and an eye / lip repositioning module. These two modules are innovative designs for multi-character scenes and detailed facial motion control.

[0036] Stitching module:

[0037] In multi-character scenes, especially in dynamic environments, traditional portrait animation techniques often face the problem of misalignment between characters and backgrounds or between characters themselves when multiple characters interact simultaneously. For example, unnatural connections may appear in areas such as the shoulders and necks of characters, resulting in an overall uncoordinated image. To address this issue, this invention designs a Stitching module that employs a small multilayer perceptron (MLP) network to calculate and adjust the deformation offsets of driving keypoints. MLP (Mobile Location Protocol) defines a series of location service standards to support the needs of various LBS services.

[0038] Specifically, the Stitching module works as follows:

[0039] Keypoint offset calculation: The Stitching module first uses input audio features and character position and posture information to accurately calculate the offset of each driving keypoint (such as shoulders, neck, etc.) using an MLP network. These offsets are used to adjust and correct the stitching effect of multiple characters in the same scene.

[0040] Natural Blending: In scenes with multiple characters, the Stitching module ensures that each character's movements are coordinated with their surroundings and the movements of other characters, especially in areas such as the shoulders and neck, avoiding visual misalignment caused by distorted movements or improper stitching. Through precise keypoint adjustments, the module ensures natural transitions between different characters and between characters and the background, resulting in a more coherent and natural visual effect.

[0041] Eye / lip repositioning module:

[0042] In the process of facial portrait animation generation, especially when fine control is required for areas such as the eyes and lips, higher precision and dynamic adjustment are often needed than for other areas. To meet this need, this invention designs an eye / lip repositioning module, which uses an independent MLP network specifically for dynamically adjusting the position of key points on the eyes and lips to ensure more natural and detailed facial expressions.

[0043] The specific implementation method is as follows:

[0044] Independent MLP Network: This module receives eye opening and lip opening / closing coefficients from audio-driven MLP networks, ensuring real-time adjustment of eye and lip movements based on intonation, rhythm, and other features contained in the audio signal. Unlike other facial regions, subtle changes in the eyes and lips often affect the naturalness of the entire facial expression, thus requiring separate optimization.

[0045] Fine-grained control: This module not only ensures precise eye and lip movements but also allows for more precise control as needed. For example, the system can ensure complete eye closure; when the "close your eyes" command appears in the audio, it automatically adjusts the eye opening coefficient to achieve highly accurate eye movements. For lip control, the module also supports adjustable lip amplitude, dynamically adjusting the degree of lip opening and closing based on parameters such as volume and speech rate in the audio to match the actual changes in pronunciation.

[0046] Detailed adjustments: This module allows users to finely control every detail of facial expressions, ensuring that the movements of the eyes and lips perfectly match the rhythm of the audio, avoiding common problems of lag or misalignment, thereby enhancing the realism and expressiveness of facial expressions.

[0047] S3: Generates digital humans based on audio-motion mapping and a high-efficiency control module.

[0048] Application scenarios of this invention:

[0049] Video conferencing scenario: In video conferencing, this invention can convert traditional static user avatars into dynamic portraits synchronized with the audio content, thereby enhancing the interactivity and realism of remote communication and avoiding the sense of alienation caused by relying solely on static avatars. The system can dynamically adjust the facial expressions of the user avatar based on the speaker's speech rhythm and emotional changes, making communication in video conferencing more natural and vivid.

[0050] Social Media Scenario: In social media, users often use static avatars or images to showcase their personal image or express emotions. Using the technology of this invention, users can convert these static avatars (such as selfies, illustrations, or artistic portraits) into audio-driven dynamic content. For example, when a user uploads a selfie, the system can generate dynamic facial expressions and movements based on the user's voice (such as emotional changes during video recording), making the selfie not just a static image, but a dynamic portrait that displays different emotional states based on changes in sound.

[0051] In the realm of entertainment creation: For animators, game developers, and content creators, producing facial animations for virtual characters is a tedious and time-consuming process. This is especially true for virtual singers and anime characters, which often require manually drawing facial movements frame by frame, a process that is not only time-consuming but also extremely difficult. The audio-driven dynamic portrait technology provided by this invention allows creators to easily generate character facial expressions and movements synchronized with sound, significantly lowering the creative barrier. For example, in creating a virtual singer, the creator only needs to input voice or song audio, and the system can automatically generate virtual character facial movements synchronized with the voice content, such as mouth opening and closing, and eye blinking. This saves creators the time of manually drawing each expression, quickly and vividly portraying the virtual singer. Simultaneously, audio-driven dynamic portraits also make the character's performance more natural and engaging.

[0052] Virtual Interaction Scenarios: With the continuous advancement of artificial intelligence and virtual reality technologies, virtual assistants and digital humans are gradually entering our daily lives. These virtual characters not only need intelligent voice recognition and feedback capabilities, but also vivid and natural facial expressions and movements to enhance the realism and approachability of virtual interactions. The audio-driven dynamic portrait technology of this invention enables virtual characters to adjust their facial movements in real time based on the user's voice commands or interactive content, making virtual interactions more human-like.

[0053] In summary, compared with the prior art, the present invention has the following advantages:

[0054] This invention significantly improves the quality of facial animation generation through a series of innovative technologies, demonstrating remarkable progress in several aspects. First, synchronization and realism are greatly enhanced. By optimizing the synchronization between audio and facial movements, especially in matching lip movements with audio rhythm, this invention controls the synchronization error to within 2 milliseconds, greatly reducing the motion lag and misalignment problems common in traditional methods. Eye movements can also be automatically adjusted according to pauses, emphasis, and other contextual elements in the audio, making the animation more realistic and natural. Subjective evaluation results show that compared with existing technologies, the realism of the animation is improved by more than 35%, and the audience's immersion and interactivity are significantly enhanced. Second, this invention achieves a breakthrough in multi-style adaptability, efficiently handling portraits of different styles such as realistic, anime, and illustration. In relevant tests, the model performs excellently on the LPIPS (Perceptual Similarity Index), with a mean value below 0.07, and the texture distortion rate is reduced by 40% compared to traditional methods. This performance ensures that the generated animation stably conforms to the characteristics of the target style, greatly expanding the application scenarios of this technology in creation and providing creators with greater freedom and flexibility. Furthermore, the balance between efficiency and quality has been optimized to the extreme, with an inference speed of 12.8 milliseconds / frame, more than 3 times faster than the traditional diffusion method (50 milliseconds / frame), while also achieving better performance in objective quality indicators. The PSNR (Peak Signal-to-Noise Ratio) reached 31.56dB, and the SSIM (Structural Similarity) was 0.765, indicating that the generated animation has a visual quality far exceeding that of traditional non-diffusion methods, with higher detail and clarity. In addition, the technology performs excellently when handling multi-character scenes, stably generating coordinated and independent facial movements between multiple characters. For portrait scenes of 2-5 people, the character movement conflict rate is controlled below 5%, and the misalignment error in splicing areas such as the shoulders is less than 1 pixel, ensuring a high degree of consistency between each character's facial movements and the interaction with the background and other characters. Overall, this invention significantly improves the generation quality and practicality of portrait animation through optimizations in synchronization, style adaptation, efficiency, quality, and multi-character processing, broadens its applicability in various application scenarios, and greatly promotes technological progress in fields such as virtual interaction, creation, and entertainment.

[0055] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A method for generating digital humans from images and audio, characterized in that, Includes the following steps: S1: The Whisper model is used to encode the features of the input audio signal to extract the temporal features in the speech. The temporal features include rhythm features, intonation change features and emotional fluctuation features. Based on the temporal features, the temporal changes of pronunciation are obtained. The temporal features are input into the action generation network based on the Transformer architecture. In the Transformer architecture, the temporal features are processed frame by frame and the correlation between different parts of the audio is identified through the self-attention mechanism so as to map the temporal features of the audio signal onto facial movements. S2: Set up the high-efficiency control module, which includes the Stitching module and the eye and lip repositioning module; S3: Generates digital humans based on audio-motion mapping and a high-efficiency control module.

2. The image and audio generation digital human method as described in claim 1, characterized in that, The Stitching module works as follows: Keypoint offset calculation: The Stitching module first takes audio features and the pose of the character's position as input, and uses an MLP network to accurately calculate the offset of each driving keypoint. Each offset is used to adjust and correct the splicing effect of multiple characters in the same scene. Natural Blending: Through precise key point adjustments, the Stitching module ensures natural transitions between different characters and between characters and the background, resulting in a more coherent and natural visual effect.

3. The image and audio generation digital human method as described in claim 1, characterized in that, The eye and lip retargeting module is implemented as follows: Set up an independent MLP network: The MLP network for the eye and lip repositioning module is trained independently and receives eye opening and lip opening / closing coefficients from audio-driven signals; Fine-grained control: The eye and lip repositioning module allows for fine-grained control as needed; Detailed adjustments: Through the eye and lip repositioning module, users can control every detail of facial expressions, ensuring that the movements of the eyes and lips are perfectly matched with the audio rhythm.

Images