Method for realizing synchronization of character expression and lip shape in video with emotion in sound and cloned digital human system

Abstract

The application discloses a method for realizing synchronization of facial expression and lip shape of a character in a video through emotion in sound and a cloned digital human system. The method for realizing synchronization of facial expression and lip shape of a character in a video through emotion in sound collects audio information and extracts multi-dimensional features of sound; a pre-trained facial expression and lip shape generation model is applied to generate corresponding facial expression parameters and lip shape parameters according to the multi-dimensional features of sound; the facial expression parameters and the lip shape parameters are fused and processed, and a continuous animation sequence is generated according to the fused parameters; the continuous animation sequence is rendered to generate video information with synchronized facial expression, lip shape and sound emotion. The method realizes highly accurate synchronization of facial expression and lip shape, improves the naturalness of the cloned digital human, and expands the application range and value of the cloned digital human.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of audio-video processing in artificial intelligence, and particularly relates to a method for realizing synchronization of facial expression and lip shape of a character in a video through emotion in sound and a cloned digital human system. BACKGROUND

[0002] In the field of artificial intelligence, the processing of human faces in videos is mainly achieved in the following three ways:

[0003] I. Manual processing method of traditional video editing software. When processing real human videos in common video editing software (such as Adobe Premiere Pro, Final Cut Pro, etc.), if synchronization of facial expression and emotional sound is to be achieved, editors mainly rely on manual operation. First, they will repeatedly listen to the audio part of the video, and judge the emotional changes of the character according to the tone, speed, volume, etc. of the sound. For example, in a dialogue video, when the sound tone is raised and the speed is accelerated, the editor will find the corresponding character frame on the video timeline, then adjust the playback speed of the video, add key frames, and use the special effect tools in the software (such as adjusting the deformation effect of facial features) to simulate the excited or excited expression of the character. For some complex scenes, the editor may need to edit and splice the video clips to ensure that the expression changes match the rhythm of the sound. For example, at a plot twist, the sound emotion changes greatly, and the editor may select appropriate character expression frames from other clips, cut and adjust them, and then insert them into the current position to achieve synchronization of expression and sound as much as possible.

[0004] II. Expression adjustment technology based on simple template matching. Some existing technologies attempt to use simple template matching methods to handle the expression synchronization problem in real human videos. First, an expression template library is established, which contains some common expression templates (such as happy, sad, angry, etc. basic expressions) and corresponding sound feature templates (such as specific tone range, speed interval, etc.). When processing the video, the system will analyze the audio in the video and extract the sound features (such as spectral features, prosodic features, etc.), and then match them with the sound feature templates in the expression template library. Once a matching template is found, the corresponding expression template is applied to the character's face in the video. For example, when the sound tone is high and the speed is fast, which matches the happy expression sound feature template, the system will add some preset happy expression effects (such as mouth up, eyes wide open, etc.) to the face of the character in the video by adjusting the parameters of the effects to adapt to the character's image in the video.

[0005] III. Preliminary Artificial Intelligence Assisted Expression Analysis and Adjustment Techniques Some existing research or technical products use artificial intelligence algorithms to analyze and adjust expressions and voices in real videos. For example, through deep learning algorithms, a model is trained to recognize the expressions and voice emotions of the characters in the video. The input of the model is the video frame sequence (containing the facial images of the characters) and the audio segment, and the output is the expression classification result (such as various emotion categories) and the voice emotion features (such as emotion intensity, etc.). In practical applications, the system first preprocesses the input real video, extracts the video frames and audio data, and then inputs them into the trained model. The model analyzes the expressions and voices according to the learned patterns, and if it detects that the expressions and voice emotions do not match, it attempts to make adjustments. A common adjustment method is to change the expressions by modifying the key feature points of the characters' faces in the video frames. For example, using techniques such as generative adversarial networks (GAN), new facial key feature point coordinates are generated according to the voice emotion features, and then these coordinates are applied to the characters' faces in the original video frames, thereby changing the characters' expressions to make them more consistent with the voice emotions.

[0006] The above three methods have the following problems:

[0007] (1) Problems of traditional video editing software manual processing method

[0008] Highly dependent on human experience and operation skills: This manual processing method requires editors to have rich video editing experience and keen observation, and the results of different editors may differ greatly, making it difficult to ensure consistency and accuracy.

[0009] Low efficiency and time-consuming: For longer videos or frequent expression and voice synchronization adjustment needs, manual operation will consume a lot of time and effort, greatly reducing the efficiency of video production.

[0010] Accuracy is difficult to guarantee: Due to human subjective judgment and manual operation errors, it is difficult to achieve accurate synchronization of expressions and voice emotions, and problems such as premature or late expression changes, and mismatch with voice emotion intensity may occur.

[0011] (2) Problems of expression adjustment technology based on simple template matching

[0012] Single and unrealistic expression templates: The expression templates used are usually simple and fixed, and cannot accurately reflect the real expression changes of real people under various complex emotions, resulting in generated expressions that look stiff and unnatural.

[0013] Limited adaptability: Only able to handle voice emotion situations that match the existing templates in the template library, and unable to make effective expression adjustments for some special voice emotion changes not included in the template library, with poor flexibility.

[0014] It is difficult to achieve personalization: different individuals may have individual differences when expressing the same emotion, and this method cannot generate personalized expressions for different individuals' characteristics, and cannot meet diversified needs.

[0015] (Three) Problems existing in the preliminary artificial intelligence assisted expression analysis and adjustment technology

[0016] Model training is difficult: a large amount of labeled data (including real video data with accurate expression and voice emotion labels) is needed to train the model, and the data collection and labeling work is costly and time-consuming. Moreover, the training effect of the model is easily affected by the quality and quantity of the data. If the data is insufficient or biased, the accuracy and generalization ability of the model will be limited.

[0017] The delicacy of expression synchronization is insufficient: although some expression and voice emotion correlation analysis and adjustment can be performed, there is still a lack of delicacy in expression change. For example, subtle changes in voice emotion may not be accurately reflected in expression, resulting in inaccurate and unnatural expression and voice emotion synchronization. Especially in the processing of real person expressions of cloned digital people, due to the richness and complexity of real person expressions, the existing technology is difficult to completely capture and reproduce the subtle expression differences corresponding to the changes in voice emotion, and cannot achieve a highly realistic expression synchronization effect. SUMMARY

[0018] The technical problem to be solved by the present application is to provide a method for realizing synchronization of expressions and lip shapes of characters in a video through voice emotion and a cloned digital person system, which solves the problem that the existing technology cannot completely capture and reproduce the subtle expression differences corresponding to the changes in voice emotion, and cannot achieve a highly realistic expression synchronization effect.

[0019] To solve the above technical problems, the present application adopts the following technical solutions:

[0020] The method for realizing synchronization of expressions and lip shapes of characters in a video through voice emotion collects audio information and extracts multi-dimensional features of the voice; applies a pre-trained expression and lip shape generation model to generate corresponding expression parameters and lip shape parameters according to the multi-dimensional features of the voice; fuses the expression parameters and lip shape parameters, and generates a continuous animation sequence according to the fused parameters; renders the continuous animation sequence to generate video information with synchronized expressions, lip shapes and voice emotions.

[0021] After collecting the audio information, first, the audio information is denoised, and then the denoised audio information is subjected to multi-dimensional feature extraction.

[0022] The multi-dimensional features include pitch features, timbre features, speech rate features, prosody features, and emotional features in the voice.

[0023] The expression and lip shape generation model comprises an expression generation submodule, a lip shape generation submodule, and a model training module. The expression generation submodule is constructed based on a generative adversarial network, with an internal generator part taking sound emotion features as input and facial expression parameters as output. The lip shape generation submodule is constructed using a variational autoencoder, with sound features as input and lip shape action parameters as output. The model training module adopts a supervised learning algorithm, taking preprocessed sound features in video data as input and expression parameters and lip shape parameters as output.

[0024] The model training module repeatedly trains the models in the expression generation submodule and the lip shape generation submodule. During the training process, the weight parameters of the models are continuously adjusted, so that the models generate appropriate expressions and lip shape actions based on the input sound features. The model training module and the corresponding model structures in the expression generation submodule and the lip shape generation submodule have a parameter update connection relationship. Finally, the trained parameters are updated to the corresponding models in a timely manner.

[0025] The cloned digital human system comprises a cloned digital human three-dimensional model and a control system. The control system applies the method to obtain video information with synchronized expressions, lip shapes, and sound emotions, and sends the information to the cloned digital human three-dimensional model for synchronized output.

[0026] The control system comprises an audio processing module, an expression and lip shape generation module, a model training module, and a fusion and rendering module. The audio processing module comprises an audio acquisition unit, a noise reduction unit, and a feature extraction unit connected in sequence. The audio acquisition unit receives external input audio signals, which are transmitted to the feature extraction unit after noise reduction for multi-dimensional feature extraction of the sound.

[0027] The audio processing module comprises an audio acquisition unit, a noise reduction unit, and a feature extraction unit connected in sequence. The audio acquisition unit receives external input audio signals, which are transmitted to the feature extraction unit after noise reduction for multi-dimensional feature extraction of the sound.

[0028] The expression and lip shape generation module is constructed based on a deep learning generation model, comprising an expression generation submodule and a lip shape generation submodule. The expression generation submodule generates facial expression parameters for the cloned digital human, and the lip shape generation submodule generates corresponding lip shape action parameters.

[0029] The model training module is used to train the deep learning generation model, comprising a data collection unit, a data preprocessing unit, and a model training unit. The data collection unit collects real person video data and labels the expressions, lip shapes, and sounds in the video. The data preprocessing unit cleans and normalizes the labeled data. The model training unit adopts a supervised learning algorithm, taking preprocessed sound features in video data as input and expression and lip shape parameters as output. The deep learning generation model in the expression and lip shape generation module is repeatedly trained until the optimal parameters of the model are obtained.

[0030] The fusion and rendering module comprises a parameter fusion unit, an animation generation unit and a rendering unit connected in sequence, the parameter fusion unit fuses the facial expression parameters and the lip shape action parameters, so that the facial expression and the lip shape action are coordinated with the facial action of the cloned digital person, the animation generation unit generates a continuous animation sequence for the face of the cloned digital person according to the fused parameters, and the cloned digital person model with the facial expression and the lip shape animation is rendered to generate a video output with the facial expression, the lip shape and the voice emotion synchronization.

[0031] The feature extraction unit analyzes the noise-reduced audio signal by using a deep learning algorithm to extract various features of the voice, including pitch, tone, speech rate, prosody, volume and emotion.

[0032] The expression generation sub-module generates facial expression parameters of muscle movements of the cloned digital person including eyes, eyebrows and cheek parts according to the voice and emotion features extracted by the audio processing module, and the lip shape generation sub-module generates corresponding lip shape action parameters including the opening and closing degree of the lips, the moving position of the mouth corner and the cooperation relationship of the lips and teeth according to the speech content and the speech rate and prosody features in the audio.

[0033] The video data collected by the data collection unit includes the facial expression, the lip shape action and the corresponding voice signal, and the labeled content includes the expression type, the corresponding relationship between the lip shape action and the speech, and the emotion features in the voice.

[0034] Compared with the prior art, the present application has the following beneficial effects:

[0035] 1. The highly accurate expression and lip shape synchronization is realized, on the one hand, the voice features are accurately mapped to the expression and lip shape changes, various features in the voice can be deeply analyzed, including but not limited to pitch, tone, speech rate, prosody, volume, etc., and these voice features are accurately mapped to the subtle changes of the facial expression of the cloned digital person (such as the blinking frequency of the eyes, the fluctuation degree of the eyebrows, the muscle movement of the cheeks, etc.) and the accurate action of the lip shape (including the opening and closing amplitude of the lips, the upward or downward angle of the mouth corner, the cooperation of the lips and teeth, etc.). For example, when the pitch in the voice is raised and the speech rate is accelerated, the model can accurately drive the face of the cloned digital person to present an excited expression, the eyes are wide open, the eyebrows are raised, the lips are quickly opened and closed, and the mouth corner is raised, realizing the high synchronization of the expression and lip shape changes with the voice emotion, avoiding the delay, misplacement or unnatural action.

[0036] According to the pronunciation characteristics of different languages, especially in scenarios involving multiple language interactions, ensure that the cloned digital person's lip movements accurately correspond to the pronunciation of different languages. Whether it's the explosive sound, friction sound in English, or tone change in Chinese, etc., through accurate analysis of the sound, the cloned digital person's lip movements and expressions can naturally match the language expression, just like a real person speaking smoothly and naturally.

[0037] On the other hand, adapt to diverse voice input scenarios, whether it's recording clear professional audio, or actual scene audio containing some environmental noise or slightly degraded sound quality (such as live interviews, remote meetings, etc.), the method of the invention can effectively extract sound features and accurately generate corresponding expressions and lip movements. For example, audio recorded in a noisy outdoor environment, the system can filter out noise interference through advanced signal processing techniques and artificial intelligence algorithms, accurately identify the speaker's voice emotion and speech content, and generate appropriate expressions and lip movement synchronization actions for the cloned digital person, ensuring its performance stability and naturalness in various complex sound environments.

[0038] 2. Improve the naturalness of the cloned digital person, on the one hand, capture subtle expression changes and enhance emotional expression, focus on simulating subtle expression changes of humans, so that the cloned digital person can exhibit similar emotional levels and subtlety to real people. In addition to common basic expressions (such as happiness, sadness, anger, surprise, etc.), subtle emotional changes corresponding to hesitation, embarrassment, shyness, etc. can also be captured and presented through precise facial muscle movement control and lip adjustment. For example, when there is a hint of hesitation in the voice, the cloned digital person's eyes may be slightly narrowed, the eyebrows may be slightly wrinkled, the lips may be slightly paused and slightly pursed, and these subtle expression changes will greatly enhance the emotional expression of the cloned digital person, making it closer to the emotional expression of real people.

[0039] Simulate the dynamic expression change rules of real people in the communication process, such as smooth transition of expressions, avoiding abrupt expression switching. When the emotion in the voice gradually changes from calm to excited, the cloned digital person's expression can gradually transition from normal to excited according to the natural emotional change pattern of humans, the movement of facial muscles and the change of lip shape present a continuous and smooth process, rather than a harsh switch, making it difficult for the audience to detect it as a digital person, as if they are communicating with a real person.

[0040] On the other hand, personalized expression generation fits different character styles, and generates expressions and lip movements with personalized characteristics according to different cloned digital human role settings or imitated real person prototypes. For example, for a cloned digital human set as cheerful and lively, its expressions and lip movements may be more exaggerated and vivid than a cloned digital human set as calm and introverted under the same voice emotion. Through learning and analysis of different character style characteristics, cloned digital humans have unique personalities in expressions and lip movements, better adapting to the needs of various application scenarios, whether in role-playing in entertainment programs or in image shaping in enterprise customer service scenarios, they can interact with users in the most natural and most character-specific way.

[0041] 3. Expanding the application range and value of cloned digital humans, mainly in the following aspects:

[0042] 1) Optimizing the application experience of the entertainment industry

[0043] In the entertainment fields of film and television production, virtual live streaming, and games, cloned digital humans can be more realistically integrated into the plot or interactive scenarios. In film and television production, cloned digital human actors can perform more naturally with synchronized expressions and lip movements, seamlessly connecting with real actors and providing audiences with more stunning visual and emotional experiences. In virtual live streaming, the cloned digital human image of the host can accurately display a variety of expressions and lip movements in real time according to the voice emotion and speech content, enhancing the interactivity and immersion with the audience, attracting more audience attention and participation, and improving the quality and influence of live streaming. In games, if non-player characters (NPCs) are played by cloned digital humans, their realistic expressions and lip movements will make the game world more lively and interesting, and players can better understand the emotions and intentions of NPCs, thus improving the playability and immersion of the game.

[0044] 2) Improving the effectiveness of education and training

[0045] In the fields of education and training, cloned digital humans can serve as virtual teachers or training instructors, better conveying knowledge and emotions through natural expressions and lip movements. For example, in language teaching, cloned digital human teachers can vividly display the changes in mouth shape and corresponding expressions according to the emotional color and voice tone of the teaching content, helping students more intuitively understand the pronunciation points and emotional connotations behind the language, and improving learning efficiency. In vocational skill training, cloned digital humans can simulate dialogues and expressions of characters in various work scenarios, allowing trainees to more realistically experience the communication in actual work, enhancing the practicality and effectiveness of training.

[0046] 3) Enhancing the quality of business services and communication

[0047] In the business field, such as customer service, product demonstration, etc., cloned digital people can communicate with customers in a more humanized way. Customer service cloned digital people can adjust their expressions and tone according to the customer's voice emotion in a timely manner. When the customer expresses dissatisfaction, the cloned digital people can respond to the customer through appropriate expressions (such as concerned eyes, slightly frowning) and sincere tone, making the customer feel understood and valued, and improving customer satisfaction. In product demonstration, cloned digital people can introduce product features and advantages through lively expressions and lip synchronization, attract the attention of consumers, better convey product information, promote sales conversion, and create greater business value for enterprises. BRIEF DESCRIPTION OF DRAWINGS

[0048] Fig. 1 The application method flow chart of the digital cloned person system of the present application.

[0049] Fig. 2 The image cloning specific method flow chart of the digital cloned person system of the present application.

[0050] Fig. 3 The voice cloning specific method flow chart of the digital cloned person system of the present application.

[0051] Fig. 4 The semantic emotion judgment and expression generation specific method flow chart of the digital cloned person system of the present application. DETAILED DESCRIPTION

[0052] The structure and working process of the present application will be further described below in conjunction with the drawings.

[0053] The present application aims to construct an audio and video processing method and system based on artificial intelligence, which is used to realize the precise synchronization of cloned digital people's expressions and lip shapes with voice emotions in videos, so that the cloned digital people can perform more naturally and realistically, like real people. The system mainly consists of an audio processing module, an expression and lip shape generation module, a model training module, and a fusion and rendering module. Each module works collaboratively to realize the complete process from voice input to cloned digital people's expression and lip shape synchronization output.

[0054] The specific scheme includes a method for realizing synchronization of video character expressions and lip shapes through voice emotions, and a cloned digital person system. In the method for realizing synchronization of video character expressions and lip shapes through voice emotions, audio information is collected and multi-dimensional features of the voice are extracted. A pre-trained expression and lip shape generation model is applied to generate corresponding expression parameters and lip shape parameters based on the multi-dimensional features of the voice. The expression parameters and lip shape parameters are fused and processed, and a continuous animation sequence is generated based on the fused parameters. The continuous animation sequence is rendered to generate video information with synchronized expressions, lip shapes, and voice emotions.

[0055] The cloned digital human system includes a cloned digital human three-dimensional model and a control system thereof, the control system applies the method to obtain video information with synchronized expression, lip shape and voice emotion, and sends the video information to the cloned digital human three-dimensional model to perform synchronized output.

[0056] Specific embodiment one,

[0057] The method for realizing synchronization of character expression and lip shape in a video with emotion in a voice includes the following steps:

[0058] Step 1, collecting audio information and extracting multi-dimensional features of the voice; a target audio signal is collected by an audio collection unit (which can be real-time recording or importing an existing audio file), the collected audio signal enters a noise reduction unit for noise reduction processing, and then multi-dimensional features of the voice are extracted by a feature extraction unit; the order of this step cannot be randomly interchanged, because the audio must be collected first to perform subsequent noise reduction and feature extraction operations, and if the order is reversed, the work cannot be normally carried out.

[0059] Step 2, applying a pre-trained expression and lip shape generation model to generate corresponding expression parameters and lip shape parameters according to the multi-dimensional features of the voice;

[0060] The expression and lip shape generation model includes an expression generation sub-module, a lip shape generation sub-module and a model training module,

[0061] The expression generation sub-module is constructed based on a generative adversarial network (GAN), the internal generator part takes voice emotion features as input and facial expression parameters as output; the lip shape generation sub-module is constructed using a variational autoencoder (VAE), taking voice features as input and lip shape action parameters as output; the work of the two sub-modules is carried out in parallel, and they provide basic parameters for the subsequent fusion, this step needs to be carried out after the completion of the first step of audio feature extraction, and relies on the voice features extracted in the first step as input.

[0062] The model training module adopts a supervised learning algorithm, taking the sound features in the preprocessed video data as input and the expression parameters and lip shape parameters as output. The data collection unit in the model training module collects and organizes real person video data in advance. After processing by the data preprocessing unit, the model training unit trains the GAN and VAE models in the expression and lip shape generation model using these data. During the training process, the model's weight parameters are continuously adjusted so that the model can generate appropriate expressions and lip shape movements based on the input sound features. The model training module has a parameter update connection relationship with the corresponding model structures in the expression generation submodule and the lip shape generation submodule. Finally, the trained parameters are updated to the corresponding models in a timely manner. This process can be performed when the system is initially built or the model is updated. The trained model has the ability to generate expression and lip shape movement parameters based on sound features. This step can be performed in parallel with the first step in terms of time sequence. However, before using the expression and lip shape generation module to generate parameters, the model needs to complete the training work.

[0063] Step 3, fuse the expression parameters and lip shape parameters, and generate a continuous animation sequence based on the fused parameters; the parameter fusion unit in the fusion and rendering module receives the expression parameters and lip shape parameters from the expression and lip shape generation module, performs fusion processing, and then passes the fused parameters to the animation generation unit. The animation generation unit generates a continuous animation sequence based on the fused parameters, allowing smooth transitions and changes in expressions and lip shape movements. This step must be performed after the expression and lip shape parameters are generated in step 3, and the order cannot be reversed, because only after the parameters are generated can fusion and animation generation operations be performed.

[0064] Step 4, render the continuous animation sequence to generate video information with synchronized expressions, lip shapes, and sound emotions.

[0065] The animation sequence data generated by the animation generation unit is transmitted to the rendering unit, which uses ray tracing technology to render the animation sequence, ultimately generating video information with synchronized expressions, lip shapes, and sound emotions.

[0066] Specific embodiment two,

[0067] The cloned digital human system includes a cloned digital human three-dimensional model and its control system. The control system uses the above method to obtain video information with synchronized expressions, lip shapes, and sound emotions, and sends it to the cloned digital human three-dimensional model for synchronized output.

[0068] The control system includes an audio processing module, an expression and lip shape generation module, a model training module, and a fusion and rendering module. Among them,

[0069] The audio processing module comprises an audio acquisition unit, a noise reduction unit and a feature extraction unit connected in sequence. The audio acquisition unit receives externally input audio signals, which are transmitted to the feature extraction unit after being de-noised by the noise reduction unit for multi-dimensional feature extraction of the sound. The audio acquisition unit uses a professional audio acquisition card (which has the ability to collect sound signals with high fidelity and can be compatible with multiple audio input sources) to receive externally input audio signals, for example, it can connect a microphone to record real-time human voice or access a stored audio file. The collected audio signals are transmitted to the noise reduction unit in real time. The noise reduction unit is designed based on an adaptive filtering algorithm (Adaptive Filtering Algorithm, an algorithm that can dynamically adjust filtering parameters according to input signal and noise characteristics to effectively remove noise), and its functions are realized through software programming. The audio acquisition unit is connected to the noise reduction unit through a data line to receive the collected raw audio signals and perform de-noising processing. The processed audio signals are then transmitted to the feature extraction unit. The feature extraction unit uses a convolutional neural network (Convolutional Neural Network, CNN, a deep learning network structure that is good at automatically extracting features from data) to build its input port connected to the output port of the noise reduction unit to receive the de-noised audio signals and perform deep feature extraction operations.

[0070] The function and operation process of the audio module are as follows:

[0071] First, the audio acquisition unit starts to acquire audio signals in real time to ensure accurate input of sound data. For example, when we want to collect the audio of a virtual customer service cloned digital person's service dialogue, we can accurately collect the customer's voice through a microphone. Then, the noise reduction unit filters the environmental noise (such as the noisy voices in the office background and the humming sound generated by the computer equipment running) that may exist in the collected audio to improve the purity of the audio signal and ensure that the subsequent feature extraction is not disturbed by noise. Finally, the feature extraction unit extracts multi-dimensional sound features from the de-noised audio, including pitch features (obtained by analyzing the frequency changes of the audio signal, such as higher frequency components corresponding to higher pitch), timbre features (distinguishing the sound characteristics of different people based on factors such as the harmonic structure of the sound), speech rate features (judging by calculating the number of syllables per unit time), prosodic features (such as through the rise and fall of speech, stress distribution, etc.), and emotional features in the sound (using the deep learning model's ability to classify and identify the emotions of the sound to determine whether it is happy, sad, or other emotions and their intensity). These extracted features will serve as an important basis for subsequent expression and lip shape generation and be transmitted to the expression and lip shape generation module.

[0072] The expression and lip shape generation module is based on a deep learning generated model, which includes an expression generation submodule and a lip shape generation submodule. The expression generation submodule generates facial expression parameters of the cloned digital person, and the lip shape generation submodule generates corresponding lip shape action parameters.

[0073] The expression generation submodule is built based on a generative adversarial network (GAN, composed of a generator and a discriminator, continuously optimized through adversarial training to generate realistic data). The internal generator part receives sound emotion features as input and outputs facial expression parameters of the cloned digital person. These parameters specifically cover eye opening angles, eyebrow bending degrees, cheek muscle contraction and relaxation degrees, etc., which are used to control the expression changes of the cloned digital person. The lip shape generation submodule is built using a variational autoencoder (VAE, a deep learning model that can learn the data latent distribution and generate data). The input is the speech features in the sound (such as phonemes, syllable information, and speech rate, rhythm, etc.), and the output is the lip shape action parameters, such as the opening amplitude of the lips, the movement coordinates of the mouth corners, and the cooperation state of the lips and teeth, etc., to accurately control the lip shape action of the cloned digital person. The output ports of the two submodules are respectively connected with the parameter fusion unit in the fusion and rendering module to transmit the generated expression parameters and lip shape parameters.

[0074] The function and operation process of this module are as follows:

[0075] After receiving the sound features from the audio processing module, the generator of the expression generation submodule generates expression parameters based on the sound emotion features. For example, if the sound contains a high-intensity happy emotion, the generator will output corresponding expression parameters to make the cloned digital person's eyes wide open, in a curved crescent shape, eyebrows raised, and cheek muscles slightly raised, showing a clear happy expression. At the same time, the lip shape generation submodule generates accurate lip shape action parameters based on the speech features, ensuring that the mouth movements perfectly match the pronunciation when the cloned digital person speaks the corresponding speech content. For example, when pronouncing the phoneme "O", the lips will naturally open in a circular shape. The parameters generated by the two submodules provide basic data support for the subsequent synchronized presentation of the cloned digital person's expression and lip shape.

[0076] The model training module is used for training a deep learning generation model, including a data collection unit, a data preprocessing unit and a model training unit. The data collection unit collects real person video data and labels the expressions, lip shapes and voices in the video. The data preprocessing unit cleans and normalizes the labeled data. The model training unit uses a supervised learning algorithm, takes the voice features in the preprocessed video data as input and the expression and lip shape parameters as output, repeatedly trains the deep learning generation model in the expression and lip shape generation module, and obtains the optimal parameters of the model until the optimal parameters are obtained.

[0077] The data collection unit is responsible for collecting real person video data from multiple channels. These data sources can be public film material libraries (such as [specific film material library name], which contains a large number of video resources of different scenes and different characters), professional recorded expression and voice data sets (such as [specific data set name, annotated in detail by relevant scientific research institutions or enterprises], etc. The collected video data contains rich expressions, lip movements and corresponding voice signals, and the expressions, lip movements and voices in each video are accurately labeled. The labeling content is detailed to expression types (such as specific smile, laughter, frown, etc.), the corresponding relationship between lip movements and voices (each phoneme corresponds to a change in lip shape), emotional characteristics in the voice (specific emotion and intensity value), etc. The data collection unit transmits the collected data to the data preprocessing unit. The data preprocessing unit uses data normalization (unifies different ranges of data to a specific interval to facilitate model training), data cleaning (removes duplicate, incorrect or incomplete data records) and other technical means to process the collected data, ensuring the quality and consistency of the data. The processed data is transmitted to the model training unit. The model training unit uses a supervised learning algorithm (such as the Back Propagation Algorithm, which adjusts the weight parameters of the neural network based on error back propagation to optimize the performance of the model), takes the voice features in the preprocessed video data as input and the expression and lip shape parameters as output, and repeatedly trains the model (i.e. the GAN and VAE models mentioned above) in the expression and lip shape generation module. During the training process, the weight parameters of the model are continuously adjusted so that the model can accurately generate appropriate expressions and lip movements based on the input voice features. The model training unit and the corresponding model structure in the expression and lip shape generation module have a parameter update connection relationship, ensuring that the trained parameters can be updated to the corresponding model in time.

[0078] The function and operation process of this module are as follows:

[0079] First, the data collection unit collects a wide range of real human video data to accumulate sufficient materials for model training, such as collecting videos of people speaking in different age groups, different genders, and different language environments, covering a variety of rich expressions and lip shape movements. Next, the data preprocessing unit standardizes the collected data, removing data that may interfere with model training effectiveness, making the input data to the model more accurate and reliable. Finally, the model training unit uses a supervised learning algorithm to learn the mapping rules between voice features and expression, lip shape parameters according to the set input-output relationship. After multiple iterations of training (such as setting the training rounds to [specific number of rounds, such as 100 rounds]), the model's prediction ability is continuously optimized until the model achieves a satisfactory accuracy rate (such as an accuracy rate of [specific accuracy rate value, such as 95%] or higher) on the validation data set (part of the collected data set aside for model validation), completing the model training process and updating the trained model parameters to the corresponding model structure of the expression and lip shape generation module, enabling it to generate expressions and lip shape movements based on actual input voice features.

[0080] The fusion and rendering module includes a parameter fusion unit, an animation generation unit, and a rendering unit connected in sequence. The parameter fusion unit fuses facial expression parameters and lip shape action parameters, making the expressions and lip shape actions consistent with the cloned digital person's facial movements. The animation generation unit generates continuous animation sequences for the cloned digital person's face based on the fused parameters. The cloned digital person model with expressions and lip shape animations is rendered to generate video output with synchronized expressions, lip shapes, and voice emotions.

[0081] The parameter fusion unit receives the expression parameters and the lip shape parameters from the expression and lip shape generation module, and performs fusion processing on the two parameters through a specific fusion algorithm (for example, a weighted fusion algorithm based on time synchronization and motion coordination, which assigns appropriate weights to the parameters of the expression and lip shape actions in the time dimension and overall visual coordination for fusion) to ensure that the expression and lip shape actions are coordinated and consistent on the cloned digital person's face, avoiding the phenomenon of disconnection or incoordination between expression and lip shape actions. The fused parameters are transmitted to the animation generation unit. The animation generation unit generates continuous animation sequences for the cloned digital person's face based on the fused parameters using keyframe interpolation (generating intermediate transition frames between the start and end keyframes to make the motion change more smooth and natural) and other animation generation techniques, so that the expression and lip shape actions can smoothly transition and change. The generated animation sequence data is transmitted to the rendering unit. The rendering unit uses Ray Tracing Technology, a rendering technology that simulates the propagation, reflection, refraction, and other physical processes of light to generate high-quality images, combined with the three-dimensional model of the cloned digital person (previously created by professional three-dimensional modeling software, the model structure contains facial bones, skin texture, and other detailed elements, such as using [specific three-dimensional modeling software name and version] to create), to render the cloned digital person model with expression and lip shape animation, and finally generate high-quality, visually realistic video output, showing the cloned digital person image with synchronized expression and lip shape and voice emotion.

[0082] The function and operation process of this module are as follows:

[0083] After receiving the expression parameters and the lip shape parameters, the parameter fusion unit performs detailed fusion operations to ensure that the cloned digital person makes expressions while the lip shape actions perfectly match them, just like the natural synchronization of expressions and mouth shapes when a real person speaks. For example, when expressing surprise while saying the "ah" syllable, the fused parameters can make the cloned digital person's eyes widen and the mouth open in a suitable circular shape, with high coordination in time and motion amplitude. Then, the animation generation unit converts the fused parameters into continuous animation sequences, giving the cloned digital person's facial movements liveliness and smoothness, allowing their expression and lip shape actions to change naturally. Finally, the rendering unit uses advanced Ray Tracing Technology to render the cloned digital person model with animation, making it more realistic in appearance, and the cloned digital person's expression and lip shape in the output video can be precisely synchronized with the voice emotion, presenting a natural and realistic visual effect.

[0084] One of the core inventions of the present scheme is the ability to simultaneously fuse multiple features in the sound (such as emotional features, speech features, etc.) and accurately map them to the cloned digital person's expressions and lip movements. Through the comprehensive analysis of multiple modalities such as pitch, timbre, speech rate, prosody, volume, and emotional intensity by the deep learning model, the synchronization of expressions and lip movements with sound in terms of time and movement amplitude is achieved. For example, in a highly emotional speech segment, the model can not only make the cloned digital person's face exhibit the corresponding expression according to the emotional features, but also ensure that the lip movements perfectly match the speech content according to the speech features, and the change rhythm of the two is completely synchronized, avoiding the problem of asynchronous or incoordination between expressions and lip movements and sound in the prior art.

[0085] For the deep learning model in the expression and lip generation module, in addition to the above-mentioned generative adversarial network (GAN) and variational autoencoder (VAE), recurrent neural networks (RNN) and their variants (such as long short-term memory network LSTM, gated recurrent unit GRU, etc.) can also be considered. RNN series models have advantages in processing time series data (such as audio signals and video frame sequences), and can better capture the dynamic relationship between sound and expressions, lip movements. For example, LSTM can remember the trend of sound features in the time series through its unique memory unit, thereby more accurately generating corresponding expressions and lip movements.

[0086] Another important invention is the ability to generate personalized expressions and lip movements according to different cloned digital person roles or real person prototypes. Through learning from a large amount of real person video data of different styles and different character features, the model can capture the individualized differences of different characters when expressing the same emotion or speech. For example, for a cloned digital person with an open personality, when expressing a happy emotion, its expressions and lip movements may be more exaggerated and lively than a cloned digital person with a stable personality, with eyes that open wider, a higher upward angle of the mouth, and more dynamic lip movements. This personalized generation technology makes the cloned digital person more realistic and unique.

[0087] In terms of feature extraction in the audio processing module, in addition to deep learning algorithms, traditional signal processing techniques combined with machine learning algorithms can also be used. For example, first use traditional methods such as mel-frequency cepstral coefficients (MFCC) to extract the basic features of the audio, and then use machine learning algorithms such as support vector machines (SVM) to classify and extract emotional features from these features. This method may have certain advantages when the amount of data is relatively small, and the computational cost is relatively low. However, its disadvantage is that it may not be as expressive as deep learning algorithms for complex sound emotions and diverse speech features.

[0088] For example, Figs. 1 to 4As shown, the specific application principle and workflow of the cloned digital human system are as follows:

[0089] Step 1, image cloning, specifically,

[0090] 1.1, image acquisition: through mobile phones, cameras or other professional equipment, clear image videos of the cloned person including different expressions and postures are taken.

[0091] 1.2, feature extraction and model creation: using image processing algorithms, the facial features of the cloned person are extracted, such as the shape, proportion and skin color of the five senses, and based on the extracted features, the basic model of the digital person is created.

[0092] 1.3, detail rendering: based on the basic outline, more delicate textures, hair, makeup and other details are drawn to enhance the realism.

[0093] Step 2, sound cloning, specifically,

[0094] 2.1, audio acquisition: record a large number of clear voice samples of the cloned person, covering various tones, speeds and emotional expressions.

[0095] 2.2, sound feature analysis: analyze the frequency spectrum, pitch, duration and other characteristics of the collected speech.

[0096] 2.3, model training: use deep learning technology to train the sound cloning model to learn the voice characteristics of the cloned person.

[0097] 2.4, speech synthesis: when inputting text, the sound cloning model generates similar voices to the cloned person based on the learned characteristics.

[0098] Step 3, semantic emotion judgment and expression generation, specifically,

[0099] 3.1, natural language processing: semantic analysis of the input text to understand its meaning and emotional tendency.

[0100] 3.2, emotion classification: use the trained emotion classification model to determine the type of emotion expressed in the text.

[0101] 3.3, expression library generation: generate a rich 2D expression library through algorithms, containing expressions of various common emotions.

[0102] 3.4, expression matching: according to the judged emotion, select the corresponding 2D expression from the expression library.

[0103] Step 4, video generation, specifically,

[0104] 4.1, synthesis and rendering: synthesize the digital human image, matching expression, generated sound and designed action, and perform rendering optimization to generate the final digital human video.

[0105] For example, the method and cloned digital human system are applied to generate a new year video. First, according to the requirements, the materials required for the new year video are obtained. According to these materials, the cloned digital human is generated. Various digital human images, expressions, sounds and designed actions that meet the new year elements are synthesized and rendered to generate a smooth video file that meets the theme.

[0106] Those skilled in the art should understand that those skilled in the art can realize variations in combination with the prior art and the above embodiments, and such variations do not affect the essential content of the scheme, which will not be described here.

[0107] It should be understood that the present scheme is not limited to the above specific embodiments, and the devices and structures not fully described should be understood as being implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the present scheme, or modify equivalent embodiments, without departing from the scope of the present scheme, which does not affect the essential content of the present scheme. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present scheme, without departing from the content of the present scheme, still belongs to the protection scope of the present scheme.

Claims

1. A method for synchronizing facial expressions and lip movements in a video by using emotions in the voice, characterized by: Collect audio information and extract multi-dimensional features of the sound; A pre-trained facial expression and lip shape generation model is applied to generate corresponding facial expression parameters and lip shape parameters based on the multi-dimensional features of the voice; the facial expression parameters and lip shape parameters are fused, and a continuous animation sequence is generated based on the fused parameters; the continuous animation sequence is rendered to generate video information with synchronized facial expression, lip shape and voice emotion; the multi-dimensional features include pitch features, timbre features, speech rate features, rhythm features and emotional features in the voice; The facial expression and lip shape generation model includes an expression generation submodule, a lip shape generation submodule, and a model training module. The expression generation submodule is built based on a generative adversarial network, and its internal generator part takes vocal emotion features as input and facial expression parameters as output. The lip shape generation submodule is built using a variational autoencoder, taking speech features in the voice as input and lip movement parameters as output. The model training module uses a supervised learning algorithm, taking the sound features in the preprocessed video data as input and the facial expression parameters and lip shape parameters as output.

2. The method for synchronizing facial expressions and lip movements in a video by using emotions in sound, as described in claim 1, is characterized in that: After collecting the audio information, the audio information is first denoised, and then multi-dimensional features are extracted from the denoised audio information.

3. The method for synchronizing facial expressions and lip movements in a video by using emotions in sound, as described in claim 1, is characterized in that: The model training module repeatedly trains the models in the expression generation submodule and the lip shape generation submodule. During the training process, the weight parameters of the model are continuously adjusted so that the model can generate appropriate expressions and lip movements based on the input sound features. There is a parameter update connection between the model training module and the corresponding model structures in the expression generation submodule and the lip shape generation submodule, and the trained parameters are updated to the corresponding models in a timely manner.

4. A cloned digital human system, characterized by: The invention includes a cloned digital human 3D model and its control system, wherein the control system uses the method described in any one of claims 1 to 3 to acquire video information with synchronized facial expressions, lip movements and vocal emotions, and sends it to the cloned digital human 3D model for synchronized output.

5. The cloned digital human system according to claim 4, characterized in that: The control system includes an audio processing module, an expression and lip shape generation module, a model training module, and a fusion and rendering module; wherein... The audio processing module includes an audio acquisition unit, a noise reduction unit, and a feature extraction unit connected in sequence. The audio acquisition unit receives externally input audio signals, which are then denoised by the noise reduction unit and transmitted to the feature extraction unit for multi-dimensional feature extraction of the sound. The expression and lip shape generation module is based on a deep learning-based generative model and includes an expression generation submodule and a lip shape generation submodule. The expression generation submodule generates facial expression parameters for the cloned digital human, and the lip shape generation submodule generates corresponding lip shape action parameters. The model training module is used to train the deep learning generative model. It includes a data collection unit, a data preprocessing unit, and a model training unit. The data collection unit collects real-person video data and annotates the facial expressions, lip shapes, and sounds in the videos. The data preprocessing unit cleans and normalizes the annotated data. The model training unit uses a supervised learning algorithm, taking the sound features in the preprocessed video data as input and the facial expression and lip shape parameters as output, and repeatedly trains the deep learning generative model in the facial expression and lip shape generation module until the optimal parameters of the model are obtained. The fusion and rendering module includes a parameter fusion unit, an animation generation unit, and a rendering unit connected in sequence. The parameter fusion unit fuses facial expression parameters and lip movement parameters to make the facial expressions and lip movements coordinated with the facial movements of the cloned digital human. The animation generation unit generates a continuous animation sequence for the face of the cloned digital human based on the fused parameters, renders the cloned digital human model with facial expressions and lip movements, and generates a video output with synchronized facial expressions, lip movements, and voice emotions.

6. The cloned digital human system according to claim 5, characterized in that: The feature extraction unit uses deep learning algorithms to analyze the denoised audio signal and extract various features of the sound, including pitch, timbre, speech rate, rhythm, volume, and emotion.

7. The cloned digital human system according to claim 6, characterized in that: The expression generation submodule generates facial expression parameters for the cloned digital human, including the muscle movements of the eyes, eyebrows, and cheeks, based on the sound and emotional features extracted by the audio processing module. The lip shape generation submodule generates corresponding lip shape action parameters, including the degree of lip opening and closing, the position of the corners of the mouth, and the coordination of the lips and teeth, based on the speech content, speech rate, and rhythm features in the audio.

8. The cloned digital human system according to claim 7, characterized in that: The video data collected by the data collection unit includes facial expressions, lip movements, and corresponding sound signals. The annotations include the type of expression, the correspondence between lip movements and speech, and the emotional features in the voice.

Images