A multi-modal digital human generation method and system

Abstract

The application discloses a kind of multimodal digital human generation method and system, including: picture generates digital human module, video generates digital human module and video translation module;It is related to the field of depth synthesis technique, the technical problem that the extensibility and flexibility of the digital human generation mode of prior art are not high is solved;The 3D face model is generated by processing reference image;Motion coefficients and first audio signal are generated first digital human based on face renderer and appearance encoder;The audio of specified duration is extracted from original video and marked as original audio;Translate text into corresponding target audio, and the target audio is synthesized into translation video with original video.The 3D face model is generated by 3DMM technology from reference image, so that the 3D face model can maintain the identity characteristics of the character and generate natural dynamic expression, which is beneficial to improve the interactive experience between the generated digital human and the user.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep synthesis, and particularly relates to a multi-modal digital person generation method and system. BACKGROUND

[0002] At present, the synthesis modes of virtual digital persons mainly include the following modes: synthesis from video and audio, synthesis from video and text. These synthesis modes are usually supported by artificial intelligence algorithms and deep learning models, and generate virtual person images that are lifelike and highly interactive.

[0003] 1. Synthesis from video and audio: this mode inputs video and audio data, uses algorithms to process and fuse sound and image, and generates a dynamic virtual person. This mode is widely used in virtual anchors, virtual customer service and other scenarios. The synthesis process involves emotion recognition of audio and accurate control of facial expressions in video, ensuring that the voice and expressions of the virtual person are synchronized, giving users a natural and smooth interactive experience.

[0004] 2. Synthesis from video and text: this mode inputs text and converts it into speech, while coordinating the lip movements and expression changes of the characters in the video. Semantic understanding of text and natural language processing (NLP) technology are the core of this mode, enabling virtual characters to generate appropriate expressions and actions based on text content, thus realizing more humanized communication.

[0005] The current synthesis technology still has some shortcomings. For example, the sound of the digital person generated by the existing technology is not synchronized with the lip movements, and the tone is not consistent with the original speaker, which brings obvious discomfort to the video viewer and produces a false feeling; these shortcomings limit the performance of virtual digital persons in higher requirement application scenarios, such as scenarios that require accurate expression of specific character features.

[0006] The present application proposes a multi-modal digital person generation method and system to solve the above technical problems. SUMMARY

[0007] The present application aims to at least solve one of the technical problems existing in the prior art; for this purpose, the present application proposes a multi-modal digital person generation method and system to solve the technical problem that the digital person generation schemes of the prior art are mostly generated from video, and in some scenarios, users can only provide pictures and audio, and the prior art is difficult to generate digital persons from pictures and audio, resulting in low expandability and flexibility of the generation mode of digital persons; in addition, the sound characteristics of the digital person generated by the prior art are quite different from those of real persons, resulting in poor interactive experience of users with digital persons.

[0008] To achieve the above object, the first aspect of the present application provides a multi-modal digital human generation system, comprising: a picture generation digital human module, a video generation digital human module and a video translation module;

[0009] The picture generation digital human module is configured to: pre-process an original image based on a SadTalker algorithm to obtain a reference image; process the reference image based on a 3DMM algorithm to obtain a 3D face model; extract and generate motion coefficients from the 3D face model based on the SadTalker algorithm; and,

[0010] convert input text into a first audio signal based on an edge-tts algorithm; generate a first digital human from the motion coefficients and the first audio signal based on a face renderer and an appearance encoder; wherein the motion coefficients comprise facial expression coefficients and head pose coefficients, and the head pose coefficients comprise head rotation coefficients and head displacement coefficients;

[0011] The video generation digital human module is configured to: process a second audio signal and a first video based on an IP_LAP algorithm to obtain a key point sequence; render the key point sequence into a second video based on the IP_LAP algorithm; convert text content into a third audio signal based on an edge-tts algorithm; synthesize the third audio signal and the second video to obtain a second digital human; wherein the key point sequence comprises key points of a mouth and a lower jaw.

[0012] The video translation module is configured to: extract an audio of a specified duration from an original video and mark it as an original audio by using an FFmpeg tool; input the original audio into a speech recognition model to generate corresponding text information; translate the text information based on a translation model to obtain translated text; convert the translated text into corresponding target audio based on an edge-tts algorithm; adjust the timbre of the target audio to the timbre of a speaker in the original video by calling a timbre cloning API; and synthesize the target audio and the original video into a translated video.

[0013] Preferably, the pre-processing of the original image comprises adjusting the resolution, size and format of the original image.

[0014] Preferably, the video translation module is in communication and / or electrical connection with the picture generation digital human module and the video generation digital human module, respectively.

[0015] Preferably, the processing of the reference image based on the 3DMM algorithm to obtain the 3D face model comprises:

[0016] extracting the reference image, inputting the reference image into a 3DMM (3D Morphable Model) algorithm, and the 3DMM algorithm obtaining the 3D face model through a formula S = S avg + αU id+βU exp The facial geometry S is calculated; where S avg For average face shape, U id U is an orthogonal basis for identity. exp Let be an orthogonal basis for facial expressions, where α is the identity coefficient greater than 0 and β is the facial expression coefficient greater than 0;

[0017] Obtain head pose coefficients: r∈SO(3), t∈R 3 And construct facial pose features; where r represents the head rotation matrix, t represents the head displacement vector, SO(3) is a three-dimensional special orthogonal group, R 3 It is a three-dimensional Euclidean space;

[0018] Integrate facial geometry and facial pose features into a 3D facial model.

[0019] This invention uses 3DMM technology to generate 3D facial models from reference images, enabling the 3D facial models to retain the identity features of the person while generating natural dynamic expressions, which is beneficial to improving the interactive experience between the subsequently generated digital human and the user.

[0020] Preferably, the extraction and generation of motion coefficients from the 3D facial model based on the Sadtalker algorithm includes:

[0021] Extract the 3D facial model, and use the ExpNet and PoseVAE modules in the Sadtalker algorithm to extract and generate the corresponding motion coefficients.

[0022] The ExpNet module uses formula β 1,…,T =Φ M (Φ A (a 1,…,T ),z blink ,β0) Analyze audio signals to generate facial expression coefficients β 1,…,T ; where Φ A For audio encoders, Φ M For the mapping network, β0 is the initial expression coefficient of the reference image, z blink To control the blink signal, T is the total number of time frames of the audio signal;

[0023] The PoseVAE module is built on a variational autoencoder framework. It generates continuous head pose coefficients by learning the mapping relationship between audio features and head movements.

[0024] The sadtalker algorithm is used to analyze the audio signal to obtain the facial expression coefficient, and the PoseVAE module in the sadtalker algorithm is used to extract and generate the head posture coefficient, so that the generated head posture can not only maintain the identity characteristics of the character, but also show natural dynamic expression changes, so that the head posture of the virtual character in the generated video is coordinated with the rhythm and content of the input audio, thereby enhancing the authenticity of visual performance.

[0025] Preferably, the edge-tts algorithm converts the input text into a first audio signal, comprising:

[0026] The edge-tts algorithm processes the input text through a natural language processing (NLP) module to extract semantic information, and generates a first audio signal based on the semantic information.

[0027] It should be noted that the audio signal will be synchronized with the generated animation frame sequence in the video synthesis stage, so that the lip movement of the virtual character is completely consistent with the generated audio content.

[0028] By introducing the edge-tts algorithm, the system can efficiently convert text to speech, ensuring that the generated virtual character is not only visually realistic, but also conveys content through sound, thereby improving the overall user experience.

[0029] Preferably, the face renderer and appearance encoder based on the motion coefficient and the first audio signal generate a first digital person, comprising:

[0030] C1: extracting motion coefficients and first audio signals;

[0031] C2: the face renderer maps the motion coefficients to a specific 3D key point space, and combines the appearance encoder to generate a plurality of video frames;

[0032] C3: synchronizing the plurality of video frames with the first audio signal to generate a first digital person containing dynamic expressions, head postures and synchronized audio.

[0033] The face renderer maps the motion coefficients to a specific 3D key point space, and combines the appearance encoder to generate a plurality of video frames, synchronizes the plurality of video frames with the audio signal to generate a first digital person containing dynamic expressions, head postures and synchronized audio, so that the obtained digital person not only visually shows a high degree of authenticity, but also realizes an immersive experience of audio-visual integration through the synchronized output of audio, which is beneficial to improve the experience of user interaction with the digital person.

[0034] Preferably, the IP_LAP algorithm-based processing of the second audio signal and the first video obtains a key point sequence, including:

[0035] A1: extracting the second audio signal and the first video;

[0036] A2: extracting audio features of the second audio signal by a Mel-spectrogram algorithm f a = Mel-Spectrogram(a) ; wherein a represents the input second audio signal; f a is a Mel-spectrogram feature vector, used to represent the spectral information of the second audio signal;

[0037] A3: extracting corresponding embedding vectors from the key points of the pose prior and the reference image according to a pose encoder Φ P and a reference encoder Φ R :

[0038] f p = Φ P (P prior ) ; f r = Φ R (r ref ) ;

[0039] wherein P prior is a pose prior, i.e. head pose information in the initial video, usually represented by key points, r ref is the key point information of the reference image; f p is an embedding vector of the pose prior, extracted by the pose encoder Φ P ; f r is an embedding vector of the reference image, extracted by the reference encoder Φ R ;

[0040] A4: adding f a , f p , f r to the position encoding and the modal encoding, and inputting them into a multi-layer Transformer module to capture the complex relationship among them through a self-attention mechanism, to generate the final mouth and lower jaw key point sequence k:

[0041] k = Transformer(f a + f p + f r ) ;

[0042] wherein k represents the generated mouth and lower jaw key point sequence; Transformer() represents a multi-layer Transformer model;

[0043] A5: loss function optimization by reconstruction loss and continuity regularization loss:

[0044]

[0045] where k gt represents the actual mouth and jaw keypoint sequence, is the reconstruction loss, used to minimize the error between the generated keypoint sequence and the true keypoint, is the continuity regularization loss, used to ensure the smoothness of the keypoint sequence in time and reduce the discontinuous changes between frames.

[0046] Preferably, the IP_LAP-based algorithm renders the keypoint sequence into a second video, including:

[0047] M1: align the reference image with the pose and expression of the target face through the alignment module, and the alignment process is:

[0048] Ι aligned = Align(Ι ref ,f p );

[0049] where Ι aligned represents the aligned reference image, Ι ref represents the input reference image, and Align() is the alignment module using the pose-embedded vector f p to transform the reference image;

[0050] M2: input the keypoint sequence into the sketch generation network to obtain the target face sketch:

[0051] S1 = SketchNet(k);

[0052] where S1 represents the generated target face sketch, and SketchNet() represents the sketch generation network;

[0053] M3: use the spatial adaptive normalization layer and adaptive instance normalization layer to fuse the aligned reference image and the target face sketch to generate the final face image Ι gen :

[0054] Ι gen = SPADE(S1, Ι aligned );

[0055] where Ι gen represents the generated face image; and SPADE() represents the spatial adaptive normalization layer, which is used to fuse the sketch and the reference image features;

[0056] M4: optimizing the generated facial image in the rendering process by perceptual loss and adversarial loss:

[0057]

[0058] wherein, represents the i-th layer of the pre-trained convolutional network; I gt represents the real facial image; is a perceptual loss, used to measure the difference between the generated facial image and the real facial image in the high-level feature space; is an adversarial loss, used to train the generator to generate images that can deceive the discriminator.

[0059] Preferably, the edge-tts algorithm based on the text content converts the third audio signal, comprising:

[0060] The edge-tts algorithm extracts semantic information from the text content through a natural language processing (NLP) module, generates a third audio signal according to the semantic information, and synchronizes the third audio signal with the mouth and jaw movement generated in the video rendering stage.

[0061] Preferably, the edge-tts algorithm based on the translated text converts the corresponding target audio, comprising:

[0062] The duration of the translated text and the original video is extracted, the translated text is converted into target audio through the edge-tts algorithm, and the playback rate of the target audio is adjusted according to the duration of the original video, so that the target audio is synchronized with the playback rate of the original video.

[0063] The second aspect of the present application provides a multi-modal digital human generation method, comprising:

[0064] S1: pre-processing the original image based on the SadTalker algorithm to obtain a reference image; processing the reference image based on the 3DMM algorithm to generate a 3D face model; extracting and generating motion coefficients from the 3D face model based on the sadtalker algorithm; converting the input text into a first audio signal based on the edge-tts algorithm; generating a first digital human based on the motion coefficients and the first audio signal based on the face renderer and the appearance encoder;

[0065] S2: processing the second audio signal and the first video based on the IP_LAP algorithm to obtain a key point sequence; rendering the key point sequence into a second video based on the IP_LAP algorithm; converting the text content into a third audio signal based on the edge-tts algorithm; and synthesizing the third audio signal with the second video to obtain a second digital human;

[0066] S3: Extract the audio of the specified duration from the original video using the FFmpeg tool and label it as the original audio; input the original audio into the speech recognition model to generate the corresponding text information; translate the text information based on the translation model to obtain the translated text; convert the translated text into the corresponding target audio based on the edge-tts algorithm, and synthesize the target audio with the original video into a translated video.

[0067] Compared with the prior art, the beneficial effects of the present application are:

[0068] The present application integrates the picture-generated digital human module, the video-generated digital human module, and the video translation module to establish an efficient and highly integrated digital human generation and translation system, which fully utilizes the latest deep learning algorithms and audio and video processing technologies in the three modules.

[0069] The picture-generated digital human module realizes the generation of dynamic 3D face from static images and voice synchronization by combining the SadTalker algorithm with the edge-tts algorithm.

[0070] The video-generated digital human module generates dynamic face video highly matched with the second video through audio driving using the IP_LAP algorithm, and supports multi-language translation and lip synchronization functions.

[0071] The video translation module not only realizes text generation and translation, but also synthesizes audio through edge-tts and voice cloning technology to ensure the consistency of translated voice and original video speaker voice.

[0072] The modular design of the overall system provides high scalability and flexibility, which can efficiently handle complex multi-language video generation and translation tasks, and significantly improves the practicality and user experience of digital human applications. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0074] Figure 1 The overall flowchart of the multi-modal digital human generation method of the present application;

[0075] Figure 2 The principle schematic diagram of the multi-modal digital human generation system of the present application;

[0076] Figure 3 The working flowchart of the picture-generated digital human module in the present application;

[0077] Figure 4 Workflow diagram of the video generation digital human module in the present application;

[0078] Figure 5 Workflow diagram of the video translation module in the present application. DETAILED DESCRIPTION

[0079] The technical solutions of the present application will be described clearly and completely below in conjunction with embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0080] Please refer to Figures 1-5 The first aspect embodiment of the present application provides a multi-modal digital human generation system, comprising: a picture generation digital human module, a video generation digital human module and a video translation module.

[0081] The picture generation digital human module is configured to: preprocess an original image based on a SadTalker algorithm to obtain a reference image; process the reference image based on a 3DMM algorithm to obtain a 3D face model; extract and generate motion coefficients from the 3D face model based on the SadTalker algorithm; convert input text into a first audio signal based on an edge-tts algorithm; generate a first digital human from the motion coefficients and the first audio signal based on a face renderer and an appearance encoder; wherein the preprocessing of the original image comprises adjusting the resolution, size and format of the original image, the motion coefficients comprise facial expression coefficients and head posture coefficients, and the head posture coefficients comprise head rotation coefficients and head displacement coefficients.

[0082] The video generation digital human module is configured to: process a second audio signal and a first video based on an IP_LAP algorithm to obtain a key point sequence; render the key point sequence into a second video based on the IP_LAP algorithm; convert text content into a third audio signal based on an edge-tts algorithm; synthesize the third audio signal and the second video to obtain a second digital human; wherein the key point sequence comprises key points of the mouth and the lower jaw.

[0083] The video translation module is configured to: extract audio of a specified time length from the original video by using an FFmpeg tool and mark the audio as original audio; input the original audio into a speech recognition model to generate corresponding text information; translate the text information based on a translation model to obtain translated text; convert the translated text into corresponding target audio based on an edge-tts algorithm; adjust the timbre of the target audio to the timbre of a speaker in the original video by calling a timbre cloning API; and combine the target audio with the original video to generate a translated video.

[0084] In this embodiment, the preprocessing of the original image includes adjusting the resolution, size and format of the original image.

[0085] In this embodiment, the video translation module is in communication and / or electrical connection with the picture generation digital human module and the video generation digital human module, respectively.

[0086] In this embodiment, the 3D face model is generated by processing the reference image based on the 3DMM algorithm, including:

[0087] The reference image is extracted and input into a 3DMM (3D Morphable Model) algorithm, and the 3DMM algorithm calculates a face geometry S by a formula S = S avg + αU id + βU exp wherein S avg is an average shape of a face, U id is an orthogonal basis of identity, and U exp is an orthogonal basis of expression, α is an identity coefficient greater than 0, and β is an expression coefficient greater than 0.

[0088] A head pose coefficient is obtained: r ∈ SO (3), t ∈ R 3 and a face pose feature is constructed; wherein r represents a head rotation matrix, t represents a head displacement vector, SO (3) is a three-dimensional special orthogonal group, and R 3 is a three-dimensional Euclidean space.

[0089] The face geometry and the face pose feature are integrated into the 3D face model.

[0090] The 3D face model is generated from the reference image by the 3DMM technology, so that the 3D face model can not only maintain the identity characteristics of the character, but also generate natural dynamic expressions, which is conducive to improving the interactive experience between the subsequently generated digital human and the user.

[0091] In this embodiment, the sadtalker algorithm is used to extract and generate motion coefficients from the 3D face model, including:

[0092] The 3D face model is extracted, and the corresponding motion coefficients are extracted and generated through the ExpNet module and the PoseVAE module in the sadtalker algorithm:

[0093] The ExpNet module generates the facial expression coefficient β by formula β 1,…,T =Φ M (Φ A (a 1,…,T ),z blink ,β0) analysis of audio signals 1,…,T ; wherein, Φ A is an audio encoder, Φ M is a mapping network, β0 is the initial expression coefficient of the reference image, z blink is a signal for controlling blinking, and T is the total number of time frames of the audio signal.

[0094] The PoseVAE module is constructed based on the variational autoencoder framework, and the continuous head posture coefficient is generated by learning the mapping relationship between the audio features and the head motion.

[0095] The ExpNet module in the sadtalker algorithm is used to analyze the audio signal to obtain the facial expression coefficient, and the PoseVAE module in the sadtalker algorithm is used to extract and generate the head posture coefficient, so that the generated head posture can not only maintain the identity characteristics of the character, but also show natural dynamic expression changes, so that the head posture of the virtual character in the generated video is coordinated with the rhythm and content of the input audio, thereby enhancing the authenticity of visual performance.

[0096] In this embodiment, the input text is converted into a first audio signal based on the edge-tts algorithm, which includes:

[0097] The edge-tts algorithm processes the input text through a natural language processing (NLP) module to extract semantic information, and the edge-tts algorithm generates a first audio signal according to the semantic information.

[0098] It should be noted that the audio signal will be synchronized with the generated animation frame sequence in the video synthesis stage, so that the lip movement of the virtual character is completely consistent with the generated audio content.

[0099] The edge-tts algorithm is introduced, and the system can realize efficient conversion from text to speech, so that the generated virtual character is not only visually realistic, but also can convey the content through the sound, thereby improving the overall user experience.

[0100] In this embodiment, the motion coefficients and the first audio signal are generated into a first digital person based on a face renderer and an appearance encoder, which includes:

[0101] C1: extracting motion coefficients and a first audio signal;

[0102] C2: a face renderer maps the motion coefficients to a specific 3D keypoint space, in combination with an appearance encoder to generate a number of video frames;

[0103] C3: synchronizing the number of video frames with the first audio signal to generate a first digital human containing dynamic expressions, head poses and synchronized audio.

[0104] The present application maps the motion coefficients to a specific 3D keypoint space through the face renderer, in combination with the appearance encoder to generate a number of video frames, synchronizes the number of video frames with the audio signal to generate a first digital human containing dynamic expressions, head poses and synchronized audio, so that the obtained digital human not only visually presents high authenticity, but also realizes immersive experience of audio-visual integration through the synchronized output of audio, which is beneficial to improve the experience of user interaction with the digital human.

[0105] In the embodiment, the second audio signal and the first video are processed based on the IP_LAP algorithm to obtain a keypoint sequence, including:

[0106] A1: extracting a second audio signal and a first video;

[0107] A2: extracting audio features of the second audio signal through a Mel-spectrogram algorithm f a =Mel-Spectrogram(a); wherein a represents the input second audio signal; f a is a Mel-spectrogram feature vector, used to represent the spectral information of the second audio signal;

[0108] A3: extracting corresponding embedding vectors from the key points of the pose prior and the reference image respectively according to a pose encoder Φ P and a reference encoder Φ R :

[0109] f p =Φ P (P prior );f r =Φ R (r ref );

[0110] wherein P prior is a pose prior, i.e. head pose information in the initial video, usually represented by key points, r ref is the key point information of the reference image; f p is an embedding vector of the pose prior, extracted by the pose encoder Φ P ; f ris the embedding vector of the reference image, which is generated by the reference encoder Φ R extracted;

[0111] A4: f a , f p , f r After adding the position encoding and the modal encoding, input into the multi-layer Transformer module, capture the complex relationship between them through the self-attention mechanism, generate the final mouth and mandible key point sequence k:

[0112] k = Transformer(f a + f p + f r );

[0113] Wherein, k represents the generated mouth and mandible key point sequence; Transformer() represents the multi-layer Transformer model;

[0114] A5: loss function optimization is carried out through reconstruction loss and continuity regularization loss:

[0115]

[0116] Wherein, k gt represents the actual mouth and mandible key point sequence, is the reconstruction loss, which is used to minimize the error between the generated key point sequence and the real key point, is the continuity regularization loss, which is used to ensure the smoothness of the key point sequence in time and reduce the discontinuous change between frames.

[0117] It's important to note that positional encoding encodes each position in the sequence using sine and cosine functions, aiming to introduce sequential information into the model. This allows the model to identify the relative positions of features in the input sequence, helping to capture temporal dynamics in audio, pose, and reference image features. This encoding method is widely used in Transformer models, enabling them to process time-series inputs and preserve and utilize the sequence and temporal dependencies of the input data. Modal encoding distinguishes input signals from different sources, such as audio, pose, and reference image features. Each type of input feature has an independent modal encoding vector, ensuring the model correctly understands the source and features of different modal inputs. Through modal encoding, Transformer models can combine information from multiple sources when generating videos, achieving more realistic facial animation and expression synchronization. Transformer models use multi-head self-attention and multilayer perceptrons (MLPs) to capture the complex relationships between different input features, ensuring that audio-driven facial animations generate realistic dynamic performances and speech synchronization effects.

[0118] In this embodiment, the keypoint sequence is rendered into a second video based on the IP_LAP algorithm, including:

[0119] M1: Aligns the reference image with the pose and expression of the target face using the alignment module. The alignment process is as follows:

[0120] Ι aligned =Align(Ι ref ,f p );

[0121] Among them, I aligned Indicates the aligned reference image, I ref The input reference image is represented by Align(), which is the alignment module that uses the pose prior embedding vector f. p Transform the reference image;

[0122] M2: Input the keypoint sequence into the sketch generation network to obtain the target facial sketch:

[0123] S1 = SketchNey(k);

[0124] Where S1 represents the generated target facial sketch, and SketchNet() represents the sketch generation network;

[0125] M3: The aligned reference image is fused with the target facial sketch using a spatial adaptive normalization layer and an adaptive instance normalization layer to generate the final facial image. gen :

[0126] Ι gen =SPADE(S1,Ι aligned );

[0127] Among them, I gen This represents the generated facial image; SPADE() represents a spatial adaptive normalization layer used to fuse sketch and reference image features.

[0128] M4: Optimizes the generated facial images during rendering using perceptual loss and adversarial loss.

[0129]

[0130] in, I represents the i-th layer of a pre-trained convolutional network; gt Represents a real facial image; Perceptual loss is used to measure the difference between the generated facial image and the real facial image in the high-level feature space. To combat the loss, the generator is trained so that the images it generates can fool the discriminator.

[0131] In this embodiment, the text content is converted into a third audio signal based on the edge-tts algorithm, including:

[0132] The edge-tts algorithm extracts semantic information from text content through a natural language processing (NLP) module, generates a third audio signal based on the semantic information, and synchronizes the third audio signal with the mouth and jaw movements generated during the video rendering stage.

[0133] In this embodiment, the translated text is converted into the corresponding target audio based on the edge-tts algorithm, including:

[0134] The translation text and the duration of the original video are extracted. The translation text is converted into target audio using the edge-tts algorithm. The playback rate of the target audio is adjusted according to the duration of the original video to synchronize the playback rate of the target audio with that of the original video.

[0135] A second aspect of the present invention provides a method for generating a multimodal digital human, comprising:

[0136] S1: Preprocess the original image based on the SadTalker algorithm to obtain a reference image; process the reference image based on the 3DMM algorithm to generate a 3D facial model; extract and generate motion coefficients from the 3D facial model based on the SadTalker algorithm; convert the input text into a first audio signal based on the edge-tts algorithm; generate a first digital human based on the motion coefficients and the first audio signal using a face renderer and appearance encoder.

[0137] S2: Process the second audio signal and the first video based on the IP_LAP algorithm to obtain a key point sequence; render the key point sequence into a second video based on the IP_LAP algorithm; convert the text content into a third audio signal based on the edge-tts algorithm; and synthesize the third audio signal with the second video to obtain a second digital person;

[0138] S3: Extract the audio of a specified duration from the original video and mark it as the original audio through the FFmpeg tool; input the original audio into a speech recognition model to generate corresponding text information; translate the text information based on a translation model to obtain translated text; and convert the translated text into corresponding target audio based on the edge-tts algorithm, and synthesize the target audio with the original video into a translated video.

[0139] Embodiment 1

[0140] Picture-generated digital person module:

[0141] Application scenario: virtual conference assistant

[0142] A certain enterprise hopes to add a virtual assistant to its remote conference platform to improve the interactivity and sense of participation of the conference. The assistant will generate a virtual image based on the personal photo of the employee and can speak and interact according to the preset content.

[0143] Specific implementation example:

[0144] Image input:

[0145] The employee uploads a personal front-facing photo to the system, ensuring that the facial features in the photo are clear.

[0146] 3D model generation:

[0147] The system automatically processes the input image and directly generates the employee's 3D face model using the SadTalker algorithm. The algorithm internally includes image processing steps such as face region recognition and feature extraction, thereby generating a 3D model highly consistent with the employee's appearance.

[0148] Expression and head movement generation:

[0149] The system automatically adds natural expressions and head movements to the virtual assistant while generating the 3D model. These expressions and movements are automatically generated based on the input text content and conference environment, ensuring that the virtual assistant's performance is natural and lively.

[0150] Speech synthesis:

[0151] The employee or administrator inputs the text content that needs to be expressed by the virtual assistant, and the system converts the text into speech through the edge-tts algorithm. The generated speech is synchronized with the 3D model, ensuring that the virtual assistant's lip movements match the speech content.

[0152] The virtual assistant generates:

[0153] Finally, the system generates a complete virtual assistant video, and the virtual assistant can participate in the meeting according to the preset content, make speeches or responses, and interact with actual participants.

[0154] In summary, through this process, this embodiment realizes the full automatic generation process from employee photos to virtual assistants, solves the problem of lack of personalized interaction in remote meetings, and improves the participation and efficiency of employees.

[0155] Embodiment 2

[0156] Video generation digital human module:

[0157] Application scenario: virtual lecturer in enterprise training

[0158] A certain enterprise hopes to introduce a virtual lecturer for its internal training courses to provide multilingual teaching services, thereby covering employees around the world and improving the accessibility and consistency of training content.

[0159] Specific implementation example:

[0160] Video input:

[0161] The training department of the enterprise records a video containing training content, and the facial expressions and lip movements of the lecturer in the video are closely related to the explanation content. This video is uploaded to the system as the basic material for generating a virtual lecturer.

[0162] Key point generation:

[0163] The system automatically extracts the key points of the lecturer's mouth and lower jaw from the video through the IP_LAP algorithm. This algorithm directly processes video frames to generate a dynamic face model synchronized with the audio. The generated model accurately reflects the lecturer's facial expressions and lip movement in the video.

[0164] Multilingual text generation:

[0165] The system performs speech recognition on the audio in the video to generate the text of the training content. Then, the system translates the text into the target language, such as Spanish, French, etc., through a translation model, ensuring that the training content can be provided in multiple languages.

[0166] Multilingual speech synthesis:

[0167] The translated text is converted into voice in the corresponding language through the edge-tts algorithm. The system automatically adjusts the rhythm and intonation of the voice according to the characteristics of different languages, ensuring consistency with the original video content. The generated voice has a natural and smooth pronunciation effect, suitable for employees of different language backgrounds around the world to understand and learn.

[0168] Multilingual virtual instructor generation:

[0169] The system synchronizes the generated voice with the 3D model of the virtual instructor, generating a multilingual version of the virtual instructor video. These videos can be directly published on the enterprise's online training platform, allowing employees to choose their familiar language for learning, thereby improving learning efficiency and content absorption effect.

[0170] In summary, through the above process, this embodiment realizes the multilingual generation of virtual instructors in enterprise training, solves the consistency and coverage problems in multilingual training within enterprises, and significantly improves the international accessibility of training content and the learning experience of employees.

[0171] Embodiment 3

[0172] Video translation module:

[0173] Application scenario: international market advertisement generation

[0174] A global enterprise plans to launch a new product and hopes to quickly translate the product advertisement video into multiple language versions for market promotion in different countries.

[0175] Specific implementation example:

[0176] Advertisement video input:

[0177] The marketing department provides a product advertisement video that has been recorded, which contains detailed product introduction and explanation.

[0178] Audio separation and text generation:

[0179] The system automatically separates the audio track from the advertisement video and uses speech recognition technology to convert the audio content into text.

[0180] Text translation:

[0181] The system translates the generated text into multiple target languages through a translation model. The translation process is fully automated to ensure quick response to the needs of different markets.

[0182] Voice synthesis and timbre cloning:

[0183] The system generates translated voice signals through the edge-tts algorithm and uses timbre cloning technology to make the generated voice similar to the voice in the original advertisement, thereby maintaining the consistency of the advertisement.

[0184] Multilingual advertisement video generation:

[0185] The system synchronizes the generated voice with the advertisement video, generating a multilingual version of the advertisement video. The lip movements of the characters in the video match the translated voice, ensuring consistency in both visual and auditory aspects. The final advertisement video can be directly uploaded to advertisement platforms in various countries for market promotion.

[0186] Through these detailed embodiments, the present application can realize efficient digital human generation and video translation by users in different application scenarios, meet diversified market demands, and improve the dissemination effect of content and user experience.

[0187] In summary, the present embodiment effectively solves the problem of mismatch between voice and visual content in multilingual advertisement generation through an automated multilingual conversion and synchronization process, helping enterprises achieve consistent and efficient brand promotion in the global market.

[0188] The above embodiments are only used to illustrate the technical method of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A multi-modal digital human generation system, comprising: The picture generation digital human module, the video generation digital human module, and the video translation module are characterized in that The picture generation digital human module is configured to preprocess an original image to obtain a reference image, process the reference image to obtain a 3D face model, generate motion coefficients based on the 3D face model, and The input text is converted into a first audio signal, and the motion coefficients and the first audio signal are generated into a first digital human based on a face renderer and an appearance encoder, wherein the motion coefficients include facial expression coefficients and head posture coefficients, and the head posture coefficients include head rotation coefficients and head displacement coefficients. The video generation digital human module is configured to process a second audio signal and a first video to obtain a key point sequence, render the key point sequence into a second video, convert text content into a third audio signal, and synthesize the third audio signal and the second video into a second digital human, wherein the key point sequence includes key points of a mouth and a lower jaw. The video translation module is configured to extract an audio of a specified duration from an original video and mark it as an original audio, input the original audio into a speech recognition model to generate corresponding text information, translate the text information to obtain translated text, convert the translated text into corresponding target audio, adjust the timbre of the target audio to the timbre of a speaker in the original video, and synthesize the target audio and the original video into a translated video. The processing of the second audio signal and the first video to obtain the key point sequence includes: A1: extracting the second audio signal and the first video; A2: by Mel-spectrogram algorithm extracting an audio feature of the second audio signal; wherein a represents the input second audio signal; is a Mel-spectrogram feature vector, used to represent the spectral information of the second audio signal; A3: According to the pose encoder and the reference encoder extracting respective embedding vectors from the key points of the pose prior and the reference image, respectively: ； ； wherein, is a pose prior, i.e. head pose information in the initial video, typically represented by keypoints, is keypoint information of the reference image; is an embedding vector of the pose prior, extracted by a pose encoder ; is an embedding vector of the reference image, extracted by a reference encoder ; A4: The , , After adding the position encoding and modal encoding, input into the multi-layer Transformer module, capture the complex relationship between them through self-attention mechanism, generate the final mouth and mandible key point sequence k: ； wherein k represents the generated key point sequence of the mouth and the lower jaw, and Transformer() represents a multi-layer Transformer model; A5: loss function optimization is performed through a reconstruction loss and a continuity regularization loss: ； ； wherein, represents a sequence of key points of the actual mouth and lower jaw, is a reconstruction loss for minimizing the error between the generated sequence of key points and the real key points, is a continuity regularization loss for ensuring the smoothness of the sequence of key points in time and reducing the discontinuous changes between frames; The rendering of the key point sequence into the second video includes: M1: aligning a reference image with the posture and expression of a target face through an alignment module, and the alignment process is: ； wherein, represents the aligned reference image, represents the input reference image, Align() is an alignment module, and the embedding vector with pose prior transforming the reference image; M2: inputting the key point sequence into a sketch generation network to obtain a target face sketch: ； wherein S1 represents the generated target face sketch, and SketchNet() represents the sketch generation network; M3: fusing the aligned reference image and the target face sketch by using a spatial adaptive normalization layer and an adaptive instance normalization layer to generate a final face image : ； wherein, denotes the generated face image; SPADE() denotes a spatially adaptive normalization layer for fusing the sketch with the reference image features; M4: optimizing the generated face image through a perception loss and an adversarial loss during the rendering process: ； ； wherein, represents the i-th layer of the pre-trained convolutional network; represents a real face image; is a perceptual loss for measuring the difference between the generated face image and the real face image in a high-level feature space; is an adversarial loss for training the generator to generate images that can deceive the discriminator; The duration of the translated text and the original video is extracted, the translated text is converted into target audio through an edge-tts algorithm, and the playback rate of the target audio is adjusted according to the duration of the original video, so that the playback rate of the target audio is synchronized with that of the original video.

2. The multi-modal digital human generation system of claim 1, wherein, The preprocessing of the original image includes adjusting the resolution, size, and format of the original image.

3. The multi-modal digital human generation system of claim 1, wherein, The processing of the reference image to obtain the 3D face model includes: The reference image is extracted and input into a 3DMM algorithm, which calculates the face geometry by formula ; wherein, is the average shape of the face, is the orthogonal basis of the identity, is the orthogonal basis of the expression, is the identity coefficient greater than 0, is the expression coefficient greater than 0;​ Obtaining head posture coefficients: And construct facial posture features; wherein, r represents the head rotation matrix, t represents the head displacement vector, For three-dimensional special orthogonal group, For three-dimensional Euclidean space; Integrating face geometry and face posture features into the 3D face model.

4. The multi-modal digital human generation system of claim 1, wherein, The generation of the motion coefficients based on the 3D face model includes: Extracting the 3D face model, and extracting and generating corresponding motion coefficients through an ExpNet module and a PoseVAE module in a sadtalker algorithm: The ExpNet module is through the formula The facial expression coefficients are generated by analyzing the audio signal ; wherein is an audio encoder, is a mapping network, is an initial expression coefficient of a reference image, is a signal for controlling blinking, and T is the total number of time frames of the audio signal; The PoseVAE module is constructed based on a variational autoencoder framework, and learns a mapping relationship between audio features and head motion to generate continuous head pose coefficients.

5. The multi-modal digital human generation system of claim 1, wherein, The input text is converted into a first audio signal, including: The edge-tts algorithm processes the input text through a natural language processing module to extract semantic information, and generates a first audio signal according to the semantic information.

6. The multi-modal digital human generation system of claim 1, wherein, The motion coefficients and the first audio signal are generated into a first digital person based on a face renderer and an appearance encoder, including: C1: Extracting motion coefficients and a first audio signal; C2: The face renderer maps the motion coefficients to a specific 3D key point space, and combines the appearance encoder to generate a plurality of video frames; C3: Synchronizing the plurality of video frames with the first audio signal to generate a first digital person containing dynamic expressions, head poses and synchronous audio.

7. A multi-modal digital human generation method, based on a multi-modal digital human generation system of any one of claims 1-6, wherein, Including: S1: Preprocessing the original image to obtain a reference image; and processing the reference image to generate a 3D face model; extracting and generating motion coefficients from the 3D face model; converting the input text into a first audio signal; generating a first digital person based on the motion coefficients and the first audio signal using a face renderer and an appearance encoder; S2: Processing the second audio signal and the first video to obtain a key point sequence; rendering the key point sequence into a second video; converting the text content into a third audio signal; and synthesizing the third audio signal and the second video to obtain a second digital person; S3: Extracting audio of a specified duration from the original video and marking it as original audio; inputting the original audio into a speech recognition model to generate corresponding text information; translating the text information to obtain translated text; Convert the translated text into the corresponding target audio, and synthesize the target audio with the original video into a translated video.

Images