Multi-modal digital human generation method and system

Abstract

The invention discloses a multi-modal digital human generation method and system. The system comprises a picture generation digital human module, a video generation digital human module and a video translation module, relates to the technical field of deep synthesis, and solves the technical problem of low expansibility and flexibility of a digital human generation mode in the prior art. The method comprises the following steps: processing a reference image to generate a 3D face model; generating a first digital human from the motion coefficient and the first audio signal based on a face renderer and an appearance encoder; extracting an audio with a specified duration from the original video and marking the audio as an original audio; and converting the translated text into a corresponding target audio, and synthesizing the target audio and the original video into a translated video. According to the invention, the reference image is generated through the 3DMM technology to obtain the 3D face model, so that the 3D face model can maintain the identity characteristics of a person and generate a natural dynamic expression, and the interactive experience between the generated digital person and a user is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep synthesis, and specifically relates to a multimodal digital human generation method and system. Background Art

[0002] At present, the synthesis modes of virtual digital humans mainly include the following: synthesis from video and audio, synthesis from video and text. These synthesis modes are usually supported by artificial intelligence algorithms and deep learning models to generate realistic and highly interactive virtual characters.

[0003] 1. Video and audio synthesis: This mode uses algorithms to process and fuse sound and images by inputting video and audio data to generate a dynamic virtual character. This mode is widely used in scenarios such as virtual anchors and virtual customer service. The synthesis process involves emotional recognition of audio and precise control of facial expressions in the video to ensure that the sound and expression of the virtual character are synchronized, giving users a natural and smooth interactive experience.

[0004] 2. Video and text synthesis: This mode inputs text and converts it into speech, while matching the mouth shape and facial expressions of the characters in the video. The semantic understanding of text and natural language processing (NLP) technology are the core of this mode, enabling the virtual character to generate appropriate expressions and actions based on the text content, thus achieving more humane communication.

[0005] The current synthesis technology still has some shortcomings. For example, the voice of the digital human generated by the existing technology is not synchronized with the lip movements and the timbre is inconsistent with the original speaker, which brings obvious discomfort to the video viewers and creates a false feeling; these shortcomings limit the performance of virtual digital humans in more demanding application scenarios, such as scenarios that require precise expression of specific character characteristics.

[0006] The present invention proposes a multimodal digital human generation method and system to solve the above technical problems. Summary of the invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a multimodal digital human generation method and system, which are used to solve the problem that most of the digital human generation solutions in the prior art generate digital humans based on videos, and in some scenarios, users can only provide pictures and audios. The prior art is difficult to generate digital humans based on pictures and audios, resulting in low scalability and flexibility in the generation method of digital humans; in addition, the sound characteristics of digital humans generated by the prior art are quite different from those of real people, resulting in a poor interactive experience between users and digital humans.

[0008] To achieve the above-mentioned purpose, the first aspect of the present invention provides a multimodal digital human generation system, comprising: a picture generation digital human module, a video generation digital human module and a video translation module;

[0009] The image generation digital human module is used to pre-process the original image based on the SadTalker algorithm to obtain a reference image; process the reference image based on the 3DMM algorithm to generate a 3D facial model; extract and generate motion coefficients from the 3D facial model based on the sadtalker algorithm; and,

[0010] Converting the input text into a first audio signal based on the edge-tts algorithm; generating a first digital human from the motion coefficient and the first audio signal based on a face renderer and an appearance encoder; wherein the motion coefficient includes a facial expression coefficient and a head posture coefficient, and the head posture coefficient includes a head rotation coefficient and a head displacement coefficient;

[0011] The video generation digital human module is used to process the second audio signal and the first video based on the IP_LAP algorithm to obtain a key point sequence; render the key point sequence into a second video based on the IP_LAP algorithm; convert the text content into a third audio signal based on the edge-tts algorithm; synthesize the third audio signal with the second video to obtain a second digital human; wherein the key point sequence includes key points of the mouth and the mandible;

[0012] The video translation module is used to extract audio of a specified length from an original video through an FFmpeg tool and mark it as original audio; input the original audio into a speech recognition model to generate corresponding text information; translate the text information based on the translation model to obtain a translated text; convert the translated text into a corresponding target audio based on an edge-tts algorithm; adjust the timbre of the target audio to the timbre of the speaker in the original video by calling a timbre cloning API; and synthesize the target audio and the original video into a translated video.

[0013] Preferably, the preprocessing of the original image includes adjusting the resolution, size and format of the original image.

[0014] Preferably, the video translation module is in communication and / or electrically connected with the picture generation digital human module and the video generation digital human module respectively.

[0015] Preferably, the step of processing the reference image based on the 3DMM algorithm to generate a 3D facial model comprises:

[0016] Extract the reference image and input it into the 3DMM (3D Morphable Model) algorithm. The 3DMM algorithm uses the formula S = S avg +αU id+βU exp The facial geometry S is calculated; where S avg is the average face shape, U id is an orthogonal basis of identity, U exp is the orthogonal basis of expression, α is the identity coefficient greater than 0, and β is the expression coefficient greater than 0;

[0017] Get the head posture coefficient: r∈SO(3),t∈R 3 And construct facial posture features; where r represents the head rotation matrix, t represents the head displacement vector, SO(3) is a three-dimensional special orthogonal group, R 3 is a three-dimensional Euclidean space;

[0018] Integrate facial geometry and facial pose features into a 3D facial model.

[0019] The present invention generates a 3D facial model from a reference image through the 3DMM technology, so that the 3D facial model can not only maintain the identity characteristics of the person, but also generate natural dynamic expressions, which is conducive to improving the interactive experience between the subsequently generated digital human and the user.

[0020] Preferably, the step of extracting and generating motion coefficients from a 3D facial model based on a sadtalker algorithm comprises:

[0021] Extract the 3D facial model, and use the ExpNet module and PoseVAE module in the sadtalker algorithm to extract and generate the corresponding motion coefficients:

[0022] The ExpNet module is constructed by formula β 1,…,T =Φ M (Φ A (a 1,…,T ),z blink ,β0) analyzes the audio signal to generate the facial expression coefficient β 1,…,T ; where Φ A is the audio encoder, Φ M is the mapping network, β0 is the initial expression coefficient of the reference image, z blink To control the blinking signal, T is the total number of time frames of the audio signal;

[0023] The PoseVAE module is built on the variational autoencoder framework and generates continuous head posture coefficients by learning the mapping relationship between audio features and head movements.

[0024] The present invention obtains facial expression coefficients by analyzing audio signals using the ExpNet module in the sadtalker algorithm, and extracts and generates head posture coefficients through the PoseVAE module in the sadtalker algorithm, thereby ensuring that the generated head posture can both maintain the identity characteristics of the character and show natural dynamic expression changes, so that in the generated video, the head posture of the virtual character is coordinated with the rhythm and content of the input audio, thereby enhancing the authenticity of the visual expression.

[0025] Preferably, the step of converting the input text into the first audio signal based on the edge-tts algorithm includes:

[0026] The edge-tts algorithm processes the input text through a natural language processing (NLP) module to extract its semantic information, and the edge-tts algorithm generates a first audio signal based on the semantic information.

[0027] It should be noted that the audio signal will be synchronized with the generated animation frame sequence during the video synthesis stage, so that the lip movements of the virtual character are completely consistent with the generated audio content.

[0028] By introducing the edge-tts algorithm, the system can achieve efficient conversion from text to speech, ensuring that the generated virtual characters are not only visually realistic but also able to convey content through sound, thereby improving the overall user experience.

[0029] Preferably, the step of generating the first digital human from the motion coefficient and the first audio signal based on the face renderer and the appearance encoder comprises:

[0030] C1: extract motion coefficients and the first audio signal;

[0031] C2: The face renderer maps the motion coefficients to a specific 3D key point space and combines it with the appearance encoder to generate several video frames;

[0032] C3: Synchronize a plurality of video frames with a first audio signal to generate a first digital human including dynamic expressions, head postures and synchronized audio.

[0033] The present invention maps motion coefficients to a specific 3D key point space through a face renderer, generates a plurality of video frames in combination with an appearance encoder, synchronizes the plurality of video frames with audio signals, and generates a first digital human including dynamic expressions, head postures, and synchronized audio, so that the obtained digital human not only exhibits a high degree of authenticity visually, but also realizes an immersive audio-visual experience through the synchronized output of audio, which is conducive to improving the user's experience of interacting with the digital human.

[0034] Preferably, the step of processing the second audio signal and the first video based on the IP_LAP algorithm to obtain a key point sequence includes:

[0035] A1: extracting the second audio signal and the first video;

[0036] A2: Through the Mel-spectrogram algorithm f a =Mel-Spectrogram(a) extracts the audio features of the second audio signal; wherein a represents the input second audio signal; f a is a Mel-spectrogram feature vector, used to represent the spectrum information of the second audio signal;

[0037] A3: According to the attitude encoder Φ P and reference encoder Φ R Extract the corresponding embedding vectors from the key points of the pose prior and the reference image respectively:

[0038] f p =Φ P (P prior );f r =Φ R (r ref );

[0039] Among them, P prior is the posture prior, i.e., the head posture information in the initial video, usually represented by key points, r ref is the key point information of the reference image; f p is the embedding vector of the pose prior, which is obtained by the pose encoder Φ P Extraction; r is the embedding vector of the reference image, obtained by the reference encoder Φ R extract;

[0040] A4: Put f a 、f p 、f r After being added to the positional encoding and modality encoding, it is input into the multi-layer Transformer module, and the complex relationship between them is captured through the self-attention mechanism to generate the final mouth and mandibular key point sequence k:

[0041] k=Transformer(f a +f p +f r );

[0042] Where k represents the generated sequence of key points of the mouth and jaw; Transformer() represents a multi-layer Transformer model;

[0043] A5: Loss function optimization through reconstruction loss and continuity regularization loss:

[0044]

[0045] Among them, k gt A sequence of key points representing the actual mouth and jaw, is the reconstruction loss, which is used to minimize the error between the generated key point sequence and the true key point. It is a continuity regularization loss, which is used to ensure the smoothness of the key point sequence in time and reduce discontinuous changes between frames.

[0046] Preferably, the rendering of the key point sequence into the second video based on the IP_LAP algorithm comprises:

[0047] M1: Align the reference image with the target face’s pose and expression through the alignment module. The alignment process is:

[0048] Ι aligned =Align(Ι ref ,f p );

[0049] Among them, aligned represents the aligned reference image, Ι ref represents the input reference image, Align() is the alignment module, using the posture prior embedding vector f p Transforming the reference image;

[0050] M2: Input the key point sequence into the sketch generation network to obtain the target facial sketch:

[0051] S1 = SketchNey(k);

[0052] Among them, S1 represents the generated target face sketch, SketchNet() represents the sketch generation network;

[0053] M3: Use the spatial adaptive normalization layer and the adaptive instance normalization layer to fuse the aligned reference image with the target face sketch to generate the final face image Ι gen :

[0054] Ι gen =SPADE(S1,Ι aligned );

[0055] Among them, gen represents the generated facial image; SPADE() represents the spatial adaptive normalization layer used to fuse the sketch with the reference image features;

[0056] M4: Optimizing generated facial images by perceptual loss and adversarial loss during rendering:

[0057]

[0058] in, Represents the i-th layer of the pre-trained convolutional network; I gt Represents a real face image; is the perceptual loss, which is used to measure the difference between the generated facial image and the real facial image in the high-level feature space; The adversarial loss is used to train the generator so that the images it generates can deceive the discriminator.

[0059] Preferably, the step of converting the text content into a third audio signal based on the edge-tts algorithm includes:

[0060] The EDGE-TTS algorithm extracts semantic information from text content through a natural language processing (NLP) module, generates a third audio signal based on the semantic information, and synchronizes the third audio signal with the mouth and jaw movements generated in the video rendering stage.

[0061] Preferably, the step of converting the translated text into the corresponding target audio based on the edge-tts algorithm includes:

[0062] The duration of the translated text and the original video is extracted, the translated text is converted into the target audio through the edge-tts algorithm, and the playback rate of the target audio is adjusted according to the duration of the original video so that the playback rate of the target audio is synchronized with that of the original video.

[0063] A second aspect of the present invention provides a multimodal digital human generation method, comprising:

[0064] S1: Preprocess the original image based on the SadTalker algorithm to obtain a reference image; process the reference image based on the 3DMM algorithm to generate a 3D facial model; extract and generate motion coefficients from the 3D facial model based on the sadtalker algorithm; convert the input text into a first audio signal based on the edge-tts algorithm; generate a first digital human from the motion coefficients and the first audio signal based on the face renderer and the appearance encoder;

[0065] S2: Processing the second audio signal and the first video based on the IP_LAP algorithm to obtain a key point sequence; rendering the key point sequence into a second video based on the IP_LAP algorithm; converting the text content into a third audio signal based on the edge-tts algorithm; synthesizing the third audio signal with the second video to obtain a second digital human;

[0066] S3: Use the FFmpeg tool to extract audio of a specified length from the original video and mark it as the original audio; input the original audio into the speech recognition model to generate corresponding text information; translate the text information based on the translation model to obtain the translated text; convert the translated text into the corresponding target audio based on the edge-tts algorithm, and synthesize the target audio and the original video into a translated video.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] The present invention establishes an efficient and highly integrated digital human generation and translation system by integrating the picture generation digital human module, the video generation digital human module and the video translation module. The system fully utilizes the latest deep learning algorithms and audio and video processing technologies in the three modules.

[0069] The image-generated digital human module combines the SadTalker algorithm with the edge-tts algorithm to achieve the generation and voice synchronization from static images to dynamic 3D faces.

[0070] The video generation digital human module uses the IP_LAP algorithm to generate dynamic face videos that are highly matched with the second video through audio drive, and supports multi-language translation and lip synchronization functions.

[0071] The video translation module not only realizes text generation and translation, but also synthesizes audio through edge-tts and timbre cloning technology to ensure the consistency of the translated voice with the timbre of the original video speaker.

[0072] The modular design of the overall system provides a high degree of scalability and flexibility, and can efficiently handle complex multi-language video generation and translation tasks, significantly improving the practicality and user experience of digital human applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 It is an overall flow chart of the multimodal digital human generation method of the present invention;

[0075] Figure 2 It is a schematic diagram of the principle of the multimodal digital human generation system of the present invention;

[0076] Figure 3 A flowchart of the digital human generation module from pictures in the present invention;

[0077] Figure 4 The flowchart of the video generation digital human module in the present invention;

[0078] Figure 5 It is a workflow diagram of the video translation module in the present invention. DETAILED DESCRIPTION

[0079] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] See also Figure 1-Figure 5 , the first aspect of the present invention provides a multimodal digital human generation system, including: a picture generation digital human module, a video generation digital human module and a video translation module;

[0081] Image generation digital human module: used for preprocessing the original image based on the SadTalker algorithm to obtain a reference image; processing the reference image based on the 3DMM algorithm to generate a 3D facial model; extracting and generating motion coefficients from the 3D facial model based on the sadtalker algorithm; converting the input text into a first audio signal based on the edge-tts algorithm; generating a first digital human from the motion coefficients and the first audio signal based on the face renderer and the appearance encoder; wherein the preprocessing of the original image includes adjusting the resolution, size and format of the original image, the motion coefficients include facial expression coefficients and head posture coefficients, and the head posture coefficients include head rotation coefficients and head displacement coefficients;

[0082] Video generation digital human module: used to process the second audio signal and the first video based on the IP_LAP algorithm to obtain a key point sequence; render the key point sequence into a second video based on the IP_LAP algorithm; convert the text content into a third audio signal based on the edge-tts algorithm; synthesize the third audio signal with the second video to obtain a second digital human; wherein the key point sequence includes key points of the mouth and the mandible;

[0083] Video translation module: used to extract audio of specified length from the original video through the FFmpeg tool and mark it as original audio; input the original audio into the speech recognition model to generate corresponding text information; translate the text information based on the translation model to obtain the translated text; convert the translated text into the corresponding target audio based on the edge-tts algorithm; adjust the timbre of the target audio to the timbre of the speaker in the original video by calling the timbre cloning API; synthesize the target audio and the original video into a translated video.

[0084] In this embodiment, preprocessing the original image includes adjusting the resolution, size and format of the original image.

[0085] In this embodiment, the video translation module communicates and / or is electrically connected to the picture generation digital human module and the video generation digital human module respectively.

[0086] In this embodiment, the reference image is processed based on the 3DMM algorithm to generate a 3D facial model, including:

[0087] Extract the reference image and input it into the 3DMM (3D Morphable Model) algorithm. The 3DMM algorithm uses the formula S = S avg +αU id +βU exp The facial geometry S is calculated; where S avg is the average face shape, U id is an orthogonal basis of identity, U exp is the orthogonal basis of expression, α is the identity coefficient greater than 0, and β is the expression coefficient greater than 0;

[0088] Get the head posture coefficient: r∈SO(3),t∈R 3 And construct facial posture features; where r represents the head rotation matrix, t represents the head displacement vector, SO(3) is a three-dimensional special orthogonal group, R 3 is a three-dimensional Euclidean space;

[0089] Integrate facial geometry and facial pose features into a 3D facial model.

[0090] The present invention generates a 3D facial model from a reference image through the 3DMM technology, so that the 3D facial model can not only maintain the identity characteristics of the person, but also generate natural dynamic expressions, which is conducive to improving the interactive experience between the subsequently generated digital human and the user.

[0091] In this embodiment, the motion coefficients are extracted and generated from the 3D facial model based on the sadtalker algorithm, including:

[0092] Extract the 3D facial model, and use the ExpNet module and PoseVAE module in the sadtalker algorithm to extract and generate the corresponding motion coefficients:

[0093] The ExpNet module is constructed by formula β 1,…,T =Φ M (Φ A (a 1,…,T ),z blink ,β0) analyzes the audio signal to generate the facial expression coefficient β 1,…,T ; where Φ A is the audio encoder, Φ M is the mapping network, β0 is the initial expression coefficient of the reference image, z blink To control the blinking signal, T is the total number of time frames of the audio signal;

[0094] The PoseVAE module is built on the variational autoencoder framework and generates continuous head posture coefficients by learning the mapping relationship between audio features and head movements.

[0095] The present invention obtains facial expression coefficients by analyzing audio signals using the ExpNet module in the sadtalker algorithm, and extracts and generates head posture coefficients through the PoseVAE module in the sadtalker algorithm, thereby ensuring that the generated head posture can both maintain the identity characteristics of the character and show natural dynamic expression changes, so that in the generated video, the head posture of the virtual character is coordinated with the rhythm and content of the input audio, thereby enhancing the authenticity of the visual expression.

[0096] In this embodiment, converting the input text into a first audio signal based on the edge-tts algorithm includes:

[0097] The edge-tts algorithm processes the input text through a natural language processing (NLP) module to extract its semantic information, and the edge-tts algorithm generates a first audio signal based on the semantic information.

[0098] It should be noted that the audio signal will be synchronized with the generated animation frame sequence during the video synthesis stage, so that the lip movements of the virtual character are completely consistent with the generated audio content.

[0099] By introducing the edge-tts algorithm, the system can achieve efficient conversion from text to speech, ensuring that the generated virtual characters are not only visually realistic but also able to convey content through sound, thereby improving the overall user experience.

[0100] In this embodiment, the first digital human is generated based on the motion coefficient and the first audio signal by the face renderer and the appearance encoder, including:

[0101] C1: extract motion coefficients and the first audio signal;

[0102] C2: The face renderer maps the motion coefficients to a specific 3D key point space and combines it with the appearance encoder to generate several video frames;

[0103] C3: Synchronize a plurality of video frames with a first audio signal to generate a first digital human including dynamic expressions, head postures and synchronized audio.

[0104] The present invention maps motion coefficients to a specific 3D key point space through a face renderer, generates a plurality of video frames in combination with an appearance encoder, synchronizes the plurality of video frames with audio signals, and generates a first digital human including dynamic expressions, head postures, and synchronized audio, so that the obtained digital human not only exhibits a high degree of authenticity visually, but also realizes an immersive audio-visual experience through the synchronized output of audio, which is conducive to improving the user's experience of interacting with the digital human.

[0105] In this embodiment, the second audio signal and the first video are processed based on the IP_LAP algorithm to obtain a key point sequence, including:

[0106] A1: extracting the second audio signal and the first video;

[0107] A2: Through the Mel-spectrogram algorithm f a =Mel-Spectrogram(a) extracts the audio features of the second audio signal; wherein a represents the input second audio signal; f a is a Mel-spectrogram feature vector, used to represent the spectrum information of the second audio signal;

[0108] A3: According to the attitude encoder Φ P and reference encoder Φ R Extract the corresponding embedding vectors from the key points of the pose prior and the reference image respectively:

[0109] f p =Φ P (P prior );f r =Φ R (r ref );

[0110] Among them, P prior is the posture prior, i.e., the head posture information in the initial video, usually represented by key points, r ref is the key point information of the reference image; f p is the embedding vector of the pose prior, which is obtained by the pose encoder Φ P Extraction; ris the embedding vector of the reference image, obtained by the reference encoder Φ R extract;

[0111] A4: Put f a 、f p 、f r After being added to the positional encoding and modality encoding, it is input into the multi-layer Transformer module, and the complex relationship between them is captured through the self-attention mechanism to generate the final mouth and mandibular key point sequence k:

[0112] k=Transformer(f a +f p +f r );

[0113] Where k represents the generated sequence of key points of the mouth and jaw; Transformer() represents a multi-layer Transformer model;

[0114] A5: Loss function optimization through reconstruction loss and continuity regularization loss:

[0115]

[0116] Among them, k gt A sequence of key points representing the actual mouth and jaw, is the reconstruction loss, which is used to minimize the error between the generated key point sequence and the true key point. It is a continuity regularization loss, which is used to ensure the smoothness of the key point sequence in time and reduce discontinuous changes between frames.

[0117] It should be noted that position encoding encodes each position in the sequence through sine and cosine functions, with the purpose of introducing the order information of the sequence into the model. In this way, the model can identify the relative position of features in the input sequence, helping to capture the temporal dynamic information in audio features, posture features, and reference image features; this encoding method is widely used in the Transformer model, enabling the model to process time series inputs, retaining and utilizing the order and temporal dependency of input data; modality encoding is used to distinguish input signals from different sources, such as audio features, posture features, and reference image features; each type of input feature has an independent modality encoding vector to ensure that the model can correctly understand the source and characteristics of different modal inputs; through modality encoding, the Transformer model can combine information from multiple sources when generating videos, and achieve more realistic facial animation and expression synchronization. The Transformer model captures the complex relationship between different input features through a multi-head self-attention mechanism and a multi-layer perceptron (MLP), ensuring that audio-driven facial animation can generate realistic dynamic performance and speech synchronization effects.

[0118] In this embodiment, rendering the key point sequence into the second video based on the IP_LAP algorithm includes:

[0119] M1: Align the reference image with the target face’s pose and expression through the alignment module. The alignment process is:

[0120] Ι aligned =Align(Ι ref ,f p );

[0121] Among them, aligned represents the aligned reference image, Ι ref represents the input reference image, Align() is the alignment module, using the posture prior embedding vector f p Transforming the reference image;

[0122] M2: Input the key point sequence into the sketch generation network to obtain the target facial sketch:

[0123] S1 = SketchNey(k);

[0124] Among them, S1 represents the generated target face sketch, SketchNet() represents the sketch generation network;

[0125] M3: Use the spatial adaptive normalization layer and the adaptive instance normalization layer to fuse the aligned reference image with the target face sketch to generate the final face image Ι gen :

[0126] Ι gen =SPADE(S1,Ι aligned );

[0127] Among them, gen represents the generated facial image; SPADE() represents the spatial adaptive normalization layer used to fuse the sketch with the reference image features;

[0128] M4: Optimizing generated facial images by perceptual loss and adversarial loss during rendering:

[0129]

[0130] in, Represents the i-th layer of the pre-trained convolutional network; I gt Represents a real face image; is the perceptual loss, which is used to measure the difference between the generated facial image and the real facial image in the high-level feature space; The adversarial loss is used to train the generator so that the images it generates can deceive the discriminator.

[0131] In this embodiment, converting the text content into a third audio signal based on the edge-tts algorithm includes:

[0132] The EDGE-TTS algorithm extracts semantic information from text content through a natural language processing (NLP) module, generates a third audio signal based on the semantic information, and synchronizes the third audio signal with the mouth and jaw movements generated in the video rendering stage.

[0133] In this embodiment, the translation text is converted into the corresponding target audio based on the edge-tts algorithm, including:

[0134] The duration of the translated text and the original video is extracted, the translated text is converted into the target audio through the edge-tts algorithm, and the playback rate of the target audio is adjusted according to the duration of the original video so that the playback rate of the target audio is synchronized with that of the original video.

[0135] A second aspect of the present invention provides a multimodal digital human generation method, comprising:

[0136] S1: Preprocess the original image based on the SadTalker algorithm to obtain a reference image; process the reference image based on the 3DMM algorithm to generate a 3D facial model; extract and generate motion coefficients from the 3D facial model based on the sadtalker algorithm; convert the input text into a first audio signal based on the edge-tts algorithm; generate a first digital human from the motion coefficients and the first audio signal based on the face renderer and the appearance encoder;

[0137] S2: Processing the second audio signal and the first video based on the IP_LAP algorithm to obtain a key point sequence; rendering the key point sequence into a second video based on the IP_LAP algorithm; converting the text content into a third audio signal based on the edge-tts algorithm; synthesizing the third audio signal with the second video to obtain a second digital human;

[0138] S3: Use the FFmpeg tool to extract audio of a specified length from the original video and mark it as the original audio; input the original audio into the speech recognition model to generate corresponding text information; translate the text information based on the translation model to obtain the translated text; convert the translated text into the corresponding target audio based on the edge-tts algorithm, and synthesize the target audio and the original video into a translated video.

[0139] Example 1

[0140] Image generation digital human module:

[0141] Application scenario: Virtual meeting assistant

[0142] A company wants to add a virtual assistant to its remote meeting platform to improve the interactivity and participation of meetings. The assistant will generate a virtual image based on the employee's personal photo and can speak and interact according to preset content.

[0143] Specific implementation use cases:

[0144] Image input:

[0145] Employees upload a personal front-facing photo into the system, ensuring that facial features are clear in the photo.

[0146] 3D model generation:

[0147] The system automatically processes the input image and directly generates a 3D facial model of the employee using the SadTalker algorithm. The algorithm includes image processing steps such as face region recognition and feature extraction to generate a 3D model that is highly consistent with the employee's appearance.

[0148] Expression and head movement generation:

[0149] While generating the 3D model, the system automatically adds natural expressions and head movements to the virtual assistant. These expressions and movements are automatically generated based on the input text content and the meeting environment, ensuring that the virtual assistant's performance is natural and vivid.

[0150] Speech Synthesis:

[0151] Employees or administrators input text content that needs to be expressed by the virtual assistant, and the system converts the text into speech through the edge-tts algorithm. The generated speech is synchronized with the 3D model to ensure that the virtual assistant's lip shape matches the speech content.

[0152] Virtual Assistant Generation:

[0153] Ultimately, the system generates a complete virtual assistant video, which can participate in the meeting, speak or respond according to the preset content, and interact with actual participants.

[0154] In summary, through this process, this embodiment realizes the fully automatic generation process from employee photos to virtual assistants, solves the problem of lack of personalized interaction in remote meetings, and improves employee participation and meeting efficiency.

[0155] Example 2

[0156] Video generation digital human module:

[0157] Application scenario: virtual instructor in corporate training

[0158] A company wants to introduce virtual instructors for its internal training courses to provide multilingual teaching services, thereby covering employees around the world and improving the accessibility and consistency of training content.

[0159] Specific implementation use cases:

[0160] Video Input:

[0161] The training department of the company recorded a video containing training content, in which the instructor's facial expressions and lip movements were closely related to the content of the lecture. The video was uploaded to the system as the basic material for generating the virtual instructor.

[0162] Key point generation:

[0163] The system automatically extracts the key points of the lecturer's mouth and jaw from the video through the IP_LAP algorithm. The algorithm directly processes the video frames and generates a dynamic face model synchronized with the audio. The generated model accurately reflects the lecturer's facial expressions and lip movements in the video.

[0164] Multilingual text generation:

[0165] The system performs speech recognition on the audio in the video and generates text for the training content. The system then uses a translation model to translate the text into the target language, such as Spanish, French, etc., to ensure that the training content can be provided in multiple languages.

[0166] Multilingual speech synthesis:

[0167] The translated text is converted into speech of the corresponding language through the edge-tts algorithm. The system automatically adjusts the rhythm and intonation of the speech according to the characteristics of different languages ​​to ensure consistency with the original video content. The generated speech has a natural and fluent pronunciation effect, which is suitable for employees with different language backgrounds around the world to understand and learn.

[0168] Multilingual virtual instructor generation:

[0169] The system synchronizes the generated voice with the 3D model of the virtual lecturer to generate multi-language versions of the virtual lecturer video. These videos can be directly published to the company's online training platform, and employees can choose the language they are familiar with to learn, thereby improving learning efficiency and content absorption.

[0170] In summary, this embodiment realizes multi-language generation of virtual instructors in enterprise training through the above process, solves the consistency and coverage problems in multi-language training within the enterprise, and significantly improves the international accessibility of training content and employees' learning experience.

[0171] Example 3

[0172] Video translation module:

[0173] Application scenario: International market advertising generation

[0174] A global enterprise plans to launch a new product and hopes to quickly translate the product advertising video into multiple languages ​​for promotion in different countries.

[0175] Specific implementation use cases:

[0176] Advertising video input:

[0177] The marketing department provides a pre-recorded product advertising video that includes detailed product introduction and explanation.

[0178] Audio Separation and Text Generation:

[0179] The system automatically separates the audio track from the ad video and uses speech recognition technology to convert the audio content into text.

[0180] Text Translation:

[0181] The system translates the generated text into multiple target languages ​​through the translation model. The translation process is fully automated, ensuring a quick response to the needs of different markets.

[0182] Speech synthesis and voice cloning:

[0183] The system generates the translated speech signal through the edge-tts algorithm and uses the timbre cloning technology to make the generated speech similar to the sound in the original advertisement, thus maintaining the consistency of the advertisement.

[0184] Multilingual Advertising Video Generation:

[0185] The system synchronizes the generated voice with the advertising video to generate a multilingual version of the advertising video. The lip movements of the characters in the video are consistent with the translated voice, ensuring the unity of vision and hearing. The final advertising video can be directly uploaded to advertising platforms in various countries for market promotion.

[0186] Through these detailed implementations, the present invention can enable users to efficiently generate digital humans and translate videos in different application scenarios, meet diverse market demands, and improve content dissemination effects and user experience.

[0187] In summary, this embodiment effectively solves the problem of mismatch between voice and visual content in multilingual advertisement generation through automated multilingual conversion and synchronization processes, helping enterprises achieve consistent and efficient brand promotion in the global market.

[0188] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multimodal digital human generation system, comprising: A picture generation digital human module, a video generation digital human module and a video translation module; characterized in that: The image generation digital human module is used to pre-process the original image to obtain a reference image; process the reference image to generate a 3D facial model; generate motion coefficients based on the 3D facial model; and, Converting input text into a first audio signal; generating a first digital human from motion coefficients and the first audio signal based on a face renderer and an appearance encoder; wherein the motion coefficients include facial expression coefficients and head posture coefficients, and the head posture coefficients include head rotation coefficients and head displacement coefficients; The video generation digital human module is used to process the second audio signal and the first video to obtain a key point sequence; render the key point sequence into a second video; convert the text content into a third audio signal; synthesize the third audio signal with the second video to obtain a second digital human; wherein the key point sequence includes key points of the mouth and the mandible; The video translation module is used to extract audio of a specified length from an original video and mark it as original audio; input the original audio into a speech recognition model to generate corresponding text information, and translate the text information to obtain a translated text; convert the translated text into a corresponding target audio; adjust the timbre of the target audio to the timbre of the speaker in the original video; and synthesize the target audio and the original video into a translated video.

2. A multimodal digital human generation system according to claim 1, characterized in that: The preprocessing of the original image includes adjusting the resolution, size and format of the original image.

3. A multimodal digital human generation system according to claim 1, characterized in that: The step of processing the reference image to generate a 3D facial model comprises: Extract the reference image and input the reference image into the 3DMM algorithm. The 3DMM algorithm uses the formula S=S avg +αU id +βU exp The facial geometry S is calculated; where S avg is the average face shape, U id is an orthogonal basis of identity, U exp is the orthogonal basis of expression, α is the identity coefficient greater than 0, and β is the expression coefficient greater than 0; Get the head posture coefficient: r∈SO(3), t∈R 3 And construct facial posture features; where r represents the head rotation matrix, t represents the head displacement vector, SO(3) is a three-dimensional special orthogonal group, R 3 is a three-dimensional Euclidean space; Integrate facial geometry and facial pose features into a 3D facial model.

4. A multimodal digital human generation system according to claim 1, characterized in that: The generating motion coefficient based on the 3D facial model comprises: Extract the 3D facial model, and use the ExpNet module and PoseVAE module in the sadtalker algorithm to extract and generate the corresponding motion coefficients: The ExpNet module is constructed by formula β 1，...，T =Φ M (Φ A (a 1，...，T ), z blink , β0) analyzes the audio signal to generate the facial expression coefficient β 1，...， T; where Φ A is the audio encoder, Φ M is the mapping network, β0 is the initial expression coefficient of the reference image, z blink To control the blinking signal, T is the total number of time frames of the audio signal; The PoseVAE module is built on the variational autoencoder framework and generates continuous head posture coefficients by learning the mapping relationship between audio features and head movements.

5. The multimodal digital human generation system according to claim 1, characterized in that: The step of converting the input text into a first audio signal comprises: The edge-tts algorithm processes the input text through a natural language processing module to extract its semantic information, and the edge-tts algorithm generates a first audio signal according to the semantic information.

6. A multimodal digital human generation system according to claim 1, characterized in that: The method generates a first digital human by using a motion coefficient and a first audio signal based on a face renderer and an appearance encoder, comprising: C1: extract motion coefficients and the first audio signal; C2: The face renderer maps the motion coefficients to a specific 3D key point space and combines it with the appearance encoder to generate several video frames; C3: Synchronize a plurality of video frames with a first audio signal to generate a first digital human including dynamic expressions, head postures and synchronized audio.

7. A multimodal digital human generation system according to claim 1, characterized in that: The step of processing the second audio signal and the first video to obtain a key point sequence includes: A1: extracting the second audio signal and the first video; A2: Through the Mel-spectrogram algorithm a =Mel-Spectrogram(a) extracts the audio features of the second audio signal; wherein a represents the input second audio signal; f a is a Mel-spectrogram feature vector, used to represent the spectrum information of the second audio signal; A3: According to the attitude encoder Φ P and reference encoder Φ R Extract the corresponding embedding vectors from the key points of the pose prior and the reference image respectively: f p =Φ P (P prior );f r =Φ R (r ref ); Among them, P prior is the posture prior, i.e., the head posture information in the initial video, usually represented by key points, r ref is the key point information of the reference image; f p is the embedding vector of the pose prior, which is obtained by the pose encoder Φ P Extraction; fr is the embedding vector of the reference image, obtained by the reference encoder Φ R extract; A4: Put f a 、f p 、f r After being added to the positional encoding and modality encoding, it is input into the multi-layer Transformer module, and the complex relationship between them is captured through the self-attention mechanism to generate the final mouth and mandibular key point sequence k: k=Transformer(f a +f p +f r ); Where k represents the generated sequence of key points of the mouth and jaw; Transformer() represents a multi-layer Transformer model; A5: Loss function optimization through reconstruction loss and continuity regularization loss: Among them, kgt represents the actual key point sequence of the mouth and mandible, is the reconstruction loss, which is used to minimize the error between the generated key point sequence and the true key point. It is a continuity regularization loss, which is used to ensure the smoothness of the key point sequence in time and reduce discontinuous changes between frames.

8. A multimodal digital human generation system according to claim 7, characterized in that: The step of rendering the key point sequence as a second video comprises: M1: Align the reference image with the target face’s pose and expression through the alignment module. The alignment process is: I aligned =Align(I ref ,f p ); Among them, I aligned represents the aligned reference image, I ref represents the input reference image, Align() is the alignment module, which transforms the reference image using the pose prior embedding vector fp; M2: Input the key point sequence into the sketch generation network to obtain the target facial sketch: S1 = SketchNey(k); Among them, S1 represents the generated target face sketch, SketchNet() represents the sketch generation network; M3: Use the spatial adaptive normalization layer and the adaptive instance normalization layer to fuse the aligned reference image with the target facial sketch to generate the final facial image Igen: I gen =SPADE(S1,I aligned ); Among them, I gen represents the generated facial image; SPADE() represents the spatial adaptive normalization layer used to fuse the sketch with the reference image features; M4: Optimizing generated facial images by perceptual loss and adversarial loss during rendering: in, represents the i-th layer of the pre-trained convolutional network; Igt represents the real facial image; is the perceptual loss, which is used to measure the difference between the generated facial image and the real facial image in the high-level feature space; The adversarial loss is used to train the generator so that the images it generates can deceive the discriminator.

9. The multimodal digital human generation system according to claim 1, characterized in that: The step of converting the translated text into the corresponding target audio includes: The duration of the translated text and the original video is extracted, the translated text is converted into the target audio through the edge-tts algorithm, and the playback rate of the target audio is adjusted according to the duration of the original video so that the playback rate of the target audio is synchronized with that of the original video.

10. A multimodal digital human generation method, based on the multimodal digital human generation system according to any one of claims 1 to 9, characterized in that: include: S1: preprocess the original image to obtain a reference image; The reference image is processed to generate a 3D facial model; motion coefficients are extracted and generated from the 3D facial model; input text is converted into a first audio signal; and a first digital human is generated from the motion coefficients and the first audio signal based on a face renderer and an appearance encoder; S2: Process the second audio signal and the first video to obtain a key point sequence; render the key point sequence into a second video; convert the text content into a third audio signal; synthesize the third audio signal with the second video to obtain a second digital human; S3: extract audio of a specified length from the original video and mark it as the original audio; input the original audio into the speech recognition model to generate corresponding text information; translate the text information to obtain the translated text; The translated text is converted into the corresponding target audio, and the target audio is combined with the original video into the translated video.

Images