A method and system for generating 3D digital humans with voice interaction

Abstract

This invention belongs to the field of 3D reconstruction technology and discloses a method and system for generating 3D digital humans with voice interaction. Based on the different languages ​​of the input target text and sampled human voice audio, this invention automatically generates new speaking audio in different languages. The use of multi-model joint estimation and a temporal loss function ensures the temporal stability and detail restoration capability of 3D human motion, facilitating accurate estimation of facial expression details and hand postures in the image. After obtaining the high-precision 3D human model, voice-driven human action and expression generation is performed, achieving precise synchronization between voice-generated actions and expressions, accurately generating facial expression movements and body posture movements that match the new speaking audio, i.e., a full-body 3D human model. Finally, a 3D neural rendering model is used to render the full-body 3D human model into a voice-interactive, realistic digital human. This invention facilitates the generation of high-precision 3D digital humans with voice interaction from a single human image input.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology and relates to a method and system for generating 3D digital humans with voice interaction. Background Technology

[0002] Digital human generation is a research hotspot in the fields of metaverse and virtual reality, with broad application prospects in game development, filmmaking, virtual try-on, virtual character design, and advertising. The goal of digital human generation is to use computer technology to create highly realistic digital humans with natural and fluid movements and expressions. Digital human generation technology involves several key issues, including digital human reconstruction, digital human dynamics, and intelligent digital human interaction.

[0003] In the field of digital human reconstruction, the main focus is on data acquisition, data representation, and reconstruction methods. Traditional digital human reconstruction input data is primarily divided into multi-view video, single-view video, and point clouds. Multi-view methods require pre-scanning of a human template model and using multiple cameras to acquire data from different angles to reduce the degrees of freedom in human deformation. However, multi-view reconstruction systems are costly, limiting their application in daily life. In recent years, deep learning-based digital human reconstruction methods have been widely studied. These methods learn prior knowledge of human shape and pose through deep neural networks, making it possible to reconstruct digital humans from monocular videos. However, due to factors such as mutual occlusion of different parts of the human body and the limitations of the monocular camera's field of view, human data is lost, increasing the research difficulty. Traditional digital human reconstruction methods typically use meshes, voxels, etc., which have the advantage of simple processing. However, these representations consume a lot of memory when processing human bodies containing a large amount of detail (clothing, face, etc.) and cannot represent color changes caused by factors such as lighting, resulting in limited realism. In recent years, the proposal of Neural Radiation Field (NeRF) has provided a new representation method for digital human reconstruction. It utilizes neural networks to map spatial coordinates to color and density, and synthesizes images through voxel rendering, exhibiting controllable memory usage and enabling realistic synthesis of images from new perspectives. Recent research combines traditional mesh representation methods with neural radiation fields, using a parameterized human mesh model as a priori deformation model of the human body within the neural radiation field, achieving high-precision digital human reconstruction. However, training the neural radiation field for these methods typically requires a long training time (one to two days). Traditional methods for reconstructing digital humans from point clouds are divided into restoring parametric and non-parametric models. Restoring parametric models often fails to capture clothing details, while non-parametric models are prone to producing unreasonable results, such as missing arms. Some methods deform parametric models to obtain clothing details, but these details (texture and geometry, etc.) are relatively coarse, making point cloud reconstruction methods difficult to use for reconstructing highly realistic digital humans.

[0004] Dynamic actuation of digital humans is a key technology for achieving natural movements and expressions in digital humans. First, a 3D model of the human body is obtained through digital human reconstruction methods. Then, actuation technology is used to make it perform the required movements and expressions. Commonly used digital human actuation methods include skeleton-based actuation and non-rigid deformation actuation methods. Skeleton-based actuation methods require creating a skeletal structure for the 3D digital human and then calculating the skeletal motion trajectory based on kinematic principles to control the digital human's movement. Non-rigid deformation methods treat the digital human model mesh as an elastic object, driving the digital human's movement by adjusting the vertex positions. While these methods can be used to actuate digital humans, efficiently and accurately generating natural, spatiotemporally coherent long sequences of movements remains a significant challenge. Manually generating 3D skeletal animation is obviously inefficient, while using motion data from real human bodies requires digital human motion capture technology. This requires the performer to complete set movements to extract motion data, and the resulting motion sequences are fixed, making it difficult to adjust to different scenes, and cannot be generated in real time. The diffusion model has been widely used in image synthesis tasks in recent years. It can generate meaningful data from noise and can also be used to generate the action sequences required for digital human driving. However, it cannot generate long action sequences at once. If multiple generated sequences are spliced ​​together, the action sequence will not be smooth.

[0005] Intelligent interaction in digital humans refers to their ability to respond and interact intelligently based on the environment and user input. Intelligent interaction typically includes text interaction, voice interaction, visual interaction, and multimodal interaction.

[0006] In summary, traditional digital human reconstruction methods suffer from high computational resource consumption and limited realism when dealing with digital humans containing a large amount of detail. While digital human dynamic driving technology can be used to drive digital humans to achieve motion effects, generating natural, spatiotemporally coherent long sequences of movements efficiently and accurately remains a significant challenge. Intelligent interaction in digital human systems also requires overcoming various technical difficulties, such as multimodal data input and multilingual processing. Summary of the Invention

[0007] The purpose of this invention is to propose a voice-interactive 3D digital human generation method to realize high-precision 3D digital human generation and voice interaction from input of a single human image, supporting functions such as voice-synchronized facial expression and gesture generation, multilingual voice dialogue, and dynamic broadcasting.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for generating a voice-interactive 3D digital human includes the following steps:

[0010] Step 1. Convert the input sampled human voice audio into a Mel spectrogram and perform sampling quantization to obtain discrete timbre features; then, based on the input target text and the obtained discrete timbre features, generate a semantic token that retains the specified timbre features; finally, convert the semantic token into a completely new spoken audio, i.e., the original audio; the original audio has the timbre features of the sampled human voice audio and the completely new spoken content of the input target text.

[0011] Step 2. Segment the input monocular video into an image sequence; use multiple models to jointly estimate an initial 3D human body parameter model from the image sequence, corresponding to the body pose, hand pose, and facial pose of the person in the image respectively; perform part-specific optimization on the initial 3D human body parameter model, and further refine the head contour, facial expression, and hand movements of the 3D human body parameter model to finally estimate the 3D human body model.

[0012] Step 3. Using a face generator, expressive facial movements are generated based on the original audio and speaker identifiers. Using a body and hand generator, realistic and diverse body and hand movements are generated based on the original audio and speaker identifiers, respectively, to obtain a full-body 3D human model corresponding to the new speaking audio. The parameters of this full-body 3D human model have facial expressions, body postures, and hand postures corresponding to the new speaking content.

[0013] Step 4. Obtain the vertex shader file of the 3D human body model using the estimated 3D human body model and image sequence; then, after shading the full-body 3D human body model using the vertex shader file, render it through the 3D neural rendering model to obtain the generated real-life 3D digital human image.

[0014] Furthermore, based on the aforementioned method for generating a voice-interactive 3D digital human, this invention also proposes a corresponding voice-interactive 3D digital human generation system, which adopts the following technical solution:

[0015] A voice-interactive 3D digital human generation system includes the following modules:

[0016] The speech synthesis module is used to convert the input sampled human voice audio into a Mel spectrogram and perform sampling quantization to obtain discrete timbre features; then, based on the input target text and the obtained discrete timbre features, a semantic token that retains the specified timbre features is generated; finally, the semantic token is converted into a completely new spoken audio, i.e., the original audio; the original audio has the timbre features of the sampled human voice audio and the completely new spoken content of the input target text;

[0017] The 3D human body parameter estimation module is used to segment the input monocular video into an image sequence; multiple models are used to jointly estimate an initial 3D human body parameter model from the image sequence, corresponding to the body pose, hand pose, and facial pose of the person in the image respectively; the initial 3D human body parameter model is optimized by part, and further refined optimization is performed on the head contour, facial expression, and hand movements of the 3D human body parameter model, and finally the 3D human body model is estimated.

[0018] The voice-driven human motion and expression generation module is used to generate expressive facial movements based on the original audio and speaker identifier using a face generator, and to generate realistic and diverse body and hand movements based on the original audio and speaker identifier input, thereby obtaining a full-body 3D human model corresponding to the new speaking audio; the parameters of the full-body 3D human model have facial expressions, body postures and hand postures corresponding to the new speaking content.

[0019] The real-person 3D digital human generation module is used to obtain the vertex coloring file of the 3D human body model from the estimated 3D human body model and image sequence; then, after coloring the full-body 3D human body model with the vertex coloring file, the generated real-person 3D digital human image is obtained by rendering through the 3D neural rendering model.

[0020] The present invention has the following advantages:

[0021] As described above, this invention relates to a method and system for generating a voice-interactive 3D digital human. Firstly, in terms of speech synthesis, this invention can automatically generate new speaking audio with the same timbre as the sampled human voice and supporting multiple languages, based on the input target text and the different languages ​​of the sampled human voice audio. This provides high-quality speech output for the digital human, significantly improving multilingual support and speech naturalness. Secondly, by using multi-model joint estimation and a temporal loss function, this invention effectively ensures the temporal stability and detail restoration capability of 3D human motion, accurately estimating facial expression details and hand postures in images. Furthermore, this invention generates human motion and facial expressions based on voice-driven methods. Benefiting from the estimated high-precision 3D human body model parameters as training data, this invention can achieve precise synchronization between voice-generated actions and facial expressions, accurately generating facial expressions and body postures that match the new spoken audio. Experiments show that the full-body movements (including body, hands, and face) generated by this invention improve realism and diversity metrics by over 200% compared to baseline methods, while maintaining good logical consistency even in complex scenarios such as hand occlusion and rapid movements. Finally, this invention abandons traditional rendering pipelines and uses a 3D neural rendering model combined with a driveable 3D human body parameter model to achieve a significantly higher quality generated 3D digital human than existing best-in-class solutions, while reducing generation time to a few seconds, significantly improving generation efficiency and visual realism. This invention can endow digital humans with highly realistic facial expressions, voice-driven motion and facial expressions, and dynamic expression capabilities. Attached Figure Description

[0022] Figure 1 This is a flowchart of the voice-interactive 3D digital human generation method in Embodiment 1 of the present invention;

[0023] Figure 2 This is a schematic diagram of the visualization results of three-dimensional human body parameter estimation in Embodiment 1 of the present invention;

[0024] Figure 3 This is a flowchart of the training process for the voice-driven body and hand motion generator in Embodiment 1 of the present invention;

[0025] Figure 4 This is a schematic diagram of the reasoning process framework of the voice-driven body and hand motion generator in Embodiment 1 of the present invention;

[0026] Figure 5 This is a schematic diagram illustrating the Chinese example results of voice-driven human body movements and facial expressions generation in Embodiment 1 of the present invention;

[0027] Figure 6 This is a schematic diagram illustrating the English example results of voice-driven human body movements and facial expressions generation in Embodiment 1 of the present invention;

[0028] Figure 7 This is a flowchart illustrating the process of generating a three-dimensional digital human in Embodiment 1 of the present invention.

[0029] Figure 8 This is a schematic diagram of the visualization effect framework for the generation of a real-person 3D digital human in Embodiment 1 of the present invention;

[0030] in, Figure 8 In the image, (a) represents a monocular image, (b) represents a shading 3D human model, and (c) represents a rendered 3D digital human image.

[0031] Figure 9 This is a schematic diagram of the generation of a 3D mesh and multi-view rendering of a real-person 3D digital human in Embodiment 1 of the present invention;

[0032] in, Figure 9 In the image, (a) represents the rendered 2D image and the corresponding 3D mesh, and (b) represents the rendered 2D images at different angles (corresponding to a 30° rotation in the direction of the image).

[0033] Figure 10 This is a schematic diagram of the interface of the voice-interactive 3D digital human generation system in Embodiment 2 of the present invention;

[0034] in, Figure 10 (a) shows the interactive interface of the 3D digital human generation module of this system; (b) shows the interactive interface of the speech synthesis module of this system; and (c) shows the interactive interface of the speech broadcast function of this system. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0036] Example 1

[0037] This embodiment 1 describes a method for generating a voice-interactive 3D digital human. The method first converts the input sampled human voice audio into a Mel spectrogram and samples and quantizes it to obtain discrete timbre features. Then, based on the input target text and the obtained discrete timbre features, a semantic token retaining the specified timbre features is generated and converted into a new speaking audio of the target human voice. Human instance segmentation is performed from a monocular video to accurately capture human movements in the video. A 3D human model containing facial motion details is obtained through parameter initialization and model parameter optimization. A pre-trained speech model generates synchronized facial movements based on the input human voice audio. The body and hand movements of the 3D human are quantized and reconstructed into two independent codebooks. Features are extracted and encoded from the new speaking audio, and the corresponding finite codebooks for the body / hands are queried and the code elements are decoded to generate body and hand movements from the input human voice audio. A 3D neural rendering model is used to render the full-body 3D human model into a voice-interactive, realistic digital human. This invention achieves rapid generation of a voice-interactive 3D digital human from a single image input, outperforming existing solutions in terms of realism, multilingual support, and generation efficiency, providing efficient technical support for scenarios such as virtual live streaming and metaverse social networking.

[0038] like Figure 1 As shown, the voice-interactive 3D digital human generation method in this embodiment includes the following steps:

[0039] Step 1. Speech synthesis.

[0040] Step 1.1. Convert the input sampled human voice audio into a Mel spectrogram, and then use a multi-layer network (i.e., FishConvNet, downsampling layer, and MP (multi-layer convolutional network)) to sample and quantize the Mel spectrogram to obtain discrete timbre features.

[0041] First, the input sampled human voice audio is converted into a Mel spectrogram. The FishConvNet feature extraction network is used to perform preliminary feature extraction on the Mel spectrogram. The extracted features are further compressed through a downsampling layer. The compressed features are input into a multi-layer convolutional network for deep feature learning. The output of the multi-layer convolutional network is processed by the application layer normalization to obtain standardized features. Finally, the standardized features are quantized into discrete timbre features through a pre-trained timbre feature encoder, VQGAN.

[0042] Step 1.2. Generate a semantic token that retains the specified timbre features based on the input target text and the obtained discrete timbre features. The input target text is the speech text to be generated.

[0043] In this embodiment, the text-to-semantic dual autoregressive inference module Dual-AR is used to generate a semantic token that retains the specified timbre features. The input of Dual-AR is the target text, the reference text, and the discrete timbre features obtained in step 1.1.

[0044] The Dual-AR module consists of two parts: a Slow-Transformer and a Fast-Transformer.

[0045] The slow Transformer accepts text input from the Dual-AR module, generates the final labeled log probabilities, and the intermediate hidden states required by the fast Transformer.

[0046] The Fast Transformer takes the relay hidden state generated by the Slow Transformer as input to generate the final codebook log probability. The discrete timbre features are combined with the final tag log probability and the codebook log probability to generate a semantic token containing speaker timbre feature information.

[0047] The reference text is the text corresponding to the sampled human voice audio (i.e., the text that needs to be read when sampling human voice audio).

[0048] To accelerate the inference process, CUDA kernel fusion can be enabled, increasing the inference speed from approximately 30 tokens / second to approximately 500 tokens / second. For GPUs that do not support bf16, half-precision mode needs to be enabled to ensure compatibility.

[0049] Step 1.3. Finally, the semantic token is converted into a completely new spoken audio, i.e., the original audio; the original audio has the timbre characteristics of the sampled human voice audio and the completely new spoken content of the input target text.

[0050] In this embodiment, the semantic token is converted into the final human voice audio using a VQGAN decoder.

[0051] Specifically, the VQGAN decoder restores the processed feature semantic tokens into a complete audio waveform, and applies post-processing techniques to remove noise and unnatural parts from the audio waveform, resulting in a completely new spoken audio.

[0052] Step 2. High-precision three-dimensional human body parameter estimation.

[0053] Step 2.1. Segment the input monocular video into an image sequence.

[0054] Input a monocular RGB video frame sequence with a resolution greater than 720p, using half-body or full-body human images. Segment the long video into short segments of less than 10 seconds each, and then extract video frames at 30fps to form an image sequence.

[0055] Step 2.2. Use multiple models to jointly estimate the initialized 3D human body parameter model from the image sequence, corresponding to the body pose, hand pose and facial pose of the person in the image.

[0056] The input single-frame RGB image is initialized with parameters, which are divided into three parts, corresponding to the body posture, hand posture, and facial posture of the person in the image, respectively; the motion of the human body is represented by the parametric 3D human body model SMPL-X model.

[0057] Using shape parameters With posture parameters It represents the shape of the human body and the movement of its various parts.

[0058] For initializing body pose parameters, the 3D human body parameter estimation model PIXIE is used to estimate body shape. Body posture For hand pose initialization, the 3D human body parameter estimation model PyMAF-X is used to estimate hand pose. The DECA model, used to estimate human head parameters, is used to generate facial expression parameters. Eye posture and jaw posture Initialize facial expressions ,Right now .

[0059] The three-dimensional human body parameter model obtained in this step is used for further optimization in step 2.3 to obtain a high-precision three-dimensional human body model.

[0060] Step 2.3. Perform part-specific optimization on the initialized 3D human body parameter model, focusing on the head contour and facial expressions. and hand gestures Further refinement and optimization yielded a high-precision 3D human body model. The optimization method combined multiple models to achieve precise and accurate 3D human body parameter estimation. During optimization, the DeepLab V3 human body mask estimation tool was used to calculate the loss between the generated body contour mask and the initialized 3D human body model to optimize the body shape β. Facial feature point information from the MediaPipe facial feature point detection tool and facial shape information from the MICA human head parameter estimation tool were used to calculate the loss between the initial 3D human body model's facial feature points and vertices to optimize the body shape β (the head portion of the body shape) and facial expression. The loss calculation was performed using hand joint feature points from the OpenPose human joint feature point detection tool and the hand joint feature points from the initialized 3D human model to optimize the hand pose. Simultaneously, facial images are rendered using 3D human model parameters and optimized facial textures. Color loss is calculated between the obtained image and the input single-frame image to better capture facial details. Combined with temporal loss, the temporal consistency of the whole body, including hand movements, is ensured, resulting in a 3D human model SMPL-X sequence with rich and realistic facial expressions and clear hand movements.

[0061] The overall losses during the optimization process are as follows: .

[0062] in The image prior at time t; This represents the loss of facial feature point information in MediaPipe. The loss between the facial shape of the 3D human model and the facial shape of MICA; The color loss between the obtained image and the input single-frame image; and This is due to the loss of smoothness in the movement of the body, hands, and joints. The loss between the body contour mask generated by DeepLab and the initial 3D human body model contour; To prevent the penalty of collision caused by the hands penetrating each other, , These are the weighting coefficients for the corresponding losses.

[0063] like Figure 2 The estimated high-precision 3D human body model (3D human body model SMPL-X sequence) is shown, which can be used as the prior training data for the motion and expression generator in step 3 and the acquisition of vertex shading files in step 4.

[0064] Step 3. Voice-driven generation of human body movements and facial expressions.

[0065] Step 3.1. Generate expressive facial movements based on the original audio and speaker identifier using a face generator. In this embodiment, the face generator uses a pre-trained speech model, Wav2vec, to achieve synchronous facial expression and movement generation.

[0066] The Wav2vec model's encoder consists of an audio feature extractor and a Transformer encoder, outputting a 768-dimensional speech representation. A linear projection layer is added on top of the encoder to reduce the feature dimension to 256.

[0067] During the inference process, the original audio code generated in step 1.3 is used as input, and the encoder encodes the original audio into audio features. ; Let t represent the t-th audio feature, where t∈(1,T), and T represents the number of features.

[0068] Audio features The speaker's identity features (which can be defined by the user, such as 'ZHANGSAN' or 'LISI') are encoded separately, and the resulting codes are concatenated and then input into the decoder.

[0069] The decoder consists of six temporal convolutional networks (TCNs), with a fully connected layer at the end. The decoder then generates facial expressions corresponding to the original input audio. .

[0070] During the training of the encoder and decoder for the face generator, the sampled human voice audio from step 1.1 and the facial expressions from the high-precision 3D human model SMPL-X sequence estimated in step 2.3 are used. The encoder and decoder are trained using mean squared error (MSE) loss function as input audio and ground truth values. In this embodiment, the face generator supports generalization to multiple languages ​​and various audio types, and is robust to unseen languages ​​and audio types.

[0071] Step 3.2. Using body and hand generators, realistic and diverse body and hand movements are generated based on the original audio and speaker identifier inputs. The body / hand generators have the same structure and each includes a feature extractor that extracts audio features, an autoregressive module that takes audio features and speaker identifiers as inputs and body / hand movement codewords as outputs, a finite codebook of body / hand movements, and a decoder that decodes the corresponding part codebook to generate motion parameters.

[0072] The body / hand generator will use the high-precision 3D human body model obtained in step 2 to generate body / hand parameters ( and The code is encoded and quantized to reconstruct two finite codebooks containing different body / hand motion codes. .in A finite codebook representing different body movement symbols. This represents the i-th symbol in the finite codebook of the body, where i represents the symbol index. This represents the number of symbols in the entire body's finite codebook. A finite codebook representing different hand gesture symbols. This represents the j-th symbol in the finite codebook of the hand, where j represents the symbol index. This indicates the number of symbols in the entire finite codebook of the hand.

[0073] To increase the diversity of predicted movements, the hand and body are separated, each corresponding to an independent encoder, finite codebook, and decoder (the encoder is only used during the training of the finite codebook and decoder). The encoders, decoders, and finite codebooks for the body and hand are trained separately, as follows: Figure 3 and Figure 4 As shown.

[0074] During training, the body / hand movements of the high-precision 3D human model estimated in step 2.3 are used. As input values, the input encoder encodes and quantizes them into a feature sequence. ;at the same time It is also used as a real value to compare with the decoded body / hand movements. Loss calculations are performed to train the encoder / decoder with a finite codebook. This represents the body / hand movements of a 3D human model at time t. This represents the quantitative characteristics of body / hand movements at time t.

[0075] The obtained feature sequence According to the formula Limited codebook in the corresponding part Find and nearest code element The codebook for mapping body / hand movements In the middle, Z represents or .in This represents the code element representing the action at time t. For the corresponding part of the finite codebook All code elements in the codebook. The codebook is generated by decoding. Decoding yields the three-dimensional human body model parameters for the corresponding body part (represented as the body or hand in this embodiment). .

[0076] use Compared with the actual input value Loss calculations are performed to simultaneously train the encoder, decoder, and finite codebooks for the corresponding body parts. and .

[0077] Because the codebooks for body and hand poses are trained separately, the representation space and combination degrees of freedom for body and hand poses can be greatly enriched. The training process of the hand / face generator, as follows... Figure 3 As shown. Training loss function. As shown below:

[0078] .in The mean squared error (MSE) represents the reconstruction loss, and sg is the stop-loss gradient operation used to calculate the codebook loss. This refers to the trade-off between 'commitment losses' and other factors.

[0079] Simultaneously, an autoregressive model is used to encode the sampled human voice audio from step 1.1 (corresponding to the high-precision 3D human body model obtained in step 2.3) with the speaker's identity features (user-defined) to obtain the codebook corresponding to the speech. .

[0080] The autoregressive model is optimized using a teacher-forcing scheme and cross-entropy loss (for body and hand movements). The training process of the autoregressive model is independent of the aforementioned encoder-decoder and finite codebook training processes. The body poses encoded by the autoregressive model are... With hand posture .

[0081] Autoregressive models should follow the following during the encoding and reasoning process: (1) Current motion The reasoning depends on the motion of past moments. (2) The movement of the hand at the current moment Should be guided by body posture This enables the autoregressive model to generate coherent body and hand postures based on the input audio, resulting in realistic hand movements that match the body posture.

[0082] During inference, 64-dimensional MFCC features are used as the audio representation. An audio feature extractor is used to extract features from the original audio generated in step 1, and an autoregressive module is used to encode the features into a codebook. (C = C) b C h ), where C b C h The codebooks represent the body and the hand, respectively, and an autoregressive module is used to encode the features into codebooks. The obtained codebook Querying the finite codebook The code elements at corresponding moments are obtained by sampling various reasonable combinations of body / hand movements. The body / hand parameters at that moment are generated by decoding the code symbols using a decoder. and ).

[0083] like Figure 4 As shown, during the inference process, the novel spoken audio generated in step 1.3 is used as input, and an autoregressive model is used to encode it into a codebook. Search for the combined codebook obtained during training. and the resulting code Codebook mapped to corresponding body parts Then, the corresponding body pose is obtained by decoding using a decoder. or hand gestures .

[0084] The parameters of the three-dimensional human body model generated in steps 3.1 and 3.2 ( By combining these steps, a full-body 3D human model corresponding to the new speaking audio generated in step 1.3 is finally obtained.

[0085] The full-body 3D human model has facial expressions that accurately correspond to the spoken content (new spoken audio). and body and hand movement posture ( and This is used to drive the subsequent rendering process of the live-action digital human.

[0086] like Figure 5 and Figure 6 Examples of voice-driven human facial expression generation in Chinese and English are shown respectively. Figure 5 and Figure 6 It has been learned that, for the same audio input, the human body motion and facial expression generation method of this invention can not only generate standard lip movements and reasonable three-dimensional human body model parameters in the original language, but also generate human body motion and facial expressions in another language, demonstrating that this invention has strong generalization in human body motion and facial expression generation and robustness to unseen languages.

[0087] Step 4. Generation of a realistic 3D digital human.

[0088] Step 4.1. Use the estimated 3D human body model and image sequence to obtain the vertex shader file of the 3D human body model.

[0089] A flowchart and visualization rendering results of the process of generating realistic 3D digital humans are shown below. Figure 7 As shown.

[0090] First, use the monocular image sequence obtained in step 2.1 (such as...) Figure 8 The high-precision 3D human body model obtained in step 2.3 (as shown in (a)) is projected onto the image sequence to obtain the RGB color of the corresponding pixel.

[0091] Then, the obtained RGB colors are saved as vertex color files according to vertex order.

[0092] Step 4.2. After coloring the full-body 3D human model using vertex shader files, the colored 3D human model looks like this. Figure 8As shown in (b), a real-life 3D digital human image is generated by rendering using a 3D neural rendering model. The rendered real-life 3D digital human image is as follows: Figure 8 As shown in (c) in the figure.

[0093] Specifically, this invention is based on the open-source UNet architecture. Based on the full-body 3D human body model and vertex color file generated in step 3, a neural rendering model is used to generate a real 3D digital human with the same movements and expressions.

[0094] A deep learning model combining deformable 3D models with neural rendering is employed. This neural rendering network (i.e., the deep learning model) consists of four downsampling (DownConv) and upsampling (UpConv) layers, with two [Conv2D, BatchNorm, ReLU] convolutional layers and a kernel size of 3. Using a shaded 3D human model as input, it can render and generate corresponding 3D digital human images based on the model's movements. A flowchart illustrating the generation process of a realistic 3D digital human is shown below. Figure 7 As shown, by fully utilizing the advantages of data-driven methods and neural radiation field technology, high-quality 3D digital humans can be generated, and 2D rendered images from any viewpoint can be obtained. The rendered 2D images and their corresponding 3D meshes are shown below. Figure 9 As shown in (a), the rendered 2D images at different angles (corresponding to a 30° rotation in the direction of the image) are as follows: Figure 9 As shown in (b) of the diagram.

[0095] This invention proposes a voice-interactive 3D digital human generation method that uses text and voice as input to drive digital human movement. Furthermore, by extracting human postures from videos or images and driving the digital human to perform corresponding actions, it enables intelligent interaction with the generated digital human through multimodal data such as text, voice, images, and video. Regarding digital avatar interaction, based on a 3D neural rendering model of the digital avatar, and driven by 3D human model parameters, it achieves digital avatar expression rendering with 3D human model parameters as input and data-driven video interaction. This invention overcomes the limitations of traditional methods in terms of detail richness, computational resource consumption, and accuracy, bringing significant progress to the field of digital human generation. Compared to traditional solutions, this invention can achieve rapid generation of a voice-interactive 3D digital human from a single image input, outperforming traditional solutions in realism, multilingual support, and generation efficiency. This invention overcomes the limitations of traditional methods in terms of voice realism, multilingual generation, generation time, and human movement accuracy, providing a more efficient, accurate, realistic, and rapid solution for the application of digital humans in fields such as the metaverse, virtual live streaming, online teaching, intelligent customer service, and cultural tourism promotion.

[0096] Example 2

[0097] Based on Embodiment 1 above, Embodiment 2 proposes a corresponding voice-interactive 3D digital human generation system—"Phantom Space". This system converts the input audio into a Mel spectrogram and obtains discrete timbre features through multi-layer network sampling and quantization. It synthesizes multilingual human voice audio through an inference module and a decoder. The input video is segmented into an image sequence, initialized with multiple models, and further optimized to obtain a high-precision 3D human body model. With a given audio input, a face generator generates expressive facial movements, a body and hand generator generates realistic and diverse body and hand movements, and a 3D neural rendering model renders the generated high-precision 3D human body model to obtain a realistic digital human image.

[0098] Specifically, the voice-interactive 3D digital human generation system in this embodiment includes the following modules:

[0099] The speech synthesis module is used to convert the input sampled human voice audio into a Mel spectrogram and perform sampling quantization to obtain discrete timbre features; then, based on the input target text and the obtained discrete timbre features, a semantic token that retains the specified timbre features is generated; finally, the semantic token is converted into a completely new spoken audio, i.e., the original audio; the original audio has the timbre features of the sampled human voice audio and the completely new spoken content of the input target text;

[0100] The 3D human body parameter estimation module is used to segment the input monocular video into an image sequence; multiple models are used to jointly estimate an initial 3D human body parameter model from the image sequence, corresponding to the body pose, hand pose, and facial pose of the person in the image respectively; the initial 3D human body parameter model is optimized by part, and further refined optimization is performed on the head contour, facial expression, and hand movements of the 3D human body parameter model, and finally the 3D human body model is estimated.

[0101] The voice-driven human motion and expression generation module is used to generate expressive facial movements based on the original audio and speaker identifier using a face generator, and to generate realistic and diverse body and hand movements based on the original audio and speaker audio input using body and hand generators, thereby obtaining a full-body 3D human model corresponding to the new speaking audio; the parameters of this full-body 3D human model have facial expressions, body postures and hand postures corresponding to the new speaking content.

[0102] The real-person 3D digital human generation module is used to obtain the vertex coloring file of the 3D human body model from the estimated 3D human body model and image sequence; then, after coloring the full-body 3D human body model with the vertex coloring file, the generated real-person 3D digital human image is obtained by rendering through the 3D neural rendering model.

[0103] It should be noted that the implementation process of the functions and roles of each functional module in the voice-interactive 3D digital human generation system is detailed in the corresponding steps of the method in Example 1, and will not be repeated here.

[0104] This invention integrates speech synthesis, high-precision 3D human body estimation, voice-driven human motion and expression generation, and a method for generating realistic 3D digital humans into a complete system. It achieves high-precision 3D digital human generation and voice interaction from a single human image input. The system can provide detailed representation and driving of human faces and features a user-friendly interface that allows all system functions to run and be displayed simultaneously. A schematic diagram of the interactive interface of this voice-interactive 3D digital human generation system is shown below. Figure 10 As shown, where, Figure 10 (a) shows the interactive interface of the 3D digital human generation module of this system, which can generate corresponding 3D digital human images according to different characters; Figure 10 (b) shows the interactive interface of the speech synthesis module of this system, which can synthesize and output a brand new speaking audio that retains the timbre of the input audio and the content of the input text based on the input audio and the specified text. Figure 10 (c) shows the interactive interface of the system's voice broadcast function. Combining speech synthesis, voice-driven facial expression generation, and a realistic 3D digital human rendering module, it enables the 3D digital human to broadcast text specified by the user. Figure 10 As can be seen, the system designed in this invention has a user-friendly interface, allowing users to see the digital human generation process in real time and interact with it.

[0105] The system of this invention can quickly reconstruct any digital human image with detailed headshot from monocular video and provides diverse and intelligent interaction methods, laying the foundation for fields such as virtual reality, virtual try-on, and virtual sports live streaming.

[0106] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A method for generating a three-dimensional digital human that can interact with voice, characterized in that, Comprising the following steps: Step 1. Convert the input sampled human voice audio into a mel-spectrogram and sample quantization to obtain discrete timbre features; then generate semantic tokens that retain specified timbre features according to the input target text and the obtained discrete timbre features; finally, convert the semantic tokens into a brand new speaking audio, i.e. the original audio; the original audio has the timbre features of the sampled human voice audio and the brand new speaking content of the input target text; Step 2. Divide the input monocular video into an image sequence; use multiple models to jointly estimate an initialized three-dimensional human body parameter model from the image sequence, corresponding to the body posture, hand posture and facial expression of the person in the image respectively; optimize the initialized three-dimensional human body parameter model by part, and further optimize the head contour, facial expression and hand posture of the three-dimensional human body parameter model, to finally estimate the three-dimensional human body model; The three-dimensional human parameter model obtained by initialization is optimized by parts, and the head is further finely optimized among facial expressions , hand postures , and body shapes , and a three-dimensional human model is estimated, and the optimization process is: The body contour mask generated from the image using the human mask estimation tool DeepLab V3 is loss calculated with the initialized three-dimensional human model to optimize the body shape β; the face feature point information of the face feature point detection tool MediaPipe and the face shape information of the human head parameter estimation model MICA are loss calculated with the face feature points and the face vertex of the initialized three-dimensional human model to optimize the body shape β and the facial expression ; the hand joint feature points of the hand joint feature point detection tool OpenPose are loss calculated with the hand joint feature points of the initialized three-dimensional human model to optimize the hand posture ; meanwhile, the three-dimensional human model parameters and the optimizable face texture are used to render the face image, the color loss between the obtained image and the input single-frame image is calculated to better capture the face details, and the time sequence loss is combined to ensure the time sequence consistency of the whole body including the hand movement, so that the three-dimensional human model SMPL-X sequence with rich and real facial expression and clear hand movement is obtained; , is the expression parameter, is the eye posture, is the jaw posture; Comprehensive loss in optimization process As follows: ; wherein is the image prior at time t; is the loss of face feature point information of MediaPipe; is the loss between the face shape of the three-dimensional human body model and the face shape of MICA; is the color loss between the obtained image and the input single-frame image; and is the smoothing loss of the body and the hand and the joint movement of the human body; is the loss between the body contour mask generated by DeepLab and the initialized three-dimensional human body model contour; is the collision penalty loss introduced to prevent the mutual penetration of the two hands, , is the weight coefficient corresponding to the loss; Step 3. Use the face generator to generate expressive facial movements based on the original audio and the speaker identifier, and use the body and hand generator to generate realistic and diverse body movements and hand movements based on the original audio and the speaker identifier input, to obtain a full-body three-dimensional human body model corresponding to the brand new speaking audio; the full-body three-dimensional human body model parameters have facial expressions, body postures and hand postures corresponding to the brand new speaking content; Step 4. Use the estimated three-dimensional human body model and the image sequence to obtain a vertex shading file for the three-dimensional human body model; Then, use the vertex shading file to color the full-body three-dimensional human body model, and render the generated real three-dimensional digital human image through a three-dimensional neural rendering model.

2. The voice-interactable three-dimensional digital human generation method of claim 1, wherein: In step 1, the process of obtaining discrete timbre features is as follows: First, convert the input sampled human voice audio into a mel-spectrogram, use the feature extraction network FishConvNet to perform preliminary feature extraction on the mel-spectrogram, further compress the extracted features through a downsampling layer, input the compressed features into a multi-layer convolutional network for deep feature learning, apply layer normalization to the output of the multi-layer convolutional network to obtain standardized features, and finally quantize the standardized features into discrete timbre features through a pre-trained timbre feature encoder VQGAN.

3. The voice-interactable three-dimensional digital human generation method of claim 1, wherein: In step 1, the Dual-AR inference module is used to generate semantic tokens that retain specified timbre features; the input of Dual-AR is the target text, the reference text and the obtained discrete timbre features; The Dual-AR module consists of two parts: a slow Transformer and a fast Transformer. The slow speed Transformer receives the text input of the Dual-AR module, generates the final token log probability and the relay hidden state required by the fast speed Transformer, the fast speed Transformer takes the relay hidden state generated by the slow speed Transformer as input, and generates the final codebook log probability; the discrete timbre feature combines the final token log probability and the codebook log probability, that is, the semantic token containing the speaker timbre feature information can be generated; Wherein the target text is the voice text to be generated, and the reference text is the text corresponding to the sampled human voice audio.

4. The voice-interactable three-dimensional digital human generation method according to claim 1, wherein, In step 1, the processed semantic token is restored to a complete audio waveform by a VQGAN decoder, and a post-processing technique is applied to eliminate noise and unnatural parts in the audio waveform, thereby obtaining a new speaker audio.

5. The voice-interactable three-dimensional digital human generation method according to claim 1, wherein, In step 2, the input single-frame image is parameterized and initialized, which is divided into three parts corresponding to the body posture, hand posture and face posture of the image; the motion of the human body is represented by a parameterized three-dimensional human body model SMPL-X model; In particular, for body pose parameter initialization, a three-dimensional human parameter estimation model, PIXIE, is used to estimate the body shape , body pose ; For hand pose initialization, a three-dimensional human parameter estimation model, PyMAF-X, is used to estimate the hand pose ; Generating expression parameters with a human head parameter estimation model DECA , eye pose , and chin pose , initializing facial expressions , .

6. The voice-interactable three-dimensional digital human generation method according to claim 1, wherein, In step 3, the face generator adopts a pre-trained speech model Wav2vec to realize synchronous face expression motion generation; the Wav2vec model includes a linear projection layer, an encoder and a decoder; the encoder is composed of an audio feature extractor and a Transformer encoder; a linear projection layer is added on the encoder to reduce the feature dimension; The original audio encoding generated in step 1 is taken as input in the inference process, and the original audio is encoded into audio features through the encoder ; wherein denotes the tth audio feature, t e (1, T) encoding the audio features with the speaker identity features I are encoded separately and the resulting encodings are concatenated before being input into the decoder; generating facial expressions corresponding to the original audio input by the decoder .

7. The voice-interactable three-dimensional digital human generation method according to claim 1, wherein, In step 3, the body / hand generator has the same structure and includes an encoder, an audio feature extractor, an autoregressive module and a decoder; the audio feature extractor is used to extract audio features; the input of the autoregressive module is the audio features and the speaker identifier, and the output of the autoregressive module is the body / hand motion code; the decoder stores a limited codebook of body / hand motion and a decoding corresponding part codebook to generate motion parameters; Using the 3D human model estimated in step 2, encode and quantize the body / hand parameters to reconstruct a finite codebook with different body / hand motion code elements or ; wherein, represents a finite codebook with different body motion code elements, represents the i-th code element in the body finite codebook, i represents the code element index, represents the number of code elements in the whole body finite codebook, represents a finite codebook with different hand motion code elements, represents the j-th code element in the hand finite codebook, j represents the code element index, represents the number of code elements in the whole hand finite codebook; The training process of the encoder, the decoder and the limited codebook of the body / hand generator is as follows: Body / hand motion of the three-dimensional human model estimated using step 2 As input values and real values, the input encoder quantizes its encoding into a feature sequence ; wherein represents the body / hand motion of the three-dimensional human model at time t, represents the quantized features of the body / hand motion at time t; the obtained feature sequence According to the formula The code closest to distance in the limited codebook Z of the corresponding part , Z represents or , mapped to the generated codebook of the body / hand motion ; wherein is the code representing the motion at time t, is all the codes in the limited codebook of the corresponding part; the generated codebook is decoded by the decoder to obtain the three-dimensional human model parameters of the corresponding part , and the loss calculation is performed with the input real value to simultaneously train the encoder, decoder and limited codebook of the body corresponding part of the body / hand generator or ; The inference process of the body / hand generator is as follows: The newly generated spoken audio is used as input and encoded into a codebook using an autoregressive model. According to the obtained codebook Find the finite codebook for the corresponding part obtained during training. or The resulting encoding is then mapped to the corresponding body part's generated codebook. Then, the corresponding body posture or hand posture is obtained by decoding using a decoder.

8. The voice-interactable three-dimensional digital human generation method according to claim 1, wherein, In step 4, the monocular image sequence obtained in step 2 and the estimated three-dimensional human body model are used to project the model vertices on the image sequence to obtain the RGB colors of the corresponding pixel points of the vertices, and the obtained RGB colors are saved as vertex color files in the order of the vertices; Based on the open-source UNet architecture, the three-dimensional human body model and the vertex color files generated in step 3 are used to generate a real three-dimensional digital human with the same motion and expression through a three-dimensional neural rendering model.

9. A voice-interactable three-dimensional digital human generation system for implementing the voice-interactable three-dimensional digital human generation method of claim 1, characterized by, The voice-interaction-enabled three-dimensional digital human generation system comprises the following modules: A speech synthesis module is configured to convert an input sampled human voice audio into a mel-spectrogram and perform sample quantization to obtain discrete timbre features; then generate semantic tokens that reserve specified timbre features according to an input target text and the obtained discrete timbre features; and finally convert the semantic tokens into a brand-new speaking audio, i.e., an original audio; the original audio has the timbre features of the sampled human voice audio and the brand-new speaking content of the input target text; A three-dimensional human body parameter estimation module is configured to segment an input monocular video into an image sequence; use multiple models in combination to estimate an initialized three-dimensional human body parameter model from the image sequence, which respectively correspond to the body posture, hand posture and facial posture of a person in the image; perform part-by-part optimization on the initialized three-dimensional human body parameter model, and further finely optimize the head contour, facial expression and hand action of the three-dimensional human body parameter model, to finally estimate a three-dimensional human body model; A voice-driven human body action and expression generation module is configured to use a face generator to generate expressive facial actions based on the original audio and the speaker identification, and use a body and hand generator to generate realistic and diverse body actions and hand actions based on the original audio and the speaker identification input, to obtain a full-body three-dimensional human body model corresponding to the brand-new speaking audio; the full-body three-dimensional human body model parameters have facial expressions, body postures and hand postures corresponding to the brand-new speaking content; A real three-dimensional digital human generation module is configured to use the estimated three-dimensional human body model and the image sequence to obtain a vertex shading file of the three-dimensional human body model; Then, the full-body three-dimensional human body model is colored using the vertex shading file, and a generated real three-dimensional digital human image is rendered through a three-dimensional neural rendering model.

Images