Digital human portrait generation method, system, equipment and medium

Abstract

The invention discloses a digital human portrait generation method, system and device and a medium, and belongs to the field of 3D digital humans.The method comprises the steps that based on an input video sequence, a camera pose, a face grid, a face feature point sequence and an audio feature sequence are obtained through extraction, and vertexes of the face grid are converted to obtain an initial Gaussian primitive set; taking the attitude prior feature points and the audio feature sequence as condition input, and predicting through a conditional variation auto-encoder to obtain a mouth feature point sequence; taking the audio feature sequence, the mouth feature point sequence and the camera pose as multi-modal features, fusing with spatial information through a multi-head cross attention mechanism, and predicting to obtain the offset of each Gaussian primitive; and the initial Gaussian primitive set is updated based on each offset, and rendering is performed through the rasterization rendering pipeline to obtain the digital human portrait image, so that the three-dimensional digital human portrait which not only meets the real-time interaction requirement but also has a highly realistic sense can be generated by implementing the method and the device.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of 3D digital people, in particular to a digital person portrait generation method, system, device and medium. BACKGROUND

[0002] Digital person portrait synthesis technology is a frontier research direction in the intersection field of computer vision and computer graphics, and its goal is to generate a virtual human image with high realism and natural interaction, which has broad application prospects in virtual anchors, film production, remote communication and metaverse, etc.

[0003] At present, although the digital person synthesis method based on three-dimensional representation can ensure multi-view consistency, it generally faces the problem of low rendering efficiency. In particular, the method using neural radiation field and other implicit representation is difficult to meet the frame rate requirement of real-time interaction due to its characteristics of computationally intensive volume rendering, which limits its application in real-time scenarios such as live broadcast and video conference. Although the explicit three-dimensional representation method appeared in recent years has improved the rendering efficiency, there are still problems such as single driving signal, insufficient facial motion details, limited synchronization accuracy of voice and lip movement when processing voice-driven dynamic digital people. SUMMARY

[0004] The present application provides a digital person portrait generation method, system, device and medium, which can solve the problem of how to realize high-precision lip synchronization and rich and natural facial expression dynamics under the premise of high rendering efficiency, so as to generate a three-dimensional digital person portrait that meets the requirements of real-time interaction and has high realism.

[0005] The present application provides a digital person portrait generation method, comprising:

[0006] Based on the input video sequence, camera pose, three-dimensional face mesh, face feature point sequence and audio feature sequence are extracted, and the vertices of the three-dimensional face mesh are converted to obtain an initial 3D Gaussian primitive set, wherein the face feature point sequence includes mouth region feature points and second region feature points except the mouth region, and the initial 3D Gaussian primitive set includes a plurality of Gaussian primitives, and each Gaussian primitive includes spatial position, rotation, scale, spherical harmonic coefficient, color and transparency attribute;

[0007] The second region feature points are taken as pose prior feature points, and the audio feature sequence is taken as a conditional input, and a mouth feature point sequence is predicted by a conditional variational autoencoder;

[0008] The audio feature sequence, the mouth feature point sequence and the camera pose are taken as multi-modal features, the multi-modal features are fused with spatial information through a multi-head cross attention mechanism, and an offset of each Gaussian cell is predicted, wherein the spatial information is a high-dimensional feature vector representation of a spatial position of the Gaussian cell obtained through multi-resolution three-plane hash coding.

[0009] The initial 3D Gaussian cell set is updated based on the offsets to obtain a dynamic 3D Gaussian field, and the dynamic 3D Gaussian field is rendered through a rasterization rendering pipeline to obtain a digital human portrait image.

[0010] The embodiment of the present application extracts the camera pose, the three-dimensional face mesh, the face feature point sequence including the mouth and the second region, and the audio feature sequence, converts the three-dimensional face mesh vertex to obtain an initial 3D Gaussian cell set including a spatial position, rotation, scale, spherical harmonic coefficient, color and transparency, and provides a complete and accurate static basis for subsequent dynamic adjustment: the three-dimensional face mesh ensures that the initial geometric structure of the Gaussian cell fits the real face, the multi-attribute Gaussian cell can restore the face details (such as skin color and transparency), has high realism, and the camera pose lays the foundation for subsequent view alignment. Taking the second region feature point as a gesture prior, combining the audio feature, predicting the mouth feature point sequence through a conditional variational autoencoder (CVAE), and ensuring the dynamic coordination of the digital human: the gesture prior constrains the non-mouth gestures (such as eyes and head) to be in order, the CVAE combines the audio to predict the mouth shape, and high-precision mouth shape synchronization is realized. Taking the audio, the mouth sequence and the camera pose as multi-modal features, predicting the dynamic offset of the Gaussian cell by fusing the multi-modal features with the spatial information obtained through multi-resolution three-plane hash coding through a multi-head cross attention mechanism, and ensuring the dynamic details of the digital human to be accurate and the view to be consistent: the multi-head cross attention realizes the deep correlation of multi-modal information (such as the audio intensity corresponding to the scale change of the mouth Gaussian cell), the spatial information coding accurately locates the position of the Gaussian cell, avoids the misplacement of face details during dynamic adjustment, and enriches the dynamic natural face expression; the camera pose is integrated into the multi-modal features to ensure that the dynamic offset matches the original video view. Based on the dynamic offset, the initial Gaussian cell is updated to obtain a dynamic 3D Gaussian field, which is rendered through a rasterization rendering pipeline to output a digital human portrait, which greatly improves the generation efficiency of the digital human portrait, can realize a high frame rate output of more than 30 frames per second, and meets the delay requirements of real-time interaction scenes (such as real-time digital human live broadcast and virtual dialogue).

[0011] Further, the conditional variational autoencoder is obtained by training an initial encoder and an initial decoder, specifically:

[0012] A training video sequence is obtained, and an audio feature training sequence and a face feature point training sequence are extracted;

[0013] The mouth feature point training sequence in the face feature point training sequence is taken as an encoding target, the posture prior feature point training sequence in the face feature point training sequence and the audio feature training sequence are taken as conditional inputs, a latent variable distribution parameter is obtained through a fully connected network in the initial encoder, and a latent variable is sampled from the latent variable distribution parameter through a reparameterization technique;

[0014] The audio feature training sequence, the posture prior feature point training sequence and the latent variable are taken as inputs, and a mouth feature point prediction sequence is obtained through prediction of the initial decoder, wherein the initial decoder adopts a Transformer cross-attention mechanism to fuse the audio feature training sequence and the posture prior feature point training sequence.

[0015] An error between the mouth feature point training sequence and the mouth feature point prediction sequence is calculated through a composite loss function, and the error is iteratively optimized to obtain the conditional variational autoencoder.

[0016] In this way, by obtaining a training video sequence, extracting an audio feature training sequence and a face feature point training sequence, taking a mouth feature point training sequence as an encoding target, and taking a posture prior feature point training sequence and an audio feature as conditional inputs, the “target (mouth shape prediction)” and “constraint (non-mouth posture, audio)” of model training are clearly defined, ensuring that the trained model meets the actual task requirements rather than being unconstrained prediction. The initial encoder uses the latent variable distribution parameter of the fully connected network to sample the latent variable through the reparameterization technique, ensuring the “reasonableness and diversity” of the latent variable: the fully connected network efficiently compresses multi-condition information, and the reparameterization avoids gradient disappearance, ensuring that the latent variable can cover different mouth shape change scenarios (such as the difference in mouth shape for different pronunciation). The initial decoder uses the Transformer cross-attention to fuse the audio and posture prior to predicting the mouth feature point sequence, strengthening the “association between multi-condition and mouth shape”: the Transformer cross-attention can capture the subtle association between the audio rhythm (such as “ah” corresponding to mouth opening) and the non-mouth posture (such as the mouth shape amplitude when nodding), avoiding the isolation of mouth shape prediction from the overall posture. Through the composite loss function to calculate the error and iteratively optimize, ensuring the “convergence and prediction accuracy” of the model training: the composite loss constrains the model from multiple dimensions (such as reconstruction accuracy, temporal fluency), avoiding prediction bias caused by a single loss (such as mouth shape freezing when only using reconstruction loss).

[0017] Further, the composite loss function is obtained by calculating a reconstruction loss, a temporal smoothing loss, a synchronization loss and a divergence loss, specifically:

[0018] A first norm difference between the mouth feature point training sequence and the mouth feature point prediction sequence is calculated to obtain the reconstruction loss.

[0019] calculate a second norm difference between the mouth feature point prediction sequence of adjacent time steps, to obtain the timing smoothing loss;

[0020] calculate the synchronization between the audio feature training sequence and the face feature point training sequence through a cross-entropy loss function, to obtain the synchronization loss;

[0021] calculate the KL divergence between the latent variable distribution and the standard normal distribution, to obtain the divergence loss.

[0022] In this way, by calculating the first norm difference (reconstruction loss) between the mouth feature point training sequence and the prediction sequence, the "accuracy" of the mouth shape is ensured: the geometric difference between the predicted mouth shape and the real mouth shape is directly constrained, avoiding the predicted mouth shape from deviating from the real shape (such as the mouth shape of "oh" being predicted as "ah"). The second norm difference (timing smoothing loss) between adjacent time step prediction sequences is calculated to ensure the "smoothness" of the mouth shape: the change amplitude of the mouth shape in adjacent frames is constrained to avoid the abrupt and jarring "mouth closed in the previous frame and mouth wide open in the next frame", and the natural dynamic transition of the mouth is restored. The cross-entropy loss calculates the synchronization between the audio and face feature point sequence (synchronization loss) to ensure that the mouth shape is "synchronized with the audio": the correlation between the audio features (such as pitch and rhythm) and the mouth shape features is quantified to avoid synchronization deviation such as "voice saying 'ah' but mouth closed". The KL divergence between the latent variable distribution and the standard normal distribution (divergence loss) is calculated to ensure the "generalization" of the model: the latent variable distribution is constrained not to be too concentrated, avoiding the model only adapting to the training data (overfitting) and still being able to stably predict the mouth shape under new audio / pose.

[0023] Further, the spatial information is a high-dimensional feature vector representation of the spatial position of the Gaussian cell obtained through multi-resolution three-plane hash coding, specifically:

[0024] project the spatial position of the Gaussian cell onto three orthogonal planes respectively to obtain the projection position corresponding to each plane, wherein each plane includes a plurality of two-dimensional grids of different resolutions;

[0025] interpolate the projection position in each two-dimensional grid to obtain the encoded feature of the projection position at each resolution;

[0026] combine the encoded features corresponding to all resolutions under the same plane to obtain the target encoded feature;

[0027] splice the target encoded features of the three orthogonal planes to obtain the spatial information.

[0028] This approach achieves "full spatial coverage" by projecting Gaussian primitive spatial locations onto three orthogonal planes, each containing multiple 2D grids of different resolutions. The three orthogonal planes (e.g., XY, YZ, XZ planes) completely represent 3D spatial locations, avoiding the spatial information loss caused by a single plane. The multi-resolution grids balance "global positioning (low resolution)" and "detailed positioning (high resolution)." Interpolation of the projected positions within each 2D grid yields encoded features at each resolution, ensuring the "accuracy" of the spatial information. Interpolation compensates for positional errors in discrete grids, making the encoded features of the projected positions more closely match the real spatial coordinates, avoiding positional deviations caused by grid discreteness. All resolution encoded features under the same plane are combined to form the target encoded features. The three-plane target features are then stitched together to form spatial information, achieving "high dimensionality and completeness" of the spatial information. The combination of multi-resolution features preserves spatial details and global correlations, while the stitching of the three planes integrates full-dimensional 3D spatial information, providing rich spatial context for subsequent fusion of multimodal features.

[0029] Furthermore, the multimodal features are fused with spatial information through a multi-head cross-attention mechanism to predict the offset of each Gaussian unit, specifically as follows:

[0030] Use the spatial information as the initial input feature;

[0031] The initial input features and the multimodal features are input into multiple sequentially connected attention layers. In each attention layer, the current input features and the multimodal features are fused through a multi-head cross-attention mechanism to obtain the current output features. A residual connection is performed between the current output features and the current input features to obtain intermediate features. The intermediate features are then subjected to a nonlinear transformation through a multilayer perceptron layer to obtain the current transformed features. A residual connection is performed between the current transformed features and the intermediate features to obtain the current fused features. This process continues until the last attention layer is reached to obtain the target fused features.

[0032] The target fusion features are input into a multilayer perceptron network to predict the offset of each Gaussian cell.

[0033] This approach prioritizes spatial information as the initial input feature, ensuring a solid foundation for spatial localization. By using spatial information as the starting point for fusion, it avoids spatial misalignment caused by multimodal feature dominance (e.g., focusing solely on audio without considering location, leading to mouth Gaussian units shifting to the cheek). In the multi-sequential attention layer, multi-head cross-attention fuses the current input with multimodal features, residual connections (input-output, intermediate-transformed), and multilayer perceptron (MLP) nonlinear transformation to achieve "deep fusion + stable training": Multi-head cross-attention simultaneously captures the multidimensional correlation between multimodal features and spatial information (e.g., camera pose → viewpoint → Gaussian unit position shift direction, audio → mouth shape → Gaussian unit scale change); Residual connections prevent gradient vanishing in deep networks, ensuring no information loss during fusion (e.g., spatial information can still accurately constrain position after multiple rounds of fusion); MLP nonlinear transformation fits the complex mapping relationship between multimodal and spatial information (e.g., different audio intensities correspond to different mouth Gaussian unit shift amplitudes). The target fusion features are input into the MLP network to predict the Gaussian offset, ensuring the "prediction accuracy" of the offset: the MLP can transform the fused high-dimensional features into specific offset parameters (such as position X direction +0.5mm, rotation +5°), which directly serve the Gaussian update.

[0034] Furthermore, the extraction of camera pose, 3D facial mesh, and facial feature point sequence based on the input video sequence specifically involves:

[0035] The video sequence is fitted using a 3D deformable model to extract the camera pose and the 3D facial mesh.

[0036] The video sequence is processed using the MediaPipe framework to extract the facial feature point sequence;

[0037] The audio feature sequence is obtained by extracting the audio track of the video sequence using an audio feature extraction model, wherein the audio feature extraction model is obtained by adjusting the wav2vec2 model based on a Chinese speech dataset.

[0038] By fitting a 3D deformable model to a video sequence to extract the camera pose and 3D facial mesh, it ensures "accurate geometric features": The 3D deformable model can dynamically adjust the template mesh according to video frames, making the output 3D facial mesh fit the real human face geometric structure, and the camera pose accurately restores the shooting perspective. The MediaPipe framework processes the video sequence to extract the facial feature point sequence, ensuring "complete and real-time facial features": MediaPipe is a mature facial feature point extraction tool that can accurately capture feature points in multiple regions such as the mouth and eyes, and adapts to the real-time nature of the video sequence. The output sequence can reflect the dynamic changes of the face. The wav2vec2 model adjusted based on the Chinese speech dataset extracts the audio feature sequence from the video audio track, ensuring "audio feature scene adaptation (Chinese)": The model optimized for Chinese speech can accurately extract the features of Chinese pronunciation (such as tone, vowel differences), avoiding the deviation of the general model in extracting Chinese speech features (such as confusing the tone features of "你" and "理").

[0039] Further, the audio feature extraction model is obtained by adjusting the wav2vec2 model based on the Chinese speech dataset, specifically:

[0040] Replace the English vocabulary of the wav2vec2 model with a Chinese vocabulary, and during the training process of the wav2vec2 model, freeze all parameters except the last two encoder layers and the CTC layer of the wav2vec2 model, and use the Chinese speech dataset to train the last two encoder layers and the CTC layer to obtain the audio feature extraction model.

[0041] By replacing the English vocabulary of the wav2vec2 model with a Chinese vocabulary, it achieves "vocabulary-level adaptation to Chinese": Solving the deviation of the original model (English) in recognizing Chinese vocabulary, enabling the model to accurately match Chinese speech with corresponding vocabulary features (such as the distinction in speech features between "苹果" and "香蕉"). Freezing all parameters except the last two encoder + CTC layers and training these two layers with the Chinese speech dataset achieves "lightweight training + knowledge reuse": Freezing parameters: Retaining the general speech feature extraction ability of the wav2vec2 pre-trained model (such as audio frequency and rhythm recognition), avoiding a huge increase in computational complexity caused by training from scratch; Training the key layers (the last two encoder + CTC layers): Optimizing only for the unique features of Chinese speech (such as tone, Chinese syllable structure), enabling the model to accurately extract Chinese speech features, and the CTC layer (connectionist temporal classification) ensures the matching of audio features with the time series.

[0042] Another embodiment of the present invention also provides a digital human portrait generation system, including: a feature processing module, a feature prediction module, an offset prediction module, and a rendering module;

[0043] The feature processing module is used to extract camera pose, 3D facial mesh, facial feature point sequence and audio feature sequence based on the input video sequence, transform the vertices of the 3D facial mesh to obtain an initial 3D Gaussian element set, wherein the facial feature point sequence includes feature points of the mouth region and feature points of a second region other than the mouth region, and the initial 3D Gaussian element set includes several Gaussian elements, and each Gaussian element includes spatial position, rotation, scale, spherical harmonic coefficient, color and transparency attributes;

[0044] The feature prediction module is used to take the feature points of the second region as prior pose feature points, combine them with the audio feature sequence as conditional input, and predict the mouth feature point sequence through a conditional variational autoencoder.

[0045] The offset prediction module is used to use the audio feature sequence, the mouth feature point sequence and the camera pose as multimodal features, and fuse the multimodal features with spatial information through a multi-head cross-attention mechanism to predict the offset of each Gaussian unit. The spatial information is a high-dimensional feature vector representation of the spatial position of the Gaussian unit obtained by multi-resolution three-plane hash encoding.

[0046] The rendering module is used to update the initial 3D Gaussian primitive set based on each of the offsets to obtain a dynamic 3D Gaussian field, and to render the dynamic 3D Gaussian field through a rasterization rendering pipeline to obtain a digital human portrait image.

[0047] This invention extracts camera pose, a 3D facial mesh, facial feature point sequences including the mouth and a second region, and audio feature sequences. It then transforms the vertices of the 3D facial mesh to obtain an initial 3D Gaussian primitive set including spatial position, rotation, scale, spherical harmonic coefficients, color, and transparency. This provides a complete and accurate static foundation for subsequent dynamic adjustments: the 3D facial mesh ensures the initial geometric structure of the Gaussian primitives conforms to a realistic human face; multi-attribute Gaussian primitives can restore facial details (such as skin color and transparency) with high realism; and the camera pose lays the groundwork for subsequent viewpoint alignment. Using the feature points of the second region as pose priors and combining them with audio features, a conditional variational autoencoder (CVAE) predicts the mouth feature point sequence, ensuring the digital human's "dynamic coordination": pose priors constrain non-mouth poses (such as eyes and head) to remain orderly; and CVAE combined with audio predicts mouth shapes, achieving high-precision lip-sync. Using audio, mouth sequences, and camera pose as multimodal features, this method predicts Gaussian pixel dynamic offsets by fusing multimodal features with spatial information obtained from multi-resolution three-plane hash encoding through multi-head cross-attention, ensuring "precise dynamic details and consistent viewpoint" for the digital human. Multi-head cross-attention achieves deep correlation of multimodal information (e.g., audio intensity corresponds to changes in the scale of Gaussian pixels in the mouth), and spatial information encoding accurately locates Gaussian pixel positions, avoiding misalignment of facial details during dynamic adjustments and enriching natural facial expression dynamics. Camera pose is integrated with multimodal features to ensure that the dynamic offset matches the original video viewpoint. The initial Gaussian pixels are updated based on the dynamic offset to obtain a dynamic 3D Gaussian field, which is then rasterized and rendered to output the digital human portrait, significantly improving the generation efficiency of the digital human portrait. It can achieve a high frame rate output of over 30 frames per second, meeting the latency requirements of real-time interactive scenarios (such as real-time digital human live streaming and virtual dialogue).

[0048] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of the digital human portrait generation method of the present invention.

[0049] Another embodiment of the present invention also provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the digital human portrait generation method of the present invention. Attached Figure Description

[0050] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is one of the flowcharts illustrating a digital human portrait generation method provided in an embodiment of the present invention;

[0052] Figure 2 This is a second schematic flowchart of a digital human portrait generation method provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of a two-stage processing framework for PoseTalker provided in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of an initialization method combining a 3D deformable model (3DMM) and multimodal features provided in an embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram illustrating the visualization result of attention score provided in an embodiment of the present invention;

[0056] Figure 6 This is a schematic diagram showing the comparison results of keyframes generated by self-driven and cross-driven methods according to an embodiment of the present invention;

[0057] Figure 7 This is a schematic diagram of the structure of a digital human portrait generation system provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0060] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0061] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0062] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0063] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0064] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0065] See Figure 1 To address the challenge of achieving high-precision lip-sync and rich, natural facial expression dynamics while maintaining high rendering efficiency in existing technologies, an embodiment of this invention provides a digital human portrait generation method, comprising:

[0066] Step S101: Based on the input video sequence, extract the camera pose, 3D facial mesh, facial feature point sequence, and audio feature sequence. Transform the vertices of the 3D facial mesh to obtain an initial 3D Gaussian unit set. The facial feature point sequence includes feature points in the mouth region and feature points in a second region other than the mouth region. The initial 3D Gaussian unit set includes several Gaussian units, and each Gaussian unit includes spatial position, rotation, scale, spherical harmonic coefficient, color, and transparency attributes.

[0067] It should be noted that the present invention proposes a method such as Figure 2 The second flowchart illustrating the digital human portrait generation method, and as shown... Figure 3The diagram shows a two-stage processing framework for PoseTalker, which combines multimodal feature-driven processing with 3D Gaussian sputtering (3DGS) rendering technology to achieve high-fidelity, audio-driven digital human portrait generation.

[0068] In this embodiment, the input data is monocular video, including audio. For example... Figure 2 As shown, based on the input monocular video sequence, video frames are extracted (to extract facial feature points, pose information, and background, etc.) and audio (to extract audio features) to obtain multimodal features. The vertices of the 3D facial mesh obtained by fitting a 3D deformable model (3DMM) are used as the spatial center points of the corresponding Gaussian elements. In terms of spatial representation, this invention uses explicit 3DGS modeling to convert each 3D facial mesh point into an initial set of 3D Gaussian elements, and each Gaussian element... Spatial position, rotation, scale, spherical harmonic coefficients, color, and transparency attributes are determined by the position parameter x, orientation parameter r (usually expressed mathematically as a rotation matrix), scaling parameter s, spherical harmonic function SH (used to encode color information), and transparency parameter α, respectively, and are represented as follows: This explicit sparse representation reduces computational complexity by 86.7% compared to NeRF's implicit volume sampling, laying the foundation for subsequent real-time rendering.

[0069] As an example of an embodiment of the present invention, the extraction of camera pose, 3D facial mesh, and facial feature point sequence based on the input video sequence specifically involves: fitting the video sequence using a 3D deformable model to extract the camera pose and the 3D facial mesh; processing the video sequence using the MediaPipe framework to extract the facial feature point sequence; and extracting the audio feature sequence from the audio track of the video sequence using an audio feature extraction model, wherein the audio feature extraction model is obtained by adjusting the wav2vec2 model based on a Chinese speech dataset.

[0070] In this embodiment, in the initial stage of digital human modeling, such as Figure 4As shown, this invention proposes an initialization method combining a 3D Deformable Model (3DMM) and multimodal features. Specifically, the 3DMM is used to fit the input monocular video frame, extracting facial information from the video frame. 3DMM coefficients are used for fitting, for example, extracting individual identity coefficients, expression coefficients, and precise 3D mesh point information, and then inferring facial mesh points to generate an initial facial geometry with individualized features. In code implementation, this is achieved by inferring the 3DMM coefficients using the `forward_geo` function. The MediaPipe facial mesh model is used to process the video frame, extracting a complete facial keypoint sequence containing 478 feature points. This sequence is divided into mouth region feature points (for supervised training) and a second region feature point (excluding the mouth) as pose prior. The optimized wav2vec2 model was used to extract audio feature vectors from the video audio track. For Chinese scenarios, the model was fine-tuned using the AISHELL-1 dataset, and its vocabulary was replaced with a 29-dimensional Pinyin vocabulary to improve the Chinese feature extraction capability.

[0071] As an example of an embodiment of the present invention, the audio feature extraction model is obtained by adjusting the wav2vec2 model based on a Chinese speech dataset. Specifically, the English vocabulary of the wav2vec2 model is replaced with a Chinese vocabulary, and during the training process of the wav2vec2 model, all parameters of the wav2vec2 model except for the last two encoder layers and the CTC layer are frozen. The last two encoder layers and the CTC layer are then trained using the Chinese speech dataset to obtain the audio feature extraction model.

[0072] In this embodiment, the mainstream audio extractor is wav2vec2, which has relatively weak information extraction capabilities for Chinese audio, limiting its future application scenarios. Therefore, this embodiment uses the AISHELL-1 Chinese dataset to fine-tune the model weights to optimize the feature extraction capabilities for Chinese audio. First, resetting the dataset requires adjusting the model's vocabulary. The vocabulary for the base weights uses Turkish characters with a dimension of 44, while for the Chinese dataset, this paper attempts to use either Chinese characters or Pinyin to construct the vocabulary. Currently, there are open-source Chinese character vocabulary weights with a dimension of 3503. A self-made Pinyin vocabulary has a dimension of 29. In subsequent comparative experiments, the Pinyin vocabulary showed the best performance on the Chinese dataset. When fine-tuning the Wav2vec2 model, the experiment freezes all parameters in the network except for the last two encoder layers and the CTC layer to reduce training time and required computational resources. The last two encoder layers (used for fine-tuning higher-level features) and the CTC layer (used to optimize the output mapping to Pinyin) are then unfrozen.

[0073] Step S102: Using the feature points of the second region as prior pose feature points, and combining them with the audio feature sequence as conditional input, the mouth feature point sequence is predicted by a conditional variational autoencoder.

[0074] In this embodiment, we can typically obtain audio clips easily. Since our model synthesizes a speaking portrait using audio and corresponding facial feature points, we need a generator to synthesize lip-sync feature points corresponding to the audio. This invention proposes a facial feature point generation method based on a conditional variational autoencoder (e.g., ...). Figure 3 (As shown in the lower left), this method uses audio features and facial pose prior feature points other than mouth shape as conditional inputs to obtain temporally smooth and style-controllable mouth movement sequences.

[0075] As an example of an embodiment of the present invention, the conditional variational autoencoder is obtained by training an initial encoder and an initial decoder, specifically: acquiring a training video sequence and extracting an audio feature training sequence and a facial feature point training sequence; using the mouth feature point training sequence in the facial feature point training sequence as the encoding target, and using the pose prior feature point training sequence in the facial feature point training sequence and the audio feature training sequence as conditional inputs, obtaining latent variable distribution parameters through a fully connected network in the initial encoder, and sampling latent variables from the latent variable distribution parameters through a reparameterization technique; using the audio feature training sequence, the pose prior feature point training sequence, and the latent variables as inputs, predicting through the initial decoder to obtain a mouth feature point prediction sequence, wherein the initial decoder uses a Transformer cross-attention mechanism to fuse the audio feature training sequence and the pose prior feature point training sequence; calculating the error between the mouth feature point training sequence and the mouth feature point prediction sequence through a composite loss function, and iteratively optimizing the error to obtain the conditional variational autoencoder.

[0076] In this embodiment, a facial feature point generator is first employed, which is trained based on multiple frames of pose prior information and multiple frames of audio information. The representation of the pose prior feature points is as follows: The input audio is represented as Finally, the generator outputs the mouth synchronization feature points as follows: Where, n o and n p These represent the number of lip and pose feature points, respectively. To improve motion smoothness, multiple frames are used as a period T, which is considered as the feature of one frame. Audio features are further extracted through two-dimensional convolution operations for model training and inference, resulting in the audio feature representation: a t =

[0077] Conv2D(m t For pose feature points, since they are essentially a one-dimensional set of points, we use one-dimensional convolution to further extract the latent representation of the feature points and obtain the feature point features: p t =Conv1D(l p t ); By analyzing audio a t and pose feature point p t Encode the audio to obtain the initial audio. and pose feature points The encoding process is accomplished through positional encoding γ(·), represented as Za =

[0078] γ(at), Zp = γ(pt). In the Enc part of the encoder, the actual mouth shape feature points are... with pose prior feature points As conditional input, the latent variable distribution parameters are obtained through a fully connected network: And latent variables are obtained through reparameterization techniques: z=μ+σ⊙ε,ε~N(0,I), and the decoder Dec part uses audio features a t Posture features p t Taking the latent variable z as input, the multimodal information is fused through the Transformer cross-attention module, and the predicted mouth shape feature points are output: The audio token is used as the key (K) and value (V), and the posture token is used as the query (Q), thus ensuring that the mouth movements are precisely adjusted according to changes in audio and are constrained by stable priors such as head posture.

[0079] It should be noted that the "extracting audio features through the wav2vec2 model" mentioned in the previous steps refers to global feature extraction for the entire audio sequence, mainly providing high-level speech semantic information (such as pronunciation content and speech style), providing speech priors for digital human-driven systems. "Extracting audio features through two-dimensional convolutional operations," on the other hand, further processes these extracted audio features to obtain local time-frequency features synchronously related to mouth movements, which are used for model training and inference. The former consists of pre-extracted global speech features, while the latter is a feature encoding specifically designed for the mouth movement generation task; their roles are complementary rather than redundant.

[0080] As an example of an embodiment of the present invention, the composite loss function is obtained by calculating reconstruction loss, temporal smoothing loss, synchronization loss, and divergence loss. Specifically, the reconstruction loss is obtained by calculating the first norm difference between the mouth feature point training sequence and the mouth feature point prediction sequence; the temporal smoothing loss is obtained by calculating the second norm difference between the mouth feature point prediction sequences at adjacent time steps; the synchronization loss is obtained by calculating the synchronization between the audio feature training sequence and the facial feature point training sequence using the cross-entropy loss function; and the divergence loss is obtained by calculating the KL divergence between the latent variable distribution and the standard normal distribution.

[0081] In this embodiment, to ensure the accuracy of the generated feature points, we designed a loss function. The loss function includes the L1 error for predicting lip feature points and the L2 loss set to ensure temporal consistency, expressed as: To better infer facial landmarks synchronized with audio, we introduce a novel loss calculation to quantify the synchronization between facial landmarks and audio. Specifically, based on the principles of CLIP and the Wav2lip algorithm, we propose a loss calculation based on 2D landmark synchronization with audio, namely... In implementation, firstly, positive and negative samples are defined. Then, two convolutional modules are used to downsample facial feature points and audio data respectively, obtaining two feature vectors of the same dimension. Finally, the cross-entropy loss (CE) is used to calculate the error, which is expressed as... In addition, increase KL divergence To ensure that the spatial distribution approximates the standard normal distribution: In summary, the composite loss is obtained, expressed as:

[0082] Step S103: The audio feature sequence, the mouth feature point sequence, and the camera pose are used as multimodal features. The multimodal features are fused with spatial information through a multi-head cross-attention mechanism to predict the offset of each Gaussian unit. The spatial information is a high-dimensional feature vector representation of the spatial position of the Gaussian unit obtained by multi-resolution three-plane hash encoding.

[0083] In this embodiment, through the design of a facial feature point generator, the present invention can achieve accurate audio-driven lip-sync generation. Subsequently, we designed a 3DGS-based renderer module (3DGS-BASED RENDERER, such as...). Figure 3(As shown in the lower right corner), this module is used to generate realistic speaker portrait images. It combines multimodal driving information composed of audio features, facial key points, and camera pose, and performs dynamic reconstruction and rendering through explicit sparse Gaussian primitive representation and a rasterization rendering pipeline. By predicting the dynamic offset of Gaussian primitives in each frame (including changes in position, scale, and color transparency), it achieves strict synchronization between mouth movements and speech, and a natural and smooth transition between facial expressions and head posture, effectively solving the problems of lacking realistic three-dimensional structure in 2D methods and the high computational cost and difficulty in real-time generation of existing implicit NeRF methods. During 3DGS rendering, to better adapt to the sampling of dynamic scenes and improve the sampling accuracy and dynamic details of spatial information, this invention employs multi-resolution three-plane encoding technology to map Gaussian primitives to sub-planes, performing spatial interpolation to capture higher details and accuracy, obtaining spatial information H(x). To generate dynamic effects, this invention utilizes a multi-head cross-attention mechanism to fuse spatial and modal features, predicting the offset of Gaussian primitive attributes. Therefore, the 3DGS Gaussian primitive is defined as: Among them, the offset of Gaussian element deformation Dynamic features (such as facial landmarks, audio, etc.) are input into the spatial attention module, which outputs the offset of each Gaussian parameter, denoted as...

[0084] In the process of fusing spatial and modal features using a multi-head cross-attention mechanism, such as Figure 5 The image shown is a visualization of the attention score, representing the attention score for dynamic features and spatial information. High attention scores are displayed in red, followed by green, with insignificant scores displayed in blue. Figure 5 It is known that the facial driving region is different for each driver. Therefore, this embodiment adopts a three-layer, staged, multi-head cross-attention mechanism, using Gaussian meta-features obtained through multi-resolution three-plane encoding as the unified fusion basis: the first layer fuses audio features and spatial features to drive lip movements in sync with speech; the second layer fuses pose prior features and spatial features to coordinate global facial expressions and head movements; the third layer fuses camera pose and spatial features to ensure cross-view consistency. Unlike existing cross-attention methods that fuse all modalities at once, this scheme calculates the principal feature weights in stages, significantly improving lip-sync accuracy, facial expression smoothness, and cross-view stability.

[0085] As an example of an embodiment of the present invention, the spatial information is a high-dimensional feature vector representation of the spatial position of the Gaussian element obtained by multi-resolution three-plane hash encoding. Specifically, the spatial position of the Gaussian element is projected onto three orthogonal planes to obtain the projection position corresponding to each plane, wherein each plane includes multiple two-dimensional grids of different resolutions; the projection position is interpolated in each of the two-dimensional grids to obtain the encoding features of the projection position at each resolution; the encoding features corresponding to all resolutions under the same plane are combined to obtain the target encoding features; the target encoding features of the three orthogonal planes are concatenated to obtain the spatial information.

[0086] In this embodiment, NeRF initially uses hash encoding to quickly sample spatial features, reducing the network's burden without compromising quality. However, for dynamic scenes, when the MLP decoder needs to process multiple dynamic features simultaneously, hash encoding collisions occur frequently, severely impacting reconstruction quality and training efficiency. Furthermore, each Gaussian element is an independent entity; during its deformation, it is generally desirable for the Gaussian deformation to influence changes in other Gaussians in the relevant region, resulting in smoother overall motion. Using a multi-resolution three-plane representation of spatial feature information can more efficiently provide deformation information to the network. Specifically, for each 3D Gaussian element, this invention projects it onto three orthogonal planes and interpolates the point into a uniformly spaced 2D grid on each plane for encoding. By interpolating each plane, each high-dimensional grid is projected into a low-dimensional feature space. Finally, these low-dimensional features are merged through a concatenation operation to form Gaussian spatial information, represented as...

[0087] As an example of an embodiment of the present invention, the step of fusing the multimodal features and spatial information through a multi-head cross-attention mechanism to predict the offset of each Gaussian unit specifically involves: using the spatial information as the initial input feature; inputting the initial input feature and the multimodal feature into multiple sequentially connected attention layers; in each attention layer, fusing the current input feature and the multimodal feature through a multi-head cross-attention mechanism to obtain the current output feature; performing a residual connection between the current output feature and the current input feature to obtain intermediate features; performing a nonlinear transformation on the intermediate features through a multilayer perceptron layer to obtain the current transformed feature; performing a residual connection between the current transformed feature and the intermediate feature to obtain the current fused feature; and so on until the last attention layer is reached to obtain the target fused feature; inputting the target fused feature into the multilayer perceptron network to predict the offset of each Gaussian unit.

[0088] In this embodiment, relying solely on audio information can only reflect the movement of the mouth, making it difficult to generate realistic and complex overall facial movements. Therefore, we need to introduce more dynamic features to drive the movement of facial regions. Facial feature points are unique and can accurately describe facial movements; some studies have already used them as driving features for facial movements. To reconstruct a more refined and continuous dynamic face, this embodiment uses multi-frame facial feature point data as feature input for a single frame to capture the details of facial movements. Furthermore, scene changes and head movements are mainly captured by spatial information, but facial feature points and audio are difficult to reflect external factors (such as changes in lighting). Therefore, drawing on the approach of NeRF, camera pose is introduced as an external driving factor, considered as the camera's viewpoint information. Finally, the dynamic features are divided into three categories: facial feature points, audio features, and camera viewpoint information.

[0089] Matching the relevance of audio features to spatial regions helps the model better predict the offset of Gaussian units (GSUs). Previous works have proposed similar methods; for example, ER-NeRF uses a linear attention mechanism to adjust the audio relevance of each spatial sampling point. However, unlike NeRF, which has a fixed and large number of sampling points, our constructed normalized Gaussian space has sparse 3DGS numbers distributed only on the face surface. Furthermore, as the video progresses, facial information at the same spatial location changes, making it impossible to fix the weight of each point relative to dynamic features. Therefore, we employ a sampling cross-attention mechanism to capture the relationship between dynamic features and spatial information.

[0090] Our spatial attention module uses spatial information as the initial token z0 = f(μ) and captures the correlation between dynamic features (facial landmarks, audio, and camera viewpoints) and spatial regions through a multi-head cross-attention mechanism (TMCA) and a multi-layer perceptron layer (TMLP).

[0091]

[0092]

[0093] Among them, l t a t and v t These represent the facial feature points, audio features, and camera viewpoint features of frame t, respectively; the output z... l t The fused features successfully capture the relationship between dynamic features and spatial information, and are used to predict the offset of Gaussian elements. Visualization of attention scores as follows Figure 5As shown, the audio focuses primarily on mouth movements, facial feature points focus on major facial motion areas (including the mouth and other facial regions), and the camera viewpoint is used to locate external factors acting on the head. Finally, we utilize multiple shallow MLP networks D θ Output the offset of each high-order cell: This offset is used to update the Gaussian unit, enabling the dynamic digital human to be driven.

[0094] Step S104: Update the initial 3D Gaussian set based on each offset to obtain a dynamic 3D Gaussian field. Render the dynamic 3D Gaussian field through the rasterization rendering pipeline to obtain a digital human portrait image.

[0095] In this embodiment, a 2-layer MLP network D is defined for each parameter. θ The Gaussian parameter offsets are predicted separately: position offset ΔH(x), scale offset Δs, rotation offset Δr, spherical harmonic coefficient offset ΔSH, and transparency offset Δα. These offsets are directly added to the original Gaussian parameters to form a new Gaussian representation. To simplify the representation, we treat the offset as a new parameter and statistically analyze all offset information, i.e. so The prediction process is performed on a per-Gaussian basis, with the offset of the network output corresponding one-to-one with the input Gaussian pixels. The updated dynamic 3D Gaussian field is then input into a customized rasterization rendering pipeline. This pipeline is designed to efficiently convert the updated, thousands of dynamic 3D Gaussian pixels into a 2D pixel image. Its core process is not traditional triangular rasterization, but rather a "Gaussian sputtering" algorithm specifically designed for Gaussian point clouds. The digital human portrait image is then rendered through the rasterization rendering pipeline.

[0096] Compared with existing technologies, the digital human synthesis method based on multimodal driving and three-dimensional Gaussian sputtering (3DGS) of the present invention has the following advantages:

[0097] 1. Enhance the realism and naturalness of digital human generation: This invention introduces multimodal driving information such as facial feature points, audio features, and camera pose, and adopts a feature fusion strategy based on cross-attention to achieve high-precision synchronization between audio and lip movements. This makes the generated digital human's expressions more natural and the head posture transitions smoother, effectively overcoming the problems of existing two-dimensional digital humans lacking three-dimensional structural support and having unnatural expressions and movements.

[0098] 2. Improved 3D representation and rendering efficiency: This invention adopts explicit sparse 3D Gaussian sputtering representation and combines it with multi-resolution three-plane hash coding technology to improve the spatial information sampling accuracy and model convergence speed. It avoids the defects of existing implicit volume rendering methods in dynamic scenes, such as large computational load and poor real-time performance, and realizes high frame rate and high fidelity 3D digital human rendering.

[0099] 3. Enhance the adaptability and generalization ability of Chinese speech-driven systems: To address the problem of insufficient adaptability of existing audio feature extractors to Chinese speech, this invention fine-tunes the wav2vec2 model based on the AISHELL-1 Chinese speech dataset and optimizes feature representation capabilities using a pinyin vocabulary. This enables the system to maintain high lip-sync accuracy and facial expression naturalness under conditions of multiple speech rates, multiple accents, and complex contexts, significantly improving its applicability and generalization performance in Chinese speech-driven scenarios.

[0100] 4. Excellent scalability and application value: The two-stage framework (facial feature point generation and dynamic Gaussian field rendering) proposed in this invention has excellent scalability, can be seamlessly integrated with existing audio driver pipelines, and supports different languages ​​and multi-speaker scenarios, providing reliable technical support for virtual human interaction, virtual anchors, film and television production and other digital human applications.

[0101] As an example of an embodiment of the present invention, quantitative evaluation was performed using different methods. Data-driven synthesis was conducted on the same identity test set, and the quality of portrait reconstruction was compared. Table 1 below shows the quality comparison results. Figure 6 The diagram shows a comparison of keyframes generated by self-driven and cross-driven algorithms, used to characterize the visual lip-sync and digital human reconstruction effects of different algorithms under the same audio. The yellow boxes indicate low-quality phenomena such as blurring, artifacts, or inaccurate lip-sync.

[0102] Table 1

[0103]

[0104]

[0105] Compared to recent best methods: (1) This invention (PoseTalker) performs well on most metrics. (2) Specifically, from the perspective of video reconstruction, PoseTalker rendering maintains the best visual quality. The Wav2lip algorithm has lower fidelity in image reconstruction and performs poorly in FID and LPIPS. Thanks to the representation capabilities of 3DGS, PoseTalker outperforms NeRF in detail reconstruction compared to GaussianTalker. In addition, PoseTalker also has a higher evaluation score on LMD due to the addition of 2D keypoint data as a driver. (3) In terms of lip-sync, Wav2lip tends to maintain the best and closest synchronization confidence score to real video because it is co-trained with SyncNet. The method in this chapter is slightly inferior, but maintains a high confidence score overall. (4) Due to the efficient rendering capabilities of 3DGS, PoseTalker's frame rate far exceeds that of NeRF-based work. However, compared to GaussianTalker, PoseTalker requires slightly more time to process keypoint data with dynamic features.

[0106] Table 2. Performance comparison of different languages ​​after fine-tuning the audio extractor

[0107]

[0108] This invention sets up three audio extraction models and fine-tuned model experiments, and compares the performance changes of Chinese and English sample datasets respectively. As shown in Table 2, the fine-tuned model has improved sensitivity to Chinese, and the mouth confidence score of the professor dataset has improved. However, the performance of English audio has decreased significantly, and the model has lost the ability to understand other audio. Therefore, in order to optimize the loss, the model focuses on reconstruction (highest quality PSNR) and ignores the mouth confidence score.

[0109] like Figure 7 As shown, based on the above method embodiments, corresponding system embodiments are provided;

[0110] An embodiment of the present invention provides a digital human portrait generation system 700, including: a feature processing module 701, a feature prediction module 702, an offset prediction module 703, and a rendering module 704;

[0111] The feature processing module 701 is used to extract camera pose, 3D facial mesh, facial feature point sequence and audio feature sequence based on the input video sequence, transform the vertices of the 3D facial mesh to obtain an initial 3D Gaussian element set, wherein the facial feature point sequence includes feature points of the mouth region and feature points of a second region other than the mouth region, and the initial 3D Gaussian element set includes a number of Gaussian elements, and each Gaussian element includes spatial position, rotation, scale, spherical harmonic coefficient, color and transparency attributes;

[0112] The feature prediction module 702 is used to take the feature points of the second region as the pose prior feature points, combine them with the audio feature sequence as the conditional input, and predict the mouth feature point sequence through a conditional variational autoencoder.

[0113] The offset prediction module 703 is used to take the audio feature sequence, the mouth feature point sequence and the camera pose as multimodal features, and fuse the multimodal features with spatial information through a multi-head cross-attention mechanism to predict the offset of each Gaussian unit. The spatial information is a high-dimensional feature vector representation of the spatial position of the Gaussian unit obtained by multi-resolution three-plane hash encoding.

[0114] The rendering module 704 is used to update the initial 3D Gaussian primitive set based on each of the offsets to obtain a dynamic 3D Gaussian field, and to render the dynamic 3D Gaussian field through a rasterization rendering pipeline to obtain a digital human portrait image.

[0115] It is understood that the above system embodiments correspond to the method embodiments of the present invention, and can implement the digital human portrait generation method provided by any of the above method embodiments of the present invention.

[0116] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0117] For ease of description and brevity, the system embodiments of the present invention include all the implementation methods described in the above embodiments of the UAV-based information recognition method, and will not be repeated here.

[0118] Based on the above embodiments of the digital human portrait generation method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the digital human portrait generation method of any embodiment of the present invention.

[0119] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0120] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0121] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0122] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the digital human portrait generation method described in any of the above-described method embodiments of the present invention.

[0123] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0124] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating digital human portraits, characterized in that, include: Based on the input video sequence, camera pose, 3D facial mesh, facial feature point sequence, and audio feature sequence are extracted. The vertices of the 3D facial mesh are transformed to obtain an initial 3D Gaussian element set. The facial feature point sequence includes feature points in the mouth region and feature points in a second region other than the mouth region. The initial 3D Gaussian element set includes several Gaussian elements, and each Gaussian element includes spatial position, rotation, scale, spherical harmonic coefficient, color, and transparency attributes. The second region feature points are used as pose prior feature points, and the audio feature sequence is used as conditional input. The mouth feature point sequence is then predicted by a conditional variational autoencoder. The audio feature sequence, the mouth feature point sequence, and the camera pose are used as multimodal features. The multimodal features are fused with spatial information through a multi-head cross-attention mechanism to predict the offset of each Gaussian unit. The spatial information is a high-dimensional feature vector representation of the spatial position of the Gaussian unit obtained by multi-resolution three-plane hash encoding. The initial 3D Gaussian primitive set is updated based on each of the aforementioned offsets to obtain a dynamic 3D Gaussian field. The dynamic 3D Gaussian field is then rendered using a rasterization rendering pipeline to obtain a digital human portrait image.

2. The digital human portrait generation method as described in claim 1, characterized in that, The conditional variational autoencoder is obtained by training an initial encoder and an initial decoder, specifically as follows: Acquire training video sequences and extract audio feature training sequences and facial feature point training sequences; The mouth feature point training sequence in the facial feature point training sequence is used as the encoding target. The pose prior feature point training sequence in the facial feature point training sequence and the audio feature training sequence are used as conditional inputs. The latent variable distribution parameters are obtained through the fully connected network in the initial encoder. The latent variables are then sampled from the latent variable distribution parameters using the reparameterization technique. Using the audio feature training sequence, the pose prior feature point training sequence, and the latent variables as input, the mouth feature point prediction sequence is obtained through the initial decoder prediction. The initial decoder uses the Transformer cross-attention mechanism to fuse the audio feature training sequence and the pose prior feature point training sequence. The error between the training sequence and the predicted sequence of mouth feature points is calculated using a composite loss function, and the error is iteratively optimized to obtain the conditional variational autoencoder.

3. The digital human portrait generation method as described in claim 2, characterized in that, The composite loss function is obtained by calculating the reconstruction loss, temporal smoothing loss, synchronization loss, and divergence loss, specifically as follows: The first norm difference between the training sequence of mouth feature points and the predicted sequence of mouth feature points is calculated to obtain the reconstruction loss; The second norm difference between the predicted sequences of mouth feature points at adjacent time steps is calculated to obtain the temporal smoothing loss; The synchronization loss is obtained by calculating the synchronicity between the audio feature training sequence and the facial feature point training sequence using the cross-entropy loss function. The KL divergence between the latent variable distribution and the standard normal distribution is calculated to obtain the divergence loss.

4. The digital human portrait generation method as described in claim 1, characterized in that, The spatial information is a high-dimensional feature vector representation of the spatial location of the Gaussian element obtained through multi-resolution three-plane hash encoding, specifically: The spatial positions of the Gaussian elements are projected onto three orthogonal planes to obtain the projection positions corresponding to each plane, wherein each plane includes multiple two-dimensional grids of different resolutions; Interpolate the projection position in each of the two-dimensional grids to obtain the encoded features of the projection position at each resolution; The target coding features are obtained by combining the coding features corresponding to all resolutions under the same plane; The spatial information is obtained by concatenating the target encoding features of the three orthogonal planes.

5. The digital human portrait generation method as described in claim 1, characterized in that, The multimodal features and spatial information are fused using a multi-head cross-attention mechanism to predict the offset of each Gaussian unit, specifically as follows: Use the spatial information as the initial input feature; The initial input features and the multimodal features are input into multiple sequentially connected attention layers. In each attention layer, the current input features and the multimodal features are fused through a multi-head cross-attention mechanism to obtain the current output features. A residual connection is performed between the current output features and the current input features to obtain intermediate features. The intermediate features are then subjected to a nonlinear transformation through a multilayer perceptron layer to obtain the current transformed features. A residual connection is performed between the current transformed features and the intermediate features to obtain the current fused features. This process continues until the last attention layer is reached to obtain the target fused features. The target fusion features are input into a multilayer perceptron network to predict the offset of each Gaussian cell.

6. The digital human portrait generation method as described in claim 1, characterized in that, The camera pose, 3D facial mesh, and facial feature point sequence are extracted from the input video sequence, specifically as follows: The video sequence is fitted using a 3D deformable model to extract the camera pose and the 3D facial mesh. The video sequence is processed using the MediaPipe framework to extract the facial feature point sequence; The audio feature sequence is obtained by extracting the audio track of the video sequence using an audio feature extraction model, wherein the audio feature extraction model is obtained by adjusting the wav2vec2 model based on a Chinese speech dataset.

7. The digital human portrait generation method as described in claim 6, characterized in that, The audio feature extraction model is obtained by adjusting the wav2vec2 model based on the Chinese speech dataset, specifically as follows: The English vocabulary of the wav2vec2 model is replaced with a Chinese vocabulary. During the training process of the wav2vec2 model, all parameters of the wav2vec2 model except for the last two encoder layers and the CTC layer are frozen. The last two encoder layers and the CTC layer are trained using the Chinese speech dataset to obtain the audio feature extraction model.

8. A digital human portrait generation system, characterized in that, include: The module consists of a feature processing module, a feature prediction module, an offset prediction module, and a rendering module. The feature processing module is used to extract camera pose, 3D facial mesh, facial feature point sequence and audio feature sequence based on the input video sequence, transform the vertices of the 3D facial mesh to obtain an initial 3D Gaussian element set, wherein the facial feature point sequence includes feature points of the mouth region and feature points of a second region other than the mouth region, and the initial 3D Gaussian element set includes several Gaussian elements, and each Gaussian element includes spatial position, rotation, scale, spherical harmonic coefficient, color and transparency attributes; The feature prediction module is used to take the feature points of the second region as prior pose feature points, combine them with the audio feature sequence as conditional input, and predict the mouth feature point sequence through a conditional variational autoencoder. The offset prediction module is used to use the audio feature sequence, the mouth feature point sequence and the camera pose as multimodal features, and fuse the multimodal features with spatial information through a multi-head cross-attention mechanism to predict the offset of each Gaussian unit. The spatial information is a high-dimensional feature vector representation of the spatial position of the Gaussian unit obtained by multi-resolution three-plane hash encoding. The rendering module is used to update the initial 3D Gaussian primitive set based on each of the offsets to obtain a dynamic 3D Gaussian field, and to render the dynamic 3D Gaussian field through a rasterization rendering pipeline to obtain a digital human portrait image.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the digital human portrait generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the digital human portrait generation method as described in any one of claims 1-7.

Images