Real-time digital human generation method and system

Abstract

The invention relates to a real-time digital person generation method and system, and the method comprises the following steps: obtaining a monocular video of a target person, and extracting 3DMM information in the monocular video; performing Gaussian point initialization on the monocular video according to the 3DMM information to obtain Gaussian parameters in a standard space; extracting voice audio features in the monocular video, and inputting the voice audio features into the audio-motion model to obtain a universal face key point motion sequence; converting the universal face key point motion sequence into a target face key point motion sequence through a projection algorithm; inputting the target face key point motion sequence and the Gaussian parameters in the standard space into a face key point guided Gaussian deformation network to obtain Gaussian deformation parameters; and rendering the Gaussian deformation parameter into a corresponding video frame through a Gaussian rasterizer, thereby obtaining a digital person video of the target person. Compared with the prior art, the digital human video generation accuracy and creation flexibility are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human generation, and in particular to a real-time digital human generation method and system. Background Art

[0002] In recent years, with the rapid development of emerging interactive technologies such as short videos, live streaming, and virtual reality, demand for more natural, vivid, and low-cost interactive experiences has increased. Traditional video content production often relies on real people, which is not only costly but also inflexible. Therefore, the emergence of voice-driven digital human technology can replace real people to a certain extent and effectively create digital avatars with diverse styles and personalization. This technology has been widely used in various fields, including live streaming, online education, intelligent customer service, film and television production, and virtual social networking, demonstrating its broad application value.

[0003] At the same time, users are placing increasingly stringent and diverse demands on voice-driven digital human technology. First, there's a demand for high-fidelity video generation. People expect synthesized video characters to exhibit highly realistic and natural details, including skin detail, facial naturalness, and even details like the inside of the mouth. Second, real-time performance is becoming increasingly important. Especially in real-time interactive scenarios like live broadcasts and video conferencing, users need to ensure that generated videos can achieve a smooth speed of at least 30 frames per second. Furthermore, as creative needs continue to expand, single voice-driven methods are no longer fully sufficient. Users are increasingly preferring multimodal control methods, such as those supporting text input and facial expression coefficient input, to enhance the creative flexibility and interactive authenticity of digital human generation technology. This has led to a pressing need for high-fidelity, accurate lip shape, real-time voice-driven digital human generation technology that supports multimodal input. For example, patent application CN119600190A discloses a method for generating speaking face videos based on Gaussian splattering. This method uses a model that generates dynamic facial expression coefficients based on audio to obtain facial expression coefficients. It then trains a Gaussian deformation network driven by these coefficients to deform the Gaussian point cloud. After rasterization, it generates an image of the target person speaking. However, this method, based on a Gaussian deformation network driven by facial expression coefficients, suffers from low accuracy and lacks analysis of the internal features of the person's mouth. This method fails to address the issues of inaccurate lip movements, facial artifacts, low fidelity, and slow generation speeds common in existing speech-driven digital human generation technologies. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a real-time digital human generation method and system, thereby improving the accuracy and creative flexibility of digital human video generation.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A real-time digital human generation method comprises the following steps:

[0007] Obtain a monocular video of the target person and extract 3DMM information from the monocular video;

[0008] Initialize the Gaussian points of the monocular video according to the 3DMM information in the monocular video to obtain Gaussian parameters in the standard space;

[0009] Extracting speech audio features from the monocular video, and inputting the speech audio features into an audio-motion model to obtain a universal facial key point motion sequence;

[0010] Converting the universal facial key point motion sequence into a target facial key point motion sequence through a projection algorithm;

[0011] Inputting the target facial key point motion sequence and the Gaussian parameters in the canonical space into a facial key point guided Gaussian deformation network to obtain Gaussian deformation parameters, wherein the Gaussian deformation parameters include facial Gaussian deformation parameters and oral cavity Gaussian deformation parameters;

[0012] The Gaussian deformation parameters are rendered into corresponding video frames through a Gaussian rasterizer, thereby obtaining a digital human video of the target person.

[0013] Furthermore, the 3DMM information includes 3DMM coefficients, facial key points and head posture sequences.

[0014] Furthermore, the monocular video is initialized with Gaussian points to obtain Gaussian parameters in the canonical space in the following specific steps:

[0015] Using an optical flow method to smooth the head posture sequence;

[0016] Fitting the face in the monocular video using a 3DMM method to obtain face vertex coordinates, and obtaining average face vertex coordinates based on the face vertex coordinates and a head posture sequence;

[0017] A preset point cloud model is used to initialize the Gaussian points inside the target person's mouth, and the initialized Gaussian points inside the target person's mouth and the average facial vertices are constructed as Gaussian parameters in a coordinate standard space.

[0018] Furthermore, the conversion formula for converting the general facial key point motion sequence into the target facial key point motion sequence through the projection algorithm is:

[0019]

[0020] Where, Xdst is the coordinate value of the target face key point, μ dst is the mean of the target face key points set, x src is the coordinate value of the universal face key point, μ src is the mean of the universal face key point set, σ src is the standard deviation of the universal face key point set, ⊙ is the element-by-element multiplication, and k is the amplitude coefficient.

[0021] Furthermore, the facial key point guided Gaussian deformation network uses the displacement of the K facial key points closest to each Gaussian point to control the offset of the Gaussian point. The offset of the Gaussian point is:

[0022]

[0023] Where Δu i is the offset of the i-th Gaussian point, K is the number of facial key points, g j is the basic weight of the j-th facial key point, w j is the weighted sum of the j-th facial key points, Δp j is the displacement of the j-th facial key point, is the offset of the i-th Gaussian point affected by the expression parameters.

[0024] Furthermore, the facial key point guided Gaussian deformation network includes a facial area multilayer perceptron and an intra-oral area multilayer perceptron. The facial area multilayer perceptron generates facial Gaussian deformation parameters of the target person based on facial key points and AU expression parameters, and the intra-oral area multilayer perceptron generates intra-oral Gaussian deformation parameters of the target person based on speech audio features and key points inside the lips.

[0025] Furthermore, the specific steps of rendering the Gaussian deformation parameters into corresponding video frames through the Gaussian rasterizer are as follows:

[0026] Passing the facial Gaussian deformation parameters through a Gaussian rasterizer to obtain a facial rendering result;

[0027] Passing the Gaussian deformation parameters of the interior of the oral cavity through a Gaussian rasterizer to obtain an interior of the oral cavity rendering result;

[0028] The facial rendering result and the oral cavity interior rendering result are fused to obtain a corresponding video frame.

[0029] Furthermore, the facial rendering result and the oral cavity interior rendering result are fused by masking:

[0030] I' full =I head ×(1-M mouth )+I mouth ×Mmouth

[0031] Where, I' full is the fusion result, I head For the face rendering result, I mouth This is the rendering result of the interior of the mouth, M mouth Mask for the inside of the mouth.

[0032] Furthermore, during the rendering process of the Gaussian rasterizer, the Gaussian rasterizer synthesizes the color of each pixel on the video frame through α-blending technology. The calculation formula of the color is:

[0033]

[0034] Where C is the color represented by the pixel, N is the number of Gaussian points, and c i is the spherical harmonic coefficient of the i-th Gaussian point, o' i is the opacity of the i-th Gaussian point, o' j is the opacity of the j-th Gaussian point.

[0035] According to another aspect of the present invention, there is provided a real-time digital human generation system, comprising:

[0036] 3DMM information extraction module, used to obtain a monocular video of the target person and extract 3DMM information from the monocular video;

[0037] A Gaussian point initialization module is used to initialize the Gaussian points of the monocular video according to the 3DMM information in the monocular video to obtain Gaussian parameters in the standard space;

[0038] A universal facial key point motion sequence acquisition module is used to extract speech and audio features from the monocular video, input the speech and audio features into an audio-motion model, and obtain a universal facial key point motion sequence;

[0039] A target human face key point motion sequence conversion module is used to convert the general human face key point motion sequence into a target human face key point motion sequence through a projection algorithm;

[0040] A Gaussian deformation parameter acquisition module is used to input the target facial key point motion sequence and the Gaussian parameters in the standard space into the facial key point guided Gaussian deformation network to obtain Gaussian deformation parameters, wherein the Gaussian deformation parameters include facial Gaussian deformation parameters and oral cavity Gaussian deformation parameters;

[0041] The digital human video construction module is used to render the Gaussian deformation parameters into corresponding video frames through a Gaussian rasterizer, thereby obtaining a digital human video of the target person.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention guides Gaussian deformation through facial key points. For each Gaussian point, the deformation is precisely controlled by the weighted sum of the displacements of K key points. In order to preserve facial expressions, the AU is introduced as the expression coefficient to ensure that facial movements unrelated to speaking are consistent with the source face. The characteristics of key points and Gaussian points as explicit expressions are utilized. By using facial key points to precisely control the deformation of Gaussian points, a natural and accurate lip shape result is obtained, effectively solving problems such as inaccurate lip shape, jitter, and artifacts, and improving the accuracy of digital human video generation.

[0044] 2. The present invention adopts a specific hierarchical generation strategy. A facial area multilayer perceptron is used to generate the target person's facial Gaussian deformation parameters based on the facial key points and AU expression parameters. An intra-oral area multilayer perceptron is used to generate the target person's intra-oral Gaussian deformation parameters based on the speech audio features and the key points on the inside of the lips. The two parts of the Gaussian deformation parameters are then rendered separately and fused to improve the fidelity of facial details and the accuracy of digital human video generation.

[0045] 3. In practical applications, the present invention accepts the speech audio of any person as input, generates a facial key point sequence through a speech-action generation model, inputs the facial key point-guided Gaussian deformation network to obtain the deformed Gaussian parameters, and finally renders it into corresponding video frames through a rasterizer to generate a speaking video corresponding to the target person. It also supports the input of facial key point sequences, thereby improving the creative flexibility of digital human generation technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a real-time digital human generation method proposed by the present invention;

[0047] Figure 2 This is a flowchart of the Gaussian initialization process;

[0048] Figure 3 Schematic diagram of the Gaussian deformation process guided by facial landmarks. DETAILED DESCRIPTION

[0049] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0050] Abbreviations involved:

[0051] 3D Morphable Model, 3DMM

[0052] Multilayer Perceptron: Multilayer Perceptron, MLP

[0053] Example 1

[0054] This embodiment provides a real-time digital human generation method. Figure 1 As shown, the following steps are included:

[0055] S1. Obtain a monocular video of the target person and extract the 3DMM information from the monocular video.

[0056] The system takes a monocular video of the target person and extracts the 3DMM information of the face from the original video. This information includes 3DMM coefficients, facial key points, and head pose sequences. 3DMM coefficients describe the shape and texture of the face. Facial key points include the location of facial landmarks such as the eyes, nose, and mouth.

[0057] S2. Initialize the Gaussian points of the monocular video according to the 3DMM information in the monocular video to obtain the Gaussian parameters in the standard space.

[0058] Initialize the Gaussian points of the monocular video and obtain the Gaussian parameters in the standard space. Figure 2 Shown, including:

[0059] The optical flow method is used to smooth the head posture sequence to reduce jitter and make the changes in head posture smoother, providing a more stable foundation for subsequent Gaussian point initialization.

[0060] The 3DMM method is used to fit faces in monocular video. Based on the previously extracted 3DMM coefficients and facial key points, the vertex coordinates of each face frame are further accurately fitted. Based on the facial vertex coordinates and the smoothed head pose sequence, the Gaussian center position is initialized as the average facial mesh vertex coordinates to obtain the average facial vertex coordinates. This is done by calculating the average of the facial vertex coordinates in the canonical space and using this average facial vertex coordinate as the initial center position of the Gaussian point.

[0061] Initialize the Gaussian points inside the target person's mouth using a preset point cloud model. This point cloud model is pre-designed based on the anatomical structure and common morphology of the oral cavity. This initialization ensures complete coverage of the target person's head (including the oral cavity), avoiding modeling gaps and ensuring a more reasonable and comprehensive distribution of Gaussian points across the entire head.

[0062] The initialized Gaussian points inside the target person's mouth and the average facial vertices are constructed as Gaussian parameters in the coordinate norm space.

[0063] Among them, the Gaussian point is expressed as:

[0064]

[0065] Where, is the Gaussian point, μ is the center position, q is the rotation parameter, s is the scaling parameter, o is the opacity, and c is the spherical harmonic coefficient.

[0066] S3. Extract speech audio features from the monocular video, input the speech audio features into the audio-motion model, and obtain a universal facial key point motion sequence.

[0067] The HuBERT audio encoder is used to extract speech audio features from monocular video. These features are then fed into a pre-trained audio-motion model, Pitch-VAE, to generate a universal facial landmark motion sequence. This model learns the mapping between speech and facial motion and outputs a universal facial landmark motion sequence corresponding to the speech audio features. The temporal resolution of the motion sequence matches the frequency of audio feature extraction. When input into the speech-motion model, the speech audio features generate a facial landmark motion sequence corresponding to the mouth shape. For example, if the speech content is "The weather is so nice today," the mouth movement in the generated facial landmark motion sequence will match the mouth shape corresponding to "The weather is so nice today." However, this motion is based on a standard face. Given the varying facial shapes and speaking habits of individuals, the amplitude of lip shape changes varies. Therefore, a projection transformation is required to convert the universal facial landmark motion sequence into a target facial landmark motion sequence, thereby guiding the deformation network to generate natural facial appearance changes.

[0068] S4. Convert the general facial key point motion sequence into the target facial key point motion sequence through a projection algorithm.

[0069] A projection algorithm based on affine transformation is used to map the universal facial keypoint motion sequence into the target face space. First, the affine transformation matrix between the universal and target face models is calculated. This matrix is ​​obtained by matching feature points and solving it using the least squares method. The affine transformation matrix is ​​then used to transform the coordinates of each keypoint in the universal facial keypoint motion sequence to obtain the target facial keypoint motion sequence.

[0070] The conversion formula for converting the general face key point motion sequence into the target face key point motion sequence through the projection algorithm is:

[0071]

[0072] Where, X dst is the coordinate value of the target face key point, μ dst is the mean of the target face key points set, x srcis the coordinate value of the universal face key point, μ src is the mean of the universal face key point set, σ src is the standard deviation of the universal face key point set, ⊙ is the element-by-element multiplication, and k is the amplitude coefficient.

[0073] S5. Input the target facial key point motion sequence and the Gaussian parameters in the standard space into the facial key point guided Gaussian deformation network to obtain Gaussian deformation parameters, which include facial Gaussian deformation parameters and oral cavity Gaussian deformation parameters.

[0074] The Gaussian deformation network guided by facial key points uses the displacement of the K facial key points closest to each Gaussian point to control the offset of the Gaussian point, such as Figure 3 As shown, the offset of the Gaussian point is:

[0075]

[0076] Where Δu i is the offset of the i-th Gaussian point, K is the number of facial key points, g j is the basic weight of the j-th facial key point, w j is the weighted sum of the j-th facial key points, Δp j is the displacement of the j-th facial key point, is the offset of the i-th Gaussian point affected by the expression parameters.

[0077] The facial landmark-guided Gaussian deformation network adopts two MLP modules, including a facial region multilayer perceptron and an oral cavity region multilayer perceptron, to train the deformation networks of the facial region and the oral cavity respectively.

[0078] For the facial region, we use facial key points and expression AU parameters to generate the target person's facial Gaussian deformation parameters. AU expression parameters are introduced into the facial region to preserve the expression information generated by facial movements unrelated to speech in the source face. AU expression parameters are obtained through OpenFace. The formula for generating facial Gaussian deformation parameters is:

[0079] G face =f MLP_face (K target ,AU)

[0080] Where G face is the facial Gaussian deformation parameter, f MLP_face is the multilayer perceptron for the facial region, K target is the target face key point motion sequence, and AU is the expression AU parameter.

[0081] For the oral cavity, we combine speech audio features with key points inside the lips to generate Gaussian deformation parameters for the target person's oral cavity. Introducing audio features inside the oral cavity improves the detail and fidelity of the dynamics inside the oral cavity.

[0082] G mouth =f MLP_mouth (K target ,F audio )

[0083] Where G mouth is the Gaussian deformation parameter inside the oral cavity, f MLP_mouth is the multilayer perceptron in the oral cavity, F audio It is the speech audio feature.

[0084] Before training, the facial landmark-guided Gaussian deformation network processes the monocular video to be trained into a format with a frame rate of 25FPS, a resolution of 512×512, and an audio sampling rate of 16kHz. During inference, any input speech audio will also be processed into 16kHz to ensure the consistency of model generation.

[0085] S6. Rendering the Gaussian deformation parameters into corresponding video frames through a Gaussian rasterizer, thereby obtaining a digital human video of the target person.

[0086] The specific steps of rendering the Gaussian deformation parameters into the corresponding video frames through the Gaussian rasterizer are:

[0087] The facial Gaussian deformation parameters are passed through the Gaussian rasterizer to obtain the facial rendering result. The Gaussian rasterizer renders the facial area based on the center position, rotation parameters, scaling parameters, opacity, and spherical harmonic coefficients of each Gaussian point.

[0088] The Gaussian deformation parameters of the interior of the oral cavity are passed through the Gaussian rasterizer to obtain the rendering result of the interior of the oral cavity;

[0089] The facial rendering results and the oral cavity interior rendering results are fused to obtain the corresponding video frames.

[0090] During the rendering process of the Gaussian rasterizer, the Gaussian rasterizer synthesizes the color of each pixel on the video frame through α-blending technology. The color calculation formula is:

[0091]

[0092] Where C is the color represented by the pixel, N is the number of Gaussian points, and c i is the spherical harmonic coefficient of the i-th Gaussian point, o' i is the opacity of the i-th Gaussian point, o' j is the opacity of the j-th Gaussian point.

[0093] The facial rendering result and the oral cavity rendering result are fused through the mask to obtain a complete video frame. The fusion process is as follows:

[0094] I' full =I head ×(1-M mouth )+I mouth ×M mouth

[0095] Where, I' full is the fusion image result, I head For the face rendering result, I mouth This is the rendering result of the interior of the mouth, M mouth Mask for the inside of the mouth.

[0096] After all video frames are rendered, the video frames and the input audio are synthesized into a digital human video with a frame rate of 25FPS and a resolution of 512×512 using the Ffmpeg tool.

[0097] In actual application, users can input a speech audio clip of any person, generate a sequence of facial key points through the speech-action generation model, input the facial key point-guided Gaussian deformation network to obtain the deformed Gaussian parameters, and finally render the corresponding video frames through the rasterizer to generate a speech video corresponding to the target person. The system also supports the input of facial key point sequences, which greatly enhances the creative flexibility of digital human generation technology. Users can easily customize digital human videos with different speech content, expressions and movements according to actual needs, meeting the application requirements of diverse scenarios.

[0098] Example 2

[0099] This embodiment provides a real-time digital human generation system, including:

[0100] 3DMM information extraction module, used to obtain a monocular video of the target person and extract 3DMM information from the monocular video;

[0101] A Gaussian point initialization module is used to initialize the Gaussian points of the monocular video according to the 3DMM information in the monocular video to obtain Gaussian parameters in the standard space;

[0102] A universal facial key point motion sequence acquisition module is used to extract speech audio features from the monocular video, input the audio features of the speech audio into an audio-motion model, and obtain a universal facial key point motion sequence;

[0103] A target human face key point motion sequence conversion module is used to convert the general human face key point motion sequence into a target human face key point motion sequence through a projection algorithm;

[0104] A Gaussian deformation parameter acquisition module is used to input the target facial key point motion sequence and the Gaussian parameters in the standard space into the facial key point guided Gaussian deformation network to obtain Gaussian deformation parameters, wherein the Gaussian deformation parameters include facial Gaussian deformation parameters and oral cavity Gaussian deformation parameters;

[0105] The digital human video construction module is used to render the Gaussian deformation parameters into corresponding video frames through a Gaussian rasterizer, thereby obtaining a digital human video of the target person.

[0106] The rest is the same as in Example 1.

[0107] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A real-time digital human generation method, characterized in that: The following steps are involved: Obtain a monocular video of the target person and extract 3DMM information from the monocular video; Initialize the Gaussian points of the monocular video according to the 3DMM information in the monocular video to obtain Gaussian parameters in the standard space; Extracting speech audio features from the monocular video, and inputting the speech audio features into an audio-motion model to obtain a universal facial key point motion sequence; Converting the universal facial key point motion sequence into a target facial key point motion sequence through a projection algorithm; Inputting the target facial key point motion sequence and the Gaussian parameters in the canonical space into a facial key point guided Gaussian deformation network to obtain Gaussian deformation parameters, wherein the Gaussian deformation parameters include facial Gaussian deformation parameters and oral cavity Gaussian deformation parameters; The Gaussian deformation parameters are rendered into corresponding video frames through a Gaussian rasterizer, thereby obtaining a digital human video of the target person.

2. The real-time digital human generation method according to claim 1, characterized in that: The 3DMM information includes 3DMM coefficients, facial key points and head posture sequences.

3. The real-time digital human generation method according to claim 2, characterized in that: The specific steps of initializing Gaussian points of the monocular video to obtain Gaussian parameters in the standard space are as follows: Using an optical flow method to smooth the head posture sequence; Fitting the face in the monocular video using a 3DMM method to obtain face vertex coordinates, and obtaining average face vertex coordinates based on the face vertex coordinates and a head posture sequence; A preset point cloud model is used to initialize the Gaussian points inside the target person's mouth, and the initialized Gaussian points inside the target person's mouth and the average facial vertices are constructed as Gaussian parameters in a coordinate standard space.

4. The real-time digital human generation method according to claim 1, characterized in that: The conversion formula for converting the general face key point motion sequence into the target face key point motion sequence through the projection algorithm is: Where, X dst is the coordinate value of the target face key point, μ dst is the mean of the target face key points set, x src is the coordinate value of the universal face key point, μ src is the mean of the universal face key point set, σ src is the standard deviation of the universal face key point set, ⊙ is the element-by-element multiplication, and k is the amplitude coefficient.

5. The real-time digital human generation method according to claim 1, characterized in that: The facial key point guided Gaussian deformation network uses the displacement of the K facial key points closest to each Gaussian point to control the offset of the Gaussian point. The offset of the Gaussian point is: Where Δu i is the offset of the i-th Gaussian point, K is the number of facial key points, g j is the basic weight of the j-th facial key point, w j is the weighted sum of the j-th facial key points, Δp j is the displacement of the j-th facial key point, is the offset of the i-th Gaussian point affected by the expression parameters.

6. The real-time digital human generation method according to claim 1, characterized in that: The facial key point guided Gaussian deformation network includes a facial area multilayer perceptron and an intraoral area multilayer perceptron. The facial area multilayer perceptron generates facial Gaussian deformation parameters of the target person based on facial key points and AU expression parameters, and the intraoral area multilayer perceptron generates intraoral Gaussian deformation parameters of the target person based on speech audio features and key points on the inside of the lips.

7. The real-time digital human generation method according to claim 1, characterized in that: The specific steps of rendering the Gaussian deformation parameters into corresponding video frames through the Gaussian rasterizer are: Passing the facial Gaussian deformation parameters through a Gaussian rasterizer to obtain a facial rendering result; Passing the Gaussian deformation parameters of the interior of the oral cavity through a Gaussian rasterizer to obtain an interior of the oral cavity rendering result; The facial rendering result and the oral cavity interior rendering result are fused to obtain a corresponding video frame.

8. The real-time digital human generation method according to claim 7, characterized in that: The facial rendering result and the oral cavity interior rendering result are fused through a mask: I' full =I head ×(1-M mouth )+I mouth ×M mouth Where, I' full is the fusion result, I head For the face rendering result, I mouth This is the rendering result of the interior of the mouth, M mouth Mask for the inside of the mouth.

9. The real-time digital human generation method according to claim 1, characterized in that: During the rendering process of the Gaussian rasterizer, the Gaussian rasterizer synthesizes the color of each pixel on the video frame through α-blending technology. The color calculation formula is: Where C is the color represented by the pixel, N is the number of Gaussian points, and c i is the spherical harmonic coefficient of the i-th Gaussian point, o' i is the opacity of the i-th Gaussian point, o' j is the opacity of the j-th Gaussian point.

10. A real-time digital human generation system, characterized in that: include; 3DMM information extraction module, used to obtain a monocular video of the target person and extract 3DMM information from the monocular video; A Gaussian point initialization module is used to initialize the Gaussian points of the monocular video according to the 3DMM information in the monocular video to obtain Gaussian parameters in the standard space; A universal facial key point motion sequence acquisition module is used to extract speech and audio features from the monocular video, input the speech and audio features into an audio-motion model, and obtain a universal facial key point motion sequence; A target human face key point motion sequence conversion module is used to convert the general human face key point motion sequence into a target human face key point motion sequence through a projection algorithm; A Gaussian deformation parameter acquisition module is used to input the target facial key point motion sequence and the Gaussian parameters in the standard space into the facial key point guided Gaussian deformation network to obtain Gaussian deformation parameters, wherein the Gaussian deformation parameters include facial Gaussian deformation parameters and oral cavity Gaussian deformation parameters; The digital human video construction module is used to render the Gaussian deformation parameters into corresponding video frames through a Gaussian rasterizer, thereby obtaining a digital human video of the target person.

Images