A human head dynamic modeling relightable gaussian mixture shape method

Abstract

The application discloses a human head dynamic modeling relightable Gaussian mixed shape method, which comprises training image collection and preprocessing, mesh mixed shape training, Gaussian mixed shape training and human head animation generation. The relightable Gaussian mixed shape expression comprises a mesh mixed shape and a Gaussian mixed shape; the mesh mixed shape is used for capturing individual characteristics and is responsible for normal and visibility calculation; the Gaussian mixed shape is responsible for coloring and final image rendering; both are composed of a neutral model representing a neutral expression and a plurality of base expression models corresponding to basic expressions. Human face and head models of arbitrary expressions are efficiently generated by linearly mixing the neutral model and the base expression models. The application expands the Gaussian mixed shape expression without reducing drawing quality and running speed, so that the Gaussian mixed shape expression supports relighting applications, and better serves the production of applications such as movies, animation production, network games and remote conferences.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of parametric human head model, head geometry and material modeling, and particularly relates to a relightable Gaussian mixture shape method for dynamic modeling of human head. BACKGROUND

[0002] Researchers have proposed various different models for representing human heads. Early work reconstructed geometry and appearance from images using explicit 3D meshes. The seminal work of Blanz and Vetter (Blanz, V. and Vetter, T., 2023. Amorphable model for the synthesis of 3D faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999) 3DMM (3D Morphable Model) modeled face geometry and appearance with a low-dimensional linear subspace. This inspired a lot of follow-up work, such as extending to nonlinear models (Tran, L. and Liu, X., 2018. Nonlinear 3d face morphable model. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7346-7355).), from videos (Garrido, P., Baumgartner, T., Vetter, T. and Blanz, V., 2013. Video-based reconstruction of animated 3D face models using an input-based morphable model. Computer Graphics Forum, 32(7), pp. 152-161.), and from scans (Alldrin, N., Zoroufy, J.H., Xu, W. and Baker, S., 2018. A 3D morphable model of face shape from 3D video. International Journal of Computer Vision, 126(4), pp. 295-318.). M., Casas, D., Valgaerts, L., Varanasi, K., Perez, P., & Theobalt, C. (2016). Reconstruction of personalized 3D face rigs from monocular video. ACM Transactions on Graphics (TOG), 35(3), 1-15.), sparse views (Cao, C, Wu, H., Weng, Y., Shao, T., & Zhou, K. (2016). Real-time facial animation with image-based dynamic avatars. ACM Transactions on Graphics, 35(4).), and even from a single image (Bai, Z., Cui, Z., Liu, X., & Tan, P. (2021). Riggable 3d face reconstruction via in-network optimization. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 6216-6225).

[0003] Implicit representations, represented by Neural Radiance Field (Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R. and Ng, R., 2021. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), pp.99-106.), have achieved remarkable results in modeling high-fidelity human heads. NeRFace (Gafni, G., Thies, J., Zollhofer, M., & Nierhaus, M. (2021). Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp.8649-8658).) and Head-NeRF (Hong, Y., Peng, B., Xiao, H., Liu, L., & Zhang, J. (2022). HeadNeRF: A real-time NeRF-based parametric head model. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp.20374-20384).) model the human head as a neural radiance field conditioned on expressions.INSTA (Zielonka, W., Bolkart, T. and Thies, J., 2023. Instant volumetric head avatars. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 4574-4584).) establishes a deformable neural radiance field with Instant Neural Graphics Primitives (InstantNGP) (Muller, T., Evans, A., Schied, C. and Keller, A., 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4), pp.1-15.). PointAvatar (Zheng, Y., Yifan, W., Wetzstein, G., Black, M.J. and Hilliges, 0., 2023. Pointavatar: Deformable point-based head avatars from videos. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 21057-21067).) is a point-based representation that utilizes multi-layer perceptrons to represent the deformation field, allowing manipulation of the point cloud according to expressions.

[0004] Recent work uses 3D Gaussian representation (Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph., 42(4), 139-1.) that balances quality and computational efficiency. One approach is to bind Gaussians to a mesh, using mesh morphing to drive the Gaussians, Gaussian Avatars (Qian, S., Kirschstein, T., Schoneveld, L., Davoli, D., Giebenhain, S., & Niesner, M. (2024). Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 20299-20309).) decomposes triangle morphing into rotation and scaling components, then applies them to the corresponding Gaussians. Another explicit representation is Gaussian Blend Shapes (Ma, S., Weng, Y., Shao, T., & Zhou, K. (2024, July). 3d gaussian blendshapes for head avatar animation. In ACM SIGGRAPH 2024 Conference Papers (pp. 1-10).), which represents a human head model with human-like expressions by linearly blending a set of expression blend shapes represented by 3D Gaussians. The advantage of Gaussian Blend Shapes is that it can capture high-frequency details caused by large facial deformations, such as wrinkles. In addition, many methods use neural networks to implicitly encode Gaussian attributes, MonoGaussian Avatar (Chen, Y., Wang, L., Li, Q., Xiao, H., Zhang, S., Yao, H., & Liu, Y. (2024, July). Monogaussianavatar: Monoculargaussian point-based head avatar. In ACM SIGGRAPH 2024 Conference Papers (pp. 1-9).) adopts a morph field similar to PointAvatar, and uses a multi-layer perceptron to predict expression-based Gaussian parameter offsets.In view of the simple expression and excellent fitting ability of Gaussian mixture shape, the present application is based on Gaussian mixture shape, which has significant advantages in performance and image quality.

[0005] For the problem of heavy illumination of human head, a large number of works have studied the heavy illumination of single human head image. Some methods (Cai, Z., Jiang, K., Chen, S. Y., Lai, Y. K., Fu, H., Shi, B., & Gao, L. (2024). Real-time 3D-aware portrait video relighting. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 6221-6231).) are committed to the heavy illumination of monocular human head video, while other methods (Guo, K., Lincoln, P., Davidson, P., Busch, J., Yu, X., Whalen, M., … & Izadi, S. (2019). The relightables: Volumetric performance capture of humans with realistic relighting. ACM Transactions on Graphics (ToG), 38(6), 1-19.) use laboratory light field (light stage) to capture the replayable heavy illumination. However, these methods cannot generate new animations, limiting their applications.

[0006] Another line of research (Saito, S., Schwartz, G., Simon, T., Li, J., & Nam, G. (2024). Relightable gaussian codec avatars. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 130-141).) implements high-fidelity relightable models that can generate new animations. However, these methods require expensive equipment and time-consuming capture processes, making it difficult to popularize among ordinary users. The most similar to the input and output and application scenarios of the present invention are PointAvatar based on point representation and FLARE (Bharadwaj, S., Zheng, Y., Hilliges, 0., Black, M. J., & Fernandez-Abrevaya, V. (2023). Flare: Fast learning of animatable and relightable mesh avatars. arXiv preprint arXiv: 2310.17519.) based on mesh. However, these methods rely on multi-layer perceptron to calculate normals and materials, in contrast, the present invention only needs linear mixing at runtime, thus having higher efficiency. In addition, existing methods either completely ignore visibility calculation or only consider visibility under directional light or point light source. Our method first calculates visibility in all directions, applicable to both diffuse and highlight components, thus achieving higher realism. SUMMARY

[0007] The present invention is aimed at the deficiencies of the prior art and proposes a relightable Gaussian mixture shape method for human head motion modeling. This method takes a segment of head video as input, optimizes the mesh mixture shape and relightable Gaussian mixture shape, and when given new environment light, expression, joint and pose parameters, it further drives the human synthesis to have higher realistic human head animation in real time. Based on the current most advanced human head model, this method expands the relightable function to better serve film, animation production, online games, remote meetings and other applications, and has high practical value.

[0008] The present invention is realized by the following technical solutions:

[0009] A relightable Gaussian mixture shape method for human head dynamic modeling, comprising the following steps:

[0010] (1) Training image collection and preprocessing: a monocular camera is used to shoot a short video of a person speaking, and the video data is processed to obtain various parameters, including: a neutral expression mesh, a set of basic expression meshes, a head foreground image of each frame, a head foreground mask, camera parameters, joint and pose parameters, and expression coefficients;

[0011] (2) Training of mesh blend shape: using the parameters of step (1), a mesh blend shape is established and initialized, an environment map is initialized, and all parameters are jointly optimized according to the image loss to achieve decoupling of material and lighting; the mesh blend shape is composed of a neutral expression mesh and a basic expression mesh;

[0012] (3) Training of re-lightable Gaussian blend shape: using step (2), a head Gaussian model and a mouth Gaussian model are established and their parameters are initialized, an environment map is initialized, and all parameters are jointly optimized according to the image loss to achieve decoupling of material and lighting; the head Gaussian model is composed of a neutral Gaussian model and a basic expression Gaussian model; each Gaussian model has seven basic attributes: position, opacity, rotation, size, diffuse reflectance coefficient, roughness, and highlight intensity;

[0013] (4) Human head animation generation: for the user-provided human head motion parameters, linearly mix the neutral expression mesh and the basic expression mesh in the mesh blend shape optimized in step (2) to generate a mesh model corresponding to the motion parameters, and calculate the normal and visibility map; for the user-provided human head motion parameters, linearly mix the neutral Gaussian model and the basic expression Gaussian model in the re-lightable Gaussian blend shape optimized in step (3), and combine them with the mouth Gaussian model to generate a Gaussian model corresponding to the motion parameters, each Gaussian in the Gaussian model obtains the normal and visibility map from the mesh model in real time, combines with the user-provided environment map and its own material parameters to calculate its own color, and draws a head animation under the new lighting conditions, new view angle and expression.

[0014] Further, the training image collection and preprocessing in step (1) includes the following sub-steps:

[0015] (1.1) Training image collection: a monocular camera is used to shoot a head video as training data; the environment lighting should be kept unchanged during shooting, and the object being shot should keep normal speaking action;

[0016] (1.2) Training image preprocessing: extract consecutive frames from the video to get a sequence of consecutive images, crop and stretch the image sequence appropriately, use a face tracker to get a neutral expression mesh, a set of basic expression meshes, camera parameters, joint and pose parameters and expression coefficients from the image sequence for each frame; use a segmentation algorithm to mask out the head region from the image to get the head foreground image and the head foreground mask.

[0017] Further, the step (2) of training the mesh blend shape includes the following sub-steps:

[0018] (2.1) Parameter representation: the neutral expression mesh vertex position in the mesh blend shape is represented as the neutral expression mesh vertex position in the FLAME model plus an offset; use a neural network to predict the basic expression mesh vertex position increment and the linear blend skinning weight from the neutral expression mesh vertex position; to predict the material parameters from the spatial position under the neutral expression;

[0019] (2.2) Define the loss function: the loss function includes image loss, opacity loss, perceptual loss function, Laplacian smoothing constraint, normal constraint, deformation constraint, gray ambient light regular constraint, material smoothing regular constraint, and the mathematical expression is as follows:

[0020] L full =λ1L rgb +λ2L α +λ3L vgg +λ4L laplacian +λ5L normal +λ6L flame +λ7L light +λ8L smooth ;

[0021]

[0022] L smooth (x c )=||ρ(x c )-ρ(x c +ε)||1+||r(x c )-r(x c +ε)‖1;

[0023] Wherein L full represents the loss function, L rgb represents the image loss, L α represents the opacity loss, L vgg represents the perceptual loss function, L laplacian represents the Laplacian smoothing constraint, L normal represents the normal constraint, Lflame denotes deformation constraints, L light denotes gray-scale ambient light regularization constraints, L smooth denotes material smoothness regularization constraints; λ1, λ2, λ3, λ4, λ5, λ6, λ7, λ8, λ e , λ w is the loss mixing weight; denotes opacity map, denotes head foreground mask, N I is the total number of image pixels; F vgg denotes the features extracted from the first four layers of the pre-trained VGG network model; P denotes the total number of mesh vertices, δ i denotes the difference of the i-th vertex; is the set of all shared edge triangle pairs, n i is the normal of the i-th triangle; and is the position delta of the FLAME vertex closest to the mesh vertex under neutral expression and the linear blend skinning weight; x c denotes the vertex of the neutral expression mesh, ε denotes the random perturbation sampled from the Gaussian distribution.

[0024] Specifically, the mesh mixed shape in the step (2) is composed of a neutral expression mesh and a base expression mesh, and the mesh mixed shape is used to generate a human head mesh representing an arbitrary pose and an arbitrary expression by linearly blending the neutral expression mesh and the base expression mesh, and performing linear blend skinning transformation on the blended mesh.

[0025] Further, the training of the re-lightable Gaussian mixed shape in the step (3) comprises the following sub-steps:

[0026] (3.1) Parameter initialization: the neutral expression mesh of the mesh mixed shape is converted into a neutral Gaussian model by sampling; the deformation gradient is extracted from the base expression mesh and applied to the neutral Gaussian model to generate a base expression Gaussian model; a mesh sheet representing teeth is generated at the corresponding position of the oral cavity of the neutral expression mesh, and the mesh sheet is converted into an oral cavity Gaussian model by sampling; the ambient light map optimized in (2) is used as the initialization of the ambient light map of step (3);

[0027] (3.2) Define the loss function: the loss function includes image loss, opacity loss, size regularization constraint, oral cavity Gaussian model regularization constraint, gray-scale ambient light regularization constraint and material smoothness loss, and the mathematical expression is as follows:

[0028] L full = λ1L rgb + λ2L α + λ3L scale + λ4Lmouth + λ5L light + λ6L smooth ;

[0029] L rgb = (1 - λ)L1+ λL D-SSIM ;

[0030]

[0031] where L full denotes the loss function, L rgb denotes the image loss, L α denotes the opacity loss, L scale denotes the size regularization constraint, L mouth denotes the mouth regularization constraint, L light denotes the grayscale ambient light regularization constraint, L smooth denotes the material smoothness loss; λ1, λ2, λ3, λ4, λ5, λ6, λ are loss mixing weights; L1denotes the mean absolute error, L D-SSIM denotes the structure difference; denotes the opacity image, denotes the head foreground mask, N I is the total number of image pixels; N denotes the total number of Gaussians, s0denotes the size threshold; SDF denotes the signed distance, V denotes the predefined mouth volume, x i denotes the position of the mouth Gaussian, N' denotes the total number of mouth Gaussians; * denotes the material property {r, g} of one of the screen space positions using Gaussian splatting, f(·) is a heuristic function;

[0032] (3.3) KNN-based nearest neighbor search interpolation: build an acceleration structure for the neutral expression mesh vertices, search for the K nearest mesh vertices to the query Gaussian, traverse all the triangles connected to these vertices, and return the position closest to the query point. Use this information to interpolate linear blend skinning weights, normals, visibility maps on the mesh vertices.

[0033] Further, the re-illuminable Gaussian mixture shape in the step (3) is composed of a group of Gaussian models, the group of Gaussian models including a head Gaussian model and a mouth Gaussian model, the head Gaussian model being used to generate a human head model representing an arbitrary pose and an arbitrary expression by linearly mixing a neutral Gaussian model and a base expression Gaussian model, and performing linear blend skinning transformation on the mixed model and the mouth Gaussian model.

[0034] Specifically, the Gaussian model in the step (3) carries material parameters, including diffuse reflectance, roughness and highlight intensity.

[0035] Specifically, the human head motion parameters in the step (4) include joint parameters, pose parameters, facial expression coefficients and camera parameters.

[0036] Specifically, the visibility map in the step (4) is obtained by a ray tracing algorithm, and a single bit is used to encode a single direction for a compact representation.

[0037] Specifically, the Gaussian color calculation in the step (4) is obtained from a physics-based rendering according to a discrete integral considering visibility; the expression is as follows:

[0038]

[0039] Where C(x i , ω o ) represents the color of the i-th Gaussian, L j is the j-th texture unit of the environment light map, J is the total number of texture pixels, is the visibility of the j-th texture unit to the i-th Gaussian, f i (ω j , ω o ) is a BRDF function, ΔΩ is a discrete solid angle covered by the texture unit; x i is the position of the i-th Gaussian; ω j is the incident direction; and ω o is the outgoing direction.

[0040] The beneficial effects of the present application are as follows:

[0041] The present model expands the existing Gaussian mixture shape to support relighting without reducing rendering quality and running speed, adds a mesh mixture shape (mesh mixture shape) corresponding to the Gaussian mixture shape to better represent the individual head geometry. The expanded Gaussian mixture shape carries material parameters: diffuse reflectance, roughness, highlight intensity, and realizes the joint optimization decoupling of lighting and material. The present model also introduces omnidirectional visibility calculation, which can realize real shadow effect, and the rendering effect is more realistic than the current most advanced relighting human head model. Thanks to the high computational efficiency of the Gaussian mixture shape, the present model leads other relighting human head models in performance with a runtime speed of 365fps. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the intermediate result and final result figure of synthesizing the first individual head animation by applying the method of the present application; wherein, Figure 1 (A) is the original image taken, Figure 1 (B) is the preprocessed head foreground image, Figure 1 (C) is the head foreground mask, Figure 1 (D) is the mesh mixture shape before optimization,Figure 1 (E) is the optimized mesh blendshape, Figure 1 (F)- Figure 1 (H) is the final result image synthesized under new lighting conditions, new expression and view angle;

[0043] Figure 2 is the intermediate result and final result image of synthesizing the head animation of the second individual using the method of the present application; wherein, Figure 2 (A) is the original image taken, Figure 2 (B) is the pre-processed head foreground image, Figure 2 (C) is the head foreground mask, Figure 2 (D) is the mesh blendshape before optimization, Figure 2 (E) is the optimized mesh blendshape, Figure 2 (F)- Figure 2 (H) is the final result image synthesized under new lighting conditions, new expression and view angle;

[0044] Figure 3 is the intermediate result and final result image of synthesizing the head animation of the third individual using the method of the present application; wherein, Figure 3 (A) is the original image taken, Figure 3 (B) is the pre-processed head foreground image, Figure 3 (C) is the head foreground mask, Figure 3 (D) is the mesh blendshape before optimization, Figure 3 (E) is the optimized mesh blendshape, Figure 3 (F)- Figure 3 (H) is the final result image synthesized under new lighting conditions, new expression and view angle. DETAILED DESCRIPTION

[0045] The core of the present application is a relightable Gaussian blendshape expression and its construction method suitable for human head motion modeling. The relightable Gaussian blendshape expression is composed of a mesh blendshape and a relightable Gaussian blendshape. Given a segment of natural speaking human head video, the corresponding mesh blendshape and relightable Gaussian blendshape are optimized, which can be used to generate human head images and animations under new lighting conditions, new view angles and expressions in real time. The relationship between the mesh blendshape and the relightable Gaussian blendshape is that the mesh blendshape provides the optimized initial value, the normal and visibility at the training and running time for the relightable Gaussian blendshape, and the final shading and rendering are completed by the Gaussian blendshape, and the two exist a one-to-one semantic correspondence.

[0046] The mesh blendshape is a mesh blendshape that is specifically optimized for a particular individual, consisting of a neutral expression mesh M0, and a set of base expression meshes {M1, M2,..., M50} corresponding to 50 basic expressions. 50 Each mesh is represented by a mesh vertex position (a P x 3 float tensor, P represents the number of mesh vertices). By linear blending, the head mesh M under any expression can be computed as: ψ

[0047]

[0048] where {ψ k} are the expression coefficients for each base expression.

[0049] In addition to facial expression control, the present application provides joint and pose parameters Θ for controlling the motion of the head, jaw, eyeballs, and eyelids, which transform the head mesh model (i.e. the vertex positions of the mesh) by linear blend skinning (LBS), specifically expressed as:

[0050]

[0051] where T j (Θ)∈R 4×4 is the motion transformation matrix for each joint (head, jaw, left eyeball, right eyeball, left eyelid, right eyelid), controlled by the joint and pose parameters Θ, W′ j ∈R P is the weight of linear blend skinning, used to determine the degree to which each head mesh vertex is affected by the motion of each joint; T′∈R P×4×4 is the blended transformation, which specifies the transformation applied to each head mesh vertex. represents a batch matrix-vector product.

[0052] The relightable Gaussian blendshape consists of a head Gaussian model B ψ and a mouth Gaussian model B m The head Gaussian model B ψ consists of a neutral Gaussian model B0and a set of base expression Gaussian models {B1, B2,..., B 50} corresponding to the base expression meshes. Each of the above models consists of a set of three-dimensional Gaussians, each of which has seven basic attributes, including four geometric attributes and three material attributes. The geometric attributes include: position x∈R 3 , opacity α∈R, rotation q (represented by a quaternion), size s∈R 3 ; the material attributes include: diffuse reflectance coefficient ρ∈R 3 ​, roughness r e R and high light intensity y e R. Each neutral Gaussian model B0is also equipped with a linear blend skinning weight W for joint and pose control; the Gaussian of neutral Gaussian model B0and the Gaussian of each basis expression Gaussian model B k There is a one-to-one correspondence between the Gaussian of each basis expression Gaussian model B k and the Gaussian of neutral Gaussian model B0. The difference of Gaussian attributes between basis expression Gaussian model B k and neutral Gaussian model B0is defined as "delta" AB k = B k - B0. Similar to mesh blend shape, the Gaussian blend shape of an arbitrary expression human head model is calculated as follows:

[0053]

[0054] Similar to mesh blend shape, the re-illuminable Gaussian blend shape is also controlled by joint and pose parameters, and the specific expression is as follows:

[0055]

[0056] T j (Θ) e R 4×4 is the motion transformation matrix, which is defined the same as in mesh blend shape. W j e R N is the linear blend skinning weight, which is used to determine the degree to which each Gaussian is affected by each joint motion, and N represents the total number of Gaussians; T e R N×4×4 is the transformed blend, which specifies the transformation applied to each Gaussian of the model; trans is an operator that transforms Gaussian attributes with a transformation matrix, and has different operations for different Gaussian attributes, and the specific expression is as follows:

[0057] x * = Tx;

[0058] α * = a;

[0059] q * = Rq;

[0060] s * = s;

[0061] ρ * = p;

[0062] r * = r;

[0063] γ * = y;

[0064] Where R = proj(T), which represents the rotation transformation component R in the affine transformation T.

[0065] The oral Gaussian model Bψ Also composed of three-dimensional Gaussian structures (possessing the same seven basic properties), it is specifically used to represent the interior of the oral cavity, such as teeth and tongue. The oral cavity Gaussian model is divided into upper and lower parts. The upper oral cavity model follows the head joint movement, while the lower oral cavity model follows the jaw joint movement. The oral cavity Gaussian model after joint motion transformation is denoted as...

[0066] Transformed Gaussian model The process also involves shading, which calculates the color of each Gaussian element under a given ambient light map L, and then uses Gaussian splatting to render a highly realistic image in real time. Shading includes the following steps: normal and visibility calculation, and physically based rendering.

[0067] (1) Normal and visibility calculation

[0068] This invention is based on the Gaussian model. Head mesh models with corresponding relationships Calculate the normals and visibility. First, calculate the head mesh model. The normal n of each vertex p p and visibility v p v p It is a 16×32 visibility map, corresponding to 512 preset directions in the world coordinate system, recording whether an event has occurred in these directions and the head mesh model. To address self-occlusion, this invention uses 512 bits, or 64 bytes, to compactly store the visibility map, where 1 indicates no occlusion and 0 indicates occlusion. Specifically, this invention uses NVIDIA's OptiX ray tracing engine to run ray tracing algorithms to calculate the aforementioned visibility.

[0069] Subsequently, this invention interpolates the normal and visibility of each Gaussian by establishing the correspondence between the Gaussian and the mesh model. Specifically, we determine the correspondence using a neutral expression mesh M0 and a neutral Gaussian model B0. Each Gaussian i starts from its position in the neutral Gaussian model B0, finds its nearest position on the neutral expression mesh M0, obtains the corresponding triangle index and centroid coordinates, and uses this information for interpolation. The vertex normals and visibility on the surface give the normal n of each Gaussian. i and visibility v i .

[0070] (2) Physically based rendering

[0071] The present application adopts 16x32 ambient light map as illumination model and modified Disney model introduced by Unreal Engine as material model. The following is the formula of bidirectional reflectance distribution function (BRDF) parameterized by diffuse reflectance coefficient p, roughness r and highlight intensity g, wherein f spec represents the highlight component of the micro-surface model modeling:

[0072]

[0073] wherein n i , p i , r i , g i respectively represent the normal, diffuse reflectance coefficient, roughness and highlight intensity of the i-th Gaussian function. We calculate the color of each Gaussian by discrete integration:

[0074]

[0075] wherein C(x i , w o ) represents the color of the i-th Gaussian, L j is the j-th texture unit of the ambient light map, and J is the total number of texture pixels (512 is adopted in the present application). is the visibility of the j-th texture unit to the i-th Gaussian. f i (w j , w o ) is the BRDF function, and D is the discrete solid angle covered by the texture unit; x i is the position of the i-th Gaussian; w j is the incident direction; and w o is the outgoing direction.

[0076] The steps of the present application will be described in detail below in combination with the accompanying drawings. Figures 1-3

[0077] Based on the re-illuminable Gaussian mixed shape of the human head, the present application proposes a re-illuminable Gaussian mixed shape method suitable for human head motion modeling, which includes the following steps: training image acquisition and preprocessing, grid mixed shape training, re-illuminable Gaussian mixed shape training, and human head animation generation. Specifically, it includes the following steps:

[0078] Training image acquisition and preprocessing

[0079] 1.1 Training image acquisition

[0080] ​This invention uses a standard monocular camera to capture a 2-3 minute video of a human head as training data. The ambient lighting must remain relatively constant during filming, and the subject must speak and move naturally.

[0081] 1.2 Training Image Preprocessing

[0082] like Figure 1 (A)- Figure 3 As shown in (A), this invention extracts images frame by frame from a video to obtain a continuous image sequence of 3000-5000 images, and then performs appropriate cropping and scaling on the image sequence. A face tracker is used to extract a FLAME (face-learned with an articulated model and expressions) neutral expression mesh m0 and 50 base expression meshes {m1, m2, ..., m} from the continuous image sequence. 50}, for each frame, the camera parameters C, joint and pose parameters Θ, and expression coefficients {ψ} k The algorithm employs video-based human body segmentation and facial semantic segmentation (to segment the head region and obtain the head foreground image). like Figure 1 (B)- Figure 3 As shown in (B), a head foreground mask (alpha mask) is generated for each image. like Figure 1 (C)- Figure 3 As shown in (C).

[0083] Training of mesh blended shapes

[0084] 2.1 Parametric Representation and Drawing of Mesh Blending Shapes Pipelines

[0085] Face trackers can output a set of deformable mesh models (FLAME models) describing the geometry of an individual's head, but their representational accuracy is limited and cannot accurately depict individual features. This invention optimizes the mesh blending shape specifically for each individual based on the FLAME model. The head mesh before optimization is shown below. Figure 1 (D)- Figure 3 As shown in (D), the optimized head mesh is as follows: Figure 1 (E)- Figure 3 As shown in (E). As previously mentioned, the parameters of the mesh blending shape include a neutral expression mesh M0 and 50 base expression meshes {M1, M2, ..., M}. 50 The vertex positions of} and the weights W′ of the linear blending skin. This invention represents the neutral expression mesh M0 as a FLAME neutral expression mesh m0 and a learnable offset d:

[0086] M o =m0+d:

[0087] The present application uses a neural network to predict {ΔM1, ΔM2,..., ΔM 50} and the weight W' of linear blend skinning, where "delta" ΔM k is defined as ΔM k = M k - M0. A neural network is used to predict the material {p, r, g} pixel by pixel, where the definition of material is exactly the same as in the Gaussian blend shape of photorealistic rendering. In addition, an ambient light map L' is introduced, which can be optimized. The introduction of material and lighting is to build a complete differentiable rendering pipeline, so that the whole pipeline can be physically based rendering, thus optimizing the geometry of the network blend shape. The material and lighting parameters in this stage are only temporary parameters in the optimization process and do not participate in the final rendering.

[0088] The complete rendering pipeline based on mesh blend shape is described as follows: first, calculate the neutral expression mesh M0 of the mesh blend shape using the neutral expression mesh and offset in FLAME; input the position of each vertex of the neutral expression mesh M0 into the neural network to get the position increment of the vertex under each base expression and the weight of linear blend skinning; use the linear blend formula and the linear blend skinning formula described above to calculate the head mesh under given parameters Rasterize the head mesh, draw the normal map and the positional map, and the value of each pixel of the positional map represents its spatial position in the neutral expression mesh M0. Input the positional map into the neural network to output the diffuse reflectance, roughness and highlight intensity of each pixel. The present application uses the method described in the "physically based rendering" described above to color each pixel, but omits the calculation of visibility, thus obtaining the rendering result of the mesh blend shape.

[0089] The specific structure of the neural network is as follows: the first layer is a fully connected layer with 3 channels as input and 128 channels as output; the second layer is a fully connected layer with 128 channels as input and 128 channels as output; the third layer is a fully connected layer with 128 channels as input and 128 channels as output; the fourth layer is a fully connected layer with 128 channels as input and 128 channels as output. From the fifth layer, the network is split into two parts, the first part predicts {ΔM kThe first part predicts the weights W' of the linearly mixed skin. The first part is a fully connected layer with 128 input channels and 150 output channels. The second part is in turn: (1) a fully connected layer with 128 input channels and 128 output channels (2) a fully connected layer with 128 input channels and 6 output channels. All network layers except the output layer use the Softplus activation with beta = 100, the output layer activation of the first part is linear activation, and the output layer activation of the second part is Softmax activation.

[0090] The specific structure of the neural network M is as follows: after a 4-order position coding, the input 3 channels are raised to 27 channels, the first layer is a fully connected layer with 27 input channels and 128 output channels; the second layer is a fully connected layer with 128 input channels and 128 output channels; the third layer is a fully connected layer with 128 input channels and 128 output channels; the fourth layer is a fully connected layer with 128 input channels and 128 output channels; the fifth layer is a fully connected layer with 128 input channels and 128 output channels; the output layer is a fully connected layer with 128 input channels and 5 output channels. Except that the activation function of the output layer is the Softplus activation with beta = 100, the activation functions of the other network layers are ReLU.

[0091] 2.2 Training of grid mixed shape

[0092] The present application jointly optimizes d and L' according to the image loss. For each frame, the rendering image and the opacity image can be drawn according to the foregoing (section 2.1), the rendering image and the head foreground image are calculated to calculate the loss, the opacity image and the head foreground mask are calculated to calculate the loss, a series of regular constraint losses are calculated, the weighted sum of the above losses is calculated for the gradient of the parameters to be optimized, and the Adam optimizer is used to optimize the parameters. The specific loss function is defined as follows:

[0093] L full = λ1L rgb + λ2L α + λ3L vgg + λ4L laplacian + λ5L normal + λ6L flame + λ7L light + λ8L smooth ;

[0094] Wherein λ1 = 1, λ2 = 2, λ3 = 0.1, λ4 = 60, λ5 = 0.1, λ6 = 5, λ7 = 0.01, λ8 = 0.01. Wherein the definition of the image loss Lr gb and the opacity loss L α is as follows:

[0095]

[0096] where I i is the rendered image, is the opacity image, is the head foreground image, is the head foreground mask, N I is the total number of image pixels. The perceptual loss function is as follows:

[0097]

[0098] where F vgg represents the features extracted from the first four layers of the pre-trained VGG network model.

[0099] The geometric regularization constraint includes Laplacian smoothing constraint L laplacia n and normal constraint L normal for smoothing the mesh and avoiding self-intersection, which is defined as follows:

[0100]

[0101] where P represents the total number of mesh vertices, represents the difference of the i-th vertex, represents the Laplacian matrix of the mesh. is the set of all pairs of triangles that share an edge, n i is the normal of the i-th triangle. The deformation constraint L flame is defined as follows:

[0102]

[0103] The parameters of the mesh are constrained by the parameters of the FLAME model, where λ e = 50, λ w = 2.5, and are the corresponding parameters on the FLAME vertex closest to the mesh vertex under the neutral expression.

[0104] The grayscale ambient light regularization constraint L light makes the ambient light color close to black and white tones:

[0105]

[0106] The material smoothing regularization constraint L smooth is defined as follows:

[0107] L smooth (x c ) = || p(x c ) - p(x c+ ε)‖1+‖r(x c ) - r(x c + ε)‖1;

[0108] where x c denotes the vertices of the neutral expression mesh, and ε denotes a random perturbation sampled from a Gaussian distribution.

[0109] Specific details of the optimization: d is initialized to 0, L' is initialized to 0.1, and initialized with Kaiming normal The batch size is 4, and the total number of iterations is 2500. d, The learning rates of d and L' are 0.001, 0.001, 0.001, and 0.01, respectively. At the 500th iteration, the mesh is upsampled once (Botsch, M., & Kobbelt, L. (2004, July). A remeshing approach to multiresolution modeling. In Proceedings of the 2004 Eurographics / ACM SIGGRAPH symposium on Geometry processing (pp. 185-192)).

[0110] Training of the re-lightable Gaussian mixture shape

[0111] 3.1 Parameter initialization

[0112] The present invention respectively trains a neutral Gaussian model B0, 50 basic expression Gaussian models {B k}, and an oral cavity Gaussian model B mand the environment light map L are initialized. For the neutral Gaussian model B0, use Poisson disk sampling to uniformly sample points on the mesh blended shape neutral expression mesh M0, which is used as the initial positions of the Gaussians. Initialize the Gaussians as isotropic Gaussians, specifically: initialize the size s as the average distance to the nearest neighbor of the three Gaussian centers, initialize the rotation q as the unit quaternion. Initialize the opacity a as 0.1, the diffuse reflectance p as 0.4, the roughness r as 1, and the highlight intensity g as 1. For each Gaussian of the neutral Gaussian model B0, search for the nearest neighbor triangle on M0, and compute the linear blend skinning weight W of the Gaussian according to the linear blend skinning weight interpolation on the triangle vertices.

[0113] For the 50 base expression Gaussian models {B k}, extract the deformation gradient from the mesh blended shape base expression meshes, i.e. the deformation of the neutral expression mesh M0 to each base expression mesh M k , and apply the deformation gradient to the neutral Gaussian model B0 to obtain the base expression Gaussian models {B k} as the initialization. Specifically, for each Gaussian G i，0 of the neutral Gaussian model B0, search for the nearest neighbor triangle on the neutral expression mesh M0, compute the affine transformation T of the triangle from M0 to M k , and apply it to the position x of the Gaussian G i，0 to generate the position of each Gaussian G k in the base expression Gaussian model B i，k . Similarly, extract the rotation component R in T, and apply it to the rotation q of the Gaussian G i，0 to generate the corresponding parameter in the Gaussian G i，k . Finally, copy the size s, the opacity a, the diffuse reflectance p, the roughness r, and the highlight intensity g in G i，0 as the corresponding parameters in G i，k , and complete the initialization.

[0114] For the mouth Gaussian model B m , predefine two mesh slices corresponding to the positions of the upper and lower teeth, and initialize the mouth Gaussian model in a similar way to the neutral Gaussian model, converting the mesh slices into a set of Gaussians. For the neutral Gaussian model and the base expression Gaussian model, the number of initialized Gaussians is 70k, and for the mouth Gaussian model, the number of initialized Gaussians is 14k (7k for the upper part and 7k for the lower part).

[0115] The present application uses the environment light map L' obtained from the training of the mesh blended shape as the initial value of the environment light map L of the training phase of the light re-illuminable Gaussian blended shape.

[0116] 3.2 Parameter optimization

[0117] The present application jointly optimizes B0, {AB k}, B m and L: first generate the Gaussian model of each frame according to the parameters of the frame, and draw the rendered image and the opacity image, calculate the loss of the rendered image and the head foreground image respectively, calculate the loss of the opacity image and the head foreground mask, calculate the size regularization constraint, the regular constraint of the oral Gaussian model, the grayscale ambient light regular constraint and the material smoothing loss, calculate the weighted sum of all the above losses and regularization constraints for the gradient of the parameters to be optimized, and use the Adam optimizer to optimize the parameters.

[0118] The rendering process is as follows: randomly extract a frame i from the training image, and reconstruct the Gaussian model of the frame. According to the aforementioned re-illuminable Gaussian mixture shape expression, linearly mix B0and {AB k} to obtain the head Gaussian model B k . ψ Then apply linearly mixed skinning using the joint and pose parameters Θ of the frame to obtain the transformed head Gaussian model and the transformed oral Gaussian model After shading, the color of each Gaussian is calculated, and Gaussian splatting technology is used to draw and into images The entire rendering pipeline is differentiable, so that the parameters B0, {AB k}, B m and L can be updated using the gradient backpropagation algorithm.

[0119] The complete loss function is defined as follows:

[0120] L full = λ1L rgb + λ2L α + λ3L scale + λ4L mouth + λ5L light + λ6L smooth ;

[0121] Where λ1=1, λ2=10, λ3=1×10 4 , λ4=100, λ5=0.01, λ6=0.02. The image loss L rgb is represented as and the combination of L1 (mean absolute error) and D-SSIM (Structural dissimilarity) loss functions calculated:

[0122] Lrgb = (1 - λ) L1+ λ L D-SSIM ;

[0123] where λ = 0.2.

[0124] opacity loss L α is defined as:

[0125]

[0126] where is the opacity image, is the aforementioned head foreground mask, N I is the total number of image pixels.

[0127] size regularization constraint L scale limits the size of the Gaussians to avoid generating extreme anisotropic Gaussians, and thus makes the distribution of Gaussians in space more uniform:

[0128]

[0129] where N is the number of Gaussians, s0= 0.001 is the size threshold.

[0130] mouth regularization constraint L mouth constrains the Gaussians of the mouth model within a predefined mouth volume. Specifically, the signed distance of each Gaussian to the boundary of the mouth volume is computed, and an L2 loss is imposed to pull the Gaussians outside the volume back:

[0131]

[0132] where N' denotes the total number of mouth Gaussians, x i denotes the position of the Gaussian, SDF denotes the signed distance, and V denotes the predefined cylindrical mouth volume.

[0133] gray ambient light regularization constraint L light makes the ambient light color close to black and white tones:

[0134]

[0135] material smoothness loss L smooth makes the optimized highlights smoother:

[0136]

[0137] Where * represents one of the material properties {r, γ} rendered to the screen using Gaussian splashing, and i and j represent the pixel coordinates in the image. f(·)=max(1-20||·||1,0) is a heuristic function that outputs zero in areas with large opacity variations, thus avoiding the calculation of the loss in unreliable areas around the contour.

[0138] Specific details of the optimization: Following the principles of Gaussian blend shapes (Ma, S., Weng, Y., Shao, T., & Zhou, K. (2024, July). 3D Gaussian blendshapes for head avatar animation. In ACM SIGGRAPH 2024 Conference Papers (pp. 1-10).), this invention uses blendshape consistency to optimize {ΔB}. k All seven Gaussian parameters are used to guarantee the basic expression Gaussian model B. k The base expression mesh M and the corresponding mesh blend shape k It exhibits semantic consistency. The training process iterates a total of 40k times, for B0 and B... m Learning rate setting: 1.6 × 10 -4 5×10 -2 1×10 -3 5×10 -3 2.5×10 -3 1×10 -3 1×10 -3 These correspond to position, opacity, rotation, size, diffuse reflection coefficient, roughness, and specular intensity, respectively. For {ΔB} k The learning rate setting for} is 3.2 × 10. -7 5×10 -5 1×10 -4 5×10 -4 2.5×10 -5 1×10 -5 1×10 -5 These correspond to position, opacity, rotation, size, diffuse reflection coefficient, roughness, and specular intensity, respectively. The learning rate for the ambient light map L is 1×10⁻⁶. -2 This invention uses a GPU-accelerated KNN algorithm to search for the nearest neighbor grid position. Specifically, it first constructs an accelerated structure composed of neutral expression grid vertices M0, searches for the four grid vertices closest to the query point, and then traverses all triangles connected to these vertices to return the position closest to the query point.

[0139] Human head animation generation. The trained relightable Gaussian mixture shape can be used to synthesize human head images and animations under new lighting conditions, new viewpoints and expressions. The user only needs to provide human head motion parameters: joint and pose parameters Θ and expression coefficients {ψ k}, camera parameters C and environment light map L, then the Gaussian model of each frame can be generated and further calculate Gaussian shading, draw with Gaussian splatting technique to get high realistic images and animations, as shown in (F-H) figure. The parameters can be edited manually by the user, or obtained from any human head video using a face tracker. Figures 1-3

[0140] Embodiment examples

[0141] The inventors implemented data acquisition in an indoor environment with a Nikon D850 single-lens reflex camera fixed on a tripod, shooting a human head video at 1080p resolution, 25 frames per second, about 3 minutes. The subject was asked to read a passage with emotion while turning his head within a suitable range to collect images of each viewpoint needed for modeling. The inventors implemented image preprocessing and model training of the present application on a server equipped with an Intel Core i9-13900KF central processor and an NVidia RTX 4090 graphics processor, and implemented real-time animation generation of the present application on a desktop computer equipped with an Intel Core i7-13700KF central processor and an NVidia RTX 4090 graphics processor. The inventors used all the parameter values listed in the specific embodiments to obtain all the experimental results shown in the accompanying Figures 1-3 The present application can smoothly draw the subject's head animation under new lighting conditions, new viewpoints and expressions at a frame rate of 365 frames per second, and allows the user to modify the lighting, rotate the viewing angle and edit the facial expression in real time. For input training images of 512*512 resolution, running the face tracker and image segmentation preprocessing images on a single NVidia RTX 4090 graphics processor takes 12 hours, and training takes 22 minutes (5 minutes to optimize the mesh mixture shape + 17 minutes to optimize the relightable Gaussian mixture shape). At runtime, for each frame, it takes about 1 millisecond to complete the parameter calculation of the mesh and Gaussian model, 0.4 milliseconds to complete the visibility calculation, 0.6 milliseconds to complete the shading of the Gaussian model, and 0.7 milliseconds to complete the drawing of the Gaussian model. (The portraits of the people in all the accompanying figures are used with the consent of the individuals.)

[0142] ​Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application embrace any and all variations, modifications, and adaptations of the application described herein, which are within the scope of the general inventive concept and include all such subject matter that is within the scope of the instant application and the claims hereafter presented.

[0143] It should be understood that the application is not limited to the precise construction here described and illustrated in the drawings and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.

Claims

1. A Gaussian mixture shape method for dynamic modeling of the human head with relighting capability, characterized in that, Includes the following steps: (1) Training image acquisition and preprocessing: Use a monocular camera to shoot a short video of a person speaking, process the video data and obtain various parameters, including: neutral expression mesh, a set of base expression mesh, head foreground image of each frame, head foreground mask, camera parameters, joint and pose parameters and expression coefficients; (2) Training of mesh blending shape: The mesh blending shape is established and initialized using the parameters in step (1), the ambient light map is initialized, and all parameters are jointly optimized according to the image loss to achieve decoupling of material and lighting; the mesh blending shape is composed of neutral expression mesh and base expression mesh; (3) Training of re-illuminated Gaussian mixture shapes: Using step (2), a head Gaussian model and an oral cavity Gaussian model are established and their parameters are initialized. The ambient light map is initialized. All parameters are jointly optimized according to the image loss to achieve decoupling of material and illumination. The head Gaussian model consists of a neutral Gaussian model and a base expression Gaussian model. Each Gaussian model has seven basic properties: position, opacity, rotation, size, diffuse reflection coefficient, roughness, and specular intensity. (4) Human head animation generation: Based on the human head motion parameters provided by the user, the neutral expression mesh and the base expression mesh in the mesh blending shape optimized in step (2) are linearly blended to generate a mesh model corresponding to the motion parameters, and the normal and visibility map are calculated; Based on the human head motion parameters provided by the user, the neutral Gaussian model and the base expression Gaussian model in the relightable Gaussian blending shape optimized in step (3) are linearly blended and combined with the oral cavity Gaussian model to generate a Gaussian model corresponding to the motion parameters. Each Gaussian in the Gaussian model obtains the normal and visibility map from the mesh model in real time, and calculates its own color by combining the ambient light map and its own material parameters provided by the user, and draws the head animation under the new lighting conditions, new perspective and expression.

2. The Gaussian mixture shape method for dynamic modeling of the human head with relighting as described in claim 1, characterized in that, The training image acquisition and preprocessing in step (1) includes the following sub-steps: (1.1) Training image acquisition: Use a monocular camera to shoot a head video as training data; the ambient lighting should remain unchanged during the shooting process, and the subject should maintain normal speaking movements; (1.2) Training image preprocessing: Extract continuous frames from the video to obtain a continuous image sequence. Perform appropriate cropping and scaling on the image sequence. Use a face tracker to obtain a neutral expression grid, a set of base expression grids, camera parameters, joint and pose parameters and expression coefficients for each frame from the image sequence. Use a segmentation algorithm to mask out the head region from the image to obtain the head foreground image and head foreground mask.

3. The Gaussian mixture shape method for dynamic modeling of the human head with relighting as described in claim 1, characterized in that, The training of the mesh blending shape in step (2) includes the following sub-steps: (2.1) Parameter representation: The position of the neutral expression mesh vertex in the mesh blending shape is represented by the position of the neutral expression mesh vertex in the FLAME model plus an offset; using a neural network Predict the base expression mesh vertex position increment and linear blending skin weights from the neutral expression mesh vertex positions; predict material parameters from the spatial position under the neutral expression using a neural network M; (2.2) Define the loss function: The loss function includes image loss, opacity loss, perceptual loss function, Laplacian smoothing constraint, normal constraint, deformation constraint, grayscale ambient light regularization constraint, and material smoothing regularization constraint. The mathematical expression is as follows: L full =λ1L rgb +λ2L α +λ3L vgg +λ4L laplacian +λ5L normal +λ6L flame +λ7L light +λ8L smooth ; L smooth (x c )=||ρ(x c )-ρ(x c +ε)||1+||r(x c )-r(x c +e)||1; Where L full L represents the loss function. rgb L represents the image loss. α L represents the loss of opacity. vgg L represents the perceptual loss function. laplacian L represents the Laplace smoothing constraint. normal L represents the normal constraint. flame L represents deformation constraint. light L represents the grayscale ambient light regularization constraint. smooth The material smoothing regularization constraint is represented by λ1, λ2, λ3, λ4, λ5, λ6, λ7, λ8, λ e , λ w It is a loss mixture weight; Indicates an opaque image. Indicates the head foreground mask, N I It is the total number of pixels in the image; F vgg This represents the features extracted from the first four layers of the pre-trained VGG network model; P represents the total number of grid vertices, and δ... i This represents the difference at the i-th vertex; It is the set of all triangle pairs that share an edge, n i It is the normal to the i-th triangle; and The position increment of the FLAME vertex closest to the mesh vertex and the linear blending skinning weights under neutral expression; x c Let represent the vertices of the neutral expression grid, and ε represent the random perturbation sampled from a Gaussian distribution.

4. The relightable Gaussian mixture shape method for dynamic modeling of the human head according to claim 1, characterized in that, The mesh blending shape in step (2) consists of a neutral expression mesh and a base expression mesh. The mesh blending shape is used to generate a human head mesh representing arbitrary poses and arbitrary expressions by linearly blending the neutral expression mesh and the base expression mesh, and performing a linear blending skinning transformation on the blended mesh.

5. The Gaussian mixture shape method for dynamic modeling of the human head with relighting as described in claim 1, characterized in that, The training of the re-illuminated Gaussian mixture shape in step (3) includes the following sub-steps: (3.1) Parameter initialization: The neutral expression mesh with mixed mesh shape is converted into a neutral Gaussian model by sampling; the deformation gradient is extracted from the base expression mesh and applied to the neutral Gaussian model to generate the base expression Gaussian model; mesh slices representing teeth are generated at the corresponding positions of the oral cavity in the neutral expression mesh, and the mesh slices representing teeth are converted into an oral cavity Gaussian model by sampling; the ambient light map optimized in (2) is used as the initialization of the ambient light map in step (3); (3.2) Define the loss function: The loss function includes image loss, opacity loss, size regularization constraint, oral Gaussian model regularization constraint, grayscale ambient light regularization constraint, and material smoothing loss. The mathematical expression is as follows: L full =λ1L rgb +λ2L α +λ3L scale +λ4L mouth +λ5L light +λ6L smooth ; L rgb =(1-λ)L1+λL D-SSIM ; Where L full L represents the loss function. rgb L represents the image loss. α L represents the loss of opacity. scale L represents the size regularization constraint. mouth L represents the oral cavity regularization constraint. light L represents the grayscale ambient light regularization constraint. smooth λ represents the material smoothing loss; λ1, λ2, λ3, λ4, λ5, λ6, and λ are the loss mixing weights; L1 represents the mean absolute error, L D-SSIM Indicates structural differences; Indicates an opaque image. Indicates the head foreground mask, N I It represents the total number of pixels in the image; N represents the total number of Gaussian pixels, s0 represents the size threshold; SDF represents the signed distance, V represents the predefined oral cavity volume, and x... i The position of the Gaussian in the oral cavity is indicated, N′ represents the total number of Gaussian in the oral cavity; * indicates one of the material properties {r, γ} rendered to the screen using Gaussian splashing, and f(·) is a heuristic function; (3.3) KNN-based nearest neighbor search interpolation: Construct an accelerated structure for neutral expression grid vertices, search for the K grid vertices closest to the query Gaussian, traverse all triangles connected to these vertices, return the position closest to the query point, and use this information to interpolate the linear blending skin weights, normals, and visibility maps on the grid vertices.

6. The Gaussian mixture shape method for dynamic modeling of the human head with relighting according to claim 1, characterized in that, The relightable Gaussian mixture shape in step (3) consists of a set of Gaussian models, including a head Gaussian model and an oral cavity Gaussian model. The head Gaussian model is used to generate a human head model representing arbitrary poses and arbitrary expressions by linearly mixing a neutral Gaussian model and a base expression Gaussian model, and performing a linear mixture skinning transformation on the mixed model and the oral cavity Gaussian model.

7. The Gaussian mixture shape method for dynamic modeling of the human head with relighting as described in claim 1, characterized in that, The Gaussian model in step (3) carries material parameters, including diffuse reflection coefficient, roughness, and specular intensity.

8. The Gaussian mixture shape method for dynamic modeling of the human head with relighting according to claim 1, characterized in that, The human head motion parameters in step (4) include joint parameters, posture parameters, facial expression coefficients, and camera parameters.

9. The Gaussian mixture shape method for dynamic modeling of the human head with relighting according to claim 1, characterized in that, The visibility map in step (4) is obtained by a ray tracing algorithm and uses a single bit to encode a compact representation of a single direction.

10. The relightable Gaussian mixture shape method for dynamic modeling of the human head according to claim 1, characterized in that, The Gaussian color calculation in step (4) is obtained by a discrete integral that takes into account visibility, based on physically based rendering; the expression is as follows: Where C(x) i ω o ) represents the color of the i-th Gaussian, L j It is the j-th texture unit in the ambient light map, where J is the total number of texture pixels. f is the visibility of the j-th texture unit to the i-th Gaussian texture. i (ω j ω o ) is the BRDF function, ΔΩ is the discrete solid angle covered by the texture unit; x i Let ω be the position of the i-th Gaussian; j ω is the incident direction; o The direction of launch.

Images