Method for carrying out animation production on digital character according to facial expression of user
Through 3D camera acquisition and point cloud processing, user-specific expression models and animation prior models are constructed, and animation parameters are calculated, which solves the problem of long cycles and high costs in the creation and animation of digital characters in the existing technology, and achieves low cost. Fast and natural facial animation production improves productivity and animation authenticity.
Patent Information
- Application Number
- CN202411243502.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-31
AI Technical Summary
The prior art requires complex acquisition systems and a lot of manual post-processing when creating and animating digital characters, resulting in long production cycles and high costs.
Using 3D cameras to collect RGB and depth maps of user faces, generate point cloud data, construct user-specific expression models and animation prior models, calculate animation parameters and drive digital characters to imitate user facial movements, and realize through point cloud fusion and gradient space optimization Accurate tracking of facial expressions.
It reduces equipment costs and production costs, simplifies post-processing, improves production efficiency, realizes more natural and real facial animation, reduces complex preparations for user participation, and promotes the development of the digital entertainment industry.
Smart Images
Figure CN119228958A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of animation production, in particular to a method for producing animation of a digital character according to the facial expression of a user. Background Art
[0002] Capturing and manipulating the geometry, appearance, and motion of the human face is a core part of modern computer animation. Digital characters are usually created by combining 3D scanning with appearance acquisition, and animated by motion capture. However, these existing methods have some disadvantages, such as requiring complex acquisition systems and extensive manual post-processing. As a result, creating and animating digital characters using existing methods requires a long cycle and considerable production costs. Summary of the invention
[0003] The present invention provides a method for animating a digital character according to the facial expression of a user, which solves the problem that the existing method for creating and animating a digital character requires a long period of time and a relatively high production cost.
[0004] To achieve the above object, the present invention provides the following technical solutions:
[0005] A method for animating a digital character based on a user's facial expression, the method comprising the following steps:
[0006] S01, data collection of 3D camera: use the 3D camera to collect the RGB image and depth image of the user's face, and obtain the 3D camera internal parameter matrix K;
[0007] S02, point cloud generation: convert the depth map and the intrinsic parameter matrix K into point cloud data;
[0008] S03, calculating animation parameters of the digital character: using the user-specific expression model and the animation prior to determine animation parameters that can be used for the digital character, wherein the animation prior is a sequence of animation parameters representing a predefined animation of the digital character;
[0009] S04, animation parameter transfer: using animation parameters to drive the digital character so that the digital character imitates the facial movements of the user;
[0010] S05. Obtain digital character animation.
[0011] As a further technical solution of the present invention, in step S03, before calculating the animation parameters of the digital character, it is necessary to construct a user-specific expression model and an animation priori model, and the step of constructing the user-specific expression model includes:
[0012] S031, 3D camera data collection, this step is the same as S01:
[0013] S032, point cloud generation, this step is consistent with S02;
[0014] S033, point cloud fusion: This step will fuse the collected point cloud sequence to obtain the user's complete facial point cloud data;
[0015] S034. Calculate a user-specific expression model: The user's complete facial point cloud data is used to modify the general expression model to obtain a user-specific expression model; the general expression model is an expression model defined for most people, and the user-specific expression model is an expression model specifically defined for the user.
[0016] As a further technical solution of the present invention, the step of point cloud fusion in S033 includes:
[0017] In the fusion process, the point cloud is explicitly expressed in the form of facets; facets have the following properties:
[0018] 1) Position: indicates the position of the surface element in space;
[0019] 2) Normal vector: describes the direction of the surface element;
[0020] 3) Radius: The radius of the facet is adaptively adjusted according to the focal length f of the 3D camera. The formula is:
[0021] 4) Visibility confidence: used to estimate the reliability of the face element, with a value range of [0,64]. If the position of the face element can be confirmed by multiple observations from different directions, it is considered to be correct and has a high visibility confidence.
[0022] When creating a new surfel, a spherical coordinate system is generated with the estimated surfel normal as the first basis vector and any vector in the surfel plane as the second basis vector;
[0023] The polar angle and azimuth angle are used to characterize the view direction, and the view directions in which the surface element has been observed are recorded in a two-dimensional binary histogram;
[0024] For this setting, a surface element has high visibility confidence if it is observed in at least 6 different view directions;
[0025] Facet operations:
[0026] 1) Added:
[0027] When: After all surfels are updated, when there is a part of the scanner depth map that is not covered by the model surfel, and the input data is valid and the confidence is high, a surfel will be added;
[0028] Specific operation: New facets are introduced only where the input data is valid and has high confidence. After each facet addition and update, the facet visibility confidence histogram is updated accordingly.
[0029] 3) Delete:
[0030] Deletion conditions: Model elements that are untrustworthy and conflict with the current scan will be deleted;
[0031] Principle: During the scanning process, many correct surfaces and some possible outliers are added. Correct surfaces are re-observed from many viewing directions to achieve high confidence values. In contrast, outliers are unlikely to be seen again. If conflicts occur, they are removed.
[0032] Additional Strategy: To keep the model lightweight, an erosion strategy is also applied, where every surfel that has not been updated in the past 30 frames and has not reached the minimum confidence value of 3 will be removed.
[0033] As a further technical solution of the present invention, step S034 comprises: calculating fusion shape parameters: in the modification process, the general expression model is represented as a fusion shape, and a set of approximate fusion shape weights are determined, the weights specifying a linear combination of the fusion shapes, representing model parameters for each predetermined facial expression, and since the facial expressions performed by the user are predefined, the approximate fusion shape weights can be determined once and remain constant for the user;
[0034] Based on this, gradient space optimization is performed based on the complete facial point cloud data of the given user to construct a set of user-specific expression models.
[0035] As a further technical solution of the present invention, the process of constructing the animation prior model is as follows:
[0036] Each animation prior is a set of animation parameters representing a predefined animation of a general model, and the animation priors can be mathematically defined so as to be included in the optimization calculation;
[0037] Animation sequences completed by animators;
[0038] We compute an animation prior model of temporal features, use a reduced-dimensional Gaussian mixture model of K Gaussian distributions with weights π_k to try to capture the local linear manifold of the expression space, and use principal component analysis in the latent space of the animation sequence A to make the learning phase of the EM algorithm more stable by retaining 99% of the total variance.
[0039] As a further technical solution of the present invention, in step S03, the step of calculating the animation parameters of the digital character includes:
[0040] After rigid registration, a time domain filter is used for filtering. The filter is based on a sliding window and dynamically adjusts the smoothing coefficient, similar to the exponentially weighted moving average method, to reduce high-frequency noise while avoiding disruptive time lags. The quaternion representation of the translation vector and the rotation is filtered independently.
[0041] The translation or quaternion vector t for the current time frame i i , through the formula
[0042] Calculate the smoothed vector
[0043] Where k is the size of the sliding window, t i-j It represents the vector from frame i to frame j, with weight w j It is expressed as:
[0044]
[0045] where H is a constant determined empirically based on the noise level of static postures;
[0046] The expression parameters of the user-specific expression model are used for calculation, since the user-specific expression model represents the facial expression of the user's face shown in the 3D depth map and the 2D image;
[0047] The first step involves a MAP estimate, letting D i =(G i , I i ) is the input data of the current frame, including the depth map G i and color image I i , trying to i Given the previously reconstructed n fused shape vector sequence X = {X 1 ,X 2 ,...,X n}, the most likely fusion shape weight X for the current frame i ∈R n , this inference problem is formulated as a maximum a posteriori MAP estimation:
[0048] x * = argmax x p(x|D,X n ), where p(·|·) represents the conditional probability. Applying Bayes’ rule, we get the following result x * = argmax x p(D|x,X n )p(x,Z n )
[0049] D and X nare independent of each other, then we get x * ≈argmax x p(D|x)p(x,X n );
[0050] The following p(x,X n ) is a prior model; to prevent unrealistic facial poses, the blend shape weights are regularized using a dynamic expression prior computed from existing blend shape animations;
[0051] The prior model is represented by the MPPCA model. Probabilistic principal component analysis (PPCA) defines the probability density function of the observed data, which is assumed to be a linear function of the latent variables and contains a mean vector and a Gaussian-distributed noise variable.
[0052] Specifically expressed: prior distribution p(x,X n ) is defined as a weighted combination of K Gaussian distributions, that is,
[0053] where π k is the weight;
[0054] Learning parameters: Use the expectation-maximization EM algorithm to learn the unknown parameters in the prior distribution based on the given blend shape animation sequence, including the mean μ k , covariance matrix Noise parameter σ k and the relative weight π k , estimated in the latent space of animation sequences and reduces the dimensionality of the training data by retaining 99% of the total variance;
[0055] Algorithm operation: Since the EM algorithm will converge to a local minimum, the algorithm was run 50 times with random initialization. In practical applications, 20 Gaussian distributions were used to model the prior distribution, and one-third of the latent space dimension was used as the dimension of PPCA.
[0056] As a further technical solution of the present invention, it is assumed that: by assuming conditional independence, the likelihood distribution is modeled as p(D|x)=p(G|x)p(I|x), where the two factors capture the alignment of the hybrid shape model with the acquired depth map and texture image respectively;
[0057] Calculation based on point cloud position: For p(G|x), it is similar to non-rigid ICP, assuming that the point plane distance of each vertex obeys Gaussian distribution, specifically expressed as:
[0058]
[0059] where V is the number of vertices in the template mesh, B∈R V×m is the blend shape matrix, Vi =(Bx) i is the i-th vertex of the reconstructed mesh, n i Yes V i The surface normal at is the corresponding nearest point in the depth map G;
[0060] Texture-based computation: For p(I|x), since the user’s facial texture is obtained when building the facial expression model, the model-based optical flow constraint can be integrated by formulating a likelihood function using a Gaussian distribution for each vertex, expressed as:
[0061]
[0062] where p i Yes V i The point projected onto image I, ▽I i Is I in p i The gradient at is the corresponding point in the rendered texture image;
[0063] Through the above calculations on both point cloud and texture, LikelihoodDistribution can take into account the information of depth map and texture image during tracking, so as to more accurately estimate the blend shape weights to achieve effective tracking of facial expressions.
[0064] As a further technical solution of the present invention, the final optimization process is also included:
[0065] Objective function: To solve the problem of x * ≈argmax x p(D|x)p(x,X n ) defines the MAP problem, which requires minimizing the negative logarithm, namely:
[0066] x * = argmax x -lnp(G|x)-lnp(I|x)-lnp(x,X n );
[0067] Simplified form: After discarding the constants, it can be written as:
[0068] x * = argmax x E geo +E im +E prior ,in
[0069] E prior =-lnp(x,X n )
[0070] Parameter setting: σ geo and σ im Used to model the noise level of the data, controlling the emphasis of the geometric and image likelihood terms relative to the prior terms. Through real-time feedback, we experimentally determine the appropriate value to achieve stable tracking performance. We use σ in all results. geo =1 and σ im =0.45;
[0071] Constraint handling: A gradient projection algorithm based on the LBFGS solver is used to ensure that the blend shape weights are between 0 and 1. Due to the effective warm start using the previous solution, the algorithm converges in less than 6 iterations. Then the correspondence between the closest point in and is updated, and the MAP estimate is recalculated. It is found that 3 iterations of this outer loop are sufficient for convergence;
[0072] Through the above optimization process, the blend shape weights can be effectively estimated, thereby achieving accurate tracking of facial expressions.
[0073] Compared with the prior art, the present invention has the following beneficial effects:
[0074] Social effect: The present invention provides a more convenient and natural way to achieve facial animation. Users do not need to wear special equipment or perform complicated preparations, and can more easily participate in the animation production of digital characters. This will help promote the development of the digital entertainment industry and bring better experience to users.
[0075] Economic effect: Compared with the existing methods that require complex acquisition systems and a lot of manual post-processing, the present invention uses low-cost real-time depth sensors, which reduces equipment costs and production costs, while reducing the time and cost of manual post-processing, improving production efficiency, and having better economic benefits.
[0076] Technical effect:
[0077] Improve tracking robustness: By decoupling the user's rigid motion from the non-rigid motion and tracking them independently, the optimization formula is simplified and the tracking accuracy and stability are improved.
[0078] Ensure the realism of animations: Use dynamic expression priors to regularize expression parameters and ensure that the reconstructed animations are in the space of real human facial expressions, allowing digital characters to more realistically mimic the user’s facial expressions.
[0079] Easy to integrate into existing systems: Facial expressions are represented as the weighted sum of fused shape meshes. The fused shape model provides a compact representation of the facial expression space, reduces the dimensionality of the optimization problem, and the output expression parameters can be directly imported into commercial animation tools, making it easy to integrate into existing animation systems / tools / production workflows. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] Figure 1 A flow chart of a method for animating a digital character according to a user's facial expression provided by the present invention. DETAILED DESCRIPTION
[0081] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.
[0082] A method for animating a digital character based on a user's facial expression, the method comprising the following steps:
[0083] S01, 3D camera data collection: Use the 3D camera related hardware to collect the RGB image and depth image of the user's face, and obtain the 3D camera internal parameter matrix K, which is a fixed parameter of the 3D camera and is provided by the manufacturer;
[0084] S02, point cloud generation: converting the depth map and the intrinsic parameter matrix K into point cloud data: converting the depth map and the intrinsic parameter matrix K into point cloud data by the following formula; Z = d, where c x , c y , f x , f y is the element of the intrinsic parameter matrix K, u, v are the image coordinates with the upper left corner of the screen as the origin, the right as the positive X axis, and the bottom as the positive Y axis, and d is the value of the depth map in the (u, v) coordinate system;
[0085] S03, calculating animation parameters of the digital character: using the user-specific expression model and the animation prior to determine animation parameters that can be used for the digital character, wherein the animation prior is a sequence of animation parameters representing a predefined animation of the digital character;
[0086] S04, animation parameter transfer: using animation parameters to drive the digital character so that the digital character imitates the facial movements of the user;
[0087] S05. Obtain digital character animation.
[0088] In this embodiment, in step S03, before calculating the animation parameters of the digital character, it is necessary to construct a user-specific expression model and an animation priori model. The step of constructing the user-specific expression model includes:
[0089] S031, 3D camera data collection, this step is the same as S01:
[0090] S032, point cloud generation, this step is consistent with S02;
[0091] S033, point cloud fusion: This step will fuse the collected point cloud sequence to obtain the user's complete facial point cloud data;
[0092] S034. Calculate a user-specific expression model: The user's complete facial point cloud data is used to modify the general expression model to obtain a user-specific expression model; the general expression model is an expression model defined for most people, and the user-specific expression model is an expression model specifically defined for the user.
[0093] In this embodiment, the step of point cloud fusion in S033 includes:
[0094] In the fusion process, the point cloud is explicitly expressed in the form of facets; facets have the following properties:
[0095] 1) Position: indicates the position of the surface element in space;
[0096] 2) Normal vector: describes the direction of the surface element;
[0097] 3) Radius: The radius of the facet is adaptively adjusted according to the focal length f of the 3D camera. The formula is:
[0098] 4) Visibility confidence: used to estimate the reliability of the face element, with a value range of [0,64]. If the position of the face element can be confirmed by multiple observations from different directions, it is considered to be correct and has a high visibility confidence.
[0099] When creating a new surfel, a spherical coordinate system is generated with the estimated surfel normal as the first basis vector and any vector in the surfel plane as the second basis vector;
[0100] The polar angle and azimuth angle are used to characterize the view direction, and the view directions in which the surface element has been observed are recorded in a two-dimensional binary histogram;
[0101] For this setting, a surface element has high visibility confidence if it is observed in at least 6 different view directions;
[0102] Facet operations:
[0103] 1) Added:
[0104] When: After all surfels are updated, when there is a part of the scanner depth map that is not covered by the model surfel, and the input data is valid and the confidence is high, a surfel will be added;
[0105] Specific operation: New facets are introduced only where the input data is valid and has high confidence. After each facet addition and update, the facet visibility confidence histogram is updated accordingly.
[0106] 4) Delete:
[0107] Deletion conditions: Model elements that are untrustworthy and conflict with the current scan will be deleted;
[0108] Principle: During the scanning process, many correct surfaces and some possible outliers are added. Correct surfaces are re-observed from many viewing directions to achieve high confidence values. In contrast, outliers are unlikely to be seen again. If conflicts occur, they are removed.
[0109] Additional Strategy: To keep the model lightweight, an erosion strategy is also applied, where every surfel that has not been updated in the past 30 frames and has not reached the minimum confidence value of 3 will be removed.
[0110] In this embodiment, step S034 includes: calculating fused shape parameters: during the modification process, the general expression model is represented as a fused shape, and a set of approximate fused shape weights are determined, the weights specifying a linear combination of the fused shapes, representing model parameters for each predetermined facial expression, and since the facial expressions performed by the user are predefined, the approximate fused shape weights can be determined once and remain constant for the user:
[0111] Based on this, gradient space optimization is performed based on the complete facial point cloud data of the given user to construct a set of user-specific expression models.
[0112] In this embodiment, the process of constructing the animation prior model is as follows:
[0113] Each animation prior is a set of animation parameters representing a predefined animation of a general model, and the animation priors can be mathematically defined so as to be included in the optimization calculation;
[0114] Animation sequences completed by animators;
[0115] Animation prior models for computing temporal features. In order to model high-dimensional animation data with the MPPCA model while still achieving real-time performance, a dimension-reduced Gaussian mixture model with K Gaussian distributions with weights π_k is used to try to capture the local linear manifold of the expression space. By using principal component analysis in the latent space of the animation sequence A, the dimensionality of the training data can be reduced. By retaining 99% of the total variance, the dimensionality can be reduced by two-thirds, making the learning phase of the EM algorithm more stable.
[0116] In this embodiment, the step of calculating the animation parameters of the digital character includes:
[0117] After rigid registration, a time domain filter is used for filtering to reduce high-frequency jitter caused by registration errors. The filter is based on a sliding window and dynamically adjusts the smoothing coefficient, similar to the exponentially weighted moving average method, to reduce high-frequency noise while avoiding disruptive time lags. The quaternion representations of the translation vector and rotation are filtered independently.
[0118] The translation or quaternion vector t for the current time frame i i , through the formula
[0119] Calculate the smoothed vector
[0120] Where k is the size of the sliding window, t i-j It represents the vector from frame i to frame j, with weight w j It is expressed as:
[0121]
[0122] where H is a constant determined empirically based on the noise level of static postures;
[0123] The expression parameters of the user-specific expression model are used for calculation, since the user-specific expression model represents the facial expression of the user's face shown in the 3D depth map and the 2D image;
[0124] The first step involves a MAP estimate, letting D i =(G i , I i ) is the input data of the current frame, including the depth map G i and color image I i , trying to i Given the previously reconstructed n fused shape vector sequence X = {X 1 ,X 2 ,...,X n}, the most likely fusion shape weight X for the current frame i ∈R n , this inference problem is formulated as a maximum a posteriori MAP estimation:
[0125] x * = argmax x p(x|D,X n ), where p(·|·) represents the conditional probability. Applying Bayes’ rule, we get the following result x * = argmax x p(D|x,X n )p(x,Z n )
[0126] D and X n are independent of each other, then we get x * ≈argmax x p(D|x)p(x,X n );
[0127] The following p(x,X n ) is a prior model; to prevent unrealistic facial poses, the blend shape weights are regularized using a dynamic expression prior computed from existing blend shape animations;
[0128] The prior model is represented by the MPPCA model. Probabilistic principal component analysis (PPCA) defines the probability density function of the observed data, which is assumed to be a linear function of the latent variables and contains a mean vector and a Gaussian-distributed noise variable.
[0129] Specifically expressed: prior distribution p(x,X n ) is defined as a weighted combination of K Gaussian distributions, that is,
[0130] where π k is the weight;
[0131] Learning parameters: Use the expectation-maximization EM algorithm to learn the unknown parameters in the prior distribution based on the given blend shape animation sequence, including the mean μ k , covariance matrix Noise parameter σ k and the relative weight π k , in order to increase the computational robustness, the estimation is performed in the latent space of the animation sequence and the dimensionality of the training data is reduced by retaining 99% of the total variance, making the learning phase of the EM algorithm more stable;
[0132] Algorithm operation: Since the EM algorithm will converge to a local minimum, the algorithm is run 50 times with random initialization to improve learning accuracy. In practical applications, 20 Gaussian distributions are used to model the prior distribution, and one-third of the latent space dimension is used as the dimension of PPCA.
[0133] In this example, the preceding one is the likelihood distribution:
[0134] Assumptions: By assuming conditional independence, the likelihood distribution is modeled as p(D|x)=p(G|x)p(I|x), where the two factors capture the alignment of the blend shape model with the acquired depth map and texture image, respectively.
[0135] Calculation based on point cloud position: For p(G|x), it is similar to non-rigid ICP, assuming that the point plane distance of each vertex obeys Gaussian distribution, specifically expressed as:
[0136]
[0137] where V is the number of vertices in the template mesh, B∈R V×m is the blend shape matrix, V i =(Bx) i is the i-th vertex of the reconstructed mesh, n i Yes V i The surface normal at is the corresponding nearest point in the depth map G;
[0138] Texture-based computation: For p(I|x), since the user’s facial texture is obtained when building the facial expression model, the model-based optical flow constraint can be integrated by formulating a likelihood function using a Gaussian distribution for each vertex, expressed as:
[0139]
[0140] where p i Yes V i The point projected onto image I, ▽I i Is I in p i The gradient at is the corresponding point in the rendered texture image;
[0141] Through the above calculations on both point cloud and texture, LikelihoodDistribution can take into account the information of depth map and texture image during tracking, so as to more accurately estimate the blend shape weights to achieve effective tracking of facial expressions.
[0142] In this embodiment, the final optimization process is:
[0143] Objective function: To solve the problem of x * ≈argmax x p(D|x)p(x,X n ) defines the MAP problem, which requires minimizing the negative logarithm, namely:
[0144] x * = argmax x -lnp(G|x)-lnp(I|x)-lnp(x,X n );
[0145] Simplified form: After discarding the constants, it can be written as:
[0146] x * = argmax x E geo +E im +E prior ,in
[0147] E prior =-lnp(x,X n )
[0148] Parameter setting: σ geo and σ im Used to model the noise level of the data, controlling the emphasis of the geometric and image likelihood terms relative to the prior terms. Through real-time feedback, we experimentally determine the appropriate value to achieve stable tracking performance. We use σ in all results. geo =1 and σ im =0.45;
[0149] Constraint handling: A gradient projection algorithm based on the LBFGS solver is used to ensure that the blend shape weights are between 0 and 1. Due to the effective warm start using the previous solution, the algorithm converges in less than 6 iterations. Then the correspondence between the closest point in and is updated, and the MAP estimate is recalculated. It is found that 3 iterations of this outer loop are sufficient for convergence;
[0150] Through the above optimization process, the blend shape weights can be effectively estimated, thereby achieving accurate tracking of facial expressions.
[0151] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0152] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for animating a digital character based on a user's facial expression, characterized in that: The method comprises the following steps: S01, data collection of 3D camera: use the 3D camera to collect the RGB image and depth image of the user's face, and obtain the 3D camera internal parameter matrix K; S02, point cloud generation: convert the depth map and the intrinsic parameter matrix K into point cloud data; S03, calculating animation parameters of the digital character: using the user-specific expression model and the animation prior to determine animation parameters that can be used for the digital character, wherein the animation prior is a sequence of animation parameters representing a predefined animation of the digital character; S04, animation parameter transfer: using animation parameters to drive the digital character so that the digital character imitates the facial movements of the user; S05. Obtain digital character animation.
2. A method for animating a digital character based on a user's facial expression according to claim 1, characterized in that: In step S02, the depth map and the intrinsic parameter matrix K are converted into point cloud data by the following formula; Z = d, where c x , c y , f x , f y is the element of the intrinsic parameter matrix K, u, v are the image coordinates with the upper left corner of the screen as the origin, the right as the positive X axis, and the bottom as the positive Y axis, and d is the value of the depth map in the (u, v) coordinate system.
3. A method for animating a digital character based on a user's facial expression according to claim 1, characterized in that: In step S03, before calculating the animation parameters of the digital character, it is necessary to construct a user-specific expression model and an animation priori model. The step of constructing the user-specific expression model includes: S031, 3D camera data collection, this step is the same as S01: S032, point cloud generation, this step is consistent with S02; S033, point cloud fusion: fuse the collected point cloud sequence to obtain the user's complete facial point cloud data; S034. Calculate a user-specific expression model: The user's complete facial point cloud data is used to modify the general expression model to obtain a user-specific expression model; the general expression model is an expression model defined for most people, and the user-specific expression model is an expression model specifically defined for the user.
4. A method for animating a digital character based on a user's facial expression according to claim 3, characterized in that: The step of point cloud fusion in S033 includes: In the fusion process, the point cloud is explicitly expressed in the form of facets; facets have the following properties: 1) Position: indicates the position of the surface element in space; 2) Normal vector: describes the direction of the surface element; 3) Radius: The radius of the facet is adaptively adjusted according to the focal length f of the 3D camera. The formula is: 4) Visibility confidence: used to estimate the reliability of the face element, with a value range of [0,64]. If the position of the face element can be confirmed by multiple observations from different directions, it is considered to be correct and has a high visibility confidence. When creating a new surfel, a spherical coordinate system is generated with the estimated surfel normal as the first basis vector and any vector in the surfel plane as the second basis vector; The polar angle and azimuth angle are used to characterize the view direction, and the view directions in which the surface element has been observed are recorded in a two-dimensional binary histogram; If a surface is observed in at least 6 different viewing directions, it has high visibility confidence; Facet operations: 1) Added: When: After all surfels are updated, when there is a part of the scanner depth map that is not covered by the model surfel, and the input data is valid and the confidence is high, a surfel will be added; 2) Delete: Deletion conditions: Model elements that are untrustworthy and conflict with the current scan will be deleted; Additional strategy: Every surfel that has not been updated in the past 30 frames and has not reached the minimum confidence value of 3 will be deleted.
5. A method for animating a digital character based on a user's facial expression according to claim 4, characterized in that: Step S034 comprises: calculating fused shape parameters: in the modification process, representing the general expression model as a fused shape, and determining a set of approximate fused shape weights, the weights specifying a linear combination of the fused shapes, representing model parameters for each predetermined facial expression, since the facial expressions performed by the user are predefined, the approximate fused shape weights can be determined once and remain constant for the user; Based on this, the gradient space optimization is performed based on the complete facial point cloud data of the given user to construct a set of user-specific expression models.
6. A method for animating a digital character based on a user's facial expression according to claim 5, characterized in that: The process of constructing the animation prior model is as follows: Animation sequences completed by animators; Calculate the animation prior model of the time domain features, using a weight π k A dimensionally reduced Gaussian mixture model of K Gaussian distributions tries to capture the local linear manifold of the expression space, and uses principal component analysis in the latent space of the animation sequence A, which makes the learning phase of the EM algorithm more stable by retaining 99% of the total variance.
7. A method for animating a digital character based on a user's facial expression according to claim 6, characterized in that: In step S03, the step of calculating the animation parameters of the digital character includes: After rigid registration, a time domain filter is used for filtering. The filter is based on a sliding window and dynamically adjusts the smoothing coefficient to independently filter the quaternion representation of the translation vector and the rotation; The translation or quaternion vector t for the current time frame i i , through the formula Calculate the smoothed vector Where k is the size of the sliding window, t i-j It represents the vector from frame i to frame j, with weight w j It is expressed as: where H is a constant determined empirically based on the noise level of static postures; The computation is performed using the animation prior and the user-specific expression model, since the user-specific expression model represents the facial expressions of the user's face shown in the 3D depth map and the 2D image; The first step involves a MAP estimate, letting D i =(G i ,I i ) is the input data of the current frame, including the depth map G i and color image I i , trying to i Given a previously reconstructed sequence of n fused shape vectors X = {X1, X2, ..., X n }, the most likely fusion shape weight X for the current frame i ∈R n , this inference problem is formulated as a maximum a posteriori MAP estimation: x * = argmax x p(x|D,X n ), where p(·|·) represents the conditional probability. Applying Bayes’ rule, we get the following result x * = arg max x p(D|x,X n )p(x,Z n ); D and X n are independent of each other, then we get x * ≈arg max x p(D|x)p(x,X n ); The following p(x,X n ) is a prior model; to prevent unrealistic facial poses, the blend shape weights are regularized using a dynamic expression prior computed from existing blend shape animations; The prior model is represented by the MPPCA model. Probabilistic principal component analysis (PPCA) defines the probability density function of the observed data, which is assumed to be a linear function of the latent variables and contains a mean vector and a Gaussian-distributed noise variable. Specifically expressed: prior distribution p(x,X n ) is defined as a weighted combination of K Gaussian distributions, that is, where π k is the weight; Learning parameters: Use the expectation-maximization EM algorithm to learn the unknown parameters in the prior distribution based on the given blend shape animation sequence, including the mean μ k , covariance matrix C k C k T , noise parameter σ k and the relative weight π k , estimated in the latent space of animation sequences and reduces the dimensionality of the training data by retaining 99% of the total variance; Algorithm operation: Since the EM algorithm will converge to a local minimum, the algorithm was run 50 times with random initialization. In practical applications, 20 Gaussian distributions were used to model the prior distribution, and one-third of the latent space dimension was used as the dimension of PPCA.
8. A method for animating a digital character based on a user's facial expression according to claim 7, characterized in that: Assumptions: By assuming conditional independence, the likelihood distribution is modeled as p(D|x)=p(G|x)p(I|x), where the two factors capture the alignment of the hybrid shape model with the acquired depth map and texture image, respectively. Calculation based on point cloud position: For p(G|x), it is similar to non-rigid ICP, assuming that the point plane distance of each vertex obeys Gaussian distribution, specifically expressed as: where V is the number of vertices in the template mesh, B∈R V×m is the blend shape matrix, V i =(Bx) i is the i-th vertex of the reconstructed mesh, n i Yes V i The surface normal at V i * is the depth map G i The corresponding nearest point in Texture-based computation: For p(I|x), since the user’s facial texture is obtained when building the facial expression model, the model-based optical flow constraint is integrated by formulating a likelihood function using a Gaussian distribution for each vertex, expressed as: where p i Yes V i The point projected onto image I, Is I in p i The gradient at is the corresponding point in the rendered texture image.
9. The method of animating a digital character according to a user's facial expression according to claim 7, characterized in that: The final optimization process is also included: Objective function: To solve the problem of x * ≈arg max x p(D|x)p(x,X n ) defines the MAP problem, which requires minimizing the negative logarithm, namely: x * =argmax x -lnp(G|x)-lnp(I|x)-lnp(x,X n ); Simplified form: After discarding the constants, it can be written as: x * = argmax x E geo +E im +E prior ,in AND prior =-lnp(x,X n ) Parameter setting: σ geo and σ im Used to model the noise level of the data, controlling the emphasis of the geometric and image likelihood terms relative to the prior terms. Through real-time feedback, we experimentally determine the appropriate value to achieve stable tracking performance. We use σ in all results. geo =1 and σ im =0.45; Constraint handling: A gradient projection algorithm based on the LBFGS solver is used to ensure that the blend shape weights are between 0 and 1. Due to the effective warm start using the previous solution, the algorithm converges in less than 6 iterations. The correspondence between the nearest points in and is then updated and the MAP estimate is recalculated. It is found that 3 iterations of this outer loop are sufficient for convergence.