Head portrait expression migration method based on parameter fitting and gradient migration

Abstract

The invention provides a head portrait expression migration method based on parameter fitting and gradient migration, and relates to the technical field of three-dimensional face animation, and the method specifically comprises the steps: inputting an audio signal into an audio driving model to generate a continuous three-dimensional grid sequence of a source role, obtaining a control parameter set, and extracting mouth motion parameters from the control parameter set; reconstructing a head portrait of a target role based on a three-dimensional Gaussian head model, generating a parameter fitting grid sequence of the target role through linear hybrid skin, and obtaining a rough expression migration result; comparing the original grid of the source role with the parameter fitting grid, and calculating deformation gradients of the original grid and the parameter fitting grid; and migrating the deformation gradient of the source role to the deformation gradient of the target role through optimization, thereby obtaining a fine grid sequence of the target role, updating the Gaussian point cloud model of the target role, and obtaining a high-precision expression migration result. According to the technical scheme, the problem that in the prior art, head portrait expression migration is difficult to give consideration to global consistency and detail reduction is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional face animation, in particular to a portrait expression transfer method based on parameter fitting and gradient migration. BACKGROUND

[0002] With the rapid development of virtual humans, digital interaction and immersive human-computer communication, three-dimensional face animation technology has gradually become an important research direction in this field. Existing methods usually use audio signals to drive the dynamic changes of three-dimensional face meshes, thereby generating expression sequences synchronized with speech. Such methods can achieve automatic synthesis of facial movements to some extent, but due to direct dependence on mesh representation, they still have deficiencies in capturing subtle expression changes and maintaining character consistency.

[0003] In recent years, parameterized face modeling techniques have been widely used in face modeling and animation, controlling expressions, poses and identities through low-dimensional parameters. However, the parameter differences between different characters often lead to inaccurate expression transfer, making it difficult to achieve natural expression mapping while maintaining individual characteristics. At the same time, while relying solely on parameterized models can ensure uniform modeling of global expressions, their expressive ability is limited, and high-frequency details such as mouth corners and wrinkles are easily lost; if direct dependence on the original mesh is used for transfer, it is difficult to maintain consistency and controllability between characters.

[0004] At the same time, three-dimensional portrait modeling methods based on Gaussian representation have received widespread attention due to their high rendering efficiency and realistic visual effects. Such methods usually bind Gaussian kernels to three-dimensional meshes to achieve controllable reconstruction and rendering of portraits. However, if audio signals are used to drive Gaussian representation models directly, a large amount of paired data is often required for training, which is costly and difficult to generalize. In contrast, using existing mesh-driven methods to achieve expression transfer can map the expressions of source characters to target characters at a lower cost, but relying solely on a single approach cannot balance global consistency and detail restoration.

[0005] Therefore, there is a need for a portrait expression transfer method based on parameter fitting and gradient migration that can achieve high-precision, natural expression transfer while maintaining character identity consistency. SUMMARY

[0006] The main purpose of the present application is to provide a portrait expression transfer method based on parameter fitting and gradient migration to solve the problem of existing technology that portrait expression transfer cannot balance global consistency and detail restoration.

[0007] To achieve the above purpose, the present application provides a portrait expression transfer method based on parameter fitting and gradient migration, which specifically includes the following steps: S1, input the audio signal into the audio driving model to generate a continuous three-dimensional mesh sequence of the source character, and fit it based on the FLAME model to obtain a set of control parameters, and extract the mouth movement parameters from the set of control parameters.

[0008] S2 reconstructs the target character's head based on a 3D Gaussian head model, combines the mouth motion parameters extracted from the source character with the initial parameters of the target character, and generates a parameter fitting mesh sequence for the target character through linear blending skinning to obtain a rough expression transfer result.

[0009] S3 calculates the deformation gradient of the source character's original mesh and the parameter-fitted mesh by comparing the two. This allows for the extraction of subtle facial expressions that were not captured by parameter fitting.

[0010] S4. The deformation gradient of the source character is transferred to the deformation gradient of the target character through optimization, thereby obtaining a fine mesh sequence of the target character and updating the Gaussian point cloud model of the target character to obtain a high-precision expression transfer result.

[0011] Furthermore, step S1 specifically includes the following steps: S1.1 Select a character from the dataset as the source character, and obtain a frame of the source character with no expression and no pose change as the initial mesh. .

[0012] S1.2, Input an audio signal Call the audio driver model, based on the audio signal. Output source character A continuous three-dimensional mesh sequence driven by audio: ; in, For frame number, Indicates the source role in the Frame grid, It is an audio-driven model.

[0013] S1.3, will As the source character's original mesh sequence, with the initial mesh FLAME model fitting was performed separately to obtain the face control parameters corresponding to the grid: ; ; in, Indicates source role In the Frame control parameters; This represents the initial control parameters of the source character.

[0014] S1.4, extracting the parameters related to the mouth movement from the and respectively, denoted as and wherein, denotes the parameters related to the mouth movement, denotes the parameters related to the mouth movement.

[0015] S1.5, normalizing the mouth parameters of each frame: ; wherein, denotes the mouth movement parameter increment of the source character in the frame relative to the neutral expression, including: the mandible movement parameter and the overall rotation parameter.

[0016] Further, step S2 specifically comprises the following steps: S2.1, training and reconstructing a Gaussian point cloud model of the target character using a three-dimensional Gaussian human head model .

[0017] S2.2, taking the corresponding three-dimensional network as the target character, obtaining a frame of expressionless and posture-changing mesh of the target character as the initial mesh , and obtaining the corresponding FLAME control parameters from the data set, i.e., the initial control parameters of the target character .

[0018] S2.3, combining the mouth movement parameter increment of the source character with the initial control parameters of the target character to obtain the control parameters of the target character in the frame: ; wherein, denotes the superposition of the mouth movement parameters.

[0019] S2.4, mapping the target character control parameters to the parameter fitting mesh sequence of the target character based on the linear blend skinning method ; wherein, is the parameter fitting mesh of the target character in the frame.

[0020] Further, step S3 specifically comprises the following steps:​ S3.1, combine the mouth movement parameter increment obtained by the source character with the initial control parameter of the source character to obtain the control parameter of the source character in the first frame wherein, the superposition of the mouth movement parameter is represented by

[0021] S3.2, based on the linear blend skinning method, map the parameter fitting mesh sequence of the target character to the parameter fitting mesh sequence of the source character; wherein, is the parameter fitting mesh of the source character in the first frame

[0022] S3.3, calculate the gradient difference between the original mesh sequence of the source character and

[0023] Further, step S3.3 is specifically as follows: Supposing the character mesh is composed of triangles, for the i-th triangle in the first frame, the edge vector matrix of the original mesh is denoted as , the edge vector matrix of the parameter fitting mesh is denoted as , and the edge vector matrix is obtained from the mesh vertex; the deformation gradient is defined as: wherein, the local deformation gradient of the original mesh of the source character in the first frame relative to the i-th triangle of the parameter fitting mesh is represented by

[0024] Further, step S4 specifically includes the following steps: S4.1, in order to transfer the deformation gradient of the source character to the target character, a gradient transfer method is adopted and a constraint vertex is randomly fixed as a constraint condition, and the deformation gradient of the target character is obtained by optimization; supposing the character mesh is composed of triangles, for the i-th triangle in the first frame, the edge vector matrix of the parameter fitting mesh of the target character is denoted as , and the set of mesh vertices is denoted as ​​​​​​​​​​​​​​​​​​​​​​​​​; the first The optimization goal of the frame is: ; wherein, denotes the deformation gradient of the source character in the first frame in the first triangle, is a constant, is a constrained vertex, is a minimum function, denotes the Frobenius norm.

[0025] S4.2, according to the deformation gradient of the target character, the target character fine mesh sequence is obtained; is the fine mesh of the target character.

[0026] S4.3, according to the target character fine mesh sequence , the related parameters of the Gaussian point cloud model of the target character are updated, and the expression migration from the source character to the target character is realized.

[0027] The present application has the following beneficial effects: The present application combines parameter fitting and gradient migration, and can realize high-precision expression migration from the source character to the target character based on Gaussian representation without additional training, not only compensates for the subtle expression changes lost in the parameter fitting process, ensures the naturalness and detail authenticity of the expression, but also maintains the identity consistency and geometric stability of the target character, has low computational overhead, good real-time performance and universality, and is suitable for efficient expression driving and interactive application in multiple characters and multiple scenes. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor. In the drawings:

[0029] Figure 1 The flow chart of the avatar expression migration method based on parameter fitting and gradient migration is shown.

[0030] Figure 2 The effect diagram of the avatar expression migration by the method proposed in the present application is shown. DETAILED DESCRIPTION ​

[0031] The technical solutions of the present application will be described clearly and completely in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] As shown in a portrait expression migration method based on parameter fitting and gradient migration, specifically comprising the following steps: Figure 1 S1, inputting an audio signal into an audio driving model (such as Imitator) to generate a continuous three-dimensional mesh sequence of a source character, fitting based on a FLAME model to obtain a control parameter set, and extracting a mouth movement parameter from the control parameter set.

[0033] S2, reconstructing a target character portrait based on a three-dimensional Gaussian human head model, combining the mouth movement parameter extracted from the source character with the initial parameter of the target character, and generating a parameter fitting mesh sequence of the target character through linear blend skinning to obtain a rough expression migration result.

[0034] S3, calculating the deformation gradient of the source character by comparing the original mesh of the source character with the parameter fitting mesh. Thus, subtle expression changes not captured by parameter fitting are extracted.

[0035] S4, migrating the deformation gradient of the source character to the deformation gradient of the target character through optimization, thereby obtaining a fine mesh sequence of the target character, and updating the Gaussian point cloud model of the target character to obtain a high-precision expression migration result.

[0036] Specifically, step S1 specifically comprises the following steps: S1.1, selecting a certain character in a data set (such as vocaset data set) as a source character, and obtaining a mesh with no expression and posture change in the source character as an initial mesh ; the mesh is generated by a parameterized face model FLAME, and the FLAME model can realize dynamic deformation of the mesh by controlling parameters.

[0037] S1.2, inputting an audio signal , calling an existing audio driving model (such as Imitator), and outputting a continuous three-dimensional mesh sequence of the source character under the audio driving according to the audio signal : ; wherein, is the frame number, represents the source character in the ​​Frame grid, It is an audio-driven model.

[0038] S1.3, will As the source character's original mesh sequence, with the initial mesh FLAME model fitting was performed separately to obtain the face control parameters corresponding to the grid: ; ; in, Indicates source role In the Frame control parameters; This represents the initial control parameters of the source character.

[0039] S1.4, from and Parameters related to mouth movements were extracted separately and denoted as follows: and ,in, express Parameters related to mouth movements, express Parameters related to mouth movements.

[0040] S1.5, In order to eliminate individual static differences, the mouth parameters of each frame are normalized: ; in, Indicates the source role in the The increment of mouth motion parameters relative to a neutral expression frame includes: jaw motion parameters and overall rotation parameters.

[0041] Specifically, step S2 includes the following steps: S2.1, using existing 3D Gaussian head models (such as Gaussian Avatars), train and reconstruct a Gaussian point cloud model of the target character. This model achieves a driveable high-fidelity Gaussian avatar by binding Gaussian points to a 3D mesh.

[0042] S2.2, will Corresponding 3D network As the target character, a single frame of mesh data with no facial expression or pose change is obtained from the target character and used as the initial mesh. And obtain the corresponding FLAME control parameters from the dataset, that is, the initial control parameters of the target character. .

[0043] S2.3, Increment the mouth motion parameters obtained from the source character. initial control parameters of the target role in combination, obtaining the target role in the first frame of the target role : ; wherein, represents the superposition of the mouth movement parameters.

[0044] S2.4, based on a linear blend skinning (LBS) method, mapping the target role control parameters to a parameter-fitted mesh sequence of the target role ; wherein, is a parameter-fitted mesh of the target role in the first frame.

[0045] Specifically, step S3 specifically comprises the following steps: S3.1, combining the mouth movement parameter increment obtained by the source role with the initial control parameters of the source role , obtaining the control parameters of the source role in the first frame of the source role : ; wherein, represents the superposition of the mouth movement parameters.

[0046] S3.2, based on a linear blend skinning method, mapping to a parameter-fitted mesh sequence of the source role ; wherein, is a parameter-fitted mesh of the source role in the first frame.

[0047] S3.3, calculating the gradient difference between the original mesh sequence of the source role and .

[0048] Specifically, step S3.3 specifically comprises: Let the role mesh be composed of triangular patches, for the triangular patch in the first frame, the edge vector matrix of the original mesh is denoted as , the edge vector matrix of the parameter-fitted mesh is denoted as , and the edge vector matrix is obtained from the mesh vertex; the deformation gradient is defined as: ; wherein, denotes the local deformation gradient of the source character in the i-th triangle of the i-th frame with respect to the parameterized mesh, which characterizes the subtle changes in the original expression that are not fully captured by the parameterized mesh.

[0049] Specifically, step S4 specifically comprises the following steps: S4.1, in order to transfer the deformation gradient of the source character to the target character, a gradient transfer method is adopted and a constraint vertex is randomly fixed as a constraint condition, the deformation gradient of the target character is obtained by optimization ; assuming that the character mesh is composed of N triangles, for the i-th triangle in the j-th frame, the edge vector matrix of the parameterized mesh of the target character is denoted as , and the set of mesh vertices is denoted as ; the optimization objective of the j-th frame is: ; wherein, denotes the deformation gradient of the source character in the i-th triangle of the j-th frame, is a constant, is the constrained vertex, is the minimum value function, denotes the Frobenius norm; the constraint condition is used to fix a certain vertex to avoid the non-uniqueness of the solution.

[0050] S4.2, according to the deformation gradient of the target character , the target character fine mesh sequence is obtained; wherein, is the fine mesh of the target character.

[0051] S4.3, according to the target character fine mesh sequence , the related parameters of the Gaussian point cloud model of the target character are updated, so as to realize the expression transfer from the source character to the target character.

[0052] ​​​​​​​​​​To verify the feasibility and superiority of this invention, the following comparative experiment was conducted. The speech-driven source characters (characters 0731 and 0809 in the VOCASET dataset) generated by the Imitator model were selected as the source character inputs, and 20 audio segments were extracted from each character, for a total of 40 audio segments.

[0053] In the experiment, the facial expression information of the source character was mapped to the target character using two transfer methods: (1) coefficient transfer method, and (2) the parameter fitting and gradient transfer combination method proposed in this invention. Simultaneously, three audio-driven models, VOCA, Faceformer, and Imitator, were compared in the experiment. By analyzing the performance of these methods in lip movement accuracy and speech synchronization, the advantages of this invention can be effectively verified. The comparison results are shown in Table 1.

[0054] Table 1 Comparison of Audio-Driven Model Metrics

[0055] Table 1 lists the different methods used for lip shaping. error( ) and lip synchronicity index ( On the surface: lip shape The error metric quantifies the spatial error between the lip vertex position in the generated mesh and the reference mesh. A lower metric indicates more accurate lip shape generation. The lip-sync metric measures the degree of synchronization between the generated lip shape and the input audio. A lower metric indicates more accurate alignment of lip movements with speech, resulting in a more natural lip shape. Experimental results show that the gradient transfer and gradient transfer combination method proposed in this invention achieves better lip shape generation. It outperforms the comparison methods in both error and lip synchronization metrics. Specifically, the lip error is as low as 0.0889, significantly better than the imitator and coefficient transfer methods, and can more accurately reproduce lip dynamics. Figure 2 Qualitative results further demonstrate that this method can more naturally preserve the subtle lip dynamics of the source character, and the generated target character has a clearer mouth shape, smoother boundaries, and a mouth opening range that is closer to the source character.

[0056] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for transferring facial expressions from avatars based on parameter fitting and gradient transfer, characterized in that, Specifically, the steps include the following: S1, input the audio signal into the audio driving model to generate a continuous three-dimensional mesh sequence of the source character, and fit it based on the FLAME model to obtain a set of control parameters, and extract the mouth movement parameters from the set of control parameters; S2, based on the 3D Gaussian head model, reconstruct the target character's head image, combine the mouth motion parameters extracted from the source character with the initial parameters of the target character, and generate the parameter fitting mesh sequence of the target character through linear blending skinning to obtain a rough expression transfer result; S3 calculates the deformation gradient of the source character's original mesh and the parameter-fitted mesh by comparing the two. This allows for the extraction of subtle facial expressions that were not captured by parameter fitting. S4. The deformation gradient of the source character is transferred to the deformation gradient of the target character through optimization, thereby obtaining a fine mesh sequence of the target character and updating the Gaussian point cloud model of the target character to obtain a high-precision expression transfer result.

2. The avatar expression transfer method based on parameter fitting and gradient transfer according to claim 1, characterized in that, Step S1 specifically includes the following steps: S1.1 Select a character from the dataset as the source character, and obtain a frame of the source character with no expression and no pose change as the initial mesh. ; S1.2, Input an audio signal Call the audio driver model, based on the audio signal. Output source character A continuous three-dimensional mesh sequence driven by audio: ； in, For frame number, Indicates the source role in the Frame grid, An audio-driven model; S1.3, will As the source character's original mesh sequence, with the initial mesh FLAME model fitting was performed separately to obtain the face control parameters corresponding to the grid: ； ； in, Indicates source role In the Frame control parameters; Indicates the initial control parameters of the source character; S1.4, from and Parameters related to mouth movements were extracted separately and denoted as follows: and ,in, express Parameters related to mouth movements, express Parameters related to mouth movements; S1.5, normalize the mouth parameters for each frame: ； in, Indicates the source role in the The increment of mouth motion parameters relative to a neutral expression frame includes: jaw motion parameters and overall rotation parameters.

3. The avatar expression transfer method based on parameter fitting and gradient transfer according to claim 1, characterized in that, Step S2 specifically includes the following steps: S2.1, Using a 3D Gaussian head model, train and reconstruct a Gaussian point cloud model of the target character. ; S2.2, will Corresponding 3D network As the target character, a single frame of mesh data with no facial expression or pose change is obtained from the target character and used as the initial mesh. And obtain the corresponding FLAME control parameters from the dataset, that is, the initial control parameters of the target character. ; S2.3, Increment the mouth motion parameters obtained from the source character. Initial control parameters of the target character Combined, the target role is obtained. In the Frame control parameters : ； in, This represents the superposition of mouth movement parameters; S2.4, based on the linear blending skinning method, adjusts the target character control parameters. Parameter fitting mesh sequence mapped to target role ;in, For target role In the The parameters of the frame are fitted to the grid.

4. The avatar expression transfer method based on parameter fitting and gradient transfer according to claim 1, characterized in that, Step S3 specifically includes the following steps: S3.1, Increment the mouth motion parameters obtained from the source character. Initial control parameters of the source character Combining these, we obtain the source character in the first place. Frame control parameters : ； in, This represents the superposition of mouth movement parameters; S3.2, based on the linear hybrid skinning method, will Parameter fitting mesh sequence mapped to source role ;in, For the source character In the Frame parameter fitting grid; S3.3, Calculate the original mesh sequence of the source role. and The gradient difference between them.

5. The avatar expression transfer method based on parameter fitting and gradient transfer according to claim 4, characterized in that, Step S3.3 specifically includes: Set the character grid by Composed of triangular facets, for the first... The first frame There are three triangles, and the edge vector matrix of the original mesh is denoted as... The edge vector matrix of the parameter-fitted mesh is denoted as The edge vector matrix is ​​obtained from the grid vertices; Deformation gradient Defined as: ； in, Indicates the source role in the The first frame of the original mesh relative to the parameter-fitted mesh The local deformation gradient of a triangle.

6. The avatar expression transfer method based on parameter fitting and gradient transfer according to claim 1, characterized in that, Step S4 specifically includes the following steps: S4.1, in order to transform the gradient of the source character Migrate to the target role, using the gradient transfer method and randomly fixing a constraint vertex. As a constraint, the deformation gradient of the target character is obtained through optimization. ; Set the character grid as follows: Composed of triangular facets, for the first... The first frame There are three triangles, and the edge vector matrix of the mesh fitted to the target character parameters is denoted as... The set of grid vertices is denoted as ;No. The optimization objective for the frame is: ； in, Indicates the source role in the The first frame The deformation gradient of a triangle, It is a constant. For constrained vertices, It is a minimum value function. Denotes the Frobenius norm; S4.2, based on the deformation gradient of the target character To obtain the fine mesh sequence of the target character ;in, For target role Fine mesh; S4.3, Based on the target character's fine mesh sequence Update the Gaussian point cloud model of the target character. The relevant parameters are used to achieve facial expression transfer from the source character to the target character.

Images