An efficient expression transfer method using basis expression space transformation
By using a base expression space transformation method and optimizing expression transfer with expression description parameters and a similarity matrix, the problem of inaccurate facial expression transfer in existing technologies is solved, achieving efficient and realistic expression transfer effects applicable to various face shapes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2022-08-18
- Publication Date
- 2026-06-09
Smart Images
Figure CN115457171B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to computer vision, and more specifically to an efficient expression transfer method employing a base expression space transformation. Background Technology
[0002] Traditional methods transfer facial expressions from an actor in a source video to an actor in a target video in real time, enabling temporary control over the facial expressions of the target actor. For example, paper 1, "Real-time expression transfer for facial reenactment," frequently involves transferring facial distortions and details with photorealistic re-rendering to the target video, making the newly synthesized expression virtually indistinguishable from the real video. To achieve this, commercially available RGB-D sensors are used to accurately capture the facial expressions of both the source and target subjects in real time. For each frame, a parameterized model for identification, expression, and skin reflectivity is combined with the input color and depth data, and scene lighting is reconstructed. For expression transfer, the differences between the source and target expressions in the parameter space are calculated, and the target parameters are modified to match the source expression. A major challenge is convincingly re-rendering the synthesized target face into the corresponding video stream. This requires careful consideration of lighting and shading design, both of which must correspond to the real-world environment. The authors demonstrate the method in a real-time setting that modifies the video conferencing source to match the facial expressions of different people (e.g., a translator) in real time.
[0003] The drawbacks of this method are as follows: Firstly, it matches the source expression by calculating the difference between two expressions in parameter space and then modifying the target's expression parameters. Secondly, this method assumes Lambert surface reflectivity and smoothly varying illumination, parameterized by spherical harmonics, which can lead to artifacts in the general environment (e.g., strong subsurface scattering, high-frequency illumination variations, or self-shadowing). Thirdly, the tracker in this method uses dense depth and color information, which allows for tight fitting but also results in a large number of residuals.
[0004] Paper 2, "Face2Face: Real-time Face Capture and Reenactment of RGBVideos," proposes a method for real-time reenactment of monocular target video sequences (e.g., YouTube videos). The source sequence is also a monocular video stream, captured in real-time using a product webcam. The goal is to animate the facial expressions of the target video by the source actor and re-render the manipulated output video in a photorealistic manner. To this end, the inadequacy of recovering facial identity from monocular videos is first addressed through non-rigid model-based binding. At runtime, dense photometric consistency measurements are used to track facial expressions in both the source and target videos. Reenactment is then achieved by rapidly and efficiently transferring deformables between the source and target. The most matching re-targeted expression of the interior of the mouth is retrieved from the target sequence and warped to produce an accurate fit. Finally, the synthesized target face is convincingly re-rendered on top of the corresponding video stream, seamlessly blending it with real-world lighting.
[0005] This method suffers from the following drawbacks: it presents challenges in scenarios where the face is obscured by long hair and beard. Furthermore, it only reconstructs and tracks a low-dimensional curved shape model (76 expression coefficients), neglecting fine-scale static and transient surface details. In very short sequences, or when the target remains stationary, we cannot learn specific oral behaviors. In such cases, temporal aliasing can be observed due to the sparse target space of the retrieved mouth samples.
[0006] In the process of facial expression transfer, the core issue is how to establish a mapping between two sets of expressions, that is, to assess the similarity of expressions. Most current methods for similarity estimation determine this by analyzing the spatial position and pose of vertices or triangles in a mesh model, which requires finding the corresponding vertices or triangles in the model. Current similarity estimation methods mainly include those based on feature points in the image, those based on model vertices, and those based on triangles. Methods based on feature points or model vertices basically estimate similarity by comparing the offset direction and size of the points. Methods based on triangles generate similar expressions by finding the corresponding triangles in the model and changing their orientation; however, this method is not very effective when the shapes of the target face and the source face differ significantly.
[0007] Therefore, how to achieve a highly efficient, ultra-realistic, and universally applicable facial expression transfer method is a problem that this invention urgently needs to solve. Summary of the Invention
[0008] In view of this, the present invention provides an efficient expression transfer method using a base expression space transformation, comprising: a preprocessing stage and a facial expression transfer stage; the preprocessing stage includes:
[0009] Obtain the source model O, which is a base expression model that includes expression meshes for various different expressions;
[0010] Facial expression reconstruction is performed on the source model O and the target model T to obtain the expression description parameters of the two; similarity estimation is performed based on the expression description parameters to obtain the expression parameter transformation matrix and similarity matrix;
[0011] The facial expression transfer stage includes: inputting a character image into a base expression model, calculating the expression description parameters of the target expression model based on the expression description parameters in the base expression model and the similarity matrix, and completing the expression transfer.
[0012] Specifically, facial expression reconstruction includes: defining a set of facial feature points and finding the corresponding vertex indices in a 3D model; for any input facial expression photo, detecting the 2D coordinates of the defined feature points, and establishing an optimization equation that minimizes the distance between the projected position of the feature points in the model onto the plane and the position of the 2D point in the image; wherein the optimization equation is expressed as follows:
[0013] E=∑||P(x v )-p v ||2
[0014] Where P is the projection matrix, x v The three-dimensional position of the v-th facial feature point can be obtained from the optimization equation, p. v It is the projected coordinate of the v-th feature point on the two-dimensional plane.
[0015] Specifically, the source model O is a neutral reference model with no facial expression, and the offsets of other facial expressions relative to the neutral reference constitute the superposition effect of different facial expressions. The offsets constitute the facial expression description parameters of different facial expressions.
[0016] Specifically, for the i-th facet, the offset of the feature points relative to the neutral expression under different expressions is δ. i As a description of the current expression, the expression difference between the source model O and the target model T is represented by the following formula:
[0017]
[0018]
[0019] Where E Ci The direction of feature point offset between the source model O and the target model T is constrained; E Di This constrains the distance of feature point offsets between the source model O and the target model T;
[0020] δOi For the expression description of the i-th facet of the source model O, δ Ti δ is the expression description of the i-th facet of model T; Di，v It is the offset of the v-th vertex in the i-th face after the transformation propagation, V i Let be all the vertices of the i-th face.
[0021] Specifically, the optimization equation for estimating the facial expression similarity of each facet is as follows:
[0022] E Si =λ M *E Di *E Ci +λ D *E Di +λ C *E Ci
[0023] Where E Ci E represents the difference in facial expressions between the source model O and the target model T. Di A reference shape is added as a second constraint to obtain accurate facial similarity estimates, λ. M , λ D , λ C This represents the weight value.
[0024] By solving the above equations for all faces, for each expression in the source model O, a similar expression of the target model T can be obtained. All similar expressions can form a set of equivalent hybrid deformation expression parameter transformation matrices. The expression parameters of the target model T corresponding to this equivalent hybrid deformation expression parameter transformation matrix, for the hybrid deformation parameters of each face, form a similarity matrix S. Through this similarity matrix, the expression parameters α of any source model O can be transformed. O Transform into the facial expression parameters α of the corresponding model T similar to the facial expression T =α O *S.
[0025] Specifically, after optimizing the smooth transition between patches, the optimization equation for expression similarity estimation is obtained as follows:
[0026] E E =E R +E O
[0027]
[0028]
[0029] Where E R For the reprojection error optimization equation, L ix represents all feature points on the i-th facet. v，i For the v-th feature point on the i-th facet, defined by the formula of the local hybrid deformation model above, p v，i P represents the two-dimensional coordinates of the feature point in the image. i Let λ be the projection matrix of the i-th facet. R For E R The weight value;
[0030] E O The equation constrains the shape of adjacent patches, where v∈S are the vertices of adjacent overlapping regions, E O The constraint is to keep the distance between the points in the overlapping part between the i-th facet and the adjacent j-th facet as close as possible.
[0031] Beneficial effects:
[0032] 1. First, by selecting the base expression, this invention effectively shortens the time required for subsequent similarity estimation, and has the characteristics of high efficiency, high realism, and universality.
[0033] 2. Secondly, by using a parameter transformation matrix, the dynamic realism of the current model's face is ensured, resulting in high fidelity.
[0034] 3. Furthermore, the use of similarity estimation and parameter transformation matrices makes the entire process highly versatile. Users of different ages, genders, and ethnicities can all achieve highly realistic and real-time expression transfer.
[0035] 4. Finally, this invention optimizes the smooth transition process between face patches, making facial expression transfer more realistic. Attached Figure Description
[0036] Figure 1 This is a flowchart of the efficient expression transfer method using base expression space transformation according to the present invention;
[0037] Figure 2 This is a schematic diagram of face model segmentation and feature points used for similarity estimation in this invention. Detailed Implementation
[0038] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0039] This invention provides an efficient expression transfer method using a base expression space transformation.
[0040] The process of this invention is as follows Figure 1 As shown, the facial expression transfer method from 2D photographs to 3D models proposed in this invention is implemented through two parts. The first part is the preprocessing section, such as... Figure 1The lower half includes: Step 1: Obtain source model O, which is a base expression model, including expression meshes under various different expressions;
[0041] In this invention, the base expression model used as the source model, such as Blendshape, is constructed by acquiring different facial expression models and building them into a series of mesh models with the same topological structure. Weighted interpolation is then used to interpolate and fuse different expressions according to their weights, thereby enabling the face model to transform between multiple different expressions. Generally, each Blendshape represents the transformation of a region of the face. During fusion, each Blendshape does not interfere with the deformation of other Blendshapes, and obtaining the weights of each Blendshape only requires focusing on the changes in the corresponding local region. Currently, the mainstream method for constructing Blendshapes is to refer to the FACS system, modeling 46 expressions as the base expressions for the Blendshape.
[0042]
[0043] Blendshape generally has two forms: one is absolute Blendshape (Absolute BlendshapeModel), which adds all expressions according to certain weights and adjusts the weights to achieve deformation between different expressions, as described in formula (1). Assuming there are K expressions in total besides the neutral expression, where α k b is the weight parameter for the k-th expression. k It is the shape of the k-th base expression, and e is the weighted blended expression shape. However, this method can only handle the fusion of different expressions, and it cannot produce the correct effect when multiple expressions need to exist at the same time; another method is offset (difference, compensation) (Delta Blendshape Model), which selects a neutral mesh model with no expression, and the offset (difference) of the other expressions relative to the neutral constitutes the blendshape. The offset blendshape can intuitively reflect the superposition effect of different expressions, which is described by formula (2), where u = b0 is the selected neutral expression, d k =b k -b0 represents the offset of the k-th emoticon relative to the neutral emoticon. This method is widely used. This invention uses an offset blendshape.
[0044]
[0045] Step 2: Reconstruct facial expressions based on the source model O to obtain facial expression description parameters for various facial expressions under the source model O;
[0046] This invention employs a single-view facial expression reconstruction method to reconstruct facial expressions, which is also a crucial step in achieving facial expression transfer. Figure 2 This diagram illustrates face model segmentation and the feature points used for subsequent similarity estimation. The feature points are symmetrically and uniformly distributed, ensuring each facet has several feature points. The displacement direction of the feature points under different expressions describes the current expression. First, a set of face feature points needs to be defined, symmetrically and uniformly distributed, ensuring each facet has several feature points, and their corresponding vertex indices are found in the 3D model. For any input facial expression image, the 2D coordinates of the defined landmark points are detected, and an optimization equation is established to minimize the distance between the projected position of the feature points in the model onto the plane and the position of the 2D points in the image. The optimization formula (energy equation) is as follows:
[0047] E=∑||P(x v )-p v || 2 (3)
[0048] Where P is the projection matrix, x v The three-dimensional position of the v-th facial feature point can be obtained from formula (3), p v It is the projected coordinate of the v-th feature point on the two-dimensional plane.
[0049] Step 3: Reconstruct facial expressions from the target model T.
[0050] Obtain the facial expression description parameters of the two; perform similarity estimation based on the facial expression description parameters to obtain the facial expression parameter transformation matrix and similarity matrix;
[0051] Facial expression transfer involves transferring the facial expressions of a source person to a target person, enabling the target face to produce the same expression. A common 3D facial expression transfer method is to directly transfer the Blendshape parameters describing the expression. However, this requires that the source and target models use the same method to construct the Blendshape, and that the base expressions used have the same meaning (semantics) and correspond one-to-one. This necessitates creating a set of consistent base expressions during model creation, and the final transfer effect is closely related to the similarity of the base expressions. This method not only requires significant work from artists but also imposes many limitations on the model. Other methods achieve transfer by finding a linear mapping space between the target and source models or by identifying similar expressions in a sequence. These methods all require finding corresponding expressions between the source and target, i.e., similarity estimation. Current similarity estimation methods mainly include those based on image feature points, model vertices, and triangles. Methods based on feature points or model vertices primarily estimate similarity by comparing the offset direction and size of points. The method based on triangles generates similar expressions by finding the triangles corresponding to the model and changing the orientation of the corresponding triangles. However, this method is not very effective when the shapes of the target face and the source face are quite different.
[0052] In this step, we assume that for the i-th facet, the offset of the feature point relative to the neutral expression is δ under different expressions. i As a description of the current expression, the expression difference between the source model O and the target model T is represented by formula (4).
[0053]
[0054] Where δ Oi For the expression description of the i-th facet of model O, δ Ti Let E be the expression description of the i-th facet of model T. Ci Only the direction of the feature point offset is constrained, but the distance of the feature point offset is not constrained. This alone is not enough to obtain an accurate facial similarity estimate. Therefore, this invention adds a reference shape as a second constraint, as shown in formula (5).
[0055]
[0056] This invention obtains a corresponding expression for each expression in model O, and after aligning the expressions with neutral expressions using Protodyakonov analysis, calculates the offset of the model vertices relative to the neutral expressions, δ. Di，v It is the offset of the v-th vertex in the i-th face after the deformation transfer, where Vi is all the vertices of the i-th face.
[0057] The final optimization equation for estimating the facial similarity of each facet is Equation (6).
[0058] E Si =λ M *E Di *E Ci +λ D *E Di +λ C *E Ci (6)
[0059] By solving the above equations for all faces, for each expression in model O, a similar expression for model T can be obtained. All similar expressions can form a set of equivalent hybrid deformation expression parameter transformation matrices. The expression parameters of model T corresponding to this equivalent hybrid deformation, for the hybrid deformation parameters of each face, form a similarity matrix S. Through this similarity matrix S, the expression parameters α of any model O can be transformed. O Transform into the facial expression parameters α of the corresponding model T similar to the facial expression T =α O *S.
[0060] The optimization equation for single-view facial expression reconstruction also needs to be adjusted for the base expression. It is necessary to consider not only minimizing the reprojection distance of feature points in each patch, but also the smooth transition between patches. The adjusted optimization equation consists of two parts, as shown in Equation (7).
[0061] E E =E R +E O (7)
[0062]
[0063] E O =λ O ∑ v∈S ∑ (i，j)∈I，i＞j ||x v，i -x v，j ||2 (9)
[0064] Where E R For the reprojection error optimization equation (energy), L i x represents all feature points on the i-th facet. v，i For the v-th feature point on the i-th facet, defined by the formula of the local hybrid deformation model above, p v，i P represents the two-dimensional coordinates of the feature point in the image. i Let λ be the projection matrix of the i-th facet. R For ER The weight value.
[0065] E O The equation (energy) constrains the shape of adjacent faces, where v∈S are the vertices of adjacent overlapping regions, E O The constraint is to keep the distance between the points in the overlapping part between the i-th facet and the adjacent j-th facet as close as possible.
[0066] Another part is the process of facial expression transfer, such as... Figure 1 The upper part includes: Step 4: Using the input facial photo of user O, the local hybrid deformation model of user O, and the hybrid deformation model of equivalent T, a single-view facial expression reconstruction is performed. A set of hybrid deformation parameters are obtained, and then facial expression transfer is performed together with the formed similarity matrix.
[0067] Specifically, in the process of transferring facial expressions from a two-dimensional image to a three-dimensional model, the image of the character corresponding to model O is input, and feature points are detected. According to the method of formula (8), E is calculated using the feature points of model O and the detected feature points. R The equivalent hybrid deformation of model T is divided in the same way, and E is substituted into formula (9). O Apply smoothing constraints by solving E E The obtained expression parameter α O And transform S into α through the similarity matrix. T By merging the separated facets, a similar expression to the transferred model T can be obtained.
[0068] This invention proposes a facial expression transfer method using a base expression space transformation. The facial region is segmented based on the muscle distribution of the face to select base expressions. Then, a method for estimating facial expression similarity based on vertex offset directions is used to estimate the similarity of base expressions between different models, obtaining a set of equivalent hybrid deformation and expression parameter transformation matrices. The transfer process of this invention is based on facial expression reconstruction. Facial feature points are detected, and reprojection errors are calculated to achieve facial expression reconstruction. The segmented facial model is constrained to have a complete facial expression through equivalent hybrid deformation. Finally, the hybrid deformation parameters of the target model are obtained through the parameter transformation matrix, and the target expression for transfer is generated.
[0069] Compared to traditional facial expression transfer methods, this invention offers three main advantages: high efficiency, high realism, and universality. First, by selecting a base expression, it effectively shortens the time required for subsequent similarity estimation. Second, through a parameter transformation matrix, it ensures the dynamic realism of the model's face, achieving high fidelity. Finally, the use of similarity estimation and the parameter transformation matrix makes the entire process highly versatile. Users of different ages, genders, and ethnicities can all achieve real-time, highly realistic expression transfer.
[0070] In summary, this method effectively reduces the time and economic costs of transferring facial expressions from 2D images to 3D face models, improves the realism of virtual human facial expressions and movements, and has the potential for wide application.
[0071] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0072] It will be apparent to those skilled in the art that the embodiments of the present invention are not limited to the details of the exemplary embodiments described above, and that the embodiments of the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the embodiments of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the embodiments of the present invention is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be encompassed within the embodiments of the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units, modules, or devices recited in the system, apparatus, or terminal claims may also be implemented by the same unit, module, or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.
[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the embodiments of the present invention should not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An efficient expression transfer method employing a base expression space transformation, characterized in that, The method includes: a preprocessing stage and a facial expression transfer stage; the preprocessing stage includes: Obtain the source model O, which is a base expression model that includes expression meshes for various different expressions; Facial expression reconstruction is performed on the source model O and the target model T to obtain the expression description parameters of the two; similarity estimation is performed based on the expression description parameters to obtain the expression parameter transformation matrix and similarity matrix; The source model O is a neutral reference with an expressionless mesh model selected as the neutral reference. The offsets of other expressions relative to the neutral reference constitute the superposition effect of different expressions. The offsets constitute the expression description parameters of different facial expressions. The facial expression transfer stage includes: inputting a character image into a base expression model, calculating the expression description parameters of the target expression model based on the expression description parameters in the base expression model and the similarity matrix, and completing the expression transfer; The efficient expression transfer method employing base expression space transformation is characterized in that, for the i-th facet, the offset of the feature points relative to the neutral expression under different expressions is δ. i As a description of the current expression, the expression difference between the source model O and the target model T is represented by the following formula: ; where E Ci The direction of feature point offset between the source model O and the target model T is constrained; E Di This constrains the distance between the feature point offsets of the source model O and the target model T; δ Oi For the expression description of the i-th facet of the source model O, δ Ti δ is the expression description of the i-th facet of model T; Di，v It is the offset of the v-th vertex in the i-th face after the transformation propagation, V i Let be all the vertices of the i-th face.
2. The efficient expression transfer method using base expression space transformation as described in claim 1, characterized in that, Facial expression reconstruction includes: defining a set of facial feature points and finding the corresponding vertex indices in a 3D model; for any input facial expression photo, detecting the 2D coordinates of the defined feature points, and establishing an optimization equation that minimizes the distance between the projected positions of the feature points in the model onto the plane and the positions of the 2D points in the image; wherein the optimization equation is expressed as follows: Where P is the projection matrix, x v The three-dimensional position of the v-th facial feature point can be obtained from the optimization equation, p. v It is the projected coordinate of the v-th feature point on the two-dimensional plane.
3. The efficient expression transfer method using base expression space transformation as described in claim 1, characterized in that, The optimization equation for estimating the facial similarity of each facet is as follows: Where E Ci E represents the difference in facial expressions between the source model O and the target model T. Di A reference shape is added as a second constraint to obtain accurate facial similarity estimates, λ. M , λ D , λ C Let α be the weight value. By solving the above equation for all faces, for each expression in the source model O, a similar expression of the target model T can be obtained. All similar expressions can form a set of equivalent hybrid deformation expression parameter transformation matrices. The expression parameters of the target model T corresponding to this equivalent hybrid deformation expression parameter transformation matrix, for the hybrid deformation parameters of each face, form a similarity matrix S. Through this similarity matrix, the expression parameters α of any source model O can be transformed. O Transform into the facial expression parameters of the corresponding model T similar to the expression .
4. The efficient expression transfer method using base expression space transformation as described in claim 3, characterized in that, After optimizing the smooth transition between patches, the optimization equation for expression similarity estimation is obtained as follows: Where E R For the reprojection error optimization equation, L i x represents all feature points on the i-th facet. v，i For the v-th feature point on the i-th facet, defined by the formula of the local hybrid deformation model above, p v，i P represents the two-dimensional coordinates of the feature point in the image. i Let λ be the projection matrix of the i-th facet. R for E R The weight value; E O The equation constrains the shape of adjacent patches, where v∈S are the vertices of adjacent overlapping regions, E O The constraint is to keep the distance between the points in the overlapping part between the i-th facet and the adjacent j-th facet as close as possible.