A facial expression binding method and device based on marker point optimization and a medium

Abstract

This invention discloses a method, apparatus, and medium for facial expression binding based on marker point optimization. The method includes: annotating facial key points, eye contour points, and expression-independent regions of the input facial model in an input facial model and an expression template; deforming the input facial model to a neutral expression of the expression template based on the annotation results; establishing a correspondence between the mesh patches of the input facial model and the expression template after deformation; and sequentially performing deformation transfer on the input facial model based on the deformation of all expressions relative to the neutral expression in the expression template to generate a blended shape corresponding to all expressions, thus completing the facial expression binding task. This invention uses eye contour points for annotation and performs eye closure optimization based on the annotation of eye contour points during subsequent deformation transfer, effectively optimizing the performance of the expression binding method for closed-eye expressions. This invention can be widely applied in the field of facial image data processing.

Description

Technical Field

[0001] This invention relates to the field of face model data processing, and in particular to a method, apparatus and medium for binding facial expressions based on marker point optimization. Background Technology

[0002] In recent years, the virtual digital human industry has developed rapidly, with virtual customer service, virtual assistants, and virtual anchors being representative applications. Facial expression binding is one of the key technologies driving virtual digital humans. It aims to generate a series of blendshapes stored in mesh form, corresponding to specific expression templates, for any input 3D facial model. The blendshapes drive the model by controlling the vertex displacement of the 3D facial model. A major mainstream method for solving facial expression binding tasks is based on deformation transfer. However, existing methods have two major drawbacks: first, they perform poorly with closed-eye expressions, failing to achieve complete eye closure; second, some expressions suffer from vertex displacement issues in expression-irrelevant regions. Both of these problems affect the driving effect of the 3D model. Summary of the Invention

[0003] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a method, device and medium for binding facial expressions based on marker point optimization.

[0004] The technical solution adopted in this invention is:

[0005] A face expression binding method based on marker optimization includes the following steps:

[0006] Obtain a 3D input face model and expression template, and annotate the facial key points, eye contour points, and expression-independent areas of the input face model in the input face model and expression template;

[0007] Based on the annotation results, the input face model is deformed into a neutral expression of the expression template;

[0008] After the input face model is deformed, a correspondence is established between the input face model and the mesh face of the expression template.

[0009] By deforming all expressions in the expression template relative to the neutral expression, deformation transfer is performed on the input face model in sequence to generate the blended shapes corresponding to all expressions, thus completing the face expression binding task.

[0010] Furthermore, the annotation of facial key points, eye contour points, and expression-independent regions of the input face model and expression template includes:

[0011] Label multiple key points containing semantic information on the input face model and expression template, and record the key point index;

[0012] Mark all the vertex sets representing the upper and lower eye sockets on the input face model and expression template, and record the vertex indices; the marked range includes the left and right eyes, and the two polylines formed by connecting the vertices of the upper and lower eye sockets need to form a closed loop, that is, the upper and lower eye contours share the vertices of the left and right corners of the eyes;

[0013] For the input face model, mark the regions that are not related to facial expressions (regions that do not participate in facial expression movements are determined by prior knowledge), and save the vertex set of these regions.

[0014] Furthermore, the step of deforming the input face model to a neutral expression from the expression template based on the annotation results includes:

[0015] Calculate the bounding box dimensions of the input face model V and the neutral expression B0 of the expression template on the x, y, and z axes in the three-dimensional coordinate system, respectively.

[0016] Scale the input face model V to the same size as the neutral expression B0 using the bounding box dimensions;

[0017] Based on the key points at the tip of the nose, perform a spatial translation operation on the input face model V, so that the input face model V is translated to align with the key points at the tip of the nose of both the input face model V and the neutral expression B0.

[0018] Using the Non-rigid Iterative Closest Point (NICP) method, combined with labeled facial key points, the input face model is V-shaped to a neutral expression B0.

[0019] Furthermore, establishing the correspondence between the input face model and the mesh patches of the expression template includes:

[0020] Construct the correspondence between the mesh patches of the input face model V and the neutral expression B0 of the expression template;

[0021] Iterate through all the mesh faces of the input face model V. For each mesh face on the input face model V, find the mesh face with the closest centroid distance to that mesh face on the neutral expression B0 and establish a correspondence.

[0022] After the traversal is complete, the corresponding relationships are as follows:

[0023] C = {(s1,t1),(s2,t2),…(s M ,t M )}

[0024] Among them, si and t i represents the grid patch indices of the input face model V and expression template B0, respectively, and M is the number of correspondences.

[0025] Furthermore, the deformation transfer of the input face model is performed sequentially on all expressions in the expression template relative to the neutral expression to generate the blended shapes corresponding to all expressions, including:

[0026] The deformation of the input face model is defined as the set of affine transformations of all triangular mesh patches;

[0027] For a triangular mesh, let the three vertices of the triangle before and after deformation be v1, v2, v3, and v4, respectively. Additionally, a fourth vertex v4 perpendicular to the triangle is defined. According to the definition of affine transformation, the affine transformation Q of this triangle can be expressed in terms of vertices, that is:

[0028]

[0029] V = [v2-v1,v3-v1,v4-v1]

[0030]

[0031] Define the affine transformations S and T for the triangular mesh patches of the expression template and the input face model, respectively. Based on the constructed list of mesh patch correspondences C, constrain the affine transformations of corresponding triangle pairs to be equal, as shown in the following expressions:

[0032]

[0033] in, and In any correspondence (s) j ,t j Affine transformation of triangle facets between the expression template and the input model.

[0034] Furthermore, this step of performing deformation transfer on the input face model also includes an eye closure optimization step:

[0035] For the input face model, the point V of the upper eye socket i Calculate the V-shape at the upper eye socket in a neutral expression. i Points with the same position ratio

[0036] In neutral expressions, focus on the lower eye socket and... The closest point in Euclidean distance

[0037] Calculate points for neutral expressions. The proportion p in the lower eye socket position;

[0038] Using the positional ratio p, calculate the points at the same positional ratio above the lower eye socket of the input face model.

[0039] Point As point V i The closest point, the optimization point and point V i The two are similar, and the expressions are as follows:

[0040]

[0041] Here, v_list is the set of vertices of the eye sockets on the input face model.

[0042] Furthermore, this step of performing deformation transfer on the input face model also includes a closed-eye smoothing optimization step:

[0043] For the input face model, a point V is located at the lower eye socket. i Calculate a point on the lower eye socket with the same proportion as the given point, representing a neutral expression.

[0044] On the template closed-eyes expression model, obtain points Position after deformation

[0045] Assuming the input face model has its eyes closed, click V. i The position after deformation is make

[0046] Calculate the dimensions of the eye sockets in the input face model and expression template: S t,x ,S t,y ,S t,z With S s,x ,S s,y ,S s,z Calculate the size ratio factor based on the dimensions;

[0047] The formula for smoothing with eyes closed is designed and expressed as follows:

[0048]

[0049]

[0050] Where v_list is the set of vertices of the lower eye socket of the input face model, and r is the size scale coefficient.

[0051] Furthermore, this step of performing deformation transfer on the input face model also includes a static region optimization step:

[0052] For areas unrelated to facial expressions, optimization is implemented to prevent displacement during deformation, thus resolving the issue of unreasonable jitter in some facial expressions. The expression is as follows:

[0053]

[0054] Where mask is the set of vertices of the expression-independent region, V i and These are the model vertices before and after deformation.

[0055] Another technical solution adopted in this invention is:

[0056] A marker-optimized facial expression binding device includes:

[0057] At least one processor;

[0058] At least one memory for storing at least one program;

[0059] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0060] Another technical solution adopted in this invention is:

[0061] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0062] The beneficial effects of this invention are: by annotating the eye contour points, and then optimizing the eye closure based on the annotated eye contour points during subsequent deformation transfer, this invention effectively optimizes the performance of the expression binding method when the eyes are closed. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0064] Figure 1 This is a flowchart illustrating the steps of a face expression binding method based on marker point optimization in an embodiment of the present invention;

[0065] Figure 2This is a schematic diagram of the annotation of 51 key points of the face in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the upper and lower eye sockets in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the eye closure optimization item in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the smoothing optimization with eyes closed in an embodiment of the present invention;

[0069] Figure 6 This is a schematic diagram of the annotation of expression-independent areas in an embodiment of the present invention;

[0070] Figure 7 This is a schematic diagram comparing the binding effect of the present invention and existing methods on closed-eye facial expressions;

[0071] Figure 8 This is a schematic diagram comparing the shaking of the present invention and existing methods in an area unrelated to the "raised eyebrow" expression; where blue represents the stationary area and red represents the area where displacement occurs. Detailed Implementation

[0072] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0073] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0074] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0075] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0076] Existing facial expression binding methods based on deformation transfer have the following problems: (1) incomplete closure when eyes are closed; (2) displacement or jitter in some facial expressions in areas unrelated to facial expressions. To address problem (1), this invention proposes an eye closure optimization term to solve the eye closure problem, and also proposes a smoothing term to make the facial model smoother and more natural when eyes are closed. To address problem (2), this invention proposes a static term for expression-independent regions, using additionally labeled unrelated regions to prevent displacement and jitter in model regions unrelated to facial expressions. Combining these two optimization terms improves the effect of facial expression binding, enabling more accurate and natural facial expressions in the process of driving virtual digital humans using expression blending shapes.

[0077] like Figure 1 As shown, this embodiment provides a face expression binding method based on marker point optimization, including the following steps:

[0078] S1. Obtain the 3D input face model and expression template, and annotate the facial key points, eye contour points, and expression-irrelevant areas of the input face model in the input face model and expression template;

[0079] S2. Based on the annotation results, deform the input face model to a neutral expression from the expression template;

[0080] S3. After the input face model is deformed, establish the correspondence between the input face model and the mesh patches of the expression template;

[0081] S4. Using the deformation of all expressions in the expression template relative to the neutral expression, perform deformation transfer on the input face model in sequence to generate the blended shapes corresponding to all expressions, thus completing the face expression binding task.

[0082] The input in this embodiment includes any 3D face model and expression template. The output is a set of mixed shapes of the input face model corresponding to all the expressions in the expression template. The input 3D face model is defined as V. The expression template is defined as B, containing one neutral expression B0 and n other expressions {B0, B ... i The goal is to generate a blending shape {V, i = 1, ..., n} for the input model V, corresponding to the n remaining facial expressions. i ,i=1,…,n}.

[0083] The above method will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0084] S1: Data Labeling

[0085] S1-1: Annotate 51 key points containing semantic information of the face for the input model (i.e., the input face model) and expression template, and record the key point indices. The location information of the key points is as follows: Figure 2 As shown.

[0086] S1-2: Mark the set of all vertices representing the upper and lower eye sockets on the input model and expression template, and record the vertex indices. The marked area includes the left and right eyes, and the two polylines connecting the vertices of the upper and lower eye sockets need to form a closed loop, that is, the upper and lower eye contours share the vertices of the left and right corners of the eyes (i.e.,...). Figure 3 (The intersection of the red and blue broken lines shown).

[0087] S1-3: Mark the regions of the input model that are not related to facial expressions (regions that do not participate in facial expression movements are determined by prior knowledge) and save the set of vertices.

[0088] S2: Input model deformation to neutral expression template

[0089] S2-1: Calculate the bounding box dimensions of the input model V and the neutral expression template B0 along the x, y, and z axes in the 3D coordinate system. Then, scale the input model V to the same size as B0 using the bounding box dimensions. Finally, perform a spatial translation operation on V using the keypoint at the tip of the nose to align the keypoints of the two models.

[0090] S2-2: Using the Non-rigid Iterative ClosestPoint (NICP) deformation method, combined with the facial key points annotated in step S1, the input model is deformed to B0.

[0091] S3: Constructing the correspondence between facets

[0092] To achieve deformation transfer, it is necessary to establish a correspondence between the mesh faces of the input model V and the expression template B0. Building upon step S2, all faces of the input model are traversed. For each face, the face on B0 with the closest centroid distance to that face is found, and a correspondence is established. After the traversal is complete, a list of correspondences is obtained:

[0093] C = {(s1,t1),(s2,t2),…(s M ,t M )}

[0094] Where s i and t i represents the face indices of the input model V and the expression template B0, respectively, and M is the number of correspondences.

[0095] S4: Use deformation transfer to complete facial expression binding

[0096] For any one of the emojis B in the emoji template i The purpose of this step, where i = 1, ..., n, is to transfer the deformation of the expression relative to the template neutral expression B0 to the input model V, so that V undergoes the same deformation, resulting in an expression similar to B0. i Mixed shapes of the same expression V i .

[0097] S4-1: Affine Transformation Constraints

[0098] The deformation of a face model, i.e., a mesh, is defined as the set of affine transformations of all triangular faces within the mesh. For a specific triangular face, the three vertices of the triangle before and after deformation are defined as v1, v2, v3, and v4, respectively. Additionally, a fourth vertex v4 perpendicular to the triangle is defined. Therefore, according to the definition of affine transformation, the affine transformation Q of this triangle can be expressed in terms of its vertices, that is:

[0099]

[0100] V = [v2-v1,v3-v1,v4-v1]

[0101]

[0102] Based on this, we define the affine transformations of the triangular faces of the expression template and the input model as S and T, respectively. Combining the face correspondence list C constructed in step S3, we constrain the affine transformations of corresponding triangle pairs to be equal, that is:

[0103]

[0104] This optimization is consistent with existing methods.

[0105] S4-2: Eye Closure Optimization Item

[0106] like Figure 4 As shown, using the upper and lower eye socket vertices marked in step S1 (the red and blue lines in the figure are the lines connecting the upper and lower eye socket vertices respectively), this embodiment designs an eye closure optimization item. For each vertex of the upper eye socket, the closest point is found on the lower eye socket line, and the proximity between the two is optimized to achieve the eye-closing effect. Specifically:

[0107] 1) For a certain labeled vertex V of the eye socket in the input model i Calculate a point on the upper eye socket of the neutral expression template that has the same proportion as the given point. (Positional proportion refers to the ratio of the length of the line segment from the leftmost point of the orbital fold to that point to the total length of the orbital fold.)

[0108] 2) On the template closed-eye expression model, the calculation of the lower eye socket and... The closest point in Euclidean distance

[0109] 3) Calculate points on the template-neutral expression model. The proportion of the lower eye socket (p)

[0110] 4) Using the positional ratio p, calculate the points at the same position ratio above the lower eye socket of the input face model.

[0111] Then point Point V i The closest point can be found by optimizing the two points to be close, that is:

[0112]

[0113] Here, v_list is the set of vertices of the eye sockets on the input face model.

[0114] S4-3: Smoothing optimization item when eyes are closed

[0115] Step S4-2 achieved the closed-eye effect of the face model, but the problem of unevenness at the closed area of ​​the upper and lower eyes still exists. This embodiment proposes the following eye-closing smoothing method. Figure 5 As shown, the goal is to make the lower eye socket point V of the input model... i The displacement vectors before and after deformation are close to the displacement vectors at points on the template with the same position ratio, as shown below:

[0116] 1) For a V-shape under the lower eyelid i Calculate a point on the neutral expression template that has the same proportions as that point.

[0117] 2) On the template of the closed-eye expression model, find Position after deformation

[0118] 3) Assume that after the input model closes its eyes, point V... i The position after deformation is Then let

[0119] 4) To eliminate the influence of different eye sizes in different face models, the translation vector of the expression template needs to be multiplied by a size scaling factor. That is, the dimensions of the eye socket in the input model and the expression template are calculated separately for the three dimensions: S t,x ,S t,y ,St,z With S s,x ,S s,y ,S s,z The size ratio coefficient is then obtained from the above dimensions.

[0120] Based on the above steps, the following optimization items can be listed:

[0121]

[0122]

[0123] Where v_list is the set of vertices of the lower eye socket of the input face model, and r is the size scale coefficient.

[0124] S4-4: Static items in expression-independent regions

[0125] Using the static area marked in step S1, i.e. Figure 6 The optimization process involves removing non-expression-related areas to prevent displacement during deformation, thus resolving the issue of unreasonable jitter in some expressions. The optimizations are as follows:

[0126]

[0127] Where mask is the set of vertices in the expression-independent region, V i and These are the model vertices before and after deformation.

[0128] S4-5: Solving for the optimization term

[0129] In summary, the above optimization item E A E C E M E S In this embodiment, an optimization item E is added during the binding process of the closed-eye expression. C E M And add optimization item E to all emoticons. S After differentiating the above optimization terms, the optimization problem can be equivalent to solving a system of linear equations. Solving the system of equations yields the deformed face model, which is the blended shape of the expression corresponding to the expression template. By sequentially performing the above equation-solving operation on all expressions in the expression template, the input face model V, corresponding to the blended shapes of all expressions in the expression template, is obtained, thus completing the face expression binding task.

[0130] See Figure 7 and Figure 8 , Figure 7 and Figure 8The following experiments compare the results of two methods: one using the method of this embodiment, and the other using existing methods, on shaking in a closed-eye expression and in an expression-independent region. The results show that the method of this invention is superior to existing methods on a closed-eye expression, achieving better closure of the upper and lower eyes while reducing the displacement of vertices in expression-independent regions.

[0131] This embodiment also provides a face expression binding device based on marker point optimization, including:

[0132] At least one processor;

[0133] At least one memory for storing at least one program;

[0134] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0135] This embodiment of the facial expression binding device based on marker point optimization can execute the facial expression binding method based on marker point optimization provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0136] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0137] This embodiment also provides a storage medium storing instructions or programs that can execute the marker-based optimized facial expression binding method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0138] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0139] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0140] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0142] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0143] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0144] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0145] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0146] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for optimizing facial expression binding based on marker points, characterized in that, Includes the following steps: Obtain a 3D input face model and expression template, and annotate the facial key points, eye contour points, and expression-independent areas of the input face model in the input face model and expression template; Based on the annotation results, the input face model is deformed into a neutral expression of the expression template; After the input face model is deformed, the correspondence between the input face model and the mesh patches of the expression template is established; By deforming all expressions in the expression template relative to the neutral expression, deformation transfer is performed on the input face model in sequence to generate the blended shapes corresponding to all expressions, thus completing the face expression binding task; The step of performing deformation transfer on the input face model also includes an eye closure optimization step: For the upper eye socket point of the input face model Calculate the distance from the point on the upper eye socket of a neutral expression. Points with the same position ratio ; On a neutral expression, the lower eye corner is obtained as the point with the smallest Euclidean distance from the point On a neutral expression, calculate the point At the position of the lower eye orbit ; Utilizing position proportion , calculating the point of the same position proportion on the upper eyelid of the input face model ; The point As the nearest point to the optimization point and the point Both approaches are as follows: wherein, is a set of vertices of the eye socket on the input face model; The step of performing deformation transfer on the input face model also includes a closed-eye smoothing optimization step: For the lower eye socket of the input face model Calculate a point on the lower eye socket with the same proportion as the given point, representing a neutral expression. ; On the template closed-eye expression model, acquire the point The position after morphing ; Let the input face model close eyes, point The position after deformation is , let ; Calculating the size of the three dimensions of the eye socket of the input face model and the expression template: With , calculating the size ratio coefficient according to the size of the dimensions; The formula for smoothing with eyes closed is designed and expressed as follows: wherein, is a set of vertices of an input face model, is a size ratio coefficient.

2. The method of claim 1, wherein, The annotation of facial key points, eye contour points, and expression-independent regions of the input face model and expression template includes: Annotate multiple key points containing semantic information for the input face model and expression template, and record the key point index; Mark all the vertex sets representing the upper and lower eye sockets on the input face model and expression template, and record the vertex indices; the marked range includes the left and right eyes, and the two polylines formed by connecting the vertices of the upper and lower eye sockets need to form a closed loop; Mark regions unrelated to facial expressions for the input face model and save the vertex set of these regions.

3. The method of claim 1, wherein, The step of deforming the input face model into a neutral expression from the expression template based on the annotation results includes: Calculate the input face model respectively and the neutral expression of the expression template The bounding box size of x, y, z axes on the three-dimensional coordinate system; Utilizing bounding box dimensions to scale input face models to and neutral expressions Same size dimensions;​ According to the key points at the tip of the nose of the face, the input face model is translated in space so that the key points at the tip of the nose of the face of both align; the input face model is translated to a neutral expression . Using a non-rigid iterative closest point morphing method, in combination with annotated facial landmarks, the input face model is morphed to a neutral expression .

4. The method of claim 1, wherein, The process of establishing the correspondence between the input face model and the mesh patches of the expression template includes: Constructing an input face model and a neutral expression of the expression template correspondence between mesh faces of both Traverse all mesh patches of the input face model , for each mesh patch on the input face model , find the mesh patch closest to the centroid of the mesh patch in the neutral expression , and establish a correspondence relationship; After the traversal is complete, the corresponding relationships are as follows: wherein, and respectively represent the mesh patch indices of the input face model and the expression template , is the number of correspondences.

5. The method of claim 1, wherein, The deformation transfer of all expressions in the expression template relative to the neutral expression is performed sequentially on the input face model to generate the blended shapes corresponding to all expressions, including: The deformation of the input face model is defined as the set of affine transformations of all triangular mesh patches; For a triangle mesh patch, define the three vertices of the triangle before morphing and the three vertices of the triangle after morphing as and respectively. Additionally, define a fourth vertex and perpendicular to the triangle. According to the definition of affine transformation, the affine transformation of the triangle is expressed in the form of vertices, i.e. The affine transformation of defining the expression template and the triangular mesh patches of the input face model is respectively defined as and According to the constructed mesh patch correspondence list , the affine transformations of the triangular pairs with the correspondence are equal, and the expression is as follows: wherein, with denotes a triangular patch affine transformation of the expression template to the input model in any corresponding relationship in the input model.

6. The method of claim 2, wherein, The step of performing deformation transfer on the input face model also includes a static region optimization step: For areas unrelated to facial expressions, optimization is implemented to prevent displacement during deformation, thus resolving the issue of unreasonable jitter in some facial expressions. The expression is as follows: wherein, is a set of expression-independent region vertices, and are model vertices before and after morphing, respectively.

7. A device for optimizing facial expression binding based on marker points, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1-6.

8. A computer readable storage medium having stored therein a program which is executable by a processor, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-6.

Images