Expression base generation method and device, equipment and medium

Through alignment and replacement techniques, the generated neutral facial mesh and expression base solve the problem of network topology mismatch, improves authenticity and nature, and ensures the retention of facial features and the fidelity of expression base.

CN119091083APending Publication Date: 2024-12-06SHINING 3D TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411145380.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When generating neutral facial mesh and expression bases in the prior art, the mismatch of network topology leads to loss of facial features, resulting in the generated neutral facial mesh and expression bases not real and natural enough.

Method used

By obtaining the preset standard mesh area in the standard expression base and the standard face mesh aligned with the initial neutral mesh of the user's face, forming a deformation mesh area, replacing the complex area in the initial neutral mesh, and generating target neutral mesh Mesh and migrate the expression state to the target neutral mesh to obtain the target expression base.

Benefits of technology

Improve the authenticity and nature of the neutral facial mesh and expression base, ensure that the target neutral mesh is more realistic and the outline is smoother, and retains the original features of the user's face, improving the fidelity and nature of the expression base Spend.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091083A_ABST
    Figure CN119091083A_ABST
Patent Text Reader

Abstract

The invention provides an expression base generation method and device, equipment and a medium. The method comprises the following steps: acquiring a standard expression base, a standard grid region preset in a standard facial grid and an initial neutral grid of a user face; aligning the standard grid region to the initial neutral grid to obtain a deformation grid region corresponding to the standard grid region; replacing a complex region corresponding to the deformation grid region in the initial neutral grid with the deformation grid region to obtain a target neutral grid of the user face; and transferring the expression state of the standard expression base to the target neutral grid to obtain a target expression base of the face of the user. By using the method, the fidelity and the naturalness of the expression base can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of three-dimensional scanning and relates to an expression base generation method, device, equipment and medium. Background Art

[0002] With the advancement of 3D expression-driven technology, its application in various fields such as virtual reality, animation creation, and film production is gradually expanding. Especially in the field of animation-driven technology, in order to vividly show the expression changes of animated characters, it is usually necessary to build realistic neutral facial meshes and expression bases.

[0003] However, there are often some problems with the neutral facial mesh obtained by scanning the user, such as the adhesion of the upper and lower lip structures and the missing internal structure of the lips. In order to solve these problems, the following two methods are mainly used in the related art to generate expression bases. The first method is to not use the neutral facial mesh obtained by scanning, but directly fit a neutral facial mesh similar to the user's face through the standard expression library, and generate an expression base based on this. The second method is to select a standard face mesh from the standard expression library, transform it to the scanned neutral face mesh, obtain the user's neutral face mesh, and directly use the expression base corresponding to the standard face mesh in the standard expression library as the user's expression base.

[0004] However, the network topology of the neutral facial mesh generated by these two methods is different from the neutral facial mesh obtained by the original scan, and because the topology of the standard human face mesh is completely adopted, the user's facial features will be lost, causing the generated neutral facial mesh and expression base to be not realistic and natural enough. Summary of the invention

[0005] In view of the above, it is necessary to provide an expression base generation method, device, equipment and medium that can solve the technical problem of poor quality of constructed neutral facial mesh and expression base due to loss of network topology and facial features.

[0006] On the one hand, the present application provides a method for generating an expression base, the method comprising: obtaining a standard expression base, a standard grid area preset in a standard facial grid, and an initial neutral grid of a user's face, aligning the standard grid area to the initial neutral grid, obtaining a deformed grid area corresponding to the standard grid area, using the deformed grid area to replace the complex area corresponding to the deformed grid area in the initial neutral grid, obtaining a target neutral grid of the user's face, migrating the expression state of the standard expression base to the target neutral grid, and obtaining a target expression base of the user's face.

[0007] In some embodiments of the present application, the standard grid area corresponds to a plurality of first coordinate points, the initial neutral grid corresponds to a plurality of second coordinate points, and the step of aligning the standard grid area to the initial neutral grid to obtain a deformed grid area corresponding to the standard grid area includes: determining a deformation field corresponding to each first coordinate point, the deformation field including a plurality of deformation control points corresponding to each first coordinate point, and determining a plurality of groups of point pairs by densely aligning the plurality of first coordinate points and the plurality of second coordinate points, wherein each group of point pairs includes first coordinate points and second coordinate points corresponding to each other, and according to each group of The initial coordinates of the first coordinate point in the point pair, the initial coordinates of the second coordinate point in each group of point pairs and the multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determine the target deformation parameters of each deformation control point, determine the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of the multiple deformation control points corresponding to each first coordinate point, and determine the deformation mesh area based on the multiple first coordinate points and the target coordinates of each first coordinate point.

[0008] In some embodiments of the present application, the method for determining the target deformation parameters of each deformation control point includes: executing a solution process for the initial deformation parameters of each deformation control point, including: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function, repeatedly executing the solution process, updating the initial deformation parameters of each deformation control point, until the number of updates to the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stopping the solution process, and determining the updated initial deformation parameters as the target deformation parameters of each deformation control point.

[0009] In some embodiments of the present application, the method for constructing the constraint function includes: constructing an objective function of target deformation parameters of multiple deformation control points corresponding to the first coordinate point in each group of point pairs according to the initial coordinates of the first coordinate point in each group of point pairs, wherein the objective function corresponds to the target coordinates of the first coordinate point in each group of point pairs, constructing a data item constraint according to the initial coordinates of the second coordinate point in each group of point pairs and the objective function, constructing a smoothing item constraint on the target deformation parameters of each deformation control point and the target deformation parameters of each deformation control point adjacent to each deformation control point, and determining the constraint function according to the data item constraint and the smoothing item constraint.

[0010] In some embodiments of the present application, the target deformation parameters of each deformation control point include a linear transformation matrix and a translation vector, and determining the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of the multiple deformation control points corresponding to each first coordinate point includes: determining the candidate coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the linear transformation matrix and translation vector of each deformation control point corresponding to each first coordinate point, and determining the target coordinates of each first coordinate point based on the multiple candidate coordinates.

[0011] In some embodiments of the present application, the use of the deformed mesh area to replace the complex area corresponding to the deformed mesh area in the initial neutral mesh to obtain the target neutral mesh of the user's face includes: determining the regional position of the complex area in the initial neutral mesh according to the target coordinates of the multiple first coordinate points, deleting the complex area from the initial neutral mesh according to the regional position, and splicing the deformed mesh area to the regional position in the initial neutral mesh to obtain the target neutral mesh.

[0012] In some embodiments of the present application, migrating the expression state of the standard expression base to the target neutral grid to obtain the target expression base of the user's face includes: matching the patches in the target neutral grid with the standard patches in the standard expression base, determining the expression deformation parameters of each patch in the target neutral grid, and transforming the target neutral grid according to the expression deformation parameters of each patch to obtain the target expression base.

[0013] In some embodiments of the present application, the method also includes: obtaining a dental model of the user, the dental model having multiple occlusion states; the target neutral grid of the user's face is aligned with the dental model, and the target expression base has a corresponding relationship with at least one occlusion state of the dental model.

[0014] On the other hand, the present application provides an expression base generation device, which includes: an acquisition module for acquiring a standard expression base, a standard grid area preset in a standard facial grid, and an initial neutral grid of a user's face; an alignment module for aligning the standard grid area to the initial neutral grid to obtain a deformed grid area corresponding to the standard grid area; a replacement module for replacing a complex area corresponding to the deformed grid area in the initial neutral grid with the deformed grid area to obtain a target neutral grid of the user's face; and a migration module for migrating the expression state of the standard expression base to the target neutral grid to obtain a target expression base of the user's face.

[0015] On the other hand, the present application provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the expression base generation method.

[0016] On the other hand, the present application provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and when the at least one instruction is executed by a processor in an electronic device, the expression base generation method is implemented.

[0017] Through the above implementation, by aligning the standard mesh area to the initial neutral mesh, the aligned deformed mesh area and the corresponding complex area of ​​the deformed mesh area in the initial neutral mesh can be matched with each other in terms of morphology, coordinate system, size, contour and spatial position. The complex area is an area in the initial neutral mesh where the structure is missing and / or does not conform to the physiological structure of the human body. Therefore, by replacing the corresponding complex area in the initial neutral mesh with the deformed mesh area, the missing of the initial neutral mesh can be filled and / or repaired, so that the morphology of the target neutral mesh is more realistic and the contour is smoother and more natural. Since the target neutral mesh is obtained by repairing the initial neutral mesh of the user's face, the network topology of the target neutral mesh can follow the network topology of the initial neutral mesh, and the original features of the user's face are retained in other areas of the target neutral mesh except for the replaced complex area. Since the target neutral mesh retains the network topology and original features of the initial neutral mesh, the accuracy of the target neutral mesh can be improved, so that the fidelity and naturalness of the expression base generated according to the target neutral mesh can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a neutral facial grid provided in an embodiment of the present application.

[0019] Figure 2A It is a schematic diagram of an expression base provided in one embodiment of the present application.

[0020] Figure 2BIt is a schematic diagram of an expression base provided in another embodiment of the present application.

[0021] Figure 2C It is a schematic diagram of an expression base provided in yet another embodiment of the present application.

[0022] Figure 3 is a schematic diagram of a neutral facial grid provided in another embodiment of the present application.

[0023] Figure 4A It is a schematic diagram of an expression base provided in one embodiment of the present application.

[0024] Figure 4B It is a schematic diagram of an expression base provided in another embodiment of the present application.

[0025] Figure 4C It is a schematic diagram of an expression base provided in yet another embodiment of the present application.

[0026] Figure 5 It is a flowchart of an expression base generation method provided in one embodiment of the present application.

[0027] Figure 6 It is a front view of a standard grid area provided in one embodiment of the present application.

[0028] Figure 7 It is a side view of a standard grid area provided in one embodiment of the present application.

[0029] Figure 8 Schematic diagram of an initial neutral grid provided in an embodiment of the present application.

[0030] Fig. 9 It is a schematic diagram of an initial neutral grid after deleting a complex area provided in an embodiment of the present application.

[0031] Fig.10 Schematic diagram of an initial neutral mesh after splicing deformed mesh regions according to an embodiment of the present application.

[0032] Fig.11 is a schematic diagram of a target neutral grid provided in an embodiment of the present application.

[0033] Fig.12 is a schematic diagram of a target neutral grid provided in another embodiment of the present application.

[0034] Fig.13A It is a schematic diagram of an expression base provided in one embodiment of the present application.

[0035] Fig. 13B It is a schematic diagram of an expression base provided in another embodiment of the present application.

[0036] Fig. 13C It is a schematic diagram of an expression base provided in yet another embodiment of the present application.

[0037] Fig.14 This is a flow chart of a method for generating a deformable mesh region provided in one embodiment of the present application.

[0038] Fig.15 It is a flowchart of a method for constructing a constraint function provided in one embodiment of the present application.

[0039] Fig.16 It is a functional module diagram of an expression base generation device provided in one embodiment of the present application.

[0040] Fig.17 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0041] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0042] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0043] With the advancement of 3D expression-driven technology, its application in various fields such as virtual reality, animation creation, and film production is gradually expanding. Especially in the field of animation-driven, in order to vividly show the expression changes of animated characters, it is usually necessary to build a realistic neutral facial mesh and expression base. The neutral face mesh is a 3D mesh model corresponding to an expressionless face, and the expression base is an expression mesh model with a specific expression state (such as blinking, opening the mouth, etc.) constructed from the neutral face mesh.

[0044] However, there are often some problems with the neutral facial mesh obtained by scanning the user, such as the adhesion of the upper and lower lip structures and the missing internal structure of the lips. In order to solve these problems, the following two methods are mainly used in the related art to generate expression bases. The first method is to not use the neutral facial mesh obtained by scanning, but directly fit a neutral facial mesh similar to the user's face through the standard expression library, and generate an expression base based on this. The second method is to select a standard face mesh from the standard expression library, transform it to the scanned neutral face mesh, obtain the user's neutral face mesh, and directly use the expression base corresponding to the standard face mesh in the standard expression library as the user's expression base.

[0045] However, the network topology of the neutral face mesh generated by these two methods is different from the neutral face mesh obtained by the original scan, and because the topology of the standard face mesh is completely adopted, the user's facial features will be lost, resulting in the generated neutral face mesh and expression base not being realistic and natural enough. Figure 1 , which is a schematic diagram of a neutral face grid provided by an embodiment of the present application. Figure 2A As shown in FIG. 1 , it is a schematic diagram of an expression base provided by an embodiment of the present application. Figure 2B As shown in FIG. 1 , it is a schematic diagram of an expression base provided by another embodiment of the present application. Figure 2C , which is a schematic diagram of an expression base provided by another embodiment of the present application. Figure 1 The neutral face mesh in is the mesh model in the standard database. Figure 2A-2C The expression base in Figure 1 The neutral facial mesh shown is constructed. Figure 1 The neutral facial mesh in the image is not complete enough (it lacks structures such as hair and ears), and is quite different from a real human face. Figure 1 A neutral face mesh in Figure 2A-2C The facial expressions in the video are not realistic and natural enough. Figure 3 , which is a schematic diagram of a neutral facial grid provided by another embodiment of the present application. Figure 4A As shown in FIG. 1 , it is a schematic diagram of an expression base provided by an embodiment of the present application. Figure 4B As shown in FIG. 1 , it is a schematic diagram of an expression base provided by another embodiment of the present application. Figure 4C , which is a schematic diagram of an expression base provided by another embodiment of the present application. Figure 3 The neutral face mesh in is another mesh model in the standard database. Figure 4A-4C The expression base in Figure 3 The neutral facial mesh shown is constructed. Figure 3 The neutral facial mesh in the image is not complete enough (missing structures such as hair and ears), not realistic enough, and has a large difference from the real face. Figure 3A neutral face mesh in Figure 4A-4C The facial expressions in the video are not realistic and natural enough.

[0046] In order to solve the above technical problems, the embodiment of the present application provides an expression base generation method, which can improve the fidelity and naturalness of the expression base. The expression base generation method provided in the embodiment of the present application can be applied to one or more electronic devices, and the electronic device can be a laptop computer, a mobile phone, a computer, a server and other devices. The present application does not limit the type of electronic device.

[0047] like Figure 5 FIG. 1 is a flowchart of a method for generating an expression base according to an embodiment of the present application. According to different requirements, the order of each step in the flowchart can be adjusted according to actual requirements, and some steps can be omitted. The method is applied to electronic devices, such as Fig.17 An electronic device 10 is shown.

[0048] S11, obtaining a standard expression base, a preset standard grid area in a standard facial grid, and an initial neutral grid of the user's face.

[0049] In some embodiments of the present application, the standard expression base, the standard facial mesh and the initial neutral mesh may be a three-dimensional mesh model, the three-dimensional mesh model includes a plurality of vertices and the connection relationship between the vertices. When a plurality of vertices constitute a face, the connection relationship between the vertices constitutes an edge in the mesh, and each vertex has a corresponding coordinate (three-dimensional coordinate). The standard expression base may be an expression base that needs to be transferred to an expression state, and the standard facial mesh may be a neutral (expressionless) three-dimensional mesh model, or a three-dimensional mesh model with a corresponding expression state (such as a smile), and the present application does not limit this. The electronic device may obtain the standard expression base and the standard facial mesh from a standard database. Various standard expression bases and standard facial meshes may be stored in the standard database, for example, the standard database may be a FaceWarehouse database and a Bosphorus Database database, etc. The above examples of the method for obtaining the standard expression base and the standard facial mesh are only examples, and are not limited to this in actual applications.

[0050] The initial neutral mesh is a three-dimensional mesh model corresponding to the expressionless user's face and does not have a corresponding expression state. The electronic device can obtain the initial neutral mesh by various methods, and the present application does not limit the method for obtaining the initial neutral mesh. For example, the electronic device can directly read the model data from the folder under the preset path as the initial neutral mesh, where the preset path can be customized. Alternatively, the electronic device can reconstruct the initial neutral mesh based on the point cloud data of the user's face, where the point cloud data of the user's face can be obtained by scanning the user's face with a face scanning device (face scanning scanner). The initial neutral mesh may include structures such as hair, ears and shoulders.

[0051] The complex area may be an area in the initial neutral grid where there are structural defects and / or areas that are inconsistent with the actual physiological state. The present application does not limit the complex areas in the initial neutral grid. For example, the complex area may be an eye area where the eyeballs rotate unnaturally and / or have structural defects, a lip area where the upper and lower lips are adhered and / or have structural defects, etc. The standard grid area corresponds to the complex area. For example, if the complex area is a lip area, the standard grid area is also a lip area, or, if the complex area includes a lip area and an eye area, the standard grid area also includes a lip area and an eye area.

[0052] In some embodiments of the present application, the electronic device can segment a standard grid area from a standard facial grid. For example, the electronic device can segment a standard grid area from a standard facial grid using a Haar feature cascade classifier (Histogram of Oriented Gradients, HOG), and output the index of edge vertices in the standard grid area. Alternatively, the standard grid area can be obtained by manual segmentation. The above examples of the segmentation method of the standard grid area are only examples and are not limited to this in practical applications.

[0053] For example, Figure 6 As shown in FIG. 1 , it is a front view of a standard grid area provided by an embodiment of the present application. Figure 7 , which is a side view of a standard grid area provided in one embodiment of the present application. Figure 6 and Figure 7 The standard mesh area in is the lip area. Figure 6 is a front view of the lip area, Figure 7 is a side view of the lip area. Figure 8 , which is a schematic diagram of an initial neutral grid provided in an embodiment of the present application. Figure 8 The initial neutral mesh in is the 3D mesh model corresponding to the expressionless user’s face. Figure 1 and Figure 3 The neutral face mesh shown, Figure 8 The initial neutral mesh in includes features such as hair and ears.

[0054] S12, aligning the standard mesh area to the initial neutral mesh to obtain a deformed mesh area corresponding to the standard mesh area.

[0055] In some embodiments of the present application, the standard grid area corresponds to a plurality of first coordinate points, and the initial neutral grid corresponds to a plurality of second coordinate points. The first coordinate point may be a vertex of the standard grid area, and the second coordinate point may be a vertex in the initial neutral grid. Both the first coordinate point and the second coordinate point have corresponding initial coordinates, and the initial coordinates may be three-dimensional coordinates.

[0056] In some embodiments of the present application, the electronic device aligns the standard grid area to the initial neutral grid to obtain a deformed grid area corresponding to the standard grid area, including: the electronic device can perform non-rigid deformation of the standard grid area relative to the initial neutral grid based on a dense alignment between multiple first coordinate points and multiple second coordinate points, thereby determining the target coordinates of each first coordinate point, transforming each first coordinate point from the corresponding initial coordinate to the corresponding target coordinate, and determining the deformed grid area based on multiple first coordinate points at the corresponding target coordinates.

[0057] The target deformation parameter of each first coordinate point may include a linear transformation matrix and a translation vector, wherein the linear transformation matrix may be a rotation matrix, the linear transformation matrix may be a 3*3 matrix, and the translation vector may be a 3*1 vector.

[0058] In this embodiment, by performing non-rigid deformation on the standard mesh area relative to the initial neutral mesh, the aligned deformed mesh area can be made to fit the complex area corresponding to the deformed mesh area in the initial neutral mesh. The aligned deformed mesh area and the corresponding complex area are in the same coordinate system, and the deformed mesh area and the corresponding complex area can have the same or similar features such as shape, size, contour, coordinates, etc.

[0059] S13, using the deformed mesh area to replace the complex area corresponding to the deformed mesh area in the initial neutral mesh, to obtain a target neutral mesh of the user's face.

[0060] In some embodiments of the present application, the electronic device uses the deformed mesh area to replace the complex area corresponding to the deformed mesh area in the initial neutral mesh to obtain a target neutral mesh of the user's face, including: the electronic device determines the regional position of the complex area in the initial neutral mesh based on the target coordinates of multiple first coordinate points, deletes the complex area from the initial neutral mesh based on the regional position, and splices the deformed mesh area to the regional position in the initial neutral mesh to obtain the target neutral mesh.

[0061] The electronic device may determine the target coordinates of multiple or all edge vertices in the multiple first coordinate points as the region position of the complex region in the initial neutral mesh. The target coordinates of the edge vertices may be acquired or called through the corresponding index.

[0062] This application does not limit the stitching method. For example, the electronic device can use a mesh fusion algorithm such as Signed Distance Field (SDF) to stitch the deformed mesh area to the area position in the initial neutral mesh, and perform operations such as hole stitching and smoothing on the stitched initial neutral mesh to obtain the target neutral area.

[0063] For example, if the standard mesh area and the complex area are lips, Fig. 9 , which is a schematic diagram of an initial neutral grid after deleting a complex area provided by an embodiment of the present application. Fig.10 , which is a schematic diagram of an initial neutral mesh after splicing deformed mesh regions provided by an embodiment of the present application. Fig.11 , which is a schematic diagram of a target neutral grid provided in one embodiment of the present application. Fig. 9 By deleting Figure 8 The lip region of the initial neutral mesh is obtained. Fig.10 By splicing the deformed lip area (deformed mesh area) to Fig. 9 The initial neutral grid in is obtained. Fig.11 Through Fig.10 The initial neutral mesh in is obtained by gap stitching and smoothing. Fig.11 In the target neutral mesh shown, all regions except the lips are from Figure 8 The initial neutral mesh in retains the network topology and original features such as hair, eyes and ears of the initial neutral mesh.

[0064] In this embodiment, since the deformed mesh area and the corresponding complex area match each other in terms of shape, coordinate system, size, outline and spatial position, replacing the corresponding complex area in the initial neutral mesh with the deformed mesh area can fill and / or repair problems such as the missing initial neutral mesh, thereby making the shape of the target neutral mesh more realistic and the outline smoother and more natural.

[0065] S14, migrating the expression state of the standard expression base to the target neutral grid to obtain the target expression base of the user's face.

[0066] In some embodiments of the present application, the electronic device migrates the expression state of the standard expression base to the target neutral grid to obtain the target expression base of the user's face, including: the electronic device matches the faces in the target neutral grid with the standard faces in the standard expression base, determines the expression state variation parameters of each face in the target neutral grid, and transforms the target neutral grid according to the expression state variation parameters of each face to obtain the target expression base.

[0067] The expression state of the target expression base is the same as the expression state of the standard expression base. The method for determining the expression deformation parameters can refer to the method for determining the target deformation parameters of each first coordinate point, and the method for transforming the target neutral grid according to the expression deformation parameters of each facet can refer to the method for transforming the initial coordinates of each first coordinate point according to the target deformation parameters of the first coordinate point, which will not be repeated in this application.

[0068] like Fig.12 , which is a schematic diagram of a target neutral grid provided by another embodiment of the present application. Fig.13A As shown in FIG. 1 , it is a schematic diagram of an expression base provided by an embodiment of the present application. Fig. 13B As shown in FIG. 1 , it is a schematic diagram of an expression base provided by another embodiment of the present application. Fig. 13C FIG. 1 is a schematic diagram of an expression base provided by another embodiment of the present application. Figure 2A-2C as well as Figure 4A-4C , Figure 13A-13C The facial expression base includes structures such as hair and ears. Fig.12 The target neutral mesh in the image is more fitted, realistic and natural, with little difference from the real face.

[0069] In this embodiment, since the target neutral mesh is obtained by repairing the initial neutral mesh of the user's face, the network topology of the target neutral mesh can follow the network topology of the initial neutral mesh, and the other regions of the target neutral mesh except the replaced complex region retain the original features of the user's face. Since the target neutral mesh retains the network topology and original features of the initial neutral mesh, the accuracy of the target neutral mesh can be improved, thereby improving the fidelity and naturalness of the expression base generated according to the target neutral mesh.

[0070] In some embodiments of the present application, a facial model of the user is obtained based on a target neutral grid and / or a target expression base of the user's face, and the electronic device can drive the facial model of the user according to the target expression base so that the driven facial model changes from the corresponding expression state to the expression state corresponding to the target expression base.

[0071] The present application does not limit the method of driving the user's facial model according to the target expression base. For example, the electronic device can perform non-rigid registration of the target expression base and the user's facial model through an iterative closest point algorithm (ICP), thereby driving the facial model to transform from the corresponding expression state to the expression state corresponding to the target expression base.

[0072] The initial neutral grid of the user's face can be obtained by scanning the user's face with a face scanning device (face scanner, tablet computer, mobile phone or AR glasses, etc.) or by simulating with a simulation algorithm. The above method for obtaining the initial neutral grid of the user's face is only an example and is not limited to this in actual applications.

[0073] In this embodiment, by driving the facial model to change from the corresponding expression state to the expression state corresponding to the target expression base, expression migration between the target expression base and the facial model can be achieved, thereby tracking the facial expression of the user.

[0074] In other embodiments of the present application, a dental model of the user is obtained, and the dental model has multiple occlusion states; the target neutral grid of the user's face is aligned with the dental model, and the target expression base has a corresponding relationship with at least one occlusion state of the dental model.

[0075] The electronic device can align the user's tooth model with the facial model to obtain an aligned model, wherein the aligned model includes the facial model and the tooth model. The electronic device can drive the aligned model according to the target expression base, and can adjust the expression state of the facial model and the occlusion state of the tooth model in the aligned model according to the target expression base, wherein the occlusion state of the tooth model can change with the change of the expression state, and the expression state of the aligned model corresponds to the expression state of the target expression base, and the occlusion state corresponds to the open and closed state corresponding to the lip structure of the target expression base. For example, when the lip structure of the target expression base is in an open state, the occlusion state of the tooth model of the aligned model can be an unoccluded state with the upper and lower teeth separated; when the lip structure of the target expression base is in a closed state, the occlusion state of the tooth model of the aligned model can be an occlusion state with the upper and lower teeth fully occluded.

[0076] Align the user's dental model with the facial model to obtain an aligned model, including:

[0077] The first facial model may not include a tooth structure area, and the second facial model includes a tooth structure area. The first rigid body area and the second rigid body area are areas in the first facial model and the second facial model that are not easily affected by changes in expression or orthodontic effects. The shapes of the first rigid body area and the second rigid body area are approximately the same. Therefore, even if the user's expression changes, so that the shapes of the second facial model and the partial area of ​​the first facial model are different, the first transformation matrix for aligning the second facial model to the first facial model can be accurately determined by matching the point cloud data of the second rigid body area with the first rigid body area. Since the second facial model includes a tooth structure area, the second transformation matrix for aligning the tooth model to the second facial model can be accurately determined by matching the point cloud data of the tooth model with the tooth structure area in the second facial model. According to the first transformation matrix and the second transformation matrix, the tooth model can be accurately aligned to the first facial model without a tooth structure area or with less tooth structure area without relying on CT scan data, so as to achieve matching of facial models with tooth models at different expressions or at different time points.

[0078] Specifically, a tooth model and a facial model of the user are obtained, wherein the facial model includes a first facial model and a second facial model; a first rigid body region in the first facial model and a second rigid body region in the second facial model are segmented; a first transformation matrix for aligning the second facial model to the first facial model is determined based on matching point cloud data of the second rigid body region with the first rigid body region; a second transformation matrix for aligning the tooth model to the second facial model is determined based on matching point cloud data of the tooth model with the tooth structure region of the second facial model; and the tooth model is aligned with the first facial model according to the first transformation matrix and the second transformation matrix.

[0079] The first facial model may be the initial neutral mesh of the user's face, thereby achieving alignment between the target neutral mesh of the user's face and the tooth model.

[0080] The tooth model may be a three-dimensional model of the user's oral cavity. For example, the tooth model may be obtained by scanning the user's oral cavity using an oral scanning device, or may be obtained by modeling using a simulation algorithm. The above examples of methods for obtaining the tooth model are only examples and are not limited thereto in practical applications.

[0081] In this embodiment, by aligning the tooth model with the facial model, the tooth details of the aligned model can be enriched, thereby improving the accuracy of the aligned model. By driving the facial model of the aligned model to transform from the original expression state to the expression state corresponding to the target expression base and the tooth model to transform from the original occlusion state to the occlusion state corresponding to the target expression base, the expression state and tooth morphology can be migrated, thereby tracking the user's facial expression and teeth.

[0082] Through the above implementation, by aligning the standard mesh area to the initial neutral mesh, the aligned deformed mesh area and the corresponding complex area of ​​the deformed mesh area in the initial neutral mesh can be matched with each other in terms of morphology, coordinate system, size, contour and spatial position. The complex area is an area in the initial neutral mesh where the structure is missing and / or does not conform to the physiological structure of the human body. Therefore, by replacing the corresponding complex area in the initial neutral mesh with the deformed mesh area, the missing of the initial neutral mesh can be filled and / or repaired, so that the morphology of the target neutral mesh is more realistic and the contour is smoother and more natural. Since the target neutral mesh is obtained by repairing the initial neutral mesh of the user's face, the network topology of the target neutral mesh can follow the network topology of the initial neutral mesh, and the original features of the user's face are retained in other areas of the target neutral mesh except for the replaced complex area. Since the target neutral mesh retains the network topology and original features of the initial neutral mesh, the accuracy of the target neutral mesh can be improved, so that the fidelity and naturalness of the expression base generated according to the target neutral mesh can be improved.

[0083] In some embodiments of the present application, Fig.14 , which is a flow chart of a method for generating a deformable mesh region provided in one embodiment of the present application.

[0084] S121, determining a deformation field corresponding to each first coordinate point, where the deformation field includes a plurality of deformation control points corresponding to each first coordinate point.

[0085] In some embodiments of the present application, the electronic device can sample multiple first coordinate points from the first coordinate points corresponding to the standard grid area as deformation control points, determine the deformation control point corresponding to each first coordinate point among the multiple deformation control points according to preset rules, and establish a corresponding relationship between each first coordinate point and the corresponding deformation control point, thereby completing the determination of the deformation field corresponding to each first coordinate point.

[0086] The number of deformation control points obtained by sampling can be customized, and this application does not impose any restrictions on this. The electronic device can use various sampling methods to sample the first coordinate point corresponding to the standard grid area to obtain multiple deformation control points. For example, the electronic device can use the nearest neighbor sampling method, the uniform sampling method, the voxel sampling method, and other methods to sample multiple first coordinate points from the first coordinate point corresponding to the standard grid area as deformation control points.

[0087] The preset rule can be customized, and the present application does not impose any restrictions on this. For example, if the preset rule is the nearest point rule, the electronic device determines the deformation control point corresponding to each first coordinate point in the multiple deformation control points according to the nearest point rule, including: the electronic device calculates the first distance between each deformation control point and each first coordinate point, and determines that the deformation control point corresponding to the first distance less than the first preset value is the deformation control point corresponding to each first coordinate point. Among them, the first distance can be Euclidean distance or geodesic distance (Geodesic) or other distance, and the present application does not impose any restrictions on this. The first preset value can be customized, and the present application does not impose any restrictions on this.

[0088] In this embodiment, a plurality of first coordinate points are sampled from the first coordinate points corresponding to the standard grid area as deformation control points, which can reduce the amount of calculation.

[0089] S122, determining a plurality of point pairs by densely registering the plurality of first coordinate points and the plurality of second coordinate points, wherein each point pair includes first coordinate points and second coordinate points corresponding to each other.

[0090] In this embodiment, dense registration of multiple first coordinate points and multiple second coordinate points can refer to the description of the nearest point method based on point cloud distance, the dense matching method based on RGB optical flow, and other methods, which will not be repeated in this application. Among them, the first coordinate points in the multiple groups of point pairs may include the multiple first coordinate points, or only some of the first coordinate points in the multiple first coordinate points, and this application does not limit this.

[0091] S123, determining the target deformation parameters of each deformation control point based on the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs, and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point.

[0092] In some embodiments of the present application, the electronic device can execute a solution process for the initial deformation parameters of each deformation control point, including: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function, repeating the above solution process, updating the initial deformation parameters of each deformation control point, until the number of updates to the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stopping the above solution process, and determining the updated initial deformation parameters as the target deformation parameters of each deformation control point.

[0093] The initial deformation parameters include a linear transformation matrix and a translation vector. The preset number of times and the preset threshold value can be customized, and this application does not limit this. For example, the preset number of times can be 20 times. The preset threshold value can be 0.1.

[0094] The electronic device may optimize the constraint function using an optimization algorithm, for example, the optimization algorithm may be a gradient descent method and a least square method, etc. The constraint residual is the value of the constraint function, and the constraint function may include a data term constraint and a smooth term constraint. The value of the data term constraint may represent the gap or difference between the target coordinates of the first coordinate point in each set of point pairs and the initial coordinates of the second coordinate point in each set of point pairs, and the value of the smooth term constraint may represent the gap or difference between the target deformation parameter of each deformation control point and the target deformation parameter of the deformation control point adjacent to each deformation control point.

[0095] The above conditions for stopping the solution process are only examples and are not limited to this in practical applications. For example, if the update time of the initial deformation parameters of each deformation control point reaches a preset time, the electronic device can stop executing the above solution process and determine the updated initial deformation parameters as the target deformation parameters of each deformation control point, wherein the preset time can be customized.

[0096] For example, if minimizing the constraint residual is the optimization goal, the calculation of the target deformation parameter of each deformation control point can refer to the following formula (1):

[0097]

[0098] Where E represents the constraint function, E d Represents data item constraints, E s represents the smoothness constraint, T jrepresents the target deformation parameter of the jth deformation control point obtained after minimizing the optimization constraint residual, A j Represents the linear transformation matrix in the target deformation parameters of the j-th deformation control point.

[0099] In this embodiment, since the deformation of each first coordinate point is often affected by the adjacent deformation control points, the constraint function is determined and minimized by using the deformation control points adjacent to each deformation control point, which can reduce errors and ensure the accuracy of the target deformation parameters.

[0100] S124, determining the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of the plurality of deformation control points corresponding to each first coordinate point.

[0101] In some embodiments of the present application, since the target deformation parameters of each deformation control point include a linear transformation matrix and a translation vector, the electronic device can determine the candidate coordinates of each first coordinate point based on the initial coordinates of each first coordinate point and the linear transformation matrix and translation vector of each deformation control point corresponding to each first coordinate point, thereby obtaining multiple candidate coordinates, and determine the target coordinates of each first coordinate point based on the multiple candidate coordinates.

[0102] Among them, the electronic device calculates the initial coordinates of each first coordinate point and the target deformation parameters of a deformation control point to obtain a candidate coordinate of each first coordinate point, and calculates the initial coordinates of each first coordinate point and the target deformation parameters of multiple deformation control points to obtain multiple candidate coordinates of each first coordinate point. The electronic device can determine the average value or weighted average value among the multiple candidate coordinates of each first coordinate point as the target coordinate of each first coordinate point. The weighted weight can be customized, and this application does not limit this.

[0103] For example, the calculation formula of the candidate coordinates can refer to formula (2):

[0104] Y=A i X+t i ; (2)

[0105] Wherein, Y represents the candidate coordinates of each first coordinate point for the i-th deformation control point among the corresponding multiple deformation control points, A i represents the linear transformation matrix in the target deformation parameter of the i-th deformation control point, X represents the initial coordinates in the target deformation parameter of each first coordinate point, t i The translation vector representing the linear transformation matrix of the i-th deformation control point.

[0106] S125: Determine a deformable mesh area based on the multiple first coordinate points and the target coordinate of each first coordinate point.

[0107] In some embodiments of the present application, the electronic device can transform each first coordinate point from the corresponding initial coordinate to the corresponding target coordinate, and determine the area formed by multiple first coordinate points on the corresponding target coordinate and the connection relationship between the multiple first coordinate points as a deformable mesh area.

[0108] like Fig.15 FIG. 1 is a flowchart of a method for constructing a constraint function provided in an embodiment of the present application, comprising the following steps:

[0109] S1231: constructing an objective function of target deformation parameters of a plurality of deformation control points corresponding to the first coordinate point in each group of point pairs according to the initial coordinates of the first coordinate point in each group of point pairs.

[0110] In some embodiments of the present application, the target function corresponds to the target coordinates of the first coordinate point in each set of point pairs. For example, the target function can refer to the following formula (3):

[0111] f(v n )=Σ m [w m (A m v n +t m )]; (3)

[0112] Among them, f(v n ) represents the objective function corresponding to the target coordinates of the first coordinate point in the nth set of point pairs, A m Represents the linear transformation matrix in the target deformation parameters of the mth deformation control point corresponding to the first coordinate point in the nth point pair, t m Represents the translation vector in the target deformation parameter of the mth deformation control point corresponding to the first coordinate point in the nth point pair, w m Indicates the preset weight corresponding to the mth deformation control point corresponding to the first coordinate point in the nth point pair.

[0113] S1232: construct data item constraints according to the initial coordinates of the second coordinate point in each group of point pairs and the objective function.

[0114] For example, the construction of data item constraints can refer to the following formula (4):

[0115]

[0116] Among them, E d Represents data item constraints, v nrepresents the initial coordinates of the first coordinate point in the nth set of point pairs, f(v n ) represents the objective function corresponding to the target coordinates of the first coordinate point in the nth set of point pairs, p n Represents the initial coordinates of the second coordinate point in the nth point pair in the dense correspondence.

[0117] In this embodiment, since the value of the data item constraint can represent the gap or difference between the target coordinates of the first coordinate point in each group of point pairs and the initial coordinates of the second coordinate point in each group of point pairs, by optimizing the data item constraint, the gap or difference between the target coordinates of the first coordinate point in each group of point pairs and the initial coordinates of the second coordinate point in each group of point pairs can be narrowed.

[0118] S1233: Construct a smoothing term constraint on a target deformation parameter of each deformation control point and a target deformation parameter of each deformation control point adjacent to each deformation control point.

[0119] In some embodiments of the present application, the electronic device may calculate a second distance between each deformation control point and other deformation control points other than each deformation control point, and determine a deformation control point corresponding to a second distance less than a second preset value as a deformation control point adjacent to each deformation control point, wherein the second distance may be a geodesic distance or other types of distances. The second preset value may be custom set, and the present application does not limit this.

[0120] For example, the construction of the smoothness term constraint can refer to the following formulas (5)-(7):

[0121]

[0122] T j ={A j , t j}; (6)

[0123] T k ={A k , t k}; (7)

[0124] Among them, E s represents the smoothness constraint, T j represents the target deformation parameter of the jth deformation control point, A j Represents the linear transformation matrix in the target deformation parameters of the j-th deformation control point, t j represents the translation vector in the target deformation parameter of the j-th deformation control point, N(j) represents all deformation control points adjacent to the j-th deformation control point, T k represents the target deformation parameter of the kth deformation control point in N(j), Ak Represents the linear transformation matrix in the target deformation parameters of the kth deformation control point, t k Represents the translation vector in the target deformation parameters of the k-th deformation control point.

[0125] In this embodiment, since the value of the smoothness term constraint can represent the gap or difference between the target deformation parameters of each deformation control point and the target deformation parameters of the deformation control points adjacent to each deformation control point, by optimizing the smoothness term constraint, the gap or difference between the target deformation parameters of each deformation control point and the target deformation parameters of the deformation control points adjacent to each deformation control point can be narrowed.

[0126] S1234, determining a constraint function according to the data item constraint and the smoothing item constraint.

[0127] In some embodiments of the present application, the method for constructing the constraint function may refer to the above formula (1).

[0128] In other embodiments of the present application, the electronic device may perform weighted operations on the data item constraints and the smoothing item constraints to obtain a constraint function.

[0129] like Fig.16 , which is a functional module diagram of an expression base generation device provided by an embodiment of the present application. The expression base generation device 11 includes an acquisition module 110, an alignment module 111, a replacement module 112 and a migration unit 113. The module / unit referred to in the present application refers to a type of expression base generation device that can be Fig.17 A series of computer readable instruction segments acquired by the processor 103 in the embodiment and capable of completing fixed functions, which are stored in Fig.17 In the memory 102 in the present embodiment, the functions of each module / unit will be described in detail in the subsequent embodiments.

[0130] The acquisition module 110 is used to acquire the standard expression base, the standard grid area preset in the standard facial grid, and the initial neutral grid of the user's face.

[0131] The alignment module 111 is used to align the standard mesh area to the initial neutral mesh to obtain the deformed mesh area corresponding to the standard mesh area.

[0132] In some embodiments of the present application, the standard grid area corresponds to multiple first coordinate points, the initial neutral grid corresponds to multiple second coordinate points, and the alignment module 111 is further used to determine the deformation field corresponding to each first coordinate point, the deformation field includes multiple deformation control points corresponding to each first coordinate point, and multiple groups of point pairs are determined by densely aligning the multiple first coordinate points and the multiple second coordinate points, wherein each group of point pairs includes mutually corresponding first coordinate points and second coordinate points, and the target deformation parameters of each deformation control point are determined according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs, and the multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, and the target coordinates of each first coordinate point are determined according to the initial coordinates of each first coordinate point and the target deformation parameters of the multiple deformation control points corresponding to each first coordinate point, and the deformation grid area is determined based on the multiple first coordinate points and the target coordinates of each first coordinate point.

[0133] In some embodiments of the present application, the alignment module 111 is also used to execute a solution process for the initial deformation parameters of each deformation control point, including: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function, repeating the solution process, updating the initial deformation parameters of each deformation control point, until the number of updates to the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stopping the solution process, and determining the updated initial deformation parameters as the target deformation parameters of each deformation control point.

[0134] In some embodiments of the present application, the alignment module 111 is further used to construct an objective function of target deformation parameters of multiple deformation control points corresponding to the first coordinate point in each group of point pairs according to the initial coordinates of the first coordinate point in each group of point pairs, wherein the objective function corresponds to the target coordinates of the first coordinate point in each group of point pairs, construct a data item constraint according to the initial coordinates of the second coordinate point in each group of point pairs and the objective function, construct a smoothing item constraint on the target deformation parameters of each deformation control point and the target deformation parameters of each deformation control point adjacent to each deformation control point, and determine the constraint function according to the data item constraint and the smoothing item constraint.

[0135] In some embodiments of the present application, the target deformation parameters of each deformation control point include a linear transformation matrix and a translation vector. The alignment module 111 is also used to determine the candidate coordinates of each first coordinate point based on the initial coordinates of each first coordinate point and the linear transformation matrix and translation vector of each deformation control point corresponding to each first coordinate point, and determine the target coordinates of each first coordinate point based on multiple candidate coordinates.

[0136] The replacement module 112 is used to replace the complex area corresponding to the deformed mesh area in the initial neutral mesh with the deformed mesh area to obtain a target neutral mesh of the user's face.

[0137] In some embodiments of the present application, the replacement module 112 is also used to determine the regional position of the complex area in the initial neutral mesh based on the target coordinates of the multiple first coordinate points, delete the complex area from the initial neutral mesh based on the regional position, and splice the deformed mesh area to the regional position in the initial neutral mesh to obtain the target neutral mesh.

[0138] The migration module 113 is used to migrate the expression state of the standard expression base to the target neutral grid to obtain the target expression base of the user's face.

[0139] In some embodiments of the present application, the migration module 113 is also used to determine the expression deformation parameters of each facet in the target neutral mesh by matching the facets in the target neutral mesh with the standard facets in the standard expression base, and transform the target neutral mesh according to the expression deformation parameters of each facet to obtain the target expression base.

[0140] In some embodiments of the present application, the acquisition module 110 is also used to acquire a dental model of the user, wherein the dental model has multiple occlusion states, the target neutral grid of the user's face is aligned with the dental model, and the target expression base has a corresponding relationship with at least one occlusion state of the dental model.

[0141] like Fig.17 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 10 can be a mobile phone, a tablet computer, a laptop computer, or other electronic device. The present embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0142] like Fig.17 As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the input / output interface 104 through the bus 105.

[0143] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), Bluetooth (Bluetooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0144] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0145] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 103. The non-volatile memory may include a disk storage device and a flash memory. For example, the flash memory may be a flash memory (Nand Flash).

[0146] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 103, the expression base generation method executed on the electronic device 10 can be implemented.

[0147] In other embodiments, Fig.17 The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0148] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0149] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned expression base generation method.

[0150] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0151] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the input / output interface 104 in the electronic device 10 .

[0152] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0153] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0154] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0155] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0157] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0159] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present application is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0160] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in this application can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A method for generating an expression base, characterized in that: The method comprises: Obtaining a standard expression base, a preset standard grid area in a standard facial grid, and an initial neutral grid of the user's face; Aligning the standard grid area to the initial neutral grid to obtain a deformed grid area corresponding to the standard grid area; Using the deformed mesh area to replace the complex area corresponding to the deformed mesh area in the initial neutral mesh, to obtain a target neutral mesh of the user's face; The expression state of the standard expression base is transferred to the target neutral grid to obtain the target expression base of the user's face.

2. The expression base generation method according to claim 1, characterized in that: The standard grid area corresponds to a plurality of first coordinate points, the initial neutral grid corresponds to a plurality of second coordinate points, and aligning the standard grid area to the initial neutral grid to obtain a deformed grid area corresponding to the standard grid area includes: Determine a deformation field corresponding to each first coordinate point, wherein the deformation field includes a plurality of deformation control points corresponding to each first coordinate point; Determine a plurality of point pairs by densely registering the plurality of first coordinate points and the plurality of second coordinate points, wherein each point pair includes first coordinate points and second coordinate points corresponding to each other; Determine a target deformation parameter of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs, and a plurality of deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and deformation control points adjacent to each deformation control point; Determining the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of a plurality of deformation control points corresponding to each first coordinate point; The deformable mesh area is determined based on the plurality of first coordinate points and the target coordinate of each of the first coordinate points.

3. The expression base generation method according to claim 2, characterized in that: The method for determining the target deformation parameters of each deformation control point includes: The process of solving the initial deformation parameters of each deformation control point includes: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs, and a plurality of deformation control points corresponding to the first coordinate point in each group of point pairs, and each deformation control point and deformation control points adjacent to each deformation control point, and determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function; Repeat the solution process to update the initial deformation parameters of each deformation control point until the number of updates of the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stop executing the solution process, and determine the updated initial deformation parameters as the target deformation parameters of each deformation control point.

4. The expression base generation method according to claim 3, characterized in that: The method for constructing the constraint function includes: constructing an objective function of target deformation parameters of a plurality of deformation control points corresponding to the first coordinate point in each set of point pairs according to the initial coordinates of the first coordinate point in each set of point pairs, wherein the objective function corresponds to the target coordinates of the first coordinate point in each set of point pairs; Constructing a data item constraint according to the initial coordinates of the second coordinate point in each set of point pairs and the objective function; Constructing a smoothing term constraint on a target deformation parameter of each deformation control point and a target deformation parameter of each deformation control point adjacent to each deformation control point; The constraint function is determined according to the data item constraint and the smoothness item constraint.

5. The expression base generation method according to claim 2, characterized in that: The target deformation parameters of each deformation control point include a linear transformation matrix and a translation vector, and determining the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of the plurality of deformation control points corresponding to each first coordinate point includes: Determine candidate coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the linear transformation matrix and translation vector of each deformation control point corresponding to each first coordinate point; According to the multiple candidate coordinates, the target coordinates of each first coordinate point are determined.

6. The expression base generation method according to claim 2, characterized in that: The step of replacing the complex area corresponding to the deformed mesh area in the initial neutral mesh with the deformed mesh area to obtain the target neutral mesh of the user's face comprises: Determining the region position of the complex region in the initial neutral grid according to the target coordinates of the plurality of first coordinate points; Deleting the complex region from the initial neutral grid according to the region position; The deformed mesh region is spliced ​​to the region position in the initial neutral mesh to obtain the target neutral mesh.

7. The expression base generation method according to any one of claims 1 to 6, characterized in that: The step of migrating the expression state of the standard expression base to the target neutral grid to obtain the target expression base of the user's face comprises: Determining the expression variation parameters of each facet in the target neutral mesh by matching the facets in the target neutral mesh with the standard facets in the standard expression base; The target neutral grid is transformed according to the expression variation parameters of each face patch to obtain the target expression base.

8. The expression base generation method according to claim 1, characterized in that: Also includes: Acquire a tooth model of a user, wherein the tooth model has multiple occlusal states; The target neutral grid of the user's face is aligned with the tooth model, and the target expression base has a corresponding relationship with at least one occlusion state of the tooth model.

9. An expression base generating device, characterized in that: The device comprises: An acquisition module, used to acquire a standard expression base, a preset standard grid area in a standard facial grid, and an initial neutral grid of the user's face; An alignment module, used for aligning the standard grid area to the initial neutral grid to obtain a deformed grid area corresponding to the standard grid area; A replacement module, configured to replace a complex area corresponding to the deformed mesh area in the initial neutral mesh with the deformed mesh area to obtain a target neutral mesh of the user's face; A migration module is used to migrate the expression state of the standard expression base to the target neutral grid to obtain the target expression base of the user's face.

10. An electronic device, characterized in that: The electronic device comprises: a memory storing at least one instruction; and A processor executes the at least one instruction to implement the expression base generation method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor in an electronic device, the expression base generation method according to any one of claims 1 to 8 is implemented.