Character facial expression segmentation method, device, computer equipment and storage medium

Abstract

This invention discloses a method, apparatus, computer device, and storage medium for character facial expression segmentation. The method includes acquiring a 3D basic face model and performing topological processing on the basic face model to obtain a face model containing a topological structure; detecting target facial feature points on the face model using a pre-trained feature point detection model; and performing deformation processing on the face model based on the target facial feature points to obtain multiple deformed facial expressions. This method achieves topological structure-based facial expression segmentation, reducing the workload of graphic designers in constructing deformed facial expressions and improving the efficiency of deformed facial expression construction.

Description

Technical Field

[0001] The present invention relates to the field of animation production technology, and in particular to a method, device, computer equipment and storage medium for splitting character facial expressions. Background Art

[0002] As a branch of virtual reality technology and computer graphics, facial modeling and expression animation has a history of more than 30 years. With the increasing application of film and television technology, game entertainment, virtual environment, remote communication, auxiliary medical treatment, criminal identification, etc., this research field has attracted more and more attention. In line with this trend of the times, the challenges of modeling the basic face model and blendshape (deformed) face model are: because the physiological structure of the face is very complex, and people are very familiar with and sensitive to the appearance of the face, it is difficult to produce realistic facial expression movements.

[0003] At present, the production of 3D (three-dimensional image) expression animation requires a large number of blendshape face models as a basis, and artists need to manually assemble and adjust each blendshape face model for a long time, which makes it very difficult to reuse the assembled model between different models. The facial muscle model is currently the most commonly used facial animation driving model, which can achieve the goal of driving the facial expression changes of the three-dimensional face by simulating the contraction and relaxation of facial muscles. However, the construction of facial muscle models in the past is not only time-consuming and labor-intensive, but also every time the 3D face model is changed, all the construction work needs to be done again, which makes it difficult for general users to customize the 3D face model they need. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, computer equipment and storage medium for character facial expression segmentation, aiming to improve the problem that it is time-consuming and labor-intensive for users to construct character facial expressions in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a method for splitting character facial expressions, which comprises:

[0006] Acquire a 3D human face basic model, and perform topology on the human face basic model to obtain a human face model including a topological structure;

[0007] Detecting the face model using a pre-trained feature point detection model to obtain target face feature points;

[0008] The face model of the human face is deformed according to the target face feature points to obtain multiple facial deformation expressions.

[0009] In a second aspect, an embodiment of the present invention provides a device for splitting character facial expressions, comprising:

[0010] A topology module is used to obtain a 3D human face basic model and perform topology on the human face basic model to obtain a human face model including a topological structure;

[0011] A detection module, used to detect the face model using a pre-trained feature point detection model to obtain target face feature points;

[0012] The deformation module is used to deform the face model of the human face according to the feature points of the target face to obtain multiple facial deformation expressions.

[0013] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for segmenting character facial expressions described in the first aspect is implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the character facial expression segmentation method described in the first aspect above.

[0015] The embodiment of the present invention provides a method, device, computer equipment and storage medium for character facial expression segmentation. The method includes obtaining a 3D human face basic model, and topologically retop ... BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0017] Figure 1A schematic diagram of a flow chart of an embodiment of a method for character facial expression segmentation provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a training process flow of a feature point detection model in a character facial expression segmentation method provided by an embodiment of the present invention;

[0019] Figure 3 for Figure 2 Schematic diagram of the process of step S230;

[0020] Figure 4 A schematic block diagram of a flow chart of another embodiment of the method for character facial expression segmentation provided by an embodiment of the present invention;

[0021] Figure 5 A schematic block diagram of a device for segmenting character facial expressions provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0024] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] See also Figure 1 , Figure 1 The present invention provides a flowchart of a method for character facial expression segmentation according to an embodiment of the present invention. The method comprises steps S110 to S130.

[0027] Step S110, obtaining a 3D human face basic model, and performing topology on the human face basic model to obtain a human face model including a topological structure;

[0028] In this embodiment, the face of the target person is scanned by a hardware scanning device to obtain the face scanning result of the target person, and a basic face model (high model) is generated according to the scanning result. As an optional implementation method, the face of the target person can be scanned by a 3D (three-dimensional) scanning device, and the basic face model of the target person is generated according to the scanning result. That is, the present application is to scan the face of the target person by a three-dimensional scanner. It should be explained that three-dimensional scanning refers to scanning the spatial shape, structure and color of an object to obtain the spatial coordinates of the surface of the object. In the embodiment of the present application, the basic face model of the performer is generated according to the scanning result, which can make the produced basic face model more accurate and real. As other implementation methods, the face of the target person can also be scanned by a stereoscopic vision camera or other equipment. After obtaining the basic face model, topology is performed according to the basic face model by zbrush (three-dimensional modeling software) to obtain a face model (low model) containing a topological structure.

[0029] Step S120, detecting the face model using a pre-trained feature point detection model to obtain target face feature points;

[0030] In this embodiment, based on the topological structure of the human face facial model, the human face facial model is detected by a pre-trained feature point detection model to obtain the facial feature points of the human face facial model.

[0031] like Figure 2 As shown, in one embodiment, the training process of the feature point detection model includes:

[0032] Step S210, obtaining a number of sample expressions;

[0033] Step S220, marking feature points of the sample expression to obtain coordinates of the sample feature points, where the sample feature points include topological feature points and muscle endpoints;

[0034] Step S230, normalizing the coordinates of the sample feature points, inputting the normalization results into an initial feature point detection model, and training the initial feature point detection model using a preset loss function to obtain a trained feature point detection model.

[0035] In this embodiment, several expressions are extracted from the expression case data constructed by the user in the past as sample expressions. Optionally, the sample expressions can also be downloaded from the Internet, or retrieved from a local database, etc. Then, the topological feature points and muscle endpoints of the sample expressions are marked with feature points to obtain the coordinates of the sample feature points; then, the coordinates of the sample feature points are normalized, and the normalized results are input into the initial feature point detection model, and the initial feature point detection model is trained using a preset loss function to obtain a trained feature point detection model.

[0036] like Figure 3 As shown, in one embodiment, step S230 includes:

[0037] Step S310: determining whether the loss value of the loss function reaches a preset loss threshold;

[0038] Step S321: If yes, output the trained initial feature point detection model as the trained feature point detection model;

[0039] Step S322: If not, adjust the model parameters until the loss value reaches a preset loss threshold.

[0040] In this embodiment, the initial feature point detection model is trained by presetting the loss function. The loss function is preferably the L2 loss function to avoid gradient explosion when the predicted value differs greatly from the standard value, which makes the training difficult to converge, improves the robustness of the model training, and improves the accuracy of facial feature point positioning.

[0041] Among them, the L2 loss function is as follows:

[0042]

[0043] In the formula, v i Indicates the standard value, t i It represents the predicted value of the initial feature point detection model, and N represents the number of standard values.

[0044] In this embodiment, it is determined whether the loss value of the loss function converges to a preset loss threshold. When the preset loss threshold is 0.01. If the loss value reaches 0.01, then the data of the standard value and the predicted value of the initial feature point detection model are very close, and the predicted value is sufficient to indicate the standard value, that is, the trained initial feature point detection model can be used to detect feature points; if the loss value does not reach 0.01, or the converged value is larger than 0.01, the model parameters are adjusted and the normalized results are re-entered for training. Among them, the model parameters include the parameters of the initial feature point detection model (i.e., the size and number of convolutional layers and other parameters).

[0045] Step S130: deforming the face model of the target person according to the feature points of the target face to obtain a plurality of deformed facial expressions.

[0046] In this embodiment, in 3D modeling software such as Maya / 3ds Max, blendshape (deformation) is used to deform the face model of the human face according to the target face feature points to obtain multiple facial deformation expressions. Blendshape is a technology for deforming a single mesh to achieve a combination of many predefined shapes and any number of shapes. For example, a single mesh is a basic shape of a default shape (such as an expressionless face), and other shapes of the basic shape are used for blending / deformation, which are different expressions (smile, frown, closed eyelids). These different expressions are collectively referred to as deformation.

[0047] like Figure 4 As shown, in one embodiment, after step S130, the following steps are included:

[0048] Step S410, creating a bone hierarchy structure that conforms to the human face model, and marking joint points according to the bone hierarchy structure to obtain target joint points;

[0049] Step S420, calculating control parameters based on the multiple facial deformation expressions and facial movement rules to obtain expression support parameters;

[0050] Step S430, setting the skin weights of the vertices bound to the bone hierarchy structure and the human face model at different target joint points, and skinning the human face model and the bone hierarchy structure according to the skin weights, and adjusting the parameters of the expression control interface according to the expression support parameters.

[0051] In this embodiment, after obtaining multiple facial deformation expressions, a bone hierarchy structure that conforms to the human face model is created, and joint points are marked according to the bone hierarchy structure to obtain target joint points; then control parameters are calculated based on multiple facial deformation expressions and facial motion laws to obtain expression support parameters for controlling expression movement; the skin weights of the vertices bound to the bone hierarchy structure and the human face model at different target joint points are set, and the human face model and the bone hierarchy structure are skinned according to the skin weights, and the parameters of the expression control interface are adjusted according to the expression support parameters. Among them, the facial motion law is obtained by analyzing the human face model through the human face motion coding system. Skinning is a three-dimensional animation term and is also used in 3D games. A production technology of three-dimensional animation. Based on the model created in the three-dimensional software, bones are added to the model. Since the bones and the model are independent of each other, in order to allow the skeleton to drive the model to produce reasonable movement, the technology of binding the model to the bones is called skinning. By adjusting the control parameters of each bone deformer in the expression control interface, the human face model is flexibly and parameterized.

[0052] Furthermore, after step S430, the method further includes:

[0053] A control expression corresponding to the expression control interface is generated through a preset control expression template, and the control expression is bound to the human face model.

[0054] In this embodiment, after adjusting the parameters of the expression control interface according to the expression support parameters, a control expression corresponding to the expression control interface is generated through a preset control expression template, and the control expression is bound to the face model. The user directly controls the deformation of the face model by adjusting the data of the control expression without calling the expression control interface, thereby improving the efficiency of controlling the deformation of the face model.

[0055] The method retopologizes the 3D face basic model to obtain a face model containing a topological structure, and then detects the face model based on the topological structure using a feature point detection model to obtain the target face feature points; finally, the face model is deformed according to the target face feature points to obtain multiple facial deformation expressions, thereby realizing facial expression splitting based on the topological structure, reducing the workload of artists in constructing facial deformation expressions, and improving the efficiency of constructing facial deformation expressions.

[0056] The embodiment of the present invention also provides a character facial expression segmentation device, which is used to execute any embodiment of the above-mentioned character facial expression segmentation method. Figure 5 , Figure 51 is a schematic block diagram of a character facial expression separation device provided by an embodiment of the present invention. The character facial expression separation device 100 can be configured in a server.

[0057] like Figure 5 As shown, the character facial expression segmentation device 100 includes a topology module 110 , a detection module 120 , and a deformation module 130 .

[0058] A topology module 110 is used to obtain a basic face model in 3D and to perform topology on the basic face model to obtain a facial model including a topological structure;

[0059] A detection module 120 is used to detect the face model using a pre-trained feature point detection model to obtain target face feature points;

[0060] The deformation module 130 is used to deform the face model of the human face according to the target face feature points to obtain multiple facial deformation expressions.

[0061] In one embodiment, the character facial expression segmentation device 100 further includes:

[0062] A creation module, used to create a bone hierarchy structure that conforms to the human face model, and identify joint points according to the bone hierarchy structure to obtain target joint points;

[0063] A calculation module, used for calculating control parameters based on the plurality of facial deformation expressions and facial movement rules to obtain expression support parameters;

[0064] A skinning module is used to set the skinning weights of the vertices bound to the bone hierarchy structure and the human face model at different target joint points, and to perform skinning on the human face model and the bone hierarchy structure according to the skinning weights, and to adjust the parameters of the expression control interface according to the expression support parameters.

[0065] In one embodiment, the character facial expression segmentation device 100 further includes:

[0066] The binding module is used to generate a control expression corresponding to the expression control interface through a preset control expression template, and to bind the control expression with the human face model.

[0067] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned character facial expression segmentation method when executing the computer program.

[0068] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the above-mentioned character facial expression segmentation method.

[0069] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0070] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be an electrical, mechanical or other form of connection.

[0071] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0072] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0073] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), disk or optical disk and other media that can store program codes.

[0074] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for splitting character facial expressions, characterized in that: include: Acquire a 3D human face basic model, and perform topology on the human face basic model to obtain a human face model including a topological structure; Detecting the face model using a pre-trained feature point detection model to obtain target face feature points; Deforming the face model of the target person according to the feature points of the target person's face to obtain a plurality of deformed facial expressions; The training process of the feature point detection model includes: obtaining a number of sample expressions; marking the feature points of the sample expressions to obtain the coordinates of the sample feature points, wherein the sample feature points include topological feature points and muscle endpoints; normalizing the coordinates of the sample feature points, inputting the normalized results into the initial feature point detection model, and training the initial feature point detection model using a preset loss function to obtain a trained feature point detection model; The method of using a preset loss function to train the initial feature point detection model to obtain the feature point detection model includes: determining whether the loss value of the loss function reaches a preset loss threshold; if so, outputting the trained initial feature point detection model as the trained feature point detection model; if not, adjusting the model parameters until the loss value reaches the preset loss threshold; The normalizing process of the coordinates of the sample feature points includes: obtaining the vertex coordinates of the sample expression, the vertex coordinates being the upper left corner coordinates, the lower left corner coordinates, the upper right corner coordinates or the lower right corner coordinates of the face frame in the sample expression; calculating the horizontal coordinate difference between the coordinates of the sample feature point and the vertex coordinates, and dividing the difference by the length value of the face frame, and calculating the vertical coordinate difference between the coordinates of the sample feature point and the vertex coordinates, and dividing the difference by the width value of the face frame.

2. The character facial expression splitting method according to claim 1, characterized in that: After deforming the face model according to the target face feature points to obtain a plurality of facial deformation expressions, the method includes: Creating a bone hierarchy structure that conforms to the human face model, and identifying joint points according to the bone hierarchy structure to obtain target joint points; Calculate control parameters based on the multiple facial deformation expressions and facial movement rules to obtain expression support parameters; The skin weights of the vertices bound to the bone hierarchy structure and the human face model at different target joint points are set, and the human face model and the bone hierarchy structure are skinned according to the skin weights, and the parameters of the expression control interface are adjusted according to the expression support parameters.

3. The character facial expression splitting method according to claim 2, characterized in that: After the skinning process is performed on the face model and the skeleton hierarchy structure, and the parameters of the expression control interface are adjusted according to the expression support parameters, the method further includes: A control expression corresponding to the expression control interface is generated through a preset control expression template, and the control expression is bound to the human face model.

4. A character facial expression splitting device, characterized in that: include: A topology module is used to obtain a 3D human face basic model and perform topology on the human face basic model to obtain a human face model including a topological structure; A detection module, used to detect the face model using a pre-trained feature point detection model to obtain target face feature points; The training process of the feature point detection model includes: obtaining a number of sample expressions; marking the feature points of the sample expressions to obtain the coordinates of the sample feature points, wherein the sample feature points include topological feature points and muscle endpoints; normalizing the coordinates of the sample feature points, inputting the normalization results into the initial feature point detection model, and training the initial feature point detection model using a preset loss function to obtain a trained feature point detection model; the training of the initial feature point detection model using the preset loss function to obtain the feature point detection model includes: judging whether the loss value of the loss function reaches a preset loss threshold; if so, Then the trained initial feature point detection model is output as the trained feature point detection model; if not, the model parameters are adjusted until the loss value reaches the preset loss threshold; the coordinates of the sample feature points are normalized, including: obtaining the vertex coordinates of the sample expression, the vertex coordinates are the upper left corner coordinates, lower left corner coordinates, upper right corner coordinates or lower right corner coordinates of the face frame in the sample expression; calculating the horizontal coordinate difference between the coordinates of the sample feature point and the vertex coordinates, and dividing it by the length value of the face frame, and calculating the vertical coordinate difference between the coordinates of the sample feature point and the vertex coordinates, and dividing it by the width value of the face frame; The deformation module is used to deform the face model of the human face according to the feature points of the target face to obtain multiple facial deformation expressions.

5. The character facial expression separation device according to claim 4, characterized in that: The character facial expression splitting device further comprises: An identification module is used to create a bone hierarchy structure that conforms to the human face model, and identify joint points according to the bone hierarchy structure to obtain target joint points; A calculation module, used for calculating control parameters based on the plurality of facial deformation expressions and facial movement rules to obtain expression support parameters; The skinning module is used to perform skinning on the human face model and the skeleton hierarchy structure, and adjust the parameters of the expression control interface according to the expression support parameters.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the character facial expression segmentation method according to any one of claims 1 to 3 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the character facial expression segmentation method as described in any one of claims 1 to 3.

Images