Face model generation method, device, storage medium and electronic device

Abstract

The present disclosure provides a method, device, storage medium and electronic device for generating a face model, and relates to the field of computer technology. The face model generation method comprises: obtaining a reference face baseline expression model and a target face sample expression model, establishing variables about the target face baseline expression model and variables about weights; adjusting the initial value of the weights according to the expression matching between the reference face weighted expression model and the sample expression model; establishing at least one energy function based on the difference between the target face weighted expression model and the sample expression model; alternately optimizing the target face baseline expression model by using the energy function under fixed weights, and optimizing the weights by using the energy function under fixed target face baseline expression model; and outputting the optimized target face baseline expression model. The present disclosure improves the generation efficiency of the face model and improves the model quality.

Description

Background Art

[0002] Face models are often used in film, television, games, animation and other fields. By controlling the face models, realistic facial movements or expressions can be presented.

[0003] In the related art, workers are usually required to manually create face models of specific characters with different expressions based on their experience. Obviously, this method requires high manpower and time costs and is inefficient. Summary of the invention

[0004] The present disclosure provides a face model generation method, a face model generation device, a computer-readable storage medium and an electronic device, thereby improving the problem of low face model generation efficiency at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to a first aspect of the present disclosure, a method for generating a face model is provided, comprising: obtaining a baseline expression model of a reference face and a sample expression model of a target face, establishing variables about the baseline expression model of the target face and variables about weights; adjusting the weights so that the expressions of the weighted expression model of the reference face and the sample expression model are successfully matched to obtain an initial value of the weight; the weighted expression model of the reference face is a model obtained by weighting the baseline expression model of the reference face by the weights; establishing at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model; the weighted expression model of the target face is a model obtained by weighting the baseline expression model of the target face by the weights; performing the following two optimizations alternately: optimizing the baseline expression model of the target face by using the energy function when the weights are fixed, and optimizing the weights by using the energy function when the baseline expression model of the target face is fixed; and outputting the optimized baseline expression model of the target face.

[0007] According to a second aspect of the present disclosure, a face model generation device is provided, comprising: a data acquisition module, configured to acquire a baseline expression model of a reference face and a sample expression model of a target face, and establish variables about the baseline expression model of the target face and variables about weights; an initial value determination module, configured to adjust the weights so that the weighted expression model of the reference face successfully matches the expression of the sample expression model to obtain the initial value of the weights; the weighted expression model of the reference face is a model obtained by weighting the baseline expression model of the reference face by the weights; an energy function establishment module, configured to establish at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model; the weighted expression model of the target face is a model obtained by weighting the baseline expression model of the target face by the weights; a data optimization module, configured to alternately perform the following two optimizations: optimizing the baseline expression model of the target face by using the energy function when the weights are fixed, and optimizing the weights by using the energy function when the baseline expression model of the target face is fixed.

[0008] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a face model according to the first aspect and possible implementation methods thereof are implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the face model generation method of the first aspect and possible implementation thereof by executing the executable instructions.

[0010] The technical solution disclosed in this disclosure has the following beneficial effects:

[0011] On the one hand, the automatic generation of the target face's benchmark expression model is realized, which greatly improves the efficiency compared with the manual production of the face model, avoids the human deviation caused by the manual production of the face model, and improves the accuracy of the model. On the other hand, the scheme can combine the expression features of the reference face's benchmark expression model with the appearance features of the target face's sample expression model to generate a benchmark expression model with the target face's appearance and the reference face's expression, which can present the facial details, enhance the model's realism, and improve the model's quality. In addition, by combining different reference face's benchmark expression models with the target face's sample expression models, the combination of different expressions and appearances can be achieved, thereby improving the diversity and flexibility of the generated model. On the other hand, the scheme sets two variables, the benchmark expression model and the weight of the target face, and adopts an alternating optimization method for iterative processing, which is conducive to accelerating the optimization process and further improving efficiency.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A flowchart of a method for generating a face model in this exemplary embodiment is shown;

[0015] Figure 2 A schematic diagram showing a reference expression model of a human face in this exemplary embodiment;

[0016] Figure 3 A schematic diagram showing a sample expression model of a target face in this exemplary embodiment;

[0017] Figure 4 A sub-flow chart showing a method for generating a face model in this exemplary embodiment;

[0018] Figure 5 A schematic diagram showing a local space corresponding to a triangular facet in this exemplary embodiment;

[0019] Figure 6 A schematic diagram showing weighting of local patch features in this exemplary embodiment is shown;

[0020] Figure 7A schematic diagram showing a reference expression model of a target face in this exemplary embodiment;

[0021] Figure 8 A schematic diagram showing the structure of a face model generating device in this exemplary embodiment is shown;

[0022] Fig. 9 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0024] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0025] In related technologies, workers usually use manual methods such as "pinching faces" to create specific characters. If you want to achieve realistic facial expression repositioning and reproduce details such as wrinkles and folds on the character's face, it will take a lot of time and be inefficient. Moreover, the resulting face model is usually less diverse and flexible.

[0026] In view of one or more of the above problems, the exemplary embodiment of the present disclosure first provides a method for generating a human face model. In the exemplary embodiment, human face is a broad concept, which can include faces of anthropomorphic animals, cartoon characters, etc. in addition to human faces. The human face model is a parameterizable and manipulable face model that can be used to create realistic digital human facial animation.

[0027] Combine the following Figure 1 The method for generating a face model is described. Figure 1 An exemplary process of a method for generating a face model is shown, comprising the following steps S110 to S150:

[0028] Step S110, obtaining a reference face baseline expression model and a target face sample expression model, and establishing variables related to the target face baseline expression model and variables related to weights;

[0029] Step S120, by adjusting the weights, the weighted expression model of the reference face is matched with the expression of the sample expression model successfully to obtain an initial value of the weight; the weighted expression model of the reference face is a model obtained by weighting the baseline expression model of the reference face by the weights;

[0030] Step S130, establishing at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model; the weighted expression model of the target face is a model obtained by weighting the reference expression model of the target face by weights;

[0031] Step S140, alternately performing the following two optimizations: optimizing the reference expression model of the target face using the energy function under the condition of fixed weights, and optimizing the weights using the energy function under the condition of fixed reference expression model of the target face;

[0032] Step S150, outputting the optimized reference expression model of the target face.

[0033] Based on the above method, on the one hand, the automatic generation of the target face's benchmark expression model is realized, which greatly improves the efficiency compared with the method of manually making the face model, avoids the human deviation caused by the manual face model, and improves the accuracy of the model. On the other hand, this scheme can combine the expression features of the benchmark expression model of the reference face with the appearance features of the sample expression model of the target face to generate a benchmark expression model with the appearance of the target face and the expression of the reference face, which can present the detailed information of the face, enhance the realism of the model, and improve the quality of the model. In addition, by combining different benchmark expression models of reference faces with sample expression models of the target face, the combination of different expressions and appearances can be realized, thereby improving the diversity and flexibility of the generated model. On the other hand, this scheme sets two variables, the benchmark expression model and weight of the target face, and adopts an alternating optimization method for iterative processing, which is conducive to accelerating the optimization process and further improving efficiency.

[0034] Below Figure 1 Each step in the procedure is described in detail.

[0035] refer to Figure 1In step S110, a baseline expression model of a reference face and a sample expression model of a target face are obtained, and variables about the baseline expression model of the target face and variables about weights are established.

[0036] Among them, the target face is a face for which a benchmark expression model is expected to be generated, and the reference face is a face for which a benchmark expression model has been generated and is used to provide reference information. The reference face and the target face can be faces of different images. The present disclosure does not limit the selection of the reference face, which can be a face of any image, for example, a face of a fixed image pre-created by a technician, or a face randomly selected from a face model database. In one embodiment, the reference face and the target face can be faces of the same type, such as both are male (or female) faces, both are faces of the same age group, both are faces of the same style of animated images, etc., so that the reference face can provide reference information that is more suitable for the target face, which is conducive to generating a high-quality benchmark expression model for the target face.

[0037] The reference expression model may be a set of representative expression models. In this exemplary embodiment, the reference expression model of the reference face may be represented as a set A = {A0, A1, ..., A n}, there are n+1 benchmark expression models in total, where n is a positive integer. Figure 2 The schematic diagram of the reference facial reference expression model is shown, including 5 reference facial reference expression models, which are respectively denoted as A0 to A4, wherein A0 is a neutral expression model, and A1 to A4 are other representative expression models.

[0038] In one embodiment, the reference face's baseline expression model may also include a base expression model and one or more offsets. The base expression model may be used as a basis for generating other base expression models, such as the initially generated base expression model. Usually, a neutral expression model may be used as the base expression model. The offset refers to the offset between the base expression model and other base expression models. Exemplarily, the reference face's baseline expression model may be represented as a set A'={A0, a1, ..., a n}, where a1=A1-A0, represents the offset between the reference expression models A0 and A1, including the offset of key facial points between A0 and A1, the offset of local face patches, the offset of key facial parts, etc. It can be seen that each offset can correspond to a specific reference expression model, that is, the offset is equivalent to the reference expression model, and the set A is equivalent to the set A'.

[0039] In general, by fusing different baseline expression models of the same face, any other expression model can be generated. jTo represent any expression model of a reference face, you can refer to the following formula:

[0040]

[0041]

[0042] Formula (1) shows that by weighting different baseline expression models, any expression model can be obtained. i represents any baseline expression model of the reference face, α ij is the weight of the base expression model. Formula (2) shows that by weighting the base expression model and the offset, any expression model can be obtained. i represents any offset, β ij is the weight for weighting the offset. Formula (2) is actually weighting different baseline expression models, so the weight β ij With α ij The meanings expressed are essentially the same, ij It is used to weight the offset, which is essentially equivalent to weighting the baseline expression model. Formula (2) can reduce the number of weights (i.e., β) by converting the baseline expression model weighting in formula (1) into the calculation method of offset weighting. ij The dimension is smaller than α ij ), which is helpful to simplify the calculation. Therefore, the following description will be based on formula (2), and the weights in the following can be β ij .

[0043] The sample expression model of the target face is used to provide samples for generating the reference expression model of the target face. It is one or more existing expression models of the target face, and can be any expression model of the target face obtained by hand or other means. The expression of the sample expression model and the reference expression model can be the same or different.

[0044] In one embodiment, the sample expression model can be a basic expression, an expression with less facial element changes, or a random face model under one or more expressions, so as to facilitate the preparation by technicians. For example, the sample expression model of the target face can be represented as a set S = {S1, S2, ..., S m}, m is a positive integer, S j Can be used to represent any of the sample expression models. Figure 3 A schematic diagram of sample expression models of the target face is shown, wherein sample expression model S1 is an expression with eyes open and mouth closed, sample expression model S2 is an expression with eyes closed and mouth closed, and sample expression model S3 is an expression with eyes open and mouth open.

[0045] The present disclosure does not limit the number of sample expression models of the target face. When the number of sample expression models is small or even only one, the exemplary embodiment can also generate a baseline expression model of the target face based on the baseline expression model of the reference face and the sample expression model of the target face, that is, generate a face model of the target face that has the same (or similar) expression as the baseline expression model of the reference face. Furthermore, any expression model of the target face can also be generated based on the baseline expression model of the target face, thereby obtaining a large number of other expression models from a small number of sample expression models of the target face.

[0046] In this exemplary embodiment, variables about the target face's baseline expression model and variables about weights can be established, thereby converting the problem of face model generation into a variable optimization problem. In one embodiment, a first variable and a second variable can be constructed. Among them, the first variable is used to represent the target face's baseline expression model, and the second variable is used to represent the above-mentioned weight. By optimizing the first variable and the second variable, the baseline expression model of the target face and the baseline expression model of the reference face have the same (or similar) expression, and the sample expression model can be obtained by weighted fusion of the target face's baseline expression model using weights.

[0047] For example, the set B = {B0, B1, ..., B n} or B'={B0,b1,…,b n} represents the target face's baseline expression model (i.e., the first variable), where B0~B n is each benchmark expression model of the target face, B0 is the basic expression model, b1~b n For the standard expression model B1~B n is the corresponding offset. ij (or α ij ) represents the weight (i.e., the second variable), which can satisfy the following conditions:

[0048]

[0049]

[0050] Where, formula (3) represents A i With B i There is an expression matching relationship, that is, the two have the same (or similar) expression or semantics, such as A4 and B4 both have open mouth expressions. j The weighted expression model of the target face is obtained by weighting the reference expression model of the target face. It can be any expression model of the target face and is different from the weighted expression model T of the reference face. jIt should be understood that the method of formula (1) (that is, not using the offset but directly using the target face's reference expression model) can also be used to calculate U j Formula (4) indicates that the target face's baseline expression model is weighted by appropriate weights, and the obtained weighted expression model of the target face is the same as or similar to the sample expression model of the target face. Therefore, the target face's baseline expression model, weights, and sample expression model can satisfy a linear relationship.

[0051] Therefore, the problem of generating a benchmark expression model for the target face can be expressed as the following mathematical problem: Under the constraints of formulas (3) and (4), optimize the first variable and the second variable, and finally obtain the optimized first variable, that is, the benchmark expression model of the target face.

[0052] Continue to refer Figure 1 In step S120, the weighted expression model of the reference face is successfully matched with the expression of the sample expression model by adjusting the weights to obtain the initial value of the weights; the weighted expression model of the reference face is a model obtained by weighting the baseline expression model of the reference face by the weights.

[0053] Among them, the weighted expression model T of the reference face j It can be calculated by formula (1) or formula (2). It can be seen that by adjusting the weight value, the weighted expression model of the reference face can be adjusted. In this exemplary embodiment, it is expected that by adjusting the weight value, the weighted expression model of the reference face and the expression of the sample expression model can be successfully matched, and the adjusted weight value is used as the initial value of the weight in subsequent processing, which will be further optimized in subsequent steps.

[0054] The weighted expression model of the reference face successfully matches the expression of the sample expression model, or the weighted expression model of the reference face has an expression matching relationship with the sample expression model, which means that the expression of the weighted expression model of the reference face is the same (or similar) as the expression of the sample expression model, and specifically, the semantics of the expression may be the same (or similar). For example, the sample expression model S2 is an expression of eyes and mouth closed. The reference face's baseline expression models A0 to A4 are weighted by appropriate weight values ​​to obtain the weighted expression model of the reference face. If the weighted expression model is also an expression of eyes and mouth closed, then it can successfully match the expression of the sample expression model S2.

[0055] Combining formulas (2) to (4), we can obtain the following relationship:

[0056]

[0057] That is to say, by adjusting the weights to appropriate values, the weighted expression model of the reference face and the sample expression model can achieve successful expression matching.

[0058] In one embodiment, for two expression models, the expression similarity between the two expression models can be calculated to determine whether the expressions of the two expression models are matched successfully. For example, when the expression similarity between the weighted expression model of the reference face and the sample expression model reaches a preset expression similarity threshold, it is determined that the expression of the weighted expression model and the sample expression model is matched successfully, and the weight value at this time is used as the initial value of the weight. Alternatively, by iteratively adjusting the weight value, the expression similarity between the weighted expression model of the reference face and the sample expression model reaches the maximum value, that is, the two expression models achieve the optimal expression match, and the weight value at this time is used as the initial value of the weight.

[0059] The present disclosure does not limit how to calculate the expression similarity. Exemplarily, the weighted expression model of the reference face (or one or more two-dimensional images of the weighted expression model) and the sample expression model (or one or more two-dimensional images of the sample expression model) can be respectively input into a pre-trained expression classification model to obtain the expression classification result corresponding to the weighted expression model and the expression classification result corresponding to the sample expression model. The expression classification result can be in the form of a vector, including probability values ​​corresponding to each standard expression; then the similarity between the expression classification result corresponding to the weighted expression model and the expression classification result corresponding to the sample expression model is calculated, that is, the expression similarity of the two expression models is obtained. Therefore, the expression similarity can characterize the degree of similarity in expression without being affected by differences in appearance. Even if the faces corresponding to the two models have different appearances, a more objective and accurate expression similarity can be calculated.

[0060] In this exemplary embodiment, the second variable can be adjusted under the constraint of formula (5), and the adjusted weight can be used as the initial value of the weight in subsequent processing, thereby providing a good basis for subsequent processing.

[0061] The above describes how to obtain the initial value of the weight. The present disclosure does not limit how to obtain the initial value of the reference expression model of the target face, and an exemplary description is given below.

[0062] In one implementation, the values ​​in the sample expression model of the target face may be adjusted, such as randomly adjusted, and the obtained expression model may be used as the initial value of the benchmark expression model of the target face.

[0063] In one embodiment, different sample expression models of the target face may be fused, such as weighted fusion of multiple sample expression models using random or preset weight values, and the obtained expression model may be used as the initial value of the benchmark expression model of the target face.

[0064] In one implementation, a general face model may be used as the initial value of the reference expression model of the target face.

[0065] In addition to determining appropriate initial values ​​for variables, you can also establish energy functions for the variables. Figure 1 In step S130, at least one energy function is established based on the difference between the weighted expression model of the target face and the sample expression model; the weighted expression model of the target face is a model obtained by weighting the baseline expression model of the target face by weights.

[0066] Among them, the weighted expression model U of the target face j It can be calculated by formula (4). The goal of optimizing variables is to satisfy the constraints of formula (3) and formula (4). Ideally, in formula (4), the weighted expression model U of the target face can be obtained. j With sample expression model S j Therefore, an energy function can be established based on the difference between the weighted expression model of the target face and the sample expression model, and the energy function can be used to optimize the first variable and the second variable.

[0067] The energy function may adopt any type of norm. In this exemplary embodiment, the F norm will be used as an example for illustration, but the present disclosure does not limit this. For example, when the energy function is used to represent the difference between vectors, the L1 norm, L2 norm, etc. of the vector difference may also be used to establish the energy function. The energy function using the F norm may be as follows:

[0068]

[0069] Among them, the sample expression model S j The weighted expression model U of the target face j Both can be in the form of matrices. The dimensions of the two matrices are the same. Subtract the two matrices, calculate the sum of the squares of each element, and then calculate the square root to get the energy function value. j with U j The greater the difference, the greater the energy function value.

[0070] Continue to refer Figure 1 In step S140, the following two optimizations are performed alternately: optimizing the baseline expression model of the target face using the energy function with fixed weights, and optimizing the weights using the energy function with fixed baseline expression model of the target face.

[0071] Among them, by optimizing the reference expression model (first variable) and weight (second variable) of the target face, the energy function value is made as small as possible, so as to be as close as possible to satisfying the constraint of formula (4). If the first variable and the second variable are optimized at the same time, it may be difficult to converge. Therefore, this exemplary embodiment alternately fixes one of the first variable and the second variable and optimizes the other, which can achieve rapid convergence and improve optimization efficiency.

[0072] It should be understood that the present disclosure can perform any number of iterative alternating optimizations on the first variable and the second variable, and the conditions for the end of the optimization may include but are not limited to: the first variable or the second variable converges; the first variable and the second variable both converge; the energy function converges; the set number of iterative rounds is reached; the first variable and the second variable are successfully verified, such as the first variable and the second variable can be verified using formulas (4) and (5); and so on.

[0073] In one embodiment, the at least one energy function established in step S130 may include a first energy function and a second energy function. Figure 4 As shown, the above-mentioned difference between the weighted expression model of the target face and the sample expression model is used to establish at least one energy function, which may include the following steps S410 and S420:

[0074] Step S410, establishing a first energy function based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model;

[0075] Step S420, establishing a second energy function based on the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model;

[0076] The above-mentioned process of optimizing the target face's reference expression model by using an energy function under a fixed weight, and optimizing the weight by using an energy function under a fixed target face's reference expression model, may include the following steps S430:

[0077] Step S430, optimizing the reference expression model of the target face by using the first energy function while fixing the weights, and optimizing the weights by using the second energy function while fixing the reference expression model of the target face.

[0078] It should be understood that step S430 includes two optimizations: optimizing the baseline expression model of the target face and optimizing the weights, and these two optimizations can be performed alternately.

[0079] Generally, the weighted expression model of the target face and the sample expression model contain more information, and part of the information can be selected to calculate the energy function, which can simplify the calculation of the energy function and improve efficiency. The first type of features and the second type of features are both used to represent part of the information in the weighted expression model of the target face or the sample expression model. In this exemplary embodiment, the goal of optimizing the variables is to make the weighted expression model of the target face equal to the sample expression model, which means that the first type of features of the weighted expression model of the target face are the same as the first type of features of the sample expression model, and the second type of features of the weighted expression model of the target face are the same as the second type of features of the sample expression model. Therefore, based on the difference between the weighted expression model of the target face and the sample expression model in the first type of features and the difference in the second type of features, energy functions can be established respectively, and the first energy function and the second energy function can be obtained accordingly.

[0080] The first type of features and the second type of features can be different information, so that the first energy function and the second energy function are different energy functions. Furthermore, when the first energy function is used to optimize the baseline expression model of the target face, and the second energy function is used to optimize the weight, different information in the weighted expression model of the target face and the sample expression model is actually used, avoiding the repeated use of the same information, which is conducive to accelerating the optimization process.

[0081] In one embodiment, the first type of features and the second type of features may be any one or more of the following features: point features, line features, and surface features. These features are described below.

[0082] Point Features

[0083] Point features can be features of any point on the weighted expression model of the target face (or the sample expression model, any features on the weighted expression model of the target face mentioned below are applicable to the sample expression model, so the sample expression model will not be described in detail). Usually, key points can be selected in the weighted expression model of the target face, and the features of the key points can be obtained as point features of the weighted expression model. The present disclosure does not limit how to select key points, the number and type of key points, etc.

[0084] In one implementation, the key points may be key points of a face in a weighted expression model of the target face, such as key points of eyes, mouth, nose, ears, facial contours, and the like.

[0085] In one embodiment, the key points may also be the key points in the local patches of the weighted expression model of the target face. A local patch refers to a two-dimensional plane unit that forms a facial model (such as the reference expression model, sample expression model, and weighted expression model in this exemplary embodiment), such as a triangular patch. Of course, the present disclosure does not limit the shape, size, etc. of the local patch. The key points of the local patch may include the vertices, center points, etc. of the local patch. Exemplarily, the weighted expression model of the target face is composed of triangular patches, and the vertices of the triangular patches may be used as key points to obtain the features of the key points.

[0086] After selecting key points, any type of features of the key points can be obtained, including features at different levels such as texture, image, and model structure. For example, algorithms such as FAST (Features From Accelerated Segment Test), BRIEF (Binary Robust Independent Elementary Features), ORB (Oriented FAST and Rotated BRIEF), SIFT (Scale-Invariant Feature Transform), and SURF (Speeded UpRobust Features) can be used to describe key point features. In addition, the above algorithms can also be used to extract key points from the weighted expression model of the target face.

[0087] In one embodiment, the key point position can also be used as a point feature. The key point position can be the position coordinate of the key point in the three-dimensional space where the weighted expression model of the target face is located, which can reflect the structural information of the weighted expression model, and the key point position is easy to obtain information, which is conducive to simplifying the processing process. Exemplarily, a certain number of key points are selected from the vertices of the triangular face of the weighted expression model of the target face, and the three-dimensional position coordinates of these key points are formed into a matrix to serve as the point feature of the weighted expression model of the target face.

[0088] Line Features

[0089] The line feature can be the feature of any line on the weighted expression model of the target face. The selected line can be the edge of a local patch, a diagonal line, a wrinkle line of the face, a line of a key part of the face, etc.

[0090] The present disclosure may use any description method to describe line features. In one embodiment, the position and posture of a line may be used as a line feature. For example, the direction of a line may be described by 3-dimensional data, and the distance from the origin to the line may be described by 1-dimensional data, so the line feature may be represented as 4-dimensional data; or, the line feature may be represented by the position coordinates of the two end points of the line, which is 6-dimensional data, and so on.

[0091] Face Features

[0092] The face feature may be a feature of any face on the weighted expression model of the target face. The selected face may be a local plane or curved surface in the weighted expression model of the target face, which is not limited in the present disclosure.

[0093] In one embodiment, the face feature may include a local patch feature. For example, the weighted expression model of the target face is composed of triangular patches, the local patches may be triangular patches, and the local patch features may be triangular patch features. The local patches are the basic components of the weighted expression model of the target face, so the local patch features are easy-to-obtain information, which helps to simplify the processing process. The local patch features may include the posture, texture features, image features, etc. of the local patches. If the weighted expression model of the target face is an unrendered model, the local patch features may be obtained based on the posture of the local patches, which can reflect the structural information of the weighted expression model.

[0094] In one embodiment, the local patch feature can be represented based on the local space corresponding to the local patch. The local space can be the space formed by the local patch and the normal vector of the local patch. Figure 5 Take the sample expression model S1 as an example to illustrate that the local face patch can be a triangular face patch. For the triangular face patch with vertices v1, v2, and v3, Figure 5 It is enlarged and displayed in the figure. The local space M corresponding to the triangle is formed by the plane where the triangle is located and the normal vector of the triangle, which can be expressed as a 3*3 matrix M=[v3-v1, v2-v1, z], where v3-v1 and v2-v1 are the two edge vectors of the triangle, and z=(v3-v1)×(v2-v1) is the normal vector of the triangle. M can be used as the feature of the triangle, that is, the local patch feature. In addition, the position of at least one vertex can be added to the above 3*3 matrix, such as adding the position coordinates of v1, to form a 3*4 matrix. This matrix can more accurately reflect the posture of the triangle and can be used as a local patch feature.

[0095] The above describes three types of features. Both the first and second types of features can be any one of the above features, or a combination of any number of features. The first and second types of features can be features of the same type but different objects. For example, both the first and second types of features are point features, but the points selected by the two can be different, such as the first type of feature is the feature of the local face vertex, and the second type of feature is the feature of the key point of the face. The first and second types of features can also be features of different types, for example, the first type of feature is a face feature, and the second type of feature is a point feature, and so on.

[0096] The first energy function is further explained below.

[0097] In one embodiment, the first type of feature may include a local patch feature, such as a local space corresponding to a triangular patch. The difference between the first type of feature of the weighted expression model of the target face and the first type of feature of the sample expression model is used to establish the first energy function, which may include the following steps:

[0098] Obtaining local patch features of a weighted expression model of the target face, and obtaining local patch features of a sample expression model;

[0099] A first energy function is established based on the difference between the local patch features of the weighted expression model of the target face and the local patch features of the sample expression model.

[0100] For example, Represents the weighted expression model U of the target face j The local patch features of The sample expression model S representing the target face j The local patch features can establish the first energy function as follows:

[0101]

[0102] It should be understood that when calculating the difference between the local patch features of the two models, the difference between the local patch features with corresponding relationships in the two models can be calculated. For example, an index (or identifier) ​​can be generated for each local patch, and then the difference between the local patch features with the same index in the weighted expression model of the target face and the sample expression model is calculated to obtain the first energy function value. For ease of expression, the index of the local patch is ignored in formula (7).

[0103] The first energy function E AIt reflects the difference between the local patch features of the weighted expression model of the target face and the local patch features of the sample expression model. Compared with the entire model, the local patch features have less information, which is conducive to reducing the amount of calculation for subsequent processing. The local patch features can more comprehensively reflect the structural information of the weighted expression model or the sample expression model of the target face. The difference in local patch features can more accurately reflect the structural difference between the weighted expression model of the target face and the sample expression model. The structural difference is essentially the difference in facial appearance and expression. In the subsequent processing, by optimizing the first variable and the second variable, the value of the first energy function is made as small as possible to ensure that the local patch features of the weighted expression model of the target face are as close to the local patch features of the sample expression model as possible, which is basically equivalent to making the weighted expression model of the target face as close to the sample expression model as possible, thereby reducing the amount of calculation while ensuring the optimization effect.

[0104] In one embodiment, the above-mentioned obtaining of the local patch features of the weighted expression model of the target face may include the following steps:

[0105] The local patch features of the benchmark expression model of the target face are weighted by the weight (second variable) to obtain the local patch features of the weighted expression model of the target face.

[0106] That is, the local patch features of the target face's baseline expression model and the local patch features of the target face's weighted expression model can satisfy a linear relationship, which can be equivalent to the linear relationship between the target face's baseline expression model and the target face's weighted expression model. For example, referring to formula (6), the target face's baseline expression model {B0, b1, ..., b n} and the weighted expression model U of the target face j The following linear relationship can be satisfied:

[0107]

[0108] Then, the baseline expression model {B0, b1, ..., b n} local patch features With the weighted expression model U j Local patch features The following linear relationship can be satisfied:

[0109]

[0110] in, Represents the local patch features of the neutral expression model B0 of the target face, Indicates the offset b of the target face i The local patch features (offset b iIt can be regarded as a baseline expression model and therefore can have local patch features).

[0111] Combination Figure 6 For further explanation. Figure 6 As shown, it is assumed that the weighted expression model U2 of the target face can be obtained by weighting the benchmark expression models B0, b1, and b2 of the target face, as can be seen from formula (8). The local patch features with the same index are extracted from the benchmark expression models B0, b1, and b2 respectively. And extract the local patch features with the same index from the weighted expression model U2 but Can be The weights are the same as the weights of the weighted expression model U2 obtained by weighting the reference expression models B0, b1, and b2. Thus, the local patch features in the weighted expression model U2 can be obtained by weighted calculation between the local patch features of the reference expression model without calculating the weighted expression model U2.

[0112] It should be understood that it is not limited to local patch features. Since there is a linear relationship between the baseline expression model and the weighted expression model, any type of features in the baseline expression model and the weighted expression model can also satisfy the linear relationship and can be calculated in a manner similar to formula (9).

[0113] Based on formula (9), the first energy function E A It is expressed as follows:

[0114]

[0115] Therefore, when calculating the first energy function value and optimizing it, there is no need to calculate the weighted expression model of the target face through the baseline expression model of the target face. Instead, the local patch features of the weighted expression model can be directly calculated through the local patch features of the baseline expression model of the target face, thereby further simplifying the calculation process and improving efficiency.

[0116] As can be seen from the above, the first energy function is used to represent the difference between the weighted expression model of the target face and the sample expression model, which can be reflected as the difference between the complete models or the difference between the first-class features of the models (such as local patch features). In the case of fixed weights, the first energy function can be used to iteratively adjust the benchmark expression model of the target face so that the first energy function is as small as possible and tends to 0, thereby obtaining a benchmark expression model of the target face that can meet the above constraints.

[0117] In one embodiment, the above-mentioned optimization of the reference expression model of the target face by using the first energy function under the condition of fixed weights may include the following steps:

[0118] Under a fixed weight, the first energy function is used to optimize the local patch features of the benchmark expression model of the target face;

[0119] The optimized target face benchmark expression model is reconstructed according to the local patch features of the optimized target face benchmark expression model.

[0120] That is, when optimizing the target face's baseline expression model, the local patch features of the target face's baseline expression model can be directly optimized. In particular, when using the first energy function established with the local patch features, such as formula (10), the local patch features are usually directly adjusted during optimization. and Then, the optimized and Reconstruct the optimized target face's baseline expression model {B0, b1, ..., b n In this way, during the optimization process, the adjustment of local patch features is a local adjustment of the model. Compared with the global adjustment of the benchmark expression model of the target face, it can improve the accuracy and efficiency of the adjustment and accelerate the convergence.

[0121] In one implementation, the baseline expression model of the target face can be reconstructed by the least square method, referring to the following formula:

[0122]

[0123] Where X represents the vertex position in the benchmark expression model of the target face to be solved (such as the vertex position of all triangular patches), including h represents the number of vertices in the model. is the optimized local patch feature (the formula is simplified and omitted ). P is used to associate X with Sparse coefficient matrix. Let the function gradient in formula (11) be zero, and the normal equation can be obtained as follows:

[0124]

[0125] Thus, the optimized benchmark expression model of the target face is obtained.

[0126] In one embodiment, the Cholesky decomposition of the sparse coefficient matrix can also be used to reconstruct the benchmark expression model of the target face, which can accelerate the reconstruction process and reduce the processing time of a single benchmark expression model from 400s to 0.22s, thereby greatly improving efficiency.

[0127] In one embodiment, the at least one energy function may include a first energy function. The at least one energy function established based on the difference between the weighted expression model of the target face and the sample expression model may include the following steps:

[0128] A first energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face; the expression gradient of the reference face is the gradient between the two benchmark expression models of the reference face, and the expression gradient of the target face is the gradient between the two benchmark expression models of the target face.

[0129] It should be understood that, compared with the first energy function in formula (7) or (10), the first energy function adds the difference between the expression gradient of the reference face and the expression gradient of the target face. That is, the first energy function may include two parts: one part is the difference between the weighted expression model of the target face and the sample expression model, which may be referred to as the first data item. The first data item may be, for example, E0 in formula (6) or E in formula (7) and (10). A , for easy distinction, the first data item is recorded as E fit1 The other part is the difference between the expression gradient of the reference face and the expression gradient of the target face, which can be called the first regularization term. The two parts are combined to form the first energy function, which can be combined in a weighted manner.

[0130] After adding the first regularization term to the first energy function, the first energy function is used to optimize the baseline expression model of the target face, so that the expression gradient of the target face can be equal to or close to the expression gradient of the reference face. As a result, the baseline expression model of the target face can learn the information in the baseline expression model of the reference face, which is conducive to achieving that the baseline expression model of the target face has the same expression as the baseline expression model of the reference face.

[0131] The expression gradient of the reference face is the gradient between the two reference expression models of the reference face. For example, the expression gradient of the reference face can be recorded as G(A i →A j ), represents any two baseline expression models A i With A j The expression gradient of the target face is the gradient between the two reference expression models of the target face. For example, the expression gradient of the reference face can be recorded as G(B i →B j ), represents any two baseline expression models B i With B j The gradient between .

[0132] In one implementation, the expression gradient may be equivalent to the above-mentioned offset.

[0133] In one embodiment, the expression gradient of the reference face may include the variation of the first type of feature between the two reference expression models of the reference face, and the expression gradient of the target face may include the variation of the first type of feature between the two reference expression models of the target face. Exemplarily, the first type of feature may include a local patch feature, such as a local space corresponding to a triangular patch. i To the baseline expression model A j , source domain (i.e. A i ) is mapped to the target domain (i.e. A j ) of the triangular patch t, ​​the change in the local patch feature can be expressed as:

[0134]

[0135] Among them, M s 、M t are the local patch features of triangle patches s and t respectively.

[0136] It can be assumed that, ideally, the expression gradient of the reference face is the same as or very close to the expression gradient of the target face, that is, G(A i →A j )≈G(B i →B j ), where i and j can be any positive integers from 0 to n. In one embodiment, when calculating the expression gradient between two reference expression models, one of the reference expression models can be fixed as a neutral expression model, and the other reference expression model can be expressed as the sum of the neutral expression model and the offset, then G(A0→A0+a i )≈G(B0→B0+b i ), where i is any positive integer from 1 to n. The expression gradient is represented by the change in local patch features. The change in the local patch features of the reference face is the same as or very close to the change in the local patch features of the target face, and the following relationship can be obtained:

[0137]

[0138]

[0139]

[0140]

[0141] in, Represents the local patch features of the neutral expression model A0 of the reference face, Indicates the offset a of the reference face i Local patch features; Represents the local patch features of the neutral expression model B0 of the target face, Indicates the offset b of the target face i Based on the difference between the two sides of “≈” in formula (14), the first regularization term can be established as follows:

[0142]

[0143]

[0144] Among them, ω i Represents the offset weight, which is used to control the proportion of the difference in the local patch feature changes corresponding to different offsets in the first regularization term. It can be calculated by the following formula:

[0145]

[0146] Among them, κ is a denominator adjustment parameter greater than 0, which is used to avoid the denominator term in formula (16) being 0; θ is an exponential parameter, which is used to adjust the degree of nonlinearity of formula (16). The values ​​of both can be determined based on experience or actual conditions. For example, κ=0.1, θ=2. In general, the larger the offset of the reference face, the larger the offset between the corresponding baseline expression model and the neutral expression model, and the smaller the corresponding offset weight. Therefore, the first regularization term tends to make the expression gradient of the reference face the same as or very close to the expression gradient of the target face when the offset is small.

[0147] In one embodiment, the first energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face, and may include the following steps:

[0148] A first energy function is established based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face.

[0149] That is, the first data item in the first energy function can be the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model. fit1 It can be E in formula (7) or (10) A Compared with the difference that the first data item adopts a complete model, this exemplary embodiment can simplify the amount of information and the calculation process of the first data item, thereby improving efficiency.

[0150] In one implementation, the first data item and the first regularization item may be weighted and combined to obtain a first energy function, as shown below:

[0151] E A =E fit1 +γE reg1 (17)

[0152] Wherein, γ is the weight of the first regularization term, which is used to adjust the proportion of the first regularization term in the first energy function to balance the first data term and the first regularization term. The value can be set according to experience or actual conditions, and the present disclosure does not limit this.

[0153] Exemplarily, the first energy function may be as follows:

[0154]

[0155] By using the first energy function, the local patch features of the target face's baseline expression model can be optimized, and then the optimized target face's baseline expression model can be reconstructed based on the optimized local patch features. For example, the algorithm of formulas (11) and (12), or the Cholesky decomposition algorithm of the sparse coefficient matrix can be used for reconstruction.

[0156] The first energy function is described above. The second energy function is further described below.

[0157] In one embodiment, the second type of features include key point features, the key points may be vertices of local patches, and the key point features may be vertex positions; the difference between the second type of features of the weighted expression model based on the target face and the second type of features of the sample expression model is used to establish the second energy function, which may include the following steps:

[0158] Obtain key point features of the weighted expression model of the target face, and obtain key point features of the sample expression model;

[0159] A second energy function is established based on the difference between the key point features of the weighted expression model of the target face and the key point features of the sample expression model.

[0160] For example, Represents the weighted expression model U of the target face j The key point features are The sample expression model S representing the target face j The key point features can establish the second energy function as follows:

[0161]

[0162] It should be understood that when calculating the difference between the key point features of the two models, the difference between the key points with corresponding relationships in the two models can be calculated. For example, an index (or identifier) ​​can be generated for each key point, and then the difference between the key point features with the same index in the weighted expression model of the target face and the sample expression model is calculated to obtain the second energy function value. For ease of expression, the index of the key point is ignored in formula (19).

[0163] The second energy function E B It reflects the difference between the key point features of the weighted expression model of the target face and the key point features of the sample expression model. Compared with the entire model, the key point features have less information, which is conducive to reducing the amount of calculation in subsequent processing. The key point features can more comprehensively reflect the structural information of the weighted expression model or the sample expression model of the target face. The difference in key point features can more accurately reflect the structural difference between the weighted expression model of the target face and the sample expression model. The structural difference is essentially the difference in facial appearance and expression. In the subsequent processing, by optimizing the first variable and the second variable, the value of the second energy function is as small as possible to ensure that the key point features of the weighted expression model of the target face are as close to the key point features of the sample expression model as possible, which is basically equivalent to making the weighted expression model of the target face as close to the sample expression model as possible, thereby reducing the amount of calculation while ensuring the optimization effect.

[0164] In one embodiment, the above-mentioned acquisition of key point features of the weighted expression model of the target face may include the following steps:

[0165] The key point features of the benchmark expression model of the target face are weighted by the weight (second variable) to obtain the key point features of the weighted expression model of the target face.

[0166] From the above formulas (8) and (9) and their related explanations, it can be seen that any type of features in the target face's baseline expression model and the target face's weighted expression model can satisfy the linear relationship between the models, which means that the key point features of the target face's baseline expression model and the key point features of the target face's weighted expression model also satisfy the linear relationship between the target face's baseline expression model and the target face's weighted expression model. For example, referring to the target face's baseline expression model {B0, b1, ..., b n} and the weighted expression model U of the target face j The linear relationship between the baseline expression model {B0, b1, …, b n Key point features of With the weighted expression model U j Key point features The following linear relationship can be satisfied:

[0167]

[0168] in, Represents the key point features of the neutral expression model B0 of the target face, Indicates the offset b of the target face i The key point features (offset b i can be considered as a baseline expression model and thus can have key point features).

[0169] Based on formula (20), the second energy function E B It is expressed as follows:

[0170]

[0171] Therefore, when calculating the value of the second energy function and performing optimization, there is no need to calculate the weighted expression model of the target face through the baseline expression model of the target face. Instead, the key point features of the weighted expression model can be directly calculated through the key point features of the baseline expression model of the target face, thereby further simplifying the calculation process and improving efficiency.

[0172] As can be seen from the above, the second energy function is used to represent the difference between the weighted expression model of the target face and the sample expression model, which can be reflected as the difference between the complete models or the difference between the second type of features of the models (such as key point features). In the case of a fixed reference expression model of the target face, the second energy function can be used to iteratively adjust the weights so that the second energy function is as small as possible and tends to 0, thereby obtaining weights that can meet the above constraints.

[0173] In one embodiment, the above-mentioned optimization of weights by using the second energy function under the condition of fixing the reference expression model of the target face may include the following steps:

[0174] The weights are optimized using a second energy function while fixing the key point features of the reference expression model of the target face.

[0175] Fixing the target face's baseline expression model means fixing the key point features of the target face's baseline expression model. In this case, optimizing the weights using the second energy function is beneficial to improving optimization efficiency.

[0176] In one embodiment, the at least one energy function may include a second energy function. The at least one energy function established based on the difference between the weighted expression model of the target face and the sample expression model may include the following steps:

[0177] A second energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the current value of the weight and the initial value of the weight.

[0178] It should be understood that, compared with the second energy function in formula (19) or (21), the second energy function adds the difference between the current value of the weight and the initial value of the weight. That is, the second energy function may include two parts: one part is the difference between the weighted expression model of the target face and the sample expression model, which may be referred to as the second data item. The second data item may be, for example, E0 in formula (6) or E0 in formula (19) or (21). B , for easy distinction, the second data item is recorded as E fit2 The other part is the difference between the current value of the weight and the initial value of the weight, which can be called the second regularization term. The two parts are combined to form the second energy function, which can be combined in a weighted manner.

[0179] Exemplarily, the second regularization term may be as follows:

[0180]

[0181] Among them, β ij represents the current value of the weight, Represents the initial value of the weight.

[0182] After adding the second regularization term to the second energy function, the weights are optimized using the second energy function, so that the weights tend to be adjusted near their initial values, that is, they will not deviate too much from the initial values, thereby better satisfying the constraints of formula (3), which is beneficial to improving the accuracy of the weights, and then in the subsequent optimization of the benchmark expression model of the target face, the benchmark expression model of the target face is achieved to have the same expression as the benchmark expression model of the reference face.

[0183] In one embodiment, the second energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the current value of the weight and the initial value of the weight, and may include the following steps:

[0184] A second energy function is established based on the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model, and the difference between the current value of the weight and the initial value of the weight.

[0185] That is, the second data item in the second energy function can be the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model. fit2 It can be E in formula (19) or (21) B Compared with the difference that the second data item adopts a complete model, this exemplary embodiment can simplify the amount of information and the calculation process of the second data item, thereby improving efficiency.

[0186] In one implementation, the second data term and the second regularization term may be weighted and combined to obtain a second energy function, as shown below:

[0187] E B =E fit2 +δE reg2 (twenty three)

[0188] Wherein, δ is the weight of the second regularization term, which is used to adjust the proportion of the second regularization term in the second energy function to balance the second data term and the second regularization term. Its value can be set according to experience or actual conditions, and the present disclosure does not limit this.

[0189] Exemplarily, the second energy function may be as follows:

[0190]

[0191] Among them, k represents the index of the vertex (such as the vertex of the triangle patch) in the sample expression model. In the benchmark expression model of the target face, the vertices have the same index; h is the number of vertices in the sample expression model. In the benchmark expression model of the target face, the number of vertices is also h. Represents the sample expression model S j The position of vertex k in represents the position of vertex k in the neutral expression model B0 of the target face, Indicates the offset b of the target face i The position of vertex k in (offset b i It can be regarded as a baseline expression model and thus can have vertices. The positions of the vertices can be used as key point features or second-class features.

[0192] In one implementation, since the weights are usually constrained to be within the range of 0 to 1, a quadratic programming method may be used to solve and optimize the weights.

[0193] In one embodiment, the above-mentioned optimization of weights by using an energy function under the condition of fixing the reference expression model of the target face may include the following steps:

[0194] When the target face's baseline expression model is fixed, the weights are optimized using the energy function and the mutually exclusive relationship between the target face's baseline expression models.

[0195] The mutually exclusive relationship between the target face's benchmark expression models means that when the target face's benchmark expression models are used to fuse and generate other expression models, some mutually exclusive benchmark expression models will not contribute to a specific expression at the same time. For example, the expressions of the jaw being tilted to the left and the jaw being tilted to the right will not appear at the same time. Therefore, when fusing and generating any other expression model, at least one of the weights of the benchmark expression model of the jaw being tilted to the left and the benchmark expression model of the jaw being tilted to the right is 0, which can be expressed as β pj β qj =0, p and q are the indexes of the baseline expression model with the jaw tilted to the left and the baseline expression model with the jaw tilted to the right, respectively, and j represents the index of any expression model. The mutually exclusive relationship can be set by the technician based on life experience. Therefore, the mutually exclusive relationship is equivalent to adding additional constraints to the weights, and the weights can be optimized using the energy function (such as the second energy function) under its constraints, which can further help accelerate the optimization process, improve efficiency, and avoid the appearance of unrealistic baseline expression models or weighted expression models.

[0196] Continue to refer Figure 1 In step S150, the optimized benchmark expression model of the target face is output.

[0197] Through optimization, the target face's benchmark expression model can have the appearance features of the sample expression model and the expression features of the reference face's benchmark expression model. In other words, the target face's benchmark expression model has the same or similar appearance as the sample expression model, and the target face's benchmark expression model has the same or similar expression as the reference face's benchmark expression model.

[0198] refer to Figure 7 As shown, based on Figure 2 The reference facial expression models A0 to A4 shown in FIG. Figure 3 The sample expression models S1~S3 of the target face shown are processed by the above steps S120 to S150 to output 5 benchmark expression models B0~B4 of the target face. It can be seen that the expressions of the benchmark expression models B0~B4 match the expressions of the benchmark expression models A0~A4 one by one, and the benchmark expression models B0~B4 have the same appearance as the sample expression models S1~S3.

[0199] In practical applications, any face can be used as a reference face to obtain baseline expression models of any number of reference faces, so that more baseline expression models can be generated based on a small number of sample expression models of the target face, and the baseline expression model of the target face has higher diversity and flexibility.

[0200] The exemplary embodiment of the present disclosure also provides a face model generation device. Figure 8As shown, the face model generating device 800 may include:

[0201] The data acquisition module 810 is configured to acquire a reference face baseline expression model and a target face sample expression model, and establish variables related to the target face baseline expression model and variables related to weights;

[0202] The initial value determination module 820 adjusts the weights so that the weighted expression model of the reference face matches the expression of the sample expression model successfully, thereby obtaining the initial value of the weights; the weighted expression model of the reference face is a model obtained by weighting the baseline expression model of the reference face by the weights;

[0203] The energy function establishment module 830 is configured to establish at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model; the weighted expression model of the target face is a model obtained by weighting the reference expression model of the target face by weights;

[0204] The data optimization module 840 is configured to alternately perform the following two optimizations: optimizing the target face's baseline expression model using an energy function with fixed weights, and optimizing the weights using an energy function with fixed target face's baseline expression model;

[0205] The model output module 850 is configured to output the optimized reference expression model of the target face.

[0206] In one embodiment, the at least one energy function includes a first energy function and a second energy function; the at least one energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, including:

[0207] Establishing a first energy function based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model;

[0208] Establishing a second energy function based on the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model;

[0209] The above-mentioned optimizing the reference expression model of the target face by using the energy function under the condition of fixed weights, and optimizing the weights by using the energy function under the condition of fixed reference expression model of the target face, include:

[0210] The first energy function is used to optimize the target face's baseline expression model while the weights are fixed, and the second energy function is used to optimize the weights while the target face's baseline expression model is fixed.

[0211] In one embodiment, the first type of features includes: local patch features; the difference between the first type of features of the weighted expression model based on the target face and the first type of features of the sample expression model is used to establish a first energy function, including:

[0212] Obtaining local patch features of a weighted expression model of the target face, and obtaining local patch features of a sample expression model;

[0213] A first energy function is established based on the difference between the local patch features of the weighted expression model of the target face and the local patch features of the sample expression model.

[0214] In one embodiment, the above-mentioned obtaining of the local patch features of the weighted expression model of the target face includes:

[0215] The local patch features of the benchmark expression model of the target face are weighted by weights to obtain the local patch features of the weighted expression model of the target face.

[0216] In one embodiment, the above-mentioned method of optimizing the reference expression model of the target face by using the first energy function under the condition of fixed weights includes:

[0217] Under a fixed weight, the first energy function is used to optimize the local patch features of the benchmark expression model of the target face;

[0218] The optimized target face benchmark expression model is reconstructed according to the local patch features of the optimized target face benchmark expression model.

[0219] In one embodiment, the second type of features includes: key point features; the difference between the second type of features of the weighted expression model based on the target face and the second type of features of the sample expression model is used to establish a second energy function, including:

[0220] Obtain key point features of the weighted expression model of the target face, and obtain key point features of the sample expression model;

[0221] A second energy function is established based on the difference between the key point features of the weighted expression model of the target face and the key point features of the sample expression model.

[0222] In one embodiment, the above-mentioned step of obtaining key point features of the weighted expression model of the target face includes:

[0223] The key point features of the benchmark expression model of the target face are weighted by weights to obtain the key point features of the weighted expression model of the target face.

[0224] In one embodiment, the above-mentioned method of optimizing weights by using the second energy function under the condition of fixing the reference expression model of the target face includes:

[0225] The weights are optimized using a second energy function while fixing the key point features of the reference expression model of the target face.

[0226] In one embodiment, the at least one energy function includes a first energy function; the at least one energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, including:

[0227] A first energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face; the expression gradient of the reference face is the gradient between the two benchmark expression models of the reference face, and the expression gradient of the target face is the gradient between the two benchmark expression models of the target face.

[0228] In one embodiment, the first energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face, including:

[0229] A first energy function is established based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face.

[0230] In one embodiment, the at least one energy function includes a second energy function; the at least one energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, including:

[0231] A second energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the current value of the weight and the initial value of the weight.

[0232] In one embodiment, the second energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the current value of the weight and the initial value of the weight, including:

[0233] A second energy function is established based on the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model, and the difference between the current value of the weight and the initial value of the weight.

[0234] In one embodiment, the above-mentioned method of optimizing weights by using an energy function under the condition of fixing the reference expression model of the target face includes:

[0235] When the target face's baseline expression model is fixed, the weights are optimized using the energy function and the mutually exclusive relationship between the target face's baseline expression models.

[0236] The specific details of each part of the above device have been described in detail in the implementation method part, so they will not be repeated here.

[0237] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In an optional embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0238] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0239] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0240] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0241] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0242] The exemplary embodiment of the present disclosure also provides an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as program codes. The processor executes the method in the exemplary embodiment by executing the executable instructions.

[0243] Reference below Fig. 9 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Fig. 9 The electronic device 900 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0244] like Fig. 9 As shown, the electronic device 900 may include: a processor 910 , a memory 920 , a bus 930 , an I / O (input / output) interface 940 , and a network adapter 950 .

[0245] The memory 920 may include a volatile memory, such as a RAM 921, a cache unit 922, and may also include a non-volatile memory, such as a ROM 923. The memory 920 may also include one or more program modules 924, such program modules 924 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 924 may include each module in the above-mentioned device 800.

[0246] The bus 930 is used to realize the connection between different components of the electronic device 900, and may include a data bus, an address bus, and a control bus.

[0247] The electronic device 900 can communicate with one or more external devices 1000 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 940 .

[0248] The electronic device 900 can communicate with one or more networks through the network adapter 950. For example, the network adapter 950 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 950 can communicate with other modules of the electronic device 900 through the bus 930.

[0249] although Fig. 9 Not shown, other hardware and / or software modules may also be provided in the electronic device 900, including but not limited to: a display, a microcode, a device driver, a redundant processor, an external disk drive array, a RAID system, a tape drive, and a data backup storage system.

[0250] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0251] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other implementations of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0252] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating a face model, characterized in that: include: Obtain a reference face's baseline expression model and a target face's sample expression model, and establish variables about the target face's baseline expression model and variables about weights; wherein the target face is a face for which a baseline expression model is expected to be generated; the reference face is a face for which a baseline expression model has been generated and is used to provide reference information; the baseline expression model is a group of representative expression models; the sample expression model of the target face is used to provide samples for generating the target face's baseline expression model, and is one or more existing expression models of the target face; By adjusting the weights, the weighted expression model of the reference face is matched successfully with the expression of the sample expression model, so as to obtain the initial value of the weights; the weighted expression model of the reference face is a model obtained by weighting the baseline expression model of the reference face by the weights; Establishing at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model; the weighted expression model of the target face is a model obtained by weighting the reference expression model of the target face by the weights; Alternately performing the following two optimizations: optimizing the reference expression model of the target face using the energy function while fixing the weights, and optimizing the weights using the energy function while fixing the reference expression model of the target face; Outputting an optimized baseline expression model of the target face; Wherein, the at least one energy function includes a first energy function and a second energy function; The establishing of at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model comprises: establishing the first energy function based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model; establishing the second energy function based on the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model; The method of optimizing the baseline expression model of the target face by using the energy function while fixing the weights, and optimizing the weights by using the energy function while fixing the baseline expression model of the target face, includes: optimizing the baseline expression model of the target face by using the first energy function while fixing the weights, and optimizing the weights by using the second energy function while fixing the baseline expression model of the target face.

2. The method according to claim 1, characterized in that The first type of features and the second type of features are any one or more of the following features: point features, line features, and surface features.

3. The method according to claim 1, characterized in that The first type of features includes: local patch features; based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model, establishing the first energy function includes: Acquire local patch features of the weighted expression model of the target face, and acquire local patch features of the sample expression model; The first energy function is established based on the difference between the local patch features of the weighted expression model of the target face and the local patch features of the sample expression model.

4. The method according to claim 3, characterized in that: The step of obtaining the local patch features of the weighted expression model of the target face includes: The local patch features of the reference expression model of the target face are weighted by the weights to obtain the local patch features of the weighted expression model of the target face.

5. The method according to claim 4, characterized in that The step of optimizing the reference expression model of the target face by using the first energy function while fixing the weights comprises: Under the condition of fixing the weight, using the first energy function to optimize the local patch features of the benchmark expression model of the target face; The optimized benchmark expression model of the target face is reconstructed according to the local patch features of the optimized benchmark expression model of the target face.

6. The method according to claim 1, characterized in that The second type of features include: key point features; the difference between the second type of features of the weighted expression model based on the target face and the second type of features of the sample expression model, establishing the second energy function, including: Obtain key point features of the weighted expression model of the target face, and obtain key point features of the sample expression model; The second energy function is established based on the difference between the key point features of the weighted expression model of the target face and the key point features of the sample expression model.

7. The method according to claim 6, characterized in that The step of obtaining key point features of the weighted expression model of the target face includes: The key point features of the reference expression model of the target face are weighted by the weights to obtain the key point features of the weighted expression model of the target face.

8. The method according to claim 7, characterized in that The step of optimizing the weight by using the second energy function while fixing the reference expression model of the target face comprises: The weight is optimized using the second energy function while fixing the key point features of the reference expression model of the target face.

9. The method according to claim 1, characterized in that: The method of establishing at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model comprises: The first energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face; the expression gradient of the reference face is the gradient between the two benchmark expression models of the reference face, and the expression gradient of the target face is the gradient between the two benchmark expression models of the target face.

10. The method according to claim 9, characterized in that The first energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face, comprising: The first energy function is established based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model, and the difference between the expression gradient of the reference face and the expression gradient of the target face.

11. The method according to claim 1, characterized in that: At least one energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, including: The second energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the current value of the weight and the initial value of the weight.

12. The method according to claim 11, characterized in that The second energy function is established based on the difference between the weighted expression model of the target face and the sample expression model, and the difference between the current value of the weight and the initial value of the weight, including: The second energy function is established based on the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model, and the difference between the current value of the weight and the initial value of the weight.

13. The method according to claim 1, characterized in that The step of optimizing the weights by using the energy function while fixing the reference expression model of the target face comprises: When the reference expression model of the target face is fixed, the weight is optimized by utilizing the mutually exclusive relationship between the energy function and the reference expression model of the target face.

14. A face model generation device, characterized in that: include: The data acquisition module is configured to acquire a reference expression model of a reference face and a sample expression model of a target face, and establish variables about the reference expression model of the target face and variables about weights; wherein the target face is a face for which a reference expression model is expected to be generated; the reference face is a face for which a reference expression model has been generated and is used to provide reference information; the reference expression model is a group of representative expression models; the sample expression model of the target face is used to provide samples for generating the reference expression model of the target face, and is one or more existing expression models of the target face; An initial value determination module is configured to adjust the weights so that the weighted expression model of the reference face matches the expression of the sample expression model successfully, so as to obtain an initial value of the weight; the weighted expression model of the reference face is a model obtained by weighting the baseline expression model of the reference face by the weights; An energy function establishment module is configured to establish at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model; the weighted expression model of the target face is a model obtained by weighting the reference expression model of the target face by the weights; The data optimization module is configured to alternately perform the following two optimizations: optimizing the reference expression model of the target face by using the energy function while the weights are fixed, and optimizing the weights by using the energy function while the reference expression model of the target face is fixed; A model output module, configured to output an optimized reference expression model of the target face; Wherein, the at least one energy function includes a first energy function and a second energy function; The establishing of at least one energy function based on the difference between the weighted expression model of the target face and the sample expression model comprises: establishing the first energy function based on the difference between the first type of features of the weighted expression model of the target face and the first type of features of the sample expression model; establishing the second energy function based on the difference between the second type of features of the weighted expression model of the target face and the second type of features of the sample expression model; The method of optimizing the baseline expression model of the target face by using the energy function while fixing the weights, and optimizing the weights by using the energy function while fixing the baseline expression model of the target face, includes: optimizing the baseline expression model of the target face by using the first energy function while fixing the weights, and optimizing the weights by using the second energy function while fixing the baseline expression model of the target face.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 13 by executing the executable instructions.

Images