Virtual object face generation method, device, equipment and readable storage medium

Abstract

The present application discloses a method, device, equipment and readable storage medium for generating a face of a virtual object, and belongs to the field of Internet technology. The method comprises: obtaining a facial bionic model of a real object by three-dimensional reconstruction, optimizing the initial model parameters of a pre-established reference facial feature model according to the facial bionic model, obtaining target model parameters, adjusting the model parameters of the virtual facial feature model according to the target model parameters, migrating the facial features represented by the target model parameters to the virtual facial feature model, and obtaining the target facial feature model of the virtual object. In the process of migrating facial features, the transition of facial features is achieved by referring to the facial feature model, avoiding the direct migration of facial features from the facial bionic model to the virtual facial feature model, and can solve the problem of abnormal deformation of the virtual facial feature model when the facial structure of the virtual object is greatly different from the facial structure of the real object.

Description

Technical Field

[0001] The present invention belongs to the field of Internet technology, and in particular relates to a method, device, equipment and readable storage medium for generating a face of a virtual object. Background Art

[0002] With the development of Internet technology, according to the following method, it has been realized to establish a virtual object similar to a real object based on the facial features of a real object, so that the virtual object has the facial features of the real object, the real object is such as a real person or animal, and the virtual object is such as a virtual three-dimensional cartoon character or animal. In the face generation process of the virtual object, a virtual facial feature model of the virtual object is first established, and a facial bionic model of the real object is established based on the facial image data of the real object face, and both the virtual facial feature model and the facial bionic model are three-dimensional (3Dimensions) models. Then, the facial features in the facial bionic model are migrated to the virtual facial feature model to obtain a target facial feature model including the facial features of the real object, so that the virtual object can have the facial features of the real object.

[0003] In the process of implementing the present application, the inventors found that there are at least the following problems in the prior art: during the face generation process, if the facial structure of the virtual object is significantly different from that of the real object, the virtual facial feature model will undergo abnormal deformation during the facial feature migration process, resulting in an abnormal facial structure of the virtual object. Summary of the invention

[0004] In view of this, the present invention provides a method, device, equipment and readable storage medium for generating a face of a virtual object, which to a certain extent solves the problem that the virtual facial feature model undergoes abnormal deformation during the face generation process, resulting in abnormal facial structure of the virtual object.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for generating a face of a virtual object, the method comprising:

[0007] Acquire facial image data of a real object;

[0008] Based on the facial image data, three-dimensionally reconstruct a facial bionic model of the real object;

[0009] According to the facial bionic model, initial model parameters of a pre-established reference facial feature model are optimized to obtain target model parameters; the target model parameters represent facial features of the real object included in the facial bionic model;

[0010] According to the target model parameters, the model parameters of the virtual facial feature model are adjusted to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain a target facial feature model; the virtual facial feature model is a facial feature model that is pre-established based on the reference facial feature model and belongs to the virtual object.

[0011] In a second aspect, an embodiment of the present application provides a face generation device for a virtual object, the device comprising:

[0012] An acquisition module, used for acquiring facial image data of a real object;

[0013] A reconstruction module, used for reconstructing a facial bionic model of the real object in three dimensions based on the facial image data;

[0014] An optimization module, configured to optimize initial model parameters of a pre-established reference facial feature model according to the facial bionic model to obtain target model parameters; the target model parameters represent facial features of the real object included in the facial bionic model;

[0015] The adjustment module is used to adjust the model parameters of the virtual facial feature model according to the target model parameters, so as to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model; the virtual facial feature model is a facial feature model of the virtual object pre-established according to the reference facial feature model.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0019] In the embodiment of the present application, the electronic device first obtains the facial image data of the real object, and based on the facial image data, three-dimensionally reconstructs the facial bionic model of the real object, and then optimizes the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain the target model parameters, and adjusts the model parameters of the virtual facial feature model according to the target model parameters, and migrates the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model of the virtual object. In the process of migrating facial features, the facial features are first migrated to the reference facial feature model, and then the facial features in the reference facial feature model are migrated to the virtual facial feature model of the virtual object, and the transition of facial features is achieved through the reference facial feature model, so as to avoid the facial features directly migrating from the facial bionic model to the virtual facial feature model, so as to solve the problem of abnormal deformation of the virtual facial feature model when the facial structure of the virtual object is greatly different from the facial structure of the real object, thereby avoiding the abnormal facial structure of the virtual object. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0021] Figure 1 is a flowchart of a method for generating a face of a virtual object provided in an embodiment of the present application;

[0022] Figure 2 is an optimized schematic diagram of a reference facial feature model provided in an embodiment of the present application;

[0023] Figure 3 is a flowchart of another method for generating a face of a virtual object provided in an embodiment of the present application;

[0024] Figure 4 is a block diagram of a face generation device for a virtual object provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0027] Figure 1 is a flowchart of a method for generating a face of a virtual object provided in an embodiment of the present application, such as Figure 1 As shown, the method may include:

[0028] Step 101: Acquire facial image data of a real object.

[0029] Step 102: Based on the facial image data, a facial bionic model of the real object is obtained by three-dimensional reconstruction.

[0030] In this embodiment, the face generation method of the virtual object can be executed by electronic devices such as personal computers, mobile phones and servers. The real object can be a real person or animal, or a real object with a face such as a real doll or statue. The facial image data is used to establish a facial bionic model of the real object, and the type and acquisition method of the facial image data can be specifically set according to the three-dimensional reconstruction method.

[0031] For example, an electronic device can obtain a facial bionic model through three-dimensional reconstruction using a binocular matching algorithm. In the three-dimensional reconstruction process, the electronic device can first use a binocular camera to shoot the facial area of ​​a real object to obtain two two-dimensional facial images of the real object, that is, facial image data, and then match the two facial images through a binocular matching algorithm to obtain depth information of the facial area. Based on the facial features and depth information in the facial images, a facial bionic model of the real object is obtained through three-dimensional reconstruction.

[0032] In one embodiment, the electronic device may use a depth camera to shoot the face area of ​​a real object to obtain facial image data including a color image and a depth image, wherein the color image includes color values ​​of three color channels, namely, a red (Red, R) channel, a green (Green, G) channel, and a blue (Blue, B) channel, and the depth image includes a depth value of a depth (D) channel, and the depth value of each pixel point in the depth image data is the distance value between the corresponding point in the face area of ​​the real object and the camera, for example, the depth value of the pixel point corresponding to the nose tip in the depth image is the distance value between the nose tip of the real object and the camera. The electronic device may reconstruct a facial bionic model of the real object in three dimensions based on the facial features included in the color image and the depth information included in the depth image.

[0033] In one embodiment, the electronic device can obtain a facial bionic model by three-dimensional reconstruction based on a two-dimensional facial image. For example, the electronic device can directly photograph the facial area of ​​a real object to obtain a two-dimensional facial image, i.e., facial image data, and then extract key points in the facial image, and the coordinates of the key points can determine the facial features of the real object. For example, the key points include multiple key points in the eye area of ​​the real object, and the multiple key points in the eye area can outline the eye contour of the real object. After extracting the key points that can characterize the facial features, the electronic device can select an average facial model from a three-dimensional deformable model (3D Morphable models, 3DMM) library, fit the multiple key points to the average facial model, and obtain the facial bionic model.

[0034] In this embodiment, the facial bionic model can be a triangular mesh model composed of a certain number of vertices, triangular face pieces and associated structures, etc. The facial bionic model includes facial features of a real object, such as the facial contour, eyebrow shape, eye shape, nose shape and mouth shape of the real object, and the layout positions of eyebrows, eyes, nose and mouth in the facial area. The reconstruction method of the facial bionic model can be set according to demand, and this embodiment does not limit this.

[0035] Step 103: Optimize the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain target model parameters.

[0036] The target model parameters characterize the facial features of the real object included in the facial bionic model. The reference facial feature model is a virtual three-dimensional mesh model designed by the user, and the reference facial feature model is used to establish a virtual facial feature model of the virtual object. Different model parameters correspond to different facial features, and the initial model parameters are the model parameters of the reference facial feature model in the initial state. In the process of establishing the virtual object, the virtual facial feature model of the virtual object can be established according to the reference facial feature model. By adjusting the model parameters of the reference facial feature model, that is, adjusting the model parameters of the virtual facial feature model, the virtual facial feature model can have different facial features.

[0037] In this embodiment, after obtaining the facial bionic model of the real object, the reference facial feature model can be fitted to the facial bionic model, thereby optimizing the initial model parameters of the reference facial feature model and obtaining the target model parameters. That is, the initial parameters of the reference facial feature model are adjusted according to the facial bionic model to migrate the facial features in the facial bionic model to the reference facial feature model, and obtain the target model parameters that can characterize the facial features. The reference facial feature model is adjusted according to the target model parameters, so that the reference facial feature model can have the facial features of the real object.

[0038] In one embodiment, a reference facial feature model is taken as an Identity blendshape model. The Identity blendshape model includes a set of feature bases (blendshape) and a basic grid. The basic grid is an average virtual facial model, and each feature base is a basic facial feature model. Each feature base has a corresponding weight. The weight of each feature base in the Identity blendshape model in an initial state is the initial model parameter. The feature base and the basic grid constitute a facial feature model. By adjusting the weight of each feature base in the Identity blendshape model, a virtual facial feature model with different facial features can be obtained.

[0039] Each feature basis in the Identity blendshape model is a three-dimensional mesh model, which is composed of a preset number of vertices, and each vertex has three-dimensional coordinates, namely, coordinates on the x-axis, y-axis, and z-axis. For example, each feature basis includes n vertices, and the Identity blendshape model may include m feature bases B 3×n and a base grid H 3×n , m and n are positive integers. The weight of each feature basis is w, then the Identity blendshape model can be expressed by formula A:

[0040] B 3×n，m × m，1 +H 3×n，1

[0041] Among them, each feature basis B 3×n is a basic matrix. The subscript 3×n indicates that the basic matrix includes the three-dimensional vectors of each vertex in the characteristic basis, namely the coordinates of the x-axis, y-axis and z-axis. m basic matrices constitute a B 3×n，m Matrix, the weights of m basic matrices form a w m，1 The basic mesh is a neutral face matrix, and the subscript 3×n represents the three-dimensional vector of each vertex in the basic mesh.

[0042] During the fitting process, an optimization function f(w) about the weight w can be set. For the optimization of the vertices of the reference facial feature model in the x-axis direction, the optimization target can be set as formula B:

[0043] ||S(B x w+H x -P x )|| 2

[0044] Among them, the subscript x represents the vector of x dimension, P x is the vector of the facial bionic model in the x dimension, and formula B represents the error between the reference facial feature model and the facial bionic model in the x dimension. S is the coefficient matrix of the x dimension. During the calculation process, the weight of each feature basis can be determined according to the coefficient matrix. The elements in S can be defined as:

[0045]

[0046] Wherein, i is the i-th row in the coefficient matrix, j is the j-th column in the coefficient matrix, and in the calculation process, the weights other than the weights on the diagonal of the matrix in the matrix corresponding to the reference facial feature model can be set to 0 according to the coefficient matrix. That is, as shown above, when i is not equal to j, the weight is 0.

[0047] By analogy, considering the components of the three dimensions x, y and z, the optimization function f(w) is:

[0048]

[0049] in, is the vector matrix of the eigenvalues ​​in three dimensions: x, y and z. is the vector matrix of the basic grid in the three dimensions of x, y and z, It is the vector matrix of the facial bionic model in three dimensions: x, y and z. A 3n×3n block matrix:

[0050]

[0051] In the optimization process, the value of the optimization function f(w) is made as small as possible, that is, the error between the reference facial feature model and the facial bionic model is made as small as possible, so that the reference facial feature model is close to the facial bionic model and has the facial features included in the facial bionic model. Specifically, the optimization function f(w) can be converted into a quadratic programming (QP) equation, and the weight w of each feature basis, i.e., the target model parameter, can be obtained by solving the QP equation.

[0052] In practical applications, the reference facial feature model can also be optimized in other ways to make the reference facial feature model close to the facial bionics model to obtain the target model parameters. The specific optimization process can be set according to needs, and this embodiment does not limit this. It should be noted that in order to facilitate obtaining accurate target model parameters, the facial bionics model and the reference facial feature model can have the same grid structure.

[0053] It should be noted that the present embodiment optimizes the initial model parameters of the reference facial feature model, which is essentially to optimize the reference facial feature model so that the reference facial feature model has the facial features in the facial bionic model. Figure 2 As shown, Figure 2 FIG. 1 is a schematic diagram of an optimization of a reference facial feature model provided by an embodiment of the present application. After reconstructing a facial bionic model 202 according to a facial image 201 of a person (real object), the initial model parameters of the reference facial feature model can be optimized according to the facial bionic model 202 to obtain target model parameters. After adjusting the reference facial feature model according to the target model parameters, the following can be obtained: Figure 2 The optimized reference facial bionic model 203 is shown. Figure 2 As shown, the optimized reference facial feature model includes the facial features of the character, and the facial features are determined by the target model parameters in the optimized reference facial feature model. This embodiment only uses the target model parameters in the target facial model to adjust the model parameters of the virtual facial feature model.

[0054] Step 104: adjust the model parameters of the virtual facial feature model according to the target model parameters to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model.

[0055] The virtual facial feature model is a facial feature model that is pre-established based on the reference facial feature model and belongs to the virtual object. In combination with the above example, if the reference facial feature model is an Identity blendshape model, the facial feature model of the virtual object can be established based on the Identity blendshape model. The specific process of establishing the virtual facial feature model based on the Identity blendshape model can be set according to the requirements, and this embodiment does not limit this.

[0056] In this embodiment, after obtaining the target parameter model, the model parameters of the virtual facial feature model can be adjusted according to the target model parameters to obtain the target facial feature model. In combination with the above example, the virtual facial feature model is established according to the reference facial feature model, the virtual facial feature model and the reference facial feature model have the same grid structure and the same model parameters, and the virtual facial features can also be expressed by formula A. After obtaining the target model parameters, that is, the weight w of each feature base in the optimized reference facial feature model, the weight w can be brought into formula A to obtain the target virtual facial feature model. Since the target model parameters are obtained by optimizing the facial bionic model, the facial features of the real object can be characterized. When the virtual facial feature model of the virtual object is adjusted according to the target model parameters, the facial features of the real object can be transferred to the virtual facial feature model of the virtual object, so that the virtual object has the facial features of the real object. According to the target model parameters, the specific process of adjusting the model parameters of the virtual facial feature model can be set according to demand, and this embodiment does not limit this.

[0057] In summary, in this embodiment, the electronic device first obtains the facial image data of the real object, and based on the facial image data, three-dimensionally reconstructs the facial bionic model of the real object, and then optimizes the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain the target model parameters, and adjusts the model parameters of the virtual facial feature model according to the target model parameters, and migrates the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model of the virtual object. In the process of migrating facial features, the facial features are first migrated to the reference facial feature model, and then the facial features in the reference facial feature model are migrated to the virtual facial feature model of the virtual object, and the transition of facial features is achieved through the reference facial feature model, so as to avoid the facial features directly migrating from the facial bionic model to the virtual facial feature model, thereby solving the problem of abnormal deformation of the virtual facial feature model when the facial structure of the virtual object is greatly different from the facial structure of the real object, thereby avoiding the abnormal facial structure of the virtual object.

[0058] Figure 3 is a flowchart of another method for generating a face of a virtual object provided in an embodiment of the present application. Figure 3 As shown, the method may include

[0059] Step 301: Acquire facial image data of a real object.

[0060] Step 302: Based on the facial image data and the average facial model, a facial bionic model is obtained by three-dimensional reconstruction.

[0061] Step 303: calibrate the reference facial feature model according to the average facial model to obtain a calibrated reference facial feature model.

[0062] The average face model is a reference face model obtained by processing the face model data set, and the structure of the calibrated reference face feature model matches the structure of the average face model.

[0063] In this embodiment, before optimizing the initial model parameters, the reference facial feature model may be calibrated first to match the structure of the reference facial feature model with that of the average facial model.

[0064] In one embodiment, a face model dataset of a real object may be established. For example, if the real object is a person, a face model dataset about a person's face may be established. In conjunction with step 102, the face model dataset may be a 3DMM library, and the average face model may be an average face model in the 3DMM library. Before optimizing the reference facial feature model, the Identity blendshape model may be calibrated according to the average face model (i.e., the average face model). For example, the size, rotation angle, and contour of the Identity blendshape model may be adjusted according to the structure of the average face model, so that the size of the Identity blendshape model is consistent with that of the average face model, the rotation angle of the Identity blendshape model is consistent with that of the average face model, and the contour of the Identity blendshape model is consistent with that of the average face model.

[0065] In this embodiment, one or more parameters in the reference facial feature model may be calibrated. During the calibration process, a corresponding deformation transfer algorithm may be selected to transfer the shape change shown by the average facial model to the reference facial feature model so that the reference facial feature model matches the structure of the average facial model. The specific method for calibrating the reference facial feature model may be set according to requirements, and this embodiment does not limit this.

[0066] In practical applications, before optimizing the reference facial feature model, the reference facial feature model is first calibrated so that the reference facial feature model has the same or similar structure as the average facial model. When the structure of the reference facial feature model matches the average facial model, more accurate target model parameters can be obtained during the optimization process, making the facial feature model represented by the target model parameters closer to the facial features of the real object.

[0067] Step 304: Optimize the initial model parameters of the calibrated reference facial feature model according to the facial bionic model to obtain target model parameters.

[0068] Optionally, the method may further include:

[0069] During the optimization process, the structure of the optimized reference facial feature model is controlled to match that of the facial bionic model.

[0070] Optionally, the method may further include:

[0071] During the optimization process, the target model parameters are controlled to be less than or equal to a preset threshold value so that the structure of the optimized reference facial feature model meets the set structural conditions.

[0072] In this embodiment, after the reference facial feature model is calibrated, the initial model parameters of the calibrated reference facial feature model can be optimized according to the facial bionic model to obtain the target model parameters. At the same time, during the optimization process, constraints can be added to make the structure of the optimized reference facial feature model match that of the facial bionic model, and to make the structure of the optimized reference facial feature model meet the set structural conditions.

[0073] For example, a structural constraint term, such as a Laplacian constraint, may be added to the optimization function f(w) to control the deformation of the reference facial feature model during the optimization process. Specifically, the Laplacian constraint in the x dimension may be set as:

[0074] α||L(B x w+H x )-δ x || 2

[0075] Where L is the n×n Laplace matrix, δ x is an n×1 vector of x-dimensional elements of Laplace coordinates in the average facial model in the facial bionic model, α is the coefficient of the regularization term in the Laplace constraint term, and α may be greater than or equal to 0 and less than or equal to 1.

[0076] At the same time, a weight constraint term can be added to the optimization function f(w), and the weight constraint term is used to make the structure of the optimized reference facial feature model meet the set structural conditions. Specifically, the weight constraint term on the x dimension can be set as:

[0077] ω||w|| 2

[0078] Wherein, β is the coefficient of the unified weight regularization term, and β can be greater than or equal to 0 and less than or equal to 1.

[0079] Combining the above example, considering the components of the three dimensions x, y and z, the optimization function f(w) can be set as:

[0080]

[0081] stw∈[0,1]

[0082] Among them, in the weight constraint term, the weight w can be restricted to be greater than or equal to 0 and less than or equal to 1 by the formula: st w∈[0,1]. A 3n×3n block matrix:

[0083]

[0084] is the x-, y-, and z-dimensional element vector in the Laplace coordinates of the average face model.

[0085] The facial bionic model can be obtained by three-dimensional reconstruction based on the average facial model and facial image data. The Laplace constraint characterizes the structural error between the reference facial feature model and the average facial model. In the optimization process, the Laplace constraint term is made as small as possible, so that the structure of the optimized reference facial feature model can be close to the structure of the facial bionic model, so that the structure of the optimized reference facial feature model can match the structure of the facial bionic model. At the same time, in the optimization process, the weight constraint term is made as small as possible, so that the weight (i.e., the target model parameter) in the optimized reference facial feature model can be less than or equal to a preset threshold. When the weight is less than the preset threshold, a large weight can be avoided, and a large deformation of the optimized reference facial feature model can be avoided, so that a strange shape of the optimized reference facial feature model can be avoided.

[0086] Similarly, the optimization function f(w) can be converted into a QP equation and the weight w of each feature basis can be obtained by solving it.

[0087] During the optimization process, the Laplace constraint term can be used to make the structure of the optimized reference facial feature model match the structure of the facial bionic model. Figure 2 As shown, when the structure of the reference facial feature model matches the structure of the facial bionics model, the optimized reference facial feature model can have more facial features in the facial bionics model, so that the target model parameters can more accurately represent the facial features of the real object.

[0088] At the same time, during the optimization process, controlling the target model parameter w to be less than or equal to a preset threshold can prevent the structure of the optimized reference facial feature model from undergoing significant changes, thereby preventing the structure of the optimized reference facial feature model from undergoing strange deformations, and allowing the target model parameters to more accurately characterize the facial features of the real object.

[0089] In practical applications, during the optimization process, only the structure matching between the optimized reference facial feature model and the facial bionic model can be controlled, or only the target model parameters can be controlled to be less than or equal to a preset threshold.

[0090] Optionally, when controlling the target model parameter to be less than or equal to a preset threshold value so that the structure of the optimized reference facial feature model meets the set structural condition, the method may include:

[0091] The target facial features in the optimized reference facial feature model are controlled to meet the set feature conditions.

[0092] In combination with the above examples, in the reference facial feature model, each feature base is a basic facial feature model, and each feature base has a corresponding weight. In the reference facial feature model, each facial feature is represented by a certain number of feature bases, for example, the eyebrow feature in the reference facial feature model can be represented by multiple feature bases. During the optimization process, corresponding coefficients can be set for multiple feature bases representing the target facial features (target facial features such as eyebrow features) in the weight constraint term, so that the eyebrow features in the optimized reference facial feature model meet the set feature conditions.

[0093] In one embodiment, the optimization function f(w) can be expressed as:

[0094]

[0095] stw∈[0,1]

[0096] Among them, C is the coefficient matrix corresponding to the β component in the weight constraint, and the elements in C can be defined as:

[0097]

[0098] Wherein, i is the i-th row in the coefficient matrix, j is the j-th column in the coefficient matrix, and in the calculation process, the weights other than the weights on the diagonal of the matrix in the matrix corresponding to the reference facial feature model can be set to 0 according to the coefficient matrix. That is, as shown above, when i is not equal to j, the weight is 0. In the optimization process, the weights in the reference facial feature model can be adjusted more finely through the coefficient matrix.

[0099] In one embodiment, for the target facial features in the reference facial feature model, corresponding coefficients can be set in the coefficient matrix for the weights of multiple feature bases representing the target facial features. During the optimization process, the deformation of the target facial feature model can be controlled, so that the target facial features in the optimized reference facial feature model can meet the set feature conditions. For example, for the eyebrow features, larger coefficients can be set in the coefficient matrix for the weights of multiple feature bases representing the eyebrow features. During the optimization process, the larger coefficients can be used to control the eyebrow features to undergo smaller deformations, so that the eyebrow features in the optimized reference facial feature model meet the set feature conditions. For another example, for the eye features in the reference facial feature model, smaller coefficients can be set in the coefficient matrix for the weights of multiple feature bases constituting the eye features. During the optimization process, the smaller coefficients can be used to control the eye features to undergo larger deformations, so that the eyes in the optimized reference facial feature model meet the set feature conditions.

[0100] It should be noted that the specific setting method of the structure constraint item and the weight constraint item can be set according to the requirements, and this embodiment does not limit this.

[0101] Step 305: Adjust the model parameters of the virtual facial feature model according to the target model parameters to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model.

[0102] In summary, in this embodiment, the electronic device first obtains the facial image data of the real object, and based on the facial image data, three-dimensionally reconstructs the facial bionic model of the real object, and then optimizes the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain the target model parameters, and adjusts the model parameters of the virtual facial feature model according to the target model parameters, and migrates the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model of the virtual object. In the process of migrating facial features, the facial features are first migrated to the reference facial feature model, and then the facial features in the reference facial feature model are migrated to the virtual facial feature model of the virtual object, and the transition of facial features is achieved through the reference facial feature model, so as to avoid the facial features directly migrating from the facial bionic model to the virtual facial feature model, thereby solving the problem of abnormal deformation of the virtual facial feature model when the facial structure of the virtual object is greatly different from the facial structure of the real object, thereby avoiding the abnormal facial structure of the virtual object.

[0103] Optionally, step 302 may be implemented in the following manner:

[0104] Based on the facial image data and the average facial model, a facial bionic model is obtained by three-dimensional reconstruction.

[0105] The average face model is a reference face model obtained by processing the face model data set.

[0106] In conjunction with step 102, the face model data set may be a 3DMM library, and the average face model may be an average face model in the 3DMM library. At this point, the method described in step 102 may be used to establish a face bionic model of a real object based on the collected two-dimensional face image of the human face and the average face model.

[0107] In practical applications, based on the average face model, a facial bionic model is obtained by three-dimensional reconstruction, and the reference facial feature model is calibrated according to the average face model, so that the structure of the calibrated reference facial feature model matches the structure of the average face model, and the facial bionic model and the calibrated reference facial feature model have the same or similar grid structure. Therefore, in the process of optimizing the initial model parameters of the calibrated reference facial feature model according to the facial bionic model, the target model parameters obtained can more accurately represent the facial features of the real object.

[0108] Optionally, the method may further include:

[0109] According to the target facial feature model, the virtual facial expression model of the virtual object is optimized to obtain the target facial expression model; the target facial expression model includes the facial features in the target facial feature model;

[0110] Based on the target facial expression model, the expression of the real object is tracked.

[0111] In this embodiment, after obtaining the target facial feature model, the facial features in the target facial feature model may be transferred to the virtual facial expression model of the virtual object, so as to track the expression of the real object according to the virtual facial expression model.

[0112] Among them, the virtual facial expression model is used to construct the facial expression of the real object, and the virtual facial expression model is, for example, an Expression Blendshape model. The Expression Blendshape model includes a set of feature bases (blendshape) and a basic grid (blendshape), the basic grid is an average face model, and each feature base is a basic facial expression model. Each feature base has a corresponding weight, and adjusting the weight of each feature base of the Expression Blendshape model can obtain an Expression Blendshape model with different expressions.

[0113] In combination with the above examples, after obtaining the target facial feature model, the Expression Blendshape model can be fitted into the target facial feature model, thereby optimizing the Expression Blendshape model and migrating the facial features of the target facial feature model into the Expression Blendshape model, so that the Expression Blendshape model has the facial features of the real object. After obtaining the Expression Blendshape model with facial features, the expression of the real object can be tracked based on the target facial expression model. The optimization process of the Expression Blendshape model can be based on the optimization process of the reference facial feature model, and the specific method of tracking the expression of the real object based on the target facial expression model can be set according to demand, and this embodiment does not limit this.

[0114] It should be noted that the face generation method of the virtual object provided in this embodiment can be implemented by an electronic device alone or by a plurality of electronic devices in cooperation. For example, the client can obtain the facial image data of the user and upload the facial image data to the server. The server can reconstruct the facial bionic model of the real object in three dimensions based on the facial image data, and optimize the initial model parameters of the reference facial feature model according to the facial bionic model to obtain the target model parameters, and send the target model parameters to the client. The client can adjust the model parameters of the virtual facial feature model according to the target model parameters to obtain the target facial feature model. The specific implementation process of the face generation method of the virtual object can be set according to the needs, and this embodiment does not limit this.

[0115] Figure 4 is a block diagram of a virtual object face generation device provided in an embodiment of the present application, such as Figure 4 As shown, the apparatus 400 may include:

[0116] The acquisition module 401 is used to acquire facial image data of a real object.

[0117] The reconstruction module 402 is used to reconstruct a facial bionic model of a real object in three dimensions based on the facial image data.

[0118] The optimization module 403 is used to optimize the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain target model parameters; the target model parameters represent the facial features of the real object included in the facial bionic model.

[0119] The adjustment module 404 is used to adjust the model parameters of the virtual facial feature model according to the target model parameters so as to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model; the virtual facial feature model is a facial feature model of the virtual object that is pre-established based on the reference facial feature model.

[0120] Optionally, the optimization module 403 is specifically used to calibrate the reference facial feature model according to the average facial model to obtain a calibrated reference facial feature model; wherein the average facial model is a reference facial model obtained by processing the facial model data set, and the structure of the calibrated reference facial feature model matches the structure of the average facial model; according to the facial bionic model, the initial model parameters of the calibrated reference facial feature model are optimized to obtain the target model parameters.

[0121] Optionally, the optimization module 403 is further used to control the structural matching between the optimized reference facial feature model and the facial bionic model.

[0122] Optionally, the optimization module 403 is further configured to control the target model parameter to be less than or equal to a preset threshold value, so that the structure of the optimized reference facial feature model meets the set structural conditions.

[0123] Optionally, the optimization module 403 is further configured to control the target facial features in the optimized reference facial feature model to meet set feature conditions.

[0124] Optionally, the reconstruction module 402 is specifically configured to obtain a facial bionic model by three-dimensional reconstruction based on the facial image data and the average facial model.

[0125] Optionally, the optimization module 403 is also used to optimize the virtual facial expression model of the virtual object according to the target facial feature model to obtain the target facial expression model; the target facial expression model includes the facial features in the target facial feature model; and based on the target facial expression model, the expression of the real object is tracked.

[0126] In the embodiment of the present application, the electronic device first obtains the facial image data of the real object, and based on the facial image data, three-dimensionally reconstructs the facial bionic model of the real object, and then optimizes the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain the target model parameters, and adjusts the model parameters of the virtual facial feature model according to the target model parameters, and migrates the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model of the virtual object. In the process of migrating facial features, the facial features are first migrated to the reference facial feature model, and then the facial features in the reference facial feature model are migrated to the virtual facial feature model of the virtual object, and the transition of facial features is achieved through the reference facial feature model, so as to avoid the facial features directly migrating from the facial bionic model to the virtual facial feature model, so as to solve the problem of abnormal deformation of the virtual facial feature model when the facial structure of the virtual object is greatly different from the facial structure of the real object, thereby avoiding the abnormal facial structure of the virtual object.

[0127] The virtual object face generation device provided in the embodiment of the present application has a functional module corresponding to the virtual object face generation method, can execute the virtual object face generation method provided in the embodiment of the present application, and can achieve the same beneficial effects.

[0128] In another embodiment of the present invention, an electronic device is also provided. The electronic device may include: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, each process of the above-mentioned virtual object face generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0129] For example, Figure 5 As shown, Figure 5 5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may specifically include: a processor 501, a storage device 502, a display screen 503 with a touch function, an input device 504, an output device 505, and a communication device 506. The number of processors 501 in the electronic device may be one or more. Figure 5 A processor 501 is taken as an example. The processor 501, storage device 502, display screen 503, input device 504, output device 505 and communication device 506 of the electronic device can be connected via a bus or other means.

[0130] In another embodiment of the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method for generating a face of a virtual object described in any of the above embodiments.

[0131] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for generating a face of a virtual object described in any one of the above embodiments.

[0132] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0133] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a face of a virtual object, characterized in that: include: Acquire facial image data of a real object; Based on the facial image data, three-dimensionally reconstruct a facial bionic model of the real object; According to the facial bionic model, initial model parameters of a pre-established reference facial feature model are optimized to obtain target model parameters; the target model parameters represent facial features of the real object included in the facial bionic model; the facial bionic model and the reference facial feature model have the same grid structure; According to the target model parameters, the model parameters of the virtual facial feature model are adjusted to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain a target facial feature model; the virtual facial feature model is a facial feature model that is pre-established based on the reference facial feature model and belongs to the virtual object.

2. The method according to claim 1, characterized in that The step of optimizing the initial model parameters of the pre-established reference facial feature model according to the facial bionic model to obtain target model parameters includes: Calibrate the reference facial feature model according to the average facial model to obtain a calibrated reference facial feature model; wherein the average facial model is a reference facial model obtained by processing the facial model data set, and the structure of the calibrated reference facial feature model matches the structure of the average facial model; According to the facial bionic model, the initial model parameters of the calibrated reference facial feature model are optimized to obtain the target model parameters.

3. The method according to claim 2, characterized in that When the initial model parameters of the calibrated reference facial feature model are optimized according to the facial bionic model to obtain the target model parameters, the method includes: The optimized reference facial feature model is controlled to match the structure of the facial bionic model.

4. The method according to claim 2, characterized in that: When the initial model parameters of the calibrated reference facial feature model are optimized according to the facial bionic model to obtain the target model parameters, the method includes: The target model parameters are controlled to be less than or equal to a preset threshold value so that the structure of the optimized reference facial feature model meets the set structural conditions.

5. The method according to claim 4, characterized in that When the target model parameter is controlled to be less than or equal to a preset threshold value so that the structure of the optimized reference facial feature model meets the set structural conditions, it includes: The target facial features in the optimized reference facial feature model are controlled to meet the set feature conditions.

6. The method according to claim 2, characterized in that The three-dimensional reconstruction based on the facial image data to obtain the facial bionic model of the real object includes: The facial bionic model is obtained by three-dimensional reconstruction based on the facial image data and the average facial model.

7. The method according to any one of claims 1 to 6, characterized in that: After adjusting the model parameters of the virtual facial feature model according to the target model parameters to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model, the method further includes: According to the target facial feature model, the virtual facial expression model of the virtual object is optimized to obtain a target facial expression model; the target facial expression model includes the facial features in the target facial feature model; Based on the target facial expression model, the expression of the real object is tracked.

8. A virtual object face generation device, characterized in that: include: An acquisition module, used for acquiring facial image data of a real object; A reconstruction module, used for three-dimensionally reconstructing a facial bionic model of the real object based on the facial image data; an optimization module, configured to optimize initial model parameters of a pre-established reference facial feature model according to the facial bionic model to obtain target model parameters; the target model parameters characterize facial features of the real object included in the facial bionic model; the facial bionic model and the reference facial feature model have the same grid structure; The adjustment module is used to adjust the model parameters of the virtual facial feature model according to the target model parameters, so as to transfer the facial features represented by the target model parameters to the virtual facial feature model to obtain the target facial feature model; the virtual facial feature model is a facial feature model of the virtual object pre-established according to the reference facial feature model.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method for generating a face of a virtual object as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the method for generating a face of a virtual object according to any one of claims 1 to 7 are implemented.

Images