Face modeling parameter generation method and device and virtual face generation method and device

By obtaining and extracting facial image features, generating virtual face images and target face modeling parameters, building an end-to-end face adjustment model, solving the problem of time-consuming and labor-intensive users in the process of pinching faces, and improving virtual faces and user satisfaction.

CN118942130APending Publication Date: 2024-11-12ZHEJIANG TMALL TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410784472.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-17
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, users need to repeatedly adjust the control parameters of facial components during face pinching, which is time-consuming and labor-intensive, and the restore degree between the generated virtual face and the face image input by the user is not high, resulting in poor user satisfaction. high.

Method used

By obtaining the target face image, feature extraction and generating virtual face images, then feature extraction of virtual face images, generating target face modeling parameters, using pre-trained face adjustment model and rendering engine to build end-to-end The face adjustment model is improved to improve the accuracy of face modeling parameters.

Benefits of technology

It simplifies the user's face-pinching operation process, improves the similarity between the virtual face and the user's input face, and improves the user's face-pinching experience and satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942130A_ABST
    Figure CN118942130A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a face modeling parameter and virtual face generation method and device, and the method comprises the steps: obtaining a target face image, carrying out the feature extraction of the target face image, and generating a virtual face image corresponding to a target face based on the extracted target face features, and performing feature extraction on the virtual face image to obtain virtual face features, and generating target face modeling parameters corresponding to the target face based on the virtual face features. According to the embodiment of the invention, the virtual face image is generated through feature extraction of the target face image, and the target face modeling parameters are obtained through prediction based on the virtual face image, so that the face pinching operation of the user is simplified, the accuracy of the face modeling parameters is effectively improved, the reduction degree between the virtual face generated through rendering and the input face is higher, and the user experience is improved. And the face pinching experience and satisfaction degree of the user are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer vision technology, and in particular, to a method and device for generating facial modeling parameters and a virtual face. Background Art

[0002] With the development of 3D reconstruction technology, in some games and digital human scenes, in order to meet the personalized customization needs of users, users are usually provided with a face adjustment function. The face adjustment process is generally called face pinching. Users can create their own virtual face image according to their preferences. In the related technology, users need to repeatedly adjust the control parameters of facial parts during the face pinching process. The whole process is time-consuming and laborious, and user satisfaction is not high. Summary of the invention

[0003] In order to improve the efficiency of face pinching and the effect of virtual face, one or more embodiments of this specification provide a method and device for generating face modeling parameters, a method and device for generating virtual face, an electronic device, a storage medium and a computer program product.

[0004] In a first aspect, one or more embodiments of this specification provide a method for generating face modeling parameters, including:

[0005] Acquire a target facial image, wherein the target facial image includes a target face;

[0006] Extracting features of the target face image, and generating a virtual face image corresponding to the target face based on the extracted target face features;

[0007] Extracting features from the virtual face image to obtain virtual face features;

[0008] Generate target face modeling parameters corresponding to the target face based on the virtual face features.

[0009] In one or more embodiments, extracting features from the target face image and generating a virtual face image corresponding to the target face based on the extracted target face features includes:

[0010] Inputting the target face image into a pre-trained face adjustment model, wherein the face adjustment model includes a virtual face module and a face parameter module;

[0011] Using the encoder of the virtual face module to extract features of the target face image to obtain features of the target face;

[0012] Inputting the target facial features into the translator of the virtual face module to obtain a facial parameter vector output by the translator;

[0013] The facial parameter vector is input into the simulator of the virtual face module to obtain the virtual face image output by the simulator.

[0014] In one or more embodiments, extracting features from the virtual face image to obtain virtual face features includes:

[0015] Inputting the virtual face image into the encoder of the face parameter module to obtain the virtual face features output by the encoder;

[0016] The generating target face modeling parameters corresponding to the target face based on the virtual face features includes:

[0017] The virtual facial features are input into the translator of the facial parameter module to obtain the target facial modeling parameters output by the translator.

[0018] In one or more embodiments, the training process of the face parameter module in the face adjustment model includes:

[0019] Acquire a first data set, wherein sample data of the first data set includes virtual face samples;

[0020] Inputting the virtual face samples of the first data set into the simulator of the face parameter module to be trained, and adjusting the network parameters of the simulator based on the loss between the output result and the virtual face samples to obtain a trained simulator;

[0021] The virtual face samples of the first data set are input into the translator of the face parameter module, and based on the loss between the output result and the virtual face samples, the network parameters of the translator are adjusted until the convergence condition is met to obtain a trained face parameter module.

[0022] In one or more embodiments, the loss between the output result and the virtual face sample image includes image loss and / or face similarity loss.

[0023] In one or more embodiments, the training process of the virtual face module in the face adjustment model includes:

[0024] Acquire a second data set, wherein sample data of the second data set includes real face samples;

[0025] The trained face parameter module is copied, the real face samples of the second data set are input into the copied face parameter module, and the network parameters of the translator in the copied face parameter module are adjusted based on the loss between the output result and the real face samples until the convergence condition is met, thereby obtaining the virtual face module.

[0026] In one or more embodiments, the loss between the output result and the real face sample includes face similarity loss, facial feature segmentation loss, and face key point loss.

[0027] In one or more embodiments, the process of acquiring the target face image includes:

[0028] Obtaining a facial image uploaded by a user, and / or obtaining a facial image captured by a camera;

[0029] Perform face alignment on the face image to obtain the target face image.

[0030] In a second aspect, one or more embodiments of this specification provide a method for generating a virtual face, including:

[0031] Obtaining target face modeling parameters corresponding to the target face, wherein the target face modeling parameters are obtained by the method described in any embodiment of the first aspect;

[0032] A virtual face is generated by rendering based on the target face modeling parameters.

[0033] In one or more embodiments, the method described in this specification further includes:

[0034] Outputting and displaying the virtual face and adjustment options corresponding to the target face modeling parameters on a display interface;

[0035] The virtual face is re-rendered based on the user's adjustment operation on the adjustment option.

[0036] In a third aspect, one or more embodiments of this specification provide a device for generating face modeling parameters, including:

[0037] An image acquisition module is configured to acquire a target face image, wherein the target face image includes a target face;

[0038] A virtual image generation module is configured to extract features from the target face image and generate a virtual face image corresponding to the target face based on the extracted target face features;

[0039] A feature extraction module is configured to extract features from the virtual face image to obtain virtual face features;

[0040] The parameter generation module is configured to generate target face modeling parameters corresponding to the target face based on the virtual face features.

[0041] In one or more embodiments, the virtual image generation module is configured to:

[0042] Inputting the target face image into a pre-trained face adjustment model, wherein the face adjustment model includes a virtual face module and a face parameter module;

[0043] Using the encoder of the virtual face module to extract features of the target face image to obtain features of the target face;

[0044] Inputting the target facial features into the translator of the virtual face module to obtain a facial parameter vector output by the translator;

[0045] The facial parameter vector is input into the simulator of the virtual face module to obtain the virtual face image output by the simulator.

[0046] In one or more embodiments, the feature extraction module is configured to:

[0047] The virtual face image is input into the encoder of the face parameter module to obtain the virtual face features output by the encoder.

[0048] In one or more embodiments, the parameter generation module is configured to:

[0049] The virtual facial features are input into the translator of the facial parameter module to obtain the target facial modeling parameters output by the translator.

[0050] In one or more embodiments, the apparatus described in this specification further includes a training module, wherein the training module is configured to:

[0051] Acquire a first data set, wherein sample data of the first data set includes virtual face samples;

[0052] Inputting the virtual face samples of the first data set into the simulator of the face parameter module to be trained, and adjusting the network parameters of the simulator based on the loss between the output result and the virtual face samples to obtain a trained simulator;

[0053] The virtual face samples of the first data set are input into the translator of the face parameter module, and based on the loss between the output result and the virtual face samples, the network parameters of the translator are adjusted until the convergence condition is met to obtain a trained face parameter module.

[0054] In one or more embodiments, the loss between the output result and the virtual face sample image includes image loss and / or face similarity loss.

[0055] In one or more embodiments, the training module is configured to:

[0056] Acquire a second data set, wherein sample data of the second data set includes real face samples;

[0057] The trained face parameter module is copied, the real face samples of the second data set are input into the copied face parameter module, and the network parameters of the translator in the copied face parameter module are adjusted based on the loss between the output result and the real face samples until the convergence condition is met, thereby obtaining the virtual face module.

[0058] In one or more embodiments, the loss between the output result and the real face sample includes face similarity loss, facial feature segmentation loss, and face key point loss.

[0059] In one or more embodiments, the image acquisition module is configured to:

[0060] Obtaining a facial image uploaded by a user, and / or obtaining a facial image captured by a camera;

[0061] Perform face alignment on the face image to obtain the target face image.

[0062] In a fourth aspect, one or more embodiments of this specification provide a device for generating a virtual face, including:

[0063] A parameter acquisition module, configured to acquire target face modeling parameters corresponding to the target face, wherein the target face modeling parameters are obtained by the method described in any implementation manner of the first aspect;

[0064] The image rendering module is configured to render and generate a virtual face based on the target face modeling parameters.

[0065] In one or more embodiments, the image rendering module is configured to:

[0066] Outputting and displaying the virtual face and adjustment options corresponding to the target face modeling parameters on a display interface;

[0067] The virtual face is re-rendered based on the user's adjustment operation on the adjustment option.

[0068] In a fifth aspect, an embodiment of the present specification provides an electronic device, including:

[0069] Processor; and

[0070] A memory storing computer instructions, wherein the computer instructions are used to enable a processor to execute the method according to any embodiment of the first aspect or the second aspect.

[0071] In a sixth aspect, an embodiment of this specification provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in any embodiment of the first aspect or the second aspect.

[0072] In a seventh aspect, an embodiment of this specification provides a computer program product, which implements the method described in any embodiment of the first aspect or the second aspect when running.

[0073] The method for generating facial modeling parameters in the implementation mode of this specification includes obtaining a target facial image, extracting features from the target facial image, generating a virtual facial image corresponding to the target face based on the extracted target facial features, extracting features from the virtual facial image to obtain virtual facial features, and generating target facial modeling parameters corresponding to the target face based on the virtual facial features. In the implementation mode of this specification, a virtual facial image is generated by extracting features from the target facial image, and target facial modeling parameters are obtained based on predictions of the virtual facial image, thereby simplifying the user's face pinching operation and effectively improving the accuracy of the parameters, so that the virtual face generated by the rendering has a higher degree of restoration than the input face, thereby improving the user's face pinching experience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of a method for generating face modeling parameters according to some implementation methods of this specification.

[0075] Figure 2 It is a schematic diagram of a method for generating face modeling parameters according to some implementation methods of this specification.

[0076] Figure 3 It is a flowchart of a method for generating face modeling parameters according to some implementation methods of this specification.

[0077] Figure 4 It is a schematic diagram of a method for generating face modeling parameters according to some implementation methods of this specification.

[0078] Figure 5 It is a flowchart of a method for generating face modeling parameters according to some implementation methods of this specification.

[0079] Figure 6 It is a schematic diagram of a method for generating face modeling parameters according to some implementation methods of this specification.

[0080] Figure 7 It is a flowchart of a method for generating face modeling parameters according to some implementation methods of this specification.

[0081] Figure 8 It is a flowchart of a method for generating a virtual face according to some implementation modes of the present specification.

[0082] Fig. 9 It is a schematic diagram of a display interface in some implementation modes of this specification.

[0083] Fig.10 It is a structural block diagram of a device for generating face modeling parameters according to some implementation methods of this specification.

[0084] Fig.11 It is a structural block diagram of a virtual face generation device in some implementation modes of this specification.

[0085] Fig.12 It is a structural block diagram of an electronic device in some implementation modes of this specification. DETAILED DESCRIPTION

[0086] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0087] It should be noted that: in other implementations, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other implementations, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other implementations; and multiple steps described in this specification may be combined into a single step for description in other implementations.

[0088] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0089] With the development of 3D reconstruction technology, in some games and digital human scenes, in order to meet the user's personalized customization needs, users are usually provided with facial adjustment functions. The facial adjustment process is generally called face pinching. Users can create their own virtual facial image according to their preferences. Moreover, as the degree of freedom of face pinching becomes higher and higher, the types of facial parts that users can adjust independently are also increasing, which can achieve a richer face pinching experience.

[0090] During the face-pinching process, users need to repeatedly adjust the control parameters of facial parts. The whole process is time-consuming and labor-intensive, and the technical threshold is high. As a result, the virtual faces that users spend a lot of time pinching are far from the virtual images they want, and user satisfaction is not high.

[0091] In order to make the face pinching process easier for users, some face pinching systems can allow users to input facial images, and automatically generate a virtual facial image similar to the user through the input facial image. For example, users can input their own selfies into the system, and the system will automatically generate a virtual facial image similar to the user's real face, which greatly simplifies user operations.

[0092] In the related technology, the face pinching algorithm process based on the input picture is mainly as follows: first, the face image input by the user is aligned, then the face features are extracted using the feature extraction network, and then the face features are sent to the translator network to predict the facial parameters, and the rendering engine generates the corresponding virtual face image based on the facial parameters.

[0093] Although this method simplifies the user's face pinching operation, it is found in actual applications that this method has low accuracy of facial parameters, resulting in a low degree of restoration of the generated virtual face image. There is still a large difference between the face image input by the user, which makes it difficult to meet the user's expectations for the virtual image.

[0094] Based on this, the embodiments of this specification provide a method and device for generating facial modeling parameters, a method and device for generating a virtual face, an electronic device, a storage medium, and a computer program product, which aim to simplify the user's face pinching operation process, improve the accuracy of facial modeling parameters, and thereby improve the similarity between the virtual face and the user input face.

[0095] For ease of understanding and explanation, some of the nouns and terms that appear in the following text of this manual are explained here:

[0096] 1. Face modeling parameters

[0097] It can also be called facial parameters or face-pinching parameters. It means that in the face-pinching algorithm, a set of facial parameters with clear physical meanings are predicted to create a three-dimensional contour for the bone-driven model. Each parameter controls the attributes of one or more facial parts. The face modeling parameters are generally a parameter vector with 1 row and n columns.

[0098] Face modeling parameters can be further divided into continuous parameters and discrete parameters. Continuous parameters are used to control the continuous details of the face, such as eyebrow length, width, and thickness. Discrete parameters are used to control discrete features of the face, such as hairstyle, eyebrow style, beard style, etc.

[0099] 2. Encoder

[0100] In a deep learning model, the encoder is a neural network whose job is to extract image features of an input image and generate a latent vector that best fits a specific description of the input image.

[0101] 3. Decoder

[0102] The decoder works with the encoder to take the latent vector generated by the encoder as input and convert the latent vector back to the original data space or perform a specific task.

[0103] 4. Translator

[0104] The translator is a neural network whose role is to, given a face image, convert it into face modeling parameters based on a deep learning algorithm.

[0105] 5. Imitator

[0106] The simulator is a neural network whose main function is to simulate the physical behavior of the rendering engine and render the corresponding virtual image based on the face modeling parameters. The simulator is used mainly because the rendering process of the rendering engine is not differentiable, so it is impossible to directly use the rendering engine for deep learning optimization. As an alternative, a deep learning network can be trained to complete the optimization process. This deep learning network is the simulator.

[0107] Based on the above, in the implementation manner of this specification, an end-to-end face adjustment model is constructed by improving the network structure of the face adjustment model, introducing the task of pre-training rendered images, and generating face modeling parameters based on the rendered images, thereby improving the accuracy of the face modeling parameters. Figure 1 The architecture diagram of the face adjustment model in some embodiments of this specification is shown below. Figure 1 Provide explanation.

[0108] See also Figure 1 As shown, in the implementation manner of this specification, the end-to-end face adjustment model includes a virtual face module 101 and a face parameter module 102. The virtual face module 101 and the face parameter module 102 can be the same network architecture, which both include an encoder, a translator and an imitator. It is worth noting that the imitator can also be removed from the face parameter module 102 during the prediction stage.

[0109] In the example of this specification, the input of the virtual face module 101 is a target face image, which refers to a real face image input by a user. The encoder of the virtual face module 101 extracts features from the target face image to obtain target face features, which are then sent to a translator to output a facial parameter vector. The facial parameter vector is input to an imitator to render and generate a virtual face image.

[0110] In the implementation of this specification, in order to improve the accuracy of the face modeling parameters, it is necessary to pre-train the face parameter module 102 based on the virtual face image. Figure 1 As shown, the input of the face parameter module 102 is a virtual face image, and the encoder of the face parameter module 102 extracts features from the virtual face image to obtain virtual face features, which are then sent to the translator to output target face modeling parameters. It can be understood that when the face parameter module 102 is trained, the target face modeling parameters output by the translator are input to the imitator to re-render and generate a virtual face image, that is, the input and output of the face parameter module 102 are both virtual face images. In this process, the input and output have a one-to-one mapping relationship, so that the target face modeling parameters predicted and output by the trained translator are face modeling parameters with a highly consistent mapping relationship with the virtual face image, that is, the target face modeling parameters are more accurate.

[0111] To further illustrate the principle of the face modeling parameter generation method disclosed in the present invention, the following first combines Figures 2 to 7 The training process of the face adjustment model is explained.

[0112] See also Figure 2 As shown, first, an initial face parameter module can be constructed. The face parameter module mainly includes an encoder, a translator and an imitator. The encoder is used for image feature extraction. For example, in one example, the encoder can use an R50 residual network combined with an FPN (Feature Pyramid Networks) network architecture. The translator is mainly used to generate face modeling parameters based on the image features output by the encoder. The translator can use, for example, an MLP (Multilayer Perceptron) network. The imitator can simulate rendering and generate a virtual face image based on the face modeling parameters generated by the translator. The network architecture of the imitator can include, for example, a decoding head network and a decoder network.

[0113] In some embodiments, an encoder is introduced to better extract image features, and multiple groups of generators (local generators) can be used in the encoder to encode image features, thereby enhancing the generalization ability of the model.

[0114] exist Figure 2Based on the face parameter module of the example, model training is first performed to obtain the trained face parameter module 102. The training process can be seen in Figure 3 As shown, it includes:

[0115] S310: Acquire a first data set.

[0116] S320. Input the virtual face samples of the first data set into the simulator of the face parameter module to be trained, and adjust the network parameters of the simulator based on the loss between the output result and the virtual face samples to obtain the trained simulator.

[0117] S330. Input the virtual face samples of the first data set into the translator of the face parameter module, and adjust the network parameters of the translator based on the loss between the output result and the virtual face sample until the convergence condition is met, thereby obtaining a trained face parameter module.

[0118] In the implementation manner of this specification, the input of the face parameter module is a virtual face image, so when training the face parameter module, the first data set used is a training data set including a large number of virtual face samples.

[0119] See also Figure 4 As shown, the virtual face samples included in the first data set can be input into the face parameter module to be trained, and the face parameter module predicts and outputs the corresponding output results. It can be understood that, ideally, the output result of the face parameter module should be the same as the input virtual face sample. Therefore, a loss function can be constructed based on the difference between the output result of the face parameter module and the virtual face sample, and the network parameters of the face parameter module can be iteratively optimized in reverse based on the loss function until the model convergence condition is met, and the training process is stopped to obtain the trained face parameter module.

[0120] In some embodiments, the loss between the output result of the face parameter module and the virtual face sample may include, for example, image loss and / or face similarity loss, where the image loss is used to constrain the similarity between the output image and the sample image, and the face similarity loss is used to constrain the similarity between the face on the output image and the face on the sample image.

[0121] In some embodiments, during the training of the face parameter module, the network parameters of the encoder and the translator may be first fixed to optimize the simulator. After the simulator training is completed, the network parameters of the simulator may be fixed to optimize the encoder and the translator.

[0122] Combination Figure 4As shown, in the process of training the face parameter module using the virtual face samples of the first data set, the virtual face samples included in the first data set can first be input into the face parameter module, and the face parameter module predicts and outputs the corresponding output results. Then, the network parameters of the encoder and the translator are fixed, and the network parameters of the simulator are iteratively tuned based on the loss between the module output results and the virtual face samples. After a preset number of iterations, the training of the simulator is completed to obtain the trained simulator. After that, the network parameters of the simulator can be fixed, and the network parameters of the translator and the encoder can be iteratively tuned based on the loss between the module output results and the virtual face samples until the model convergence conditions are met, such as when the number of iterations reaches the preset number of rounds, the model training process can be completed to obtain the trained face parameter module 102.

[0123] It can be understood that since the translator of the facial parameter module is trained based on virtual facial samples, there is a highly consistent mapping relationship between the virtual facial image and the facial modeling parameters. Therefore, the facial modeling parameters predicted and output by the trained facial parameter module 102 based on the virtual facial image are highly accurate.

[0124] After the face parameter module 102 is obtained through training, the virtual face module 101 can be trained based on the trained face parameter module 102. Figure 5 Provide explanation.

[0125] like Figure 5 As shown, in some embodiments, the face modeling parameter generation method exemplified in this specification, the training process of the virtual face module 101 includes:

[0126] S510: Acquire a second data set.

[0127] S520, copy the trained face parameter module, input the real face samples of the second data set into the copied face parameter module, and adjust the network parameters of the translator in the copied face parameter module based on the loss between the output result and the real face sample until the convergence condition is met, thereby obtaining a virtual face module.

[0128] See also Figure 6 As shown, after the face parameter module 102 is obtained through training by the aforementioned method process, the face parameter module 102 can be model copied to obtain a network module that retains all parameters and weights of the face parameter module.

[0129] In the implementation manner of this specification, after the replication model is trained, what is obtained is the virtual face module 101. Therefore, when training the replication model, the second data set used is a training data set including a large number of real face samples.

[0130] See also Figure 6 As shown, the real face samples included in the second data set can be input into the replication model, and the replication model predicts and outputs the corresponding output results. It can be understood that, ideally, the output results of the replication model should be very close to the input real face samples. Therefore, a loss function can be constructed based on the difference between the output results of the replication model and the real face samples, and the network parameters of the translator of the replication model can be iteratively optimized in reverse based on the loss function until the model convergence condition is met, and the model training process is stopped. At this time, the model obtained is the virtual face module 101.

[0131] In some embodiments, the loss between the output result of the copied model and the real face sample may include, for example, face similarity loss, facial features segmentation loss, and face key point loss. It is worth noting that by introducing the facial features segmentation loss, the problem of insufficient fitting ability of facial details can be effectively alleviated, and the face fitting ability of the model can be improved.

[0132] After the virtual face module 101 and the face parameter module 102 are trained based on the aforementioned implementation steps, see Figure 1 As shown in , by connecting the output of the virtual face module 101 to the input of the face parameter module 102, an end-to-end face adjustment model in this specification can be constructed.

[0133] It can be understood that the end-to-end face adjustment model in the implementation manner of this specification is used to output the target face modeling parameters, so the imitator of the face parameter module 102 can be removed in the prediction stage. Those skilled in the art can understand this and this specification will not elaborate on it.

[0134] Based on the above face adjustment model, the following Figure 7 The method for generating face modeling parameters according to the implementation mode of this specification is described.

[0135] like Figure 7 As shown, in some implementations, the method for generating face modeling parameters in the example of this specification includes:

[0136] S710: Obtain a target face image.

[0137] S720: extract features from the target face image, and generate a virtual face image corresponding to the target face based on the extracted target face features.

[0138] S730: Extract features from the virtual face image to obtain virtual face features.

[0139] S740: Generate target face modeling parameters corresponding to the target face based on the virtual face features.

[0140] In the implementation manner of this specification, the target facial image refers to a real facial image that includes the target face. The target facial image can be uploaded by the user or collected in real time by a camera of an electronic device. This specification does not impose any restrictions on this.

[0141] For example, in one example, the user expects the generated virtual face image to be close to his or her real face, so that the user can call the camera of the electronic device to take a real-time selfie, or upload his or her own selfie from the album as the target face image.

[0142] For example, in another example, the user expects the generated virtual facial image to be similar to a favorite character, so the user can upload the facial image of the character as the target facial image.

[0143] In some embodiments, in order to improve the quality of facial images, facial images uploaded by users may have problems such as angle rotation and center offset, etc., and facial alignment can be performed on the facial images uploaded by users to obtain target facial images. Specifically, face detection can be performed on the facial images uploaded by users first, and then key feature points in the face, such as key points of facial features, can be located. Then, similarity transformations can be performed on the key feature points, including rotation, scaling, translation, etc., so that the key feature points are aligned to the reference. Finally, the facial area is cropped according to the aligned feature points to obtain the target facial image. The process of facial alignment can be understood and fully implemented with reference to related technologies, and this specification will not be repeated.

[0144] Combination Figure 1 As shown, in the implementation mode of the present specification, after the face alignment of the face image is performed to obtain the target face image, the target face image can be input into the end-to-end face adjustment model. First, the target face image is input into the encoder of the virtual face module 101, and the encoder of the virtual face module 101 obtains the target face features of the target face by extracting features from the target face image. For example, the target face features can be the latent vector of the W+ space. Then, the target face features are sent to the translator of the virtual face module 101, and the translator generates a facial parameter vector based on the target face features. Thereafter, the facial parameter vector is sent to the simulator of the virtual face module 101 to render and generate the corresponding virtual face image.

[0145] In the implementation mode of this specification, the virtual face image is continuously sent to the face parameter module 102, and the encoder of the face parameter module 102 extracts features from the virtual face image to obtain virtual face features, for example, the virtual face features can be latent vectors in the W+ space. Then, the virtual face features are sent to the translator of the face parameter module 102, and the translator generates and outputs target face modeling parameters based on the virtual face features.

[0146] In combination with the foregoing, it can be understood that the face parameter module 102 is a network module trained based on virtual face samples. Since there is a highly consistent mapping relationship between the virtual face image and the face modeling parameters, the face modeling parameters predicted and output by the face parameter module 102 based on the virtual face image are highly accurate, that is, the target face modeling parameters can accurately express the facial attributes of the virtual face image.

[0147] In the implementation manner of this specification, after obtaining the target face modeling parameters, a virtual face can be generated by rendering based on the target face modeling parameters. The following text of this specification describes the rendering process of the virtual face.

[0148] In some embodiments, this specification provides a method for generating a virtual face, such as Figure 8 As shown, the method includes:

[0149] S810: Obtain target face modeling parameters corresponding to the target face.

[0150] S820: Generate a virtual face by rendering based on the target face modeling parameters.

[0151] In the implementation manner of this specification, the target face modeling parameters refer to the face modeling parameters obtained by the face modeling parameter determination method in any of the above implementation manners, combined with Figure 1 As shown, after obtaining the target face modeling parameters, the rendering engine can generate a virtual face based on the target face modeling parameters.

[0152] It is worth noting that in the implementation manner of this specification, the target face modeling parameters can be face modeling parameters that are common to 2D face pinching scenes and 3D face pinching scenes. Therefore, the virtual face generation method of this specification can also be compatible with the face modeling parameters of 3D face pinching scenes. For example, the 3D algorithm can be connected to the method of this specification, and the rendering engine can generate a corresponding virtual face based on the face modeling parameters of the 3D algorithm.

[0153] From the above, it can be seen that in the implementation of this specification, by being compatible with 2D and 3D face modeling parameters, it is possible to support the fusion of 2D and 3D face pinching algorithm results, thereby improving the scalability and robustness of the method.

[0154] In some embodiments, in consideration of the scenario in which the user may fine-tune the virtual face generated by the system, after rendering and generating the virtual face, the virtual face and adjustment options corresponding to the target face modeling parameters may be output and displayed on the display interface of the electronic device.

[0155] For example, in one example, the electronic device displays an interface such as Fig. 9As shown, a virtual face is displayed on the display interface, and adjustment options for the target face modeling parameters are displayed next to the virtual face. In this example, the adjustment options are displayed in the form of sliding bars. If the user is satisfied with the virtual face generated by the system, he can directly click on the Save Image button below.

[0156] If the user is not satisfied with the generated virtual face and wants to fine-tune it, he can adjust the adjustment options, such as Fig. 9 In the example, the user can click on the facial part they want to adjust, and then adjust the facial modeling parameters by dragging the slider. After the user adjusts the facial modeling parameters, the rendering engine re-renders and displays the virtual face based on the adjusted facial modeling parameters. The user can adjust the virtual face to their satisfaction and save it.

[0157] From the above, it can be seen that in the implementation of this specification, by introducing an encoder in the face adjustment model to better encode image features, it is possible to learn the feature mapping of real people to virtual faces, so that the generated virtual faces are more restored to real faces, and support the generation of virtual faces under custom attributes or categories. Moreover, by being compatible with 2D and 3D face modeling parameters, it is possible to support the fusion of 2D and 3D face pinching algorithm results, thereby improving the scalability and robustness of the method, and using an end-to-end face adjustment model to achieve real-time operation of virtual face images. The average single-image inference time is only 45ms after actual measurement, and the algorithm is highly efficient. In addition, during the training process of the face adjustment model, the problem of insufficient fitting ability of facial details can be effectively alleviated by introducing facial features segmentation loss, further improving the accuracy of face modeling parameters, and improving user face pinching experience and satisfaction.

[0158] In some embodiments, this specification provides a device for generating face modeling parameters, such as Fig.10 As shown, the device comprises:

[0159] The image acquisition module 10 is configured to acquire a target face image, wherein the target face image includes a target face;

[0160] The virtual image generation module 20 is configured to extract features of the target face image and generate a virtual face image corresponding to the target face based on the extracted target face features;

[0161] The feature extraction module 30 is configured to extract features from the virtual face image to obtain virtual face features;

[0162] The parameter generation module 40 is configured to generate target face modeling parameters corresponding to the target face based on the virtual face features.

[0163] In one or more embodiments, the virtual image generation module 20 is configured to:

[0164] Inputting the target face image into a pre-trained face adjustment model, wherein the face adjustment model includes a virtual face module and a face parameter module;

[0165] Using the encoder of the virtual face module to extract features of the target face image to obtain features of the target face;

[0166] Inputting the target facial features into the translator of the virtual face module to obtain a facial parameter vector output by the translator;

[0167] The facial parameter vector is input into the simulator of the virtual face module to obtain the virtual face image output by the simulator.

[0168] In one or more embodiments, the feature extraction module 30 is configured to:

[0169] The virtual face image is input into the encoder of the face parameter module to obtain the virtual face features output by the encoder.

[0170] In one or more embodiments, the parameter generation module 40 is configured to:

[0171] The virtual facial features are input into the translator of the facial parameter module to obtain the target facial modeling parameters output by the translator.

[0172] In one or more embodiments, the generating device described in this specification further includes a training module, wherein the training module is configured to:

[0173] Acquire a first data set, wherein sample data of the first data set includes virtual face samples;

[0174] Inputting the virtual face samples of the first data set into the simulator of the face parameter module to be trained, and adjusting the network parameters of the simulator based on the loss between the output result and the virtual face samples to obtain a trained simulator;

[0175] The virtual face samples of the first data set are input into the translator of the face parameter module, and based on the loss between the output result and the virtual face samples, the network parameters of the translator are adjusted until the convergence condition is met to obtain a trained face parameter module.

[0176] In one or more embodiments, the loss between the output result and the virtual face sample image includes image loss and / or face similarity loss.

[0177] In one or more embodiments, the training module is configured to:

[0178] Acquire a second data set, wherein sample data of the second data set includes real face samples;

[0179] The trained face parameter module is copied, the real face samples of the second data set are input into the copied face parameter module, and the network parameters of the translator in the copied face parameter module are adjusted based on the loss between the output result and the real face samples until the convergence condition is met, thereby obtaining the virtual face module.

[0180] In one or more embodiments, the loss between the output result and the real face sample includes face similarity loss, facial feature segmentation loss, and face key point loss.

[0181] In one or more embodiments, the image acquisition module 10 is configured to:

[0182] Obtaining a facial image uploaded by a user, and / or obtaining a facial image captured by a camera;

[0183] Perform face alignment on the face image to obtain the target face image.

[0184] In some embodiments, this specification provides a device for generating a virtual face, such as Fig.11 As shown, the device comprises:

[0185] A parameter acquisition module 50 is configured to acquire target face modeling parameters corresponding to the target face, wherein the target face modeling parameters are obtained by the face modeling parameter generation method described in any of the above embodiments;

[0186] The image rendering module 60 is configured to render and generate a virtual face based on the target face modeling parameters.

[0187] In one or more embodiments, the image rendering module 60 is configured to:

[0188] Outputting and displaying the virtual human face and the target human face modeling parameters on a display interface;

[0189] The virtual face is re-rendered based on the user's adjustment operation on the target face modeling parameters.

[0190] In some implementations, this specification example provides an electronic device, including:

[0191] Processor; and

[0192] A memory stores computer instructions, wherein the computer instructions are used to enable a processor to execute the method described in any of the above embodiments.

[0193] In some embodiments, the present specification provides an example storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in any of the above embodiments.

[0194] In some implementations, the examples in this specification provide a computer program product, which implements the method of any of the aforementioned implementations when running.

[0195] Fig.12 is a schematic structural diagram of a device provided by an exemplary embodiment. Fig.12 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other scenarios. One or more embodiments of this specification may be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0196] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0197] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0198] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0199] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0200] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A method for generating face modeling parameters, characterized in that: include: Acquire a target facial image, wherein the target facial image includes a target face; Extracting features of the target face image, and generating a virtual face image corresponding to the target face based on the extracted target face features; Extracting features from the virtual face image to obtain virtual face features; Generate target face modeling parameters corresponding to the target face based on the virtual face features.

2. The method according to claim 1, characterized in that: The step of extracting features from the target face image and generating a virtual face image corresponding to the target face based on the extracted target face features includes: Inputting the target face image into a pre-trained face adjustment model, wherein the face adjustment model includes a virtual face module and a face parameter module; Using the encoder of the virtual face module to extract features of the target face image to obtain features of the target face; Inputting the target facial features into the translator of the virtual face module to obtain a facial parameter vector output by the translator; The facial parameter vector is input into the simulator of the virtual face module to obtain the virtual face image output by the simulator.

3. The method according to claim 2, characterized in that The step of extracting features from the virtual face image to obtain virtual face features includes: Inputting the virtual face image into the encoder of the face parameter module to obtain the virtual face features output by the encoder; The generating target face modeling parameters corresponding to the target face based on the virtual face features includes: The virtual facial features are input into the translator of the facial parameter module to obtain the target facial modeling parameters output by the translator.

4. The method according to claim 3, characterized in that The training process of the face parameter module in the face adjustment model includes: Acquire a first data set, wherein sample data of the first data set includes virtual face samples; Inputting the virtual face samples of the first data set into the simulator of the face parameter module to be trained, and adjusting the network parameters of the simulator based on the loss between the output result and the virtual face samples to obtain a trained simulator; The virtual face samples of the first data set are input into the translator of the face parameter module, and based on the loss between the output result and the virtual face samples, the network parameters of the translator are adjusted until the convergence condition is met to obtain a trained face parameter module.

5. The method according to claim 4, characterized in that The loss between the output result and the virtual face sample includes image loss and / or face similarity loss.

6. The method according to claim 4, characterized in that The training process of the virtual face module in the face adjustment model includes: Acquire a second data set, wherein sample data of the second data set includes real face samples; The trained face parameter module is copied, the real face samples of the second data set are input into the copied face parameter module, and the network parameters of the translator in the copied face parameter module are adjusted based on the loss between the output result and the real face samples until the convergence condition is met, thereby obtaining the virtual face module.

7. The method according to claim 6, characterized in that The loss between the output result and the real face sample includes face similarity loss, facial features segmentation loss and face key point loss.

8. The method according to claim 1, characterized in that The acquiring of the target face image comprises: Obtaining a facial image uploaded by a user, and / or obtaining a facial image captured by a camera; Perform face alignment on the face image to obtain the target face image.

9. A method for generating a virtual face, characterized in that: include: Obtaining target face modeling parameters corresponding to the target face, wherein the target face modeling parameters are obtained based on the method described in any one of claims 1 to 8; A virtual face is generated by rendering based on the target face modeling parameters.

10. The method according to claim 9, characterized in that Also includes: Outputting and displaying the virtual face and adjustment options corresponding to the target face modeling parameters on a display interface; The virtual face is re-rendered based on the user's adjustment operation on the adjustment option.

11. A device for generating face modeling parameters, characterized in that: include: An image acquisition module is configured to acquire a target face image, wherein the target face image includes a target face; A virtual image generation module is configured to extract features from the target face image and generate a virtual face image corresponding to the target face based on the extracted target face features; A feature extraction module is configured to extract features from the virtual face image to obtain virtual face features; The parameter generation module is configured to generate target face modeling parameters corresponding to the target face based on the virtual face features.

12. A device for generating a virtual face, characterized in that: include: A parameter acquisition module, configured to acquire target face modeling parameters corresponding to the target face, wherein the target face modeling parameters are obtained based on the method described in any one of claims 1 to 8; The image rendering module is configured to render and generate a virtual face based on the target face modeling parameters.

13. An electronic device, characterized in that: include: processor; and A memory storing computer instructions, wherein the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 10.

14. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 10.

15. A computer program product, characterized in that The computer program product implements the method according to any one of claims 1 to 10 when running.

Citation Information

Patent Citations

  • Virtual face generation method and device, equipment and storage medium

    CN114155587A

  • Image generation model training method and device, image generation method and device, medium and equipment

    CN114863214A

  • Face model construction method, face image generation method, equipment and storage medium

    CN114972661A

  • Face pinching parameter generation method and device, electronic equipment and storage medium

    CN115527255A

  • Cascaded domain bridging for image generation

    CN117876558A