Method for training skeleton parameter generation model and generating three-dimensional face model

Abstract

The invention provides a method and device for training a skeleton parameter generation model and generating a three-dimensional face model, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC, digital humans, intelligent e-commerce and the like. The training method of the skeleton parameter generation model comprises the steps of obtaining a first training set; constructing a neural network model comprising an extraction module, an initial conversion module and a generation module; adjusting parameters of an initial conversion module in the neural network model by using the sample face image to obtain a conversion module; and obtaining a skeleton parameter generation model according to the extraction module and the conversion module. The three-dimensional face model generation method comprises the steps of obtaining a to-be-processed face image; inputting a to-be-processed face image into the skeleton parameter generation model, and obtaining a target skeleton parameter according to an output result of the skeleton parameter generation model; and obtaining a three-dimensional face model corresponding to the to-be-processed face image according to the target skeleton parameter.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to AIGC, digital human, smart e-commerce and other scenarios. Provided are a method, device, electronic device and readable storage medium for training a skeleton parameter generation model and generating a three-dimensional face model. Background Art

[0002] With the rapid development of mobile terminals and computer technology, the usage rate of games, virtual social applications and other applications has gradually increased. In games and virtual social applications, users usually need to personalize their virtual images, which has created a great demand for "face pinching", which specifically refers to creating a three-dimensional face model of the virtual image. Since the creation of a three-dimensional face model requires the corresponding bone parameters, how to obtain the bone parameters more accurately and quickly is a technical problem that needs to be solved urgently. Summary of the invention

[0003] According to a first aspect of the present disclosure, a training method for a bone parameter generation model is provided, comprising: obtaining a first training set, wherein the first training set comprises a plurality of sample face images; constructing a neural network model comprising an extraction module, an initial conversion module and a generation module, wherein the extraction module is used to extract face image features from an input face image, the initial conversion module is used to obtain bone parameters according to the face image features, and the generation module is used to generate a simulated face image according to the bone parameters; using the sample face images to adjust the parameters of the initial conversion module in the neural network model to obtain a conversion module; and obtaining a bone parameter generation model according to the extraction module and the conversion module.

[0004] According to a second aspect of the present disclosure, a method for generating a three-dimensional face model is provided, comprising: obtaining a face image to be processed; inputting the face image to be processed into a bone parameter generation model, and obtaining target bone parameters according to an output result of the bone parameter generation model; and obtaining a three-dimensional face model corresponding to the face image to be processed according to the target bone parameters.

[0005] According to a third aspect of the present disclosure, a training device for a bone parameter generation model is provided, comprising: a first acquisition unit, used to acquire a first training set, wherein the first training set includes multiple sample face images; a construction unit, used to construct a neural network model including an extraction module, an initial conversion module and a generation module, wherein the extraction module is used to extract face image features from an input face image, the initial conversion module is used to obtain bone parameters according to the face image features, and the generation module is used to generate a simulated face image according to the bone parameters; a first training unit, used to adjust the parameters of the initial conversion module in the neural network model using the sample face images to obtain a conversion module; and a first processing unit, used to obtain a bone parameter generation model according to the extraction module and the conversion module.

[0006] According to a fourth aspect of the present disclosure, a device for generating a three-dimensional face model is provided, comprising: a second acquisition unit, used to acquire a face image to be processed; a second processing unit, used to input the face image to be processed into a bone parameter generation model, and obtain target bone parameters according to an output result of the bone parameter generation model; and a generation unit, used to obtain a three-dimensional face model corresponding to the face image to be processed according to the target bone parameters.

[0007] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0008] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0009] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0013] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0016] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0017] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0018] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0019] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0020] Fig. 9 It is a block diagram of an electronic device used to implement the training method of the skeleton parameter generation model or the generation method of the three-dimensional face model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and mechanisms is omitted in the following description.

[0022] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the training method of the skeleton parameter generation model of this embodiment specifically includes the following steps:

[0023] S101, obtaining a first training set, where the first training set includes a plurality of sample face images;

[0024] S102, constructing a neural network model including an extraction module, an initial conversion module and a generation module, wherein the extraction module is used to extract facial image features from an input facial image, the initial conversion module is used to obtain bone parameters according to the facial image features, and the generation module is used to generate a simulated facial image according to the bone parameters;

[0025] S103, using the sample face image to adjust the parameters of the initial conversion module in the neural network model to obtain a conversion module;

[0026] S104: Obtain a skeleton parameter generation model according to the extraction module and the conversion module.

[0027] The training method of the skeleton parameter generation model of the present embodiment can achieve the purpose of training the initial conversion module in the neural network model only based on sample facial images by adding a generation module for generating simulated facial images based on skeleton parameters in the neural network model, so that the training process of the initial conversion module does not need to obtain more complex training data, thereby simplifying the training steps of the skeleton parameter generation model and improving the training efficiency of the skeleton parameter generation model.

[0028] After completing the training of the initial conversion module to obtain the conversion module, this embodiment can use the bone parameter generation model including the conversion module to obtain the corresponding bone parameters according to the input face image, which can improve the efficiency and accuracy of obtaining the bone parameters.

[0029] The multiple sample face images obtained by executing S101 in this embodiment are images including only the face parts of different users, and the face parts included in the sample face images are the front faces of the users. It can be understood that this embodiment obtains the user's permission when obtaining the sample face images corresponding to different users and using the sample face images.

[0030] When executing S101 to obtain sample facial images, the present embodiment may first obtain user images corresponding to different users, which may include body, limbs, etc. in addition to the facial part; then, a plurality of sample facial images may be extracted from different user images by a face alignment method; in the present embodiment, face alignment specifically refers to the process of extracting and standardizing faces from user images, with the purpose of aligning key points (such as eyes, nose, mouth, etc.) of different extracted facial images to the same position, thereby improving the accuracy and consistency of subsequent processing.

[0031] It is understandable that, when executing S101, this embodiment may also directly obtain images of facial parts corresponding to different users as a plurality of sample facial images.

[0032] In this embodiment, after executing S101 to obtain the first training set, S102 is executed to construct a neural network model including an extraction module, an initial conversion module and a generation module.

[0033] In the neural network model constructed by executing S102 in this embodiment, the extraction module is used to extract facial image features from the input facial image (during the training process, the input facial image is a sample facial image); this embodiment can use the feature extraction module in the trained facial recognition model as the extraction module in the constructed neural network model.

[0034] In the neural network model constructed by executing S102 in this embodiment, the initial conversion module is used to obtain the skeleton parameters of the user's face in the input face image according to the face image features output by the extraction module.

[0035] In this embodiment, the bone parameters are parameters that characterize the bone features of different parts of the human face (eyes, mouth, nose, eyebrows, ears, etc.), and the bone features of different parts include size features, shape features, position features, etc. of different parts; the bone parameters can be used in three-dimensional (3D) modeling software, game engines, etc. to create a realistic three-dimensional face model.

[0036] In the neural network model constructed by executing S102 in this embodiment, the generation module is used to generate a simulated face image according to the bone parameters output by the initial conversion module, and the simulated face image only includes the face part of the virtual image.

[0037] In this embodiment, the simulated facial image generated by the generation module is a facial image corresponding to the virtual image of the user in the input facial image; specifically, the face in the generated simulated facial image has a high degree of similarity with the face in the sample facial image.

[0038] The generation module in this embodiment can be obtained by pre-training, or can be obtained by real-time training when building a neural network model. The training process of the generation module is described in detail below.

[0039] After executing S102 to construct a neural network model including an extraction module, an initial conversion module and a generation module, this embodiment executes S103 to adjust the parameters of the initial conversion module in the training neural network model using the sample face image to obtain a conversion module.

[0040] When executing S103, this embodiment only uses multiple sample face images to adjust the parameters of the initial conversion module in the neural network model, and does not adjust the parameters of the extraction module and the generation module in the neural network model.

[0041] Specifically, when executing S103, the present embodiment uses the sample face image to adjust the parameters of the initial conversion module in the training neural network model to obtain the conversion module. The implementation method that can be adopted is: inputting the sample face image into the neural network model to obtain a simulated face image output by the neural network model; obtaining a first target loss function value based on the sample face image and the simulated face image; and adjusting the parameters of the initial conversion module in the neural network model based on the obtained first target loss function value to obtain the conversion module.

[0042] When executing S103, this embodiment can stop adjusting the parameters of the initial conversion module to obtain the conversion module when it is determined that the first target loss function value reaches a preset value; or stop adjusting the parameters of the initial conversion module to obtain the conversion module when it is determined that the number of times the first target loss function value is obtained exceeds a preset number.

[0043] Among them, when executing S103 to input the sample facial image into the neural network model and obtain the simulated facial image output by the neural network model, the implementation method that can be adopted in this embodiment is: input the sample facial image into the extraction module in the neural network model to obtain the sample facial image features output by the extraction module; input the obtained sample facial image features into the initial conversion module in the neural network model to obtain the sample bone parameters output by the initial conversion module; input the obtained sample bone parameters into the generation module in the neural network model to obtain the simulated facial image output by the generation module.

[0044] When executing S103 in this embodiment to obtain the first target loss function value based on the sample face image and the simulated face image, the pixel error between the segmentation map of the sample face image and the segmentation map of the simulated face image can be used as the first target loss function value; the error between the key point coordinates in the sample face image and the simulated face image can be used as the first target loss function value; the pixel error and the error between the key point coordinates can also be calculated simultaneously as the first target loss function value.

[0045] After executing S103 to obtain the conversion module, this embodiment executes S104 to obtain a skeleton parameter generation model according to the extraction module and the conversion module.

[0046] After completing the training of the initial conversion module to obtain the conversion module, this embodiment can use the extraction module and the conversion module in the neural network model to obtain the bone parameter generation model.

[0047] The present embodiment executes the skeleton parameter generation model obtained in S104, and can output the skeleton parameters of the user's face in the input face image according to the input face image, and then use the obtained skeleton parameters to generate a three-dimensional face model of the virtual image of the corresponding user.

[0048] Therefore, the bone parameter generation model obtained in this embodiment can be applied to scenes such as games, metaverse, AR (Augmented Reality), VR (Virtual Reality), etc., so that users can create their own virtual images more quickly and accurately.

[0049] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown in , the generation module in this embodiment can be trained in the following way:

[0050] S201, obtaining a second training set, wherein the second training set includes a plurality of candidate skeleton parameters and annotated face images corresponding to the plurality of candidate skeleton parameters;

[0051] S202, inputting the candidate skeleton parameters into an initial generation module to obtain a predicted face image output by the initial generation module;

[0052] S203, obtaining a second objective loss function value according to the labeled face image and the predicted face image;

[0053] S204. Adjust the parameters of the initial generation module according to the second objective loss function value to obtain a generation module.

[0054] That is to say, this embodiment uses candidate bone parameters and their corresponding annotated facial images to train the initial generation module, thereby obtaining a generation module. The obtained generation module can output the corresponding facial image based on the input bone parameters, and the facial image output by the generation module corresponds to the virtual image.

[0055] In the second training set obtained by executing S201 in this embodiment, the multiple candidate bone parameters can be manually input, or multiple randomly constructed bone parameters can be used as the multiple candidate bone parameters.

[0056] In the second training set obtained by executing S201 in this embodiment, the annotated facial image corresponding to the candidate skeletal parameters can be manually input, or the candidate skeletal parameters can be input into the rendering engine of the three-dimensional face model (such as the game engine Unity, etc.), and the annotated facial image corresponding to the input candidate skeletal parameters can be obtained according to the output result of the rendering engine, for example, the frontal image of the three-dimensional face model output by the rendering engine is captured as the annotated facial image.

[0057] That is to say, this embodiment can obtain annotated facial images corresponding to different candidate bone parameters through a rendering engine, and then use the candidate bone parameters and their corresponding annotated facial images to complete the training of the initial generation module, which can reduce the cost of obtaining annotated facial images and improve the training efficiency of the initial generation module.

[0058] When executing S203 in this embodiment to obtain the second target loss function value based on the annotated face image and the predicted face image, the pixel error between the segmentation map of the annotated face image and the segmentation map of the predicted face image can be used as the second target loss function value; the error between the coordinates of key points in the annotated face image and the predicted face image can also be used as the second target loss function value; and the error between the pixel error and the error between the key point coordinates can also be calculated simultaneously as the second target loss function value.

[0059] When executing S204, this embodiment can stop adjusting the parameters of the initial generation module to obtain the generation module when it is determined that the second objective loss function value reaches a preset value; or stop adjusting the parameters of the initial generation module to obtain the generation module when it is determined that the number of times the second objective loss function value is obtained exceeds a preset number.

[0060] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure. Figure 3 As shown in , in this embodiment, when executing “obtaining a first target loss function value according to the sample face image and the simulated face image”, the following steps can be adopted:

[0061] S301, inputting the simulated face image into the extraction module to obtain the simulated face image features output by the extraction module;

[0062] S302, inputting the simulated face image features into the initial conversion module to obtain simulated bone parameters output by the initial conversion module;

[0063] S303, obtaining a first loss function value according to the sample bone parameters corresponding to the sample face image and the simulated bone parameters;

[0064] S304. Obtain the first target loss function value according to the first loss function value.

[0065] That is to say, this embodiment will also obtain the first target loss function value for adjusting the parameters of the initial conversion module based on the simulated bone parameters corresponding to the simulated facial image, so that the first target loss function value can include the constraint part of the bone parameters, thereby improving the adjustment accuracy and efficiency of the parameters of the initial conversion module.

[0066] When executing S303 , this embodiment may calculate the error between the sample bone parameter and the bone parameter corresponding to the same part in the simulated bone parameter to obtain the first loss function value.

[0067] When executing S304 in this embodiment, the first loss function value may be directly used as the first target loss function value.

[0068] When executing S304 and obtaining the first target loss function value according to the first loss function value, this embodiment can also obtain the second loss function value according to the pixel error between the segmentation map of the sample face image and the segmentation map of the simulated face image and / or the error between the coordinates of key points in the sample face image and the simulated face image, and then obtain the first target loss function value according to the first loss function value and the second loss function value.

[0069] That is to say, this embodiment can further enrich the types of loss functions in the first objective loss function value, so that the adjustment process of the parameters of the initial conversion module is subject to more constraints, further improving the adjustment accuracy of the parameters of the initial conversion module.

[0070] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 As shown in , this embodiment shows a structural diagram of the neural network model when training to obtain a bone parameter generation model: in the neural network model, it includes an extraction module, an initial conversion module and a generation module; after the sample face image is input into the neural network model, the extraction module obtains the sample face image features according to the sample face image, the initial conversion module obtains the sample bone parameters according to the sample face image features, and the generation module obtains the simulated face image according to the sample bone parameters; after completing the training of the initial conversion module to obtain the conversion module, the extraction module and the conversion module in the neural network model are extracted as the bone parameter generation model.

[0071] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5 As shown in , the method for generating a 3D face model of this embodiment specifically includes the following steps:

[0072] S501, obtaining a face image to be processed;

[0073] S502: Input the face image to be processed into a bone parameter generation model, and obtain target bone parameters according to the output result of the bone parameter generation model:

[0074] S503: Obtain a three-dimensional face model corresponding to the face image to be processed according to the target skeleton parameters.

[0075] This embodiment obtains the target bone parameters corresponding to the facial image to be processed by generating a model with bone parameters obtained through pre-training, which can improve the accuracy and efficiency of obtaining the target bone parameters, and further improve the accuracy and efficiency of obtaining a three-dimensional facial model based on the target bone parameters.

[0076] The method for generating a 3D face model provided in this embodiment can create a 3D face model of a virtual image that matches the face image based on the face image; the method can be applied to the client, for example, when the user wants to "pinch the face" of a game character, the face image can be input into the game software built into the client, and the game software generates a 3D face model similar to the face based on the face image. Therefore, the method for generating a 3D face model provided in this embodiment can reduce the complexity of "pinching the face" and improve the accuracy and efficiency of "pinching the face".

[0077] The facial image to be processed obtained by executing S501 in this embodiment is an image that only includes the facial part of the corresponding user, and the facial part of the user is the front face of the user; if the obtained image is not a facial image, this embodiment can obtain an image of the corresponding facial part from the image by image capture or face alignment methods.

[0078] In this embodiment, when executing S502 and inputting the face image to be processed into the bone parameter generation model to obtain the target bone parameters, the face image to be processed is first input into the extraction module in the bone parameter generation model to obtain the target face image features output by the extraction module; then the target face image features are input into the conversion module in the bone parameter generation model to obtain the target bone parameters output by the conversion module.

[0079] The three-dimensional face model corresponding to the face image to be processed generated by executing S503 in this embodiment is the user's "face pinching" result. The three-dimensional face model can be used to create a virtual image of the corresponding user in the game or metaverse; the user can rotate, enlarge, stretch, translate and other operations on the generated three-dimensional face model to modify or view the facial details of the three-dimensional face model at different angles.

[0080] When executing S503, this embodiment can input the target bone parameters output by the bone parameter generation model into a rendering engine (such as the game engine Unity) for rendering, thereby obtaining a three-dimensional face model of the corresponding user.

[0081] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure. Figure 6As shown in the embodiment, a flowchart is shown when a skeletal parameter generation model is used to generate a three-dimensional face model; the acquired face image to be processed is input into the skeletal parameter generation model to obtain the target skeletal parameters output by the skeletal parameter generation model; the target skeletal parameters are input into the rendering engine to obtain the three-dimensional face model of the virtual image corresponding to the user output by the rendering engine.

[0082] Figure 7 is a schematic diagram according to the seventh embodiment of the present disclosure. Figure 7 As shown, the training device 700 for generating a skeleton parameter model in this embodiment includes:

[0083] A first acquisition unit 701 is used to acquire a first training set, where the first training set includes a plurality of sample face images;

[0084] A construction unit 702 is used to construct a neural network model including an extraction module, an initial conversion module and a generation module, wherein the extraction module is used to extract facial image features from an input facial image, the initial conversion module is used to obtain bone parameters according to the facial image features, and the generation module is used to generate a simulated facial image according to the bone parameters;

[0085] The first training unit 703 is used to adjust the parameters of the initial conversion module in the neural network model using the sample face image to obtain a conversion module;

[0086] The first processing unit 704 is used to obtain a skeleton parameter generation model according to the extraction module and the conversion module.

[0087] The multiple sample face images acquired by the first acquisition unit 701 are images including only the face parts of different users, and the face parts included in the sample face images are the front faces of the users. It can be understood that this embodiment obtains the user's permission when acquiring the sample face images corresponding to different users and using the sample face images.

[0088] When acquiring sample facial images, the first acquisition unit 701 may first acquire user images corresponding to different users, which may include body, limbs, etc. in addition to the facial part; then, a plurality of sample facial images may be extracted from different user images by a facial alignment method; in this embodiment, facial alignment specifically refers to the process of extracting and standardizing faces from user images, with the purpose of aligning key points (such as eyes, nose, mouth, etc.) of different extracted facial images to the same position, thereby improving the accuracy and consistency of subsequent processing.

[0089] It is understandable that the first acquiring unit 701 may also directly acquire images of facial parts corresponding to different users as a plurality of sample facial images.

[0090] In this embodiment, after the first acquisition unit 701 acquires the first training set, the construction unit 702 constructs a neural network model including an extraction module, an initial conversion module and a generation module.

[0091] In the neural network model constructed by the construction unit 702, the extraction module is used to extract facial image features from the input facial image (during the training process, the input facial image is a sample facial image); this embodiment can use the feature extraction module in the trained face recognition model as the extraction module in the constructed neural network model.

[0092] In the neural network model constructed by the construction unit 702, the initial conversion module is used to obtain the skeleton parameters of the user's face in the input face image according to the face image features output by the extraction module.

[0093] In this embodiment, the bone parameters are parameters that characterize the bone features of different parts of the human face (eyes, mouth, nose, eyebrows, ears, etc.), and the bone features of different parts include size features, shape features, position features, etc. of different parts; the bone parameters can be used in three-dimensional (3D) modeling software, game engines, etc. to create a realistic three-dimensional face model.

[0094] In the neural network model constructed by the construction unit 702, the generation module is used to generate a simulated face image according to the bone parameters output by the initial conversion module, and the simulated face image only includes the face part of the virtual image.

[0095] In this embodiment, the simulated facial image generated by the generation module is a facial image corresponding to the virtual image of the user in the input facial image; specifically, the face in the generated simulated facial image has a high degree of similarity with the face in the sample facial image.

[0096] The generation module in this embodiment may be obtained by pre-training, or may be obtained by real-time training when building a neural network model.

[0097] The training device 700 of the bone parameter generation model of this embodiment may also include a second training unit 705, which is used to train the generation module in the following manner: obtain a second training set, the second training set including multiple candidate bone parameters and annotated face images corresponding to the multiple candidate bone parameters; input the candidate bone parameters into the initial generation module to obtain a predicted face image output by the initial generation module; obtain a second target loss function value based on the annotated face image and the predicted face image; adjust the parameters of the initial generation module according to the second target loss function value to obtain the generation module.

[0098] In the second training set acquired by the second training unit 705, the multiple candidate bone parameters may be manually input, or multiple randomly constructed bone parameters may be used as the multiple candidate bone parameters.

[0099] In the second training set obtained by the second training unit 705, the annotated facial images corresponding to the candidate skeletal parameters can be manually input, or the candidate skeletal parameters can be input into the rendering engine of the three-dimensional face model (such as the game engine Unity, etc.), and the annotated facial images corresponding to the input candidate skeletal parameters can be obtained according to the output results of the rendering engine. For example, the frontal image of the three-dimensional face model output by the rendering engine is captured as the annotated facial image.

[0100] That is to say, the second training unit 705 can obtain annotated facial images corresponding to different candidate bone parameters through the rendering engine, and then use the candidate bone parameters and their corresponding annotated facial images to complete the training of the initial generation module, which can reduce the cost of obtaining annotated facial images and improve the training efficiency of the initial generation module.

[0101] When the second training unit 705 obtains the second target loss function value based on the annotated face image and the predicted face image, the pixel error between the segmentation map of the annotated face image and the segmentation map of the predicted face image can be used as the second target loss function value; the error between the key point coordinates in the annotated face image and the predicted face image can be used as the second target loss function value; and the error between the pixel error and the key point coordinates can be simultaneously calculated as the second target loss function value.

[0102] The second training unit 705 can stop adjusting the parameters of the initial generation module to obtain the generation module when it is determined that the second target loss function value reaches a preset value; it can also stop adjusting the parameters of the initial generation module to obtain the generation module when it is determined that the number of times the second target loss function value is obtained exceeds a preset number.

[0103] In this embodiment, after the construction unit 702 constructs the neural network model including the extraction module, the initial conversion module and the generation module, the first training unit 703 uses the sample face image to adjust the parameters of the initial conversion module in the training neural network model to obtain the conversion module.

[0104] The first training unit 703 only uses a plurality of sample face images to adjust the parameters of the initial conversion module in the neural network model, but does not adjust the parameters of the extraction module and the generation module in the neural network model.

[0105] Specifically, when the first training unit 703 uses the sample face image to adjust the parameters of the initial conversion module in the training neural network model to obtain the conversion module, the implementation method that can be adopted is: input the sample face image into the neural network model to obtain a simulated face image output by the neural network model; obtain a first target loss function value based on the sample face image and the simulated face image; adjust the parameters of the initial conversion module in the neural network model based on the obtained first target loss function value to obtain the conversion module.

[0106] The first training unit 703 may stop adjusting the parameters of the initial conversion module to obtain the conversion module when it is determined that the first target loss function value reaches a preset value; or may stop adjusting the parameters of the initial conversion module to obtain the conversion module when it is determined that the number of times the first target loss function value is obtained exceeds a preset number.

[0107] Among them, when the first training unit 703 inputs the sample face image into the neural network model to obtain the simulated face image output by the neural network model, the implementation method that can be adopted is: input the sample face image into the extraction module in the neural network model to obtain the sample face image features output by the extraction module; input the obtained sample face image features into the initial conversion module in the neural network model to obtain the sample bone parameters output by the initial conversion module; input the obtained sample bone parameters into the generation module in the neural network model to obtain the simulated face image output by the generation module.

[0108] When the first training unit 703 obtains the first target loss function value based on the sample face image and the simulated face image, the pixel error between the segmentation map of the sample face image and the segmentation map of the simulated face image can be used as the first target loss function value; the error between the key point coordinates in the sample face image and the simulated face image can be used as the first target loss function value; the pixel error and the error between the key point coordinates can also be calculated simultaneously as the first target loss function value.

[0109] When the first training unit 703 obtains the first target loss function value based on the sample face image and the simulated face image, the following steps can be adopted: inputting the simulated face image into the extraction module to obtain the simulated face image features output by the extraction module; inputting the simulated face image features into the initial conversion module to obtain the simulated bone parameters output by the initial conversion module; obtaining the first loss function value based on the sample bone parameters and the simulated bone parameters corresponding to the sample face image; and obtaining the first target loss function value based on the first loss function value.

[0110] That is to say, the first training unit 703 will also obtain the first target loss function value for adjusting the parameters of the initial conversion module based on the simulated bone parameters of the corresponding simulated facial image, so that the first target loss function value can include the constraint part of the bone parameters, thereby improving the adjustment accuracy and efficiency of the parameters of the initial conversion module.

[0111] The first training unit 703 may calculate the error between the sample bone parameters and the bone parameters corresponding to the same part in the simulated bone parameters to obtain the first loss function value.

[0112] When the first training unit 703 obtains the first target loss function value based on the first loss function value, it can also obtain the second loss function value based on the pixel error between the segmentation map of the sample face image and the segmentation map of the simulated face image and / or the error between the coordinates of key points in the sample face image and the simulated face image, and then obtain the first target loss function value based on the first loss function value and the second loss function value.

[0113] That is to say, the first training unit 703 can further enrich the types of loss functions in the first objective loss function value, so that the adjustment process of the parameters of the initial conversion module is subject to more constraints, further improving the adjustment accuracy of the parameters of the initial conversion module.

[0114] In this embodiment, after the first training unit 703 obtains the conversion module, the first processing unit 704 obtains the skeleton parameter generation model according to the extraction module and the conversion module.

[0115] After completing the training of the initial conversion module to obtain the conversion module, this embodiment can use the extraction module and the conversion module in the neural network model to obtain the bone parameter generation model.

[0116] The skeleton parameter generation model obtained by the first processing unit 704 can output the skeleton parameters of the user's face in the corresponding face image according to the input face image, and then use the obtained skeleton parameters to generate a three-dimensional face model of the corresponding user's virtual image.

[0117] Therefore, the bone parameter generation model obtained in this embodiment can be applied to scenes such as games, metaverse, AR (Augmented Reality), VR (Virtual Reality), etc., so that users can create their own virtual images more quickly and accurately.

[0118] Figure 8 is a schematic diagram according to the eighth embodiment of the present disclosure. Figure 8 As shown in FIG. 8 , the device 800 for generating a 3D face model of this embodiment includes:

[0119] The second acquisition unit 801 is used to acquire a face image to be processed;

[0120] The second processing unit 802 is used to input the face image to be processed into the bone parameter generation model, and obtain the target bone parameters according to the output result of the bone parameter generation model:

[0121] The generating unit 803 is used to obtain a three-dimensional face model corresponding to the face image to be processed according to the target bone parameters.

[0122] The facial image to be processed acquired by the second acquisition unit 801 is an image that only includes the facial part of the corresponding user, and the facial part of the user is the front face of the user; if the acquired image is not a facial image, the second acquisition unit 801 can acquire the image of the corresponding facial part from the image by image capture or face alignment.

[0123] When the second processing unit 802 inputs the face image to be processed into the bone parameter generation model to obtain the target bone parameters, it first inputs the face image to be processed into the extraction module in the bone parameter generation model to obtain the target face image features output by the extraction module; then inputs the target face image features into the conversion module in the bone parameter generation model to obtain the target bone parameters output by the conversion module.

[0124] The three-dimensional face model corresponding to the face image to be processed generated by the generation unit 803 is the user's "face pinching" result. The three-dimensional face model can be used to create a virtual image of the corresponding user in the game or metaverse.

[0125] The generating unit 803 may input the target bone parameters output by the bone parameter generating model into a rendering engine (eg, the game engine Unity) for rendering, thereby obtaining a three-dimensional face model of the corresponding user.

[0126] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0127] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0128] like Fig. 9As shown, it is a block diagram of an electronic device for training a skeletal parameter generation model or a generation method of a three-dimensional face model according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0129] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0130] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0131] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training of the skeleton parameter generation model or the generation method of the three-dimensional face model. For example, in some embodiments, the training of the skeleton parameter generation model or the generation method of the three-dimensional face model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908.

[0132] In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by computing unit 901, one or more steps of the training of the skeleton parameter generation model or the generation method of the three-dimensional face model described above can be performed. Alternatively, in other embodiments, computing unit 901 can be configured to perform the training of the skeleton parameter generation model or the generation method of the three-dimensional face model by any other appropriate means (e.g., by means of firmware).

[0133] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable skeletal parameter generation model training or three-dimensional face model generation device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0135] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0137] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0138] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a bone parameter generation model, comprising: Acquire a first training set, wherein the first training set includes a plurality of sample face images; Constructing a neural network model including an extraction module, an initial conversion module and a generation module, wherein the extraction module is used to extract facial image features from an input facial image, the initial conversion module is used to obtain bone parameters according to the facial image features, and the generation module is used to generate a simulated facial image according to the bone parameters; Using the sample face image to adjust the parameters of the initial conversion module in the neural network model to obtain a conversion module; According to the extraction module and the conversion module, a bone parameter generation model is obtained.

2. The method according to claim 1, wherein: The generation module is trained in the following way: Acquire a second training set, wherein the second training set includes a plurality of candidate skeleton parameters and annotated face images corresponding to the plurality of candidate skeleton parameters; Inputting the candidate skeleton parameters into an initial generation module to obtain a predicted face image output by the initial generation module; Obtaining a second objective loss function value according to the labeled face image and the predicted face image; The parameters of the initial generation module are adjusted according to the second objective loss function value to obtain the generation module.

3. The method according to claim 1, wherein: The step of using the sample face image to adjust the parameters of the initial conversion module in the neural network model to obtain the conversion module includes: Inputting the sample face image into the neural network model to obtain a simulated face image output by the neural network model; Obtaining a first target loss function value according to the sample face image and the simulated face image; According to the first objective loss function value, the parameters of the initial conversion module are adjusted to obtain the conversion module.

4. The method according to claim 3, wherein: The step of inputting the sample face image into the neural network model to obtain a simulated face image output by the neural network model comprises: Inputting the sample face image into the extraction module to obtain features of the sample face image output by the extraction module; Inputting the sample face image features into the initial conversion module to obtain the sample bone parameters output by the initial conversion module; The sample bone parameters are input into the generation module to obtain a simulated face image output by the generation module.

5. The method according to claim 3, wherein: The obtaining of a first target loss function value according to the sample face image and the simulated face image comprises: Inputting the simulated face image into the extraction module to obtain the simulated face image features output by the extraction module; Inputting the simulated face image features into the initial conversion module to obtain simulated bone parameters output by the initial conversion module; Obtaining a first loss function value according to the sample bone parameters corresponding to the sample face image and the simulated bone parameters; The first target loss function value is obtained according to the first loss function value.

6. The method according to claim 5, wherein: The obtaining the first target loss function value according to the first loss function value includes: Obtaining a second loss function value according to a pixel error between a segmentation map of the sample face image and a segmentation map of the simulated face image and / or an error between key point coordinates in the sample face image and the simulated face image; The first target loss function value is obtained according to the first loss function value and the second loss function value.

7. A method for generating a three-dimensional face model, comprising: Obtain the face image to be processed; The face image to be processed is input into the bone parameter generation model, and the target bone parameters are obtained according to the output result of the bone parameter generation model: Obtaining a three-dimensional face model corresponding to the face image to be processed according to the target bone parameters; Wherein, the bone parameter generation model is trained according to any one of the methods in claims 1-6.

8. The method according to claim 7, wherein: The step of inputting the face image to be processed into a bone parameter generation model and obtaining target bone parameters according to an output result of the bone parameter generation model comprises: Inputting the face image to be processed into the extraction module in the skeleton parameter generation model to obtain the target face image features output by the extraction module; The target face image features are input into a conversion module in the bone parameter generation model to obtain target bone parameters output by the conversion module.

9. A training device for generating a skeleton parameter model, comprising: A first acquisition unit, configured to acquire a first training set, wherein the first training set includes a plurality of sample face images; A construction unit, used to construct a neural network model including an extraction module, an initial conversion module and a generation module, wherein the extraction module is used to extract facial image features from an input facial image, the initial conversion module is used to obtain bone parameters according to the facial image features, and the generation module is used to generate a simulated facial image according to the bone parameters; A first training unit, configured to adjust the parameters of an initial conversion module in the neural network model using the sample face image to obtain a conversion module; The first processing unit is used to obtain a skeleton parameter generation model according to the extraction module and the conversion module.

10. The apparatus according to claim 9, further comprising a second training unit, configured to train the generation module in the following manner: Acquire a second training set, wherein the second training set includes a plurality of candidate skeleton parameters and annotated face images corresponding to the plurality of candidate skeleton parameters; Inputting the candidate skeleton parameters into an initial generation module to obtain a predicted face image output by the initial generation module; Obtaining a second objective loss function value according to the labeled face image and the predicted face image; The parameters of the initial generation module are adjusted according to the second objective loss function value to obtain the generation module.

11. The device according to claim 9, wherein: When the first training unit uses the sample face image to adjust the parameters of the initial conversion module in the neural network model to obtain the conversion module, specifically performing: Inputting the sample face image into the neural network model to obtain a simulated face image output by the neural network model; Obtaining a first target loss function value according to the sample face image and the simulated face image; According to the first objective loss function value, the parameters of the initial conversion module are adjusted to obtain the conversion module.

12. The device according to claim 11, wherein When the first training unit inputs the sample face image into the neural network model to obtain the simulated face image output by the neural network model, the first training unit specifically performs: Inputting the sample face image into the extraction module to obtain features of the sample face image output by the extraction module; Inputting the sample face image features into the initial conversion module to obtain the sample bone parameters output by the initial conversion module; The sample bone parameters are input into the generation module to obtain a simulated face image output by the generation module.

13. The device according to claim 11, wherein: When the first training unit obtains the first target loss function value according to the sample face image and the simulated face image, the first training unit specifically performs: Inputting the simulated face image into the extraction module to obtain the simulated face image features output by the extraction module; Inputting the simulated face image features into the initial conversion module to obtain simulated bone parameters output by the initial conversion module; Obtaining a first loss function value according to the sample bone parameters corresponding to the sample face image and the simulated bone parameters; The first target loss function value is obtained according to the first loss function value.

14. The device according to claim 13, wherein: When the first training unit obtains the first target loss function value according to the first loss function value, the first training unit specifically performs: Obtaining a second loss function value according to a pixel error between a segmentation map of the sample face image and a segmentation map of the simulated face image and / or an error between key point coordinates in the sample face image and the simulated face image; The first target loss function value is obtained according to the first loss function value and the second loss function value.

15. A device for generating a three-dimensional face model, comprising: A second acquisition unit, used for acquiring a face image to be processed; The second processing unit is used to input the face image to be processed into the bone parameter generation model, and obtain the target bone parameters according to the output result of the bone parameter generation model: A generating unit, used for obtaining a three-dimensional face model corresponding to the face image to be processed according to the target bone parameters; Wherein, the bone parameter generation model is obtained by training according to any one of the devices in claims 9-14.

16. The device according to claim 15, wherein: When the processing unit inputs the face image to be processed into the bone parameter generation model and obtains the target bone parameters according to the output result of the bone parameter generation model, the processing unit specifically executes: Inputting the face image to be processed into the extraction module in the skeleton parameter generation model to obtain the target face image features output by the extraction module; The target face image features are input into a conversion module in the bone parameter generation model to obtain target bone parameters output by the conversion module.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images