Training method and device for face reconstruction model, face reconstruction method and device, electronic device and readable storage medium

Abstract

A training method and device for a face reconstruction model, a face reconstruction method and device, an electronic device and a readable storage medium. The training method for the face reconstruction model includes obtaining training data (101); inputting the first face image into a first network model to obtain a second face image (102); inputting the target face image and the second face image into a second network model to obtain a discrimination result (103); obtaining a first loss function corresponding to the first network model, and adjusting the parameters of the first network model according to the first loss function (104); obtaining a second loss function corresponding to the second network model, and adjusting the parameters of the second network model according to the second loss function (105); performing the above steps alternately to train the first network model and the second network model in turn (106); and using the trained first network model as a face reconstruction model (107). This solution can improve the accuracy and clarity of face reconstruction.

Description

Technical Field

[0001] The present disclosure relates to the technical field of face reconstruction, and in particular to a training method and device for a face reconstruction model, an electronic device, and a readable storage medium. Background Art

[0002] Face reconstruction refers to the technology of reconstructing the facial image of a person included in video data based on video data. It is widely used in aspects such as person recognition and tracking. Among related technologies, the accuracy and clarity of face reconstruction are relatively poor. Summary of the invention

[0003] The disclosed embodiments provide a method and device for training a face reconstruction model, and a method and device for face reconstruction to solve the problem of poor accuracy and clarity in face reconstruction.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for training a face reconstruction model, comprising the following steps:

[0005] Acquire training data, the training data including a target facial image and a first facial image corresponding to the target facial image, wherein the definition of the first facial image is smaller than the definition of the target facial image;

[0006] Inputting the first face image into a first network model to obtain a second face image, wherein the first network model is a generative network model that takes the face image as input and takes a reconstructed image of the input face image as output;

[0007] Inputting the target face image and the second face image into a second network model to obtain a discrimination result, wherein the second network model is a discrimination network model that takes the face image as input and takes a discrimination result of the authenticity of the input face image as output, and the discrimination result includes a discrimination result of the authenticity of the whole face image and the authenticity of the local features of the input face image;

[0008] Obtaining a first loss function corresponding to the first network model, and adjusting parameters of the first network model according to the first loss function;

[0009] Obtaining a second loss function corresponding to the second network model, and adjusting parameters of the second network model according to the second loss function;

[0010] Perform the above steps alternately to perform model training on the first network model and the second network model in turn;

[0011] The trained first network model is used as a face reconstruction model, wherein when the training is completed, the values ​​of the first loss function and the second loss function both reach corresponding target thresholds.

[0012] In some embodiments, the second loss function includes a first discriminative adversarial loss, and the second network model includes a global discriminative subnetwork;

[0013] The obtaining a second loss function corresponding to the second network model includes:

[0014] Marking the second face image as false, marking the target face image as true, inputting the second face image and the target face image into the global discriminant subnetwork respectively, and obtaining a first discriminant result and a second discriminant result respectively;

[0015] The first discrimination adversarial loss is obtained according to the first discrimination result and the second discrimination result.

[0016] In some embodiments, the second loss function includes a second discriminant adversarial loss and a third discriminant adversarial loss, and the second network model further includes an eye discriminant subnetwork and a mouth discriminant subnetwork;

[0017] The obtaining a second loss function corresponding to the second network model includes:

[0018] Obtaining a corresponding first eye image and a first mouth image according to the second face image;

[0019] Obtaining a corresponding second eye image and a second mouth image according to the target face image;

[0020] Mark the first eye image and the first mouth image as false, mark the second eye image and the second mouth image as true, input the first eye image and the second eye image into the eye discrimination subnetwork respectively, and output a third discrimination result and a fourth discrimination result respectively; input the first mouth image and the second mouth image into the mouth discrimination subnetwork respectively, and output a fifth discrimination result and a sixth discrimination result respectively;

[0021] Obtaining a second discrimination adversarial loss according to the third discrimination result and the fourth discrimination result;

[0022] The third discrimination adversarial loss is obtained according to the fifth discrimination result and the sixth discrimination result.

[0023] In some embodiments, the first loss function includes a first sub-loss and a second sub-loss;

[0024] The obtaining a first loss function corresponding to the first network model includes:

[0025] Acquire a first face part map and a second face part map corresponding to the target face image;

[0026] parsing the second facial image to obtain a third facial part map and a fourth facial part map corresponding to the second facial image, wherein the first facial part map and the second facial part map correspond to different regions of the same facial image, the first facial part map and the third facial part map correspond to the same region of different facial images, and the second facial part map and the fourth facial part map correspond to the same region of different facial images;

[0027] Obtaining the first sub-loss according to a difference between the first face part map and the third face part map;

[0028] The second sub-loss is obtained according to a difference between the second face part map and the fourth face part map.

[0029] In some embodiments, the first face part map includes a facial feature image of a face image, and the second face part map includes a skin image of the face image.

[0030] In some embodiments, the first loss function includes a third sub-loss;

[0031] The obtaining of the first loss function corresponding to the first network model further includes:

[0032] Acquire first feature point data corresponding to the target face image;

[0033] Analyze the second facial image to obtain second feature point data corresponding to the second facial image;

[0034] The third sub-loss is obtained according to the difference between the first feature point data and the second feature point data.

[0035] In some embodiments, the first feature point data includes a heat map of the target facial image, and the second feature point data includes a heat map of the second facial image, wherein the heat map includes one or more of a left eye heat map, a right eye heat map, a nose heat map, a mouth heat map, and a facial contour heat map of the facial image.

[0036] In some embodiments, the first loss function includes a fourth sub-loss;

[0037] The obtaining of the first loss function corresponding to the first network model further includes:

[0038] Obtaining a first feature vector corresponding to the target face image;

[0039] Obtaining a second feature vector corresponding to the second facial image;

[0040] The fourth sub-loss is obtained according to the difference between the first eigenvector and the second eigenvector.

[0041] In some embodiments, the first loss function includes a fifth sub-loss;

[0042] The obtaining of the first loss function corresponding to the first network model further includes:

[0043] The fifth sub-loss is obtained according to the difference between the target face image and the second face image.

[0044] In some embodiments, the first loss function includes one or more of a sixth sub-loss and a seventh sub-loss;

[0045] The obtaining of the first loss function corresponding to the first network model further includes:

[0046] using a perceptual loss according to a difference between the eye region image of the target face image and the eye region image of the second face image as the sixth sub-loss; and / or

[0047] A perceptual loss according to the difference between the mouth area image of the target face image and the mouth area image of the second face image is used as the seventh sub-loss.

[0048] In some embodiments, the first loss function includes an eighth sub-loss;

[0049] The obtaining of the first loss function corresponding to the first network model further includes:

[0050] The eighth sub-loss is obtained according to the generative adversarial loss between the first network model and the second network model, wherein the second network model includes one or more of a global discriminant subnetwork, an eye discriminant subnetwork, and a mouth discriminant subnetwork, and the generative adversarial loss is obtained by marking the second face image output by the first network model as true, and then inputting the second face image into one or more of the global discriminant subnetwork, the eye discriminant subnetwork, and the mouth discriminant subnetwork to obtain a discrimination result, and the loss is determined according to the obtained discrimination result.

[0051] In a second aspect, the present disclosure provides a face reconstruction method, comprising the following steps:

[0052] Get the input image;

[0053] The input image is input into a face reconstruction model to obtain a face reconstruction image, wherein the face reconstruction model is obtained by model training using any face reconstruction model training method in the first aspect.

[0054] In a third aspect, the present disclosure provides a training device for a face reconstruction model, including:

[0055] A training data acquisition module, used to acquire training data, wherein the training data includes a target facial image and a first facial image corresponding to the target facial image, wherein the definition of the first facial image is smaller than the definition of the target facial image;

[0056] A first input module, used for inputting the first face image into a first network model to obtain a second face image, wherein the first network model is a generative network model that takes the face image as input and takes a reconstructed image of the input face image as output;

[0057] A second input module, used to input the target face image and the second face image into a second network model to obtain a discrimination result, wherein the second network model is a discrimination network model that takes the face image as input and takes the discrimination result of the authenticity of the input face image as output, and the discrimination result includes the discrimination result of the authenticity of the whole face image and the authenticity of the local features of the input face image;

[0058] A first loss function acquisition module, used to acquire a first loss function corresponding to the first network model, and adjust parameters of the first network model according to the first loss function;

[0059] A second loss function acquisition module, used to acquire a second loss function corresponding to the second network model, and adjust the parameters of the second network model according to the second loss function;

[0060] A training module, used for performing model training on the first network model and the second network model in turn;

[0061] A face reconstruction model confirmation module is used to use the trained first network model as the face reconstruction model, wherein, when the training is completed, the values ​​of the first loss function and the second loss function both reach corresponding target thresholds.

[0062] In a fourth aspect, an embodiment of the present disclosure provides a face reconstruction device, comprising:

[0063] An input image acquisition module, used for acquiring an input image;

[0064] The input module inputs the input image into a face reconstruction model to obtain a face reconstruction image, wherein the face reconstruction model is obtained by model training using any face reconstruction model training method in the first aspect.

[0065] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the computer program implements the steps of the training method for a face reconstruction model as described in any one of the first aspects, or implements the steps of the face reconstruction method as described in the second aspect.

[0066] In a fifth aspect, an embodiment of the present disclosure provides a readable storage medium having a computer program stored thereon. When the computer program is executed by the processor, the computer program implements the steps of the training method for a face reconstruction model as described in any one of the first aspects, or implements the steps of the face reconstruction method as described in the second aspect.

[0067] The disclosed embodiment trains the first network model and the second network model by establishing a generative adversarial network including the first network model and the second network model, wherein the second network model includes the discrimination results of the overall authenticity and the authenticity of the local features, which helps to improve the accuracy of the judgment of the output result of the first network model, thereby improving the accuracy of the reconstruction model completed by the training for reconstructing the face image, and also helps to increase the iteration speed, thereby improving the model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0069] Figure 1 is a flow chart of a method for training a face reconstruction model provided by an embodiment of the present disclosure;

[0070] Figure 2 is a flow chart of a face reconstruction method provided by an embodiment of the present disclosure;

[0071] Figure 3 is a structural diagram of a training device for a face reconstruction model provided by an embodiment of the present disclosure;

[0072] Figure 4 It is a structural diagram of a face reconstruction device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0074] The disclosed embodiment provides a method for training a face reconstruction model.

[0075] like Figure 1 As shown, in one embodiment, the training method of the face reconstruction model includes the following steps:

[0076] Step 101: Obtain training data.

[0077] The training data in this embodiment is also called a training set, and the training data includes a target face image and a first face image, and the training data includes a target face image and a first face image corresponding to the target face image, and the definition of the first face image is smaller than the definition of the target face image. The face image can be in the form of video data or a photo.

[0078] In this embodiment, the target facial image and the first facial image can be directly provided by training data, or only the target facial image, that is, a facial image with higher clarity, can be provided, and then the clarity of the target facial image can be reduced to generate the first facial image, which is also called degrading the target facial image. After degradation, the clarity of the target facial image is reduced, thereby obtaining a first facial image with less clarity than the target facial image.

[0079] In the embodiments of the present disclosure, "clarity" may refer to the clarity of each detail and its boundary in the image. The higher the clarity, the better the perception effect of the human eye. The clarity of the output image is higher than the clarity of the input image. For example, it means that the input image is processed by the image processing method provided by the embodiments of the present disclosure, such as denoising, super-resolution and / or deblurring, so that the processed output image is clearer than the input image.

[0080] In one embodiment, the target facial image can be degraded by adding noise to the image, Gaussian blurring, adjusting the brightness and contrast of the image, scaling the image, deforming the image, and adding motion blurring effects to the image.

[0081] It should be understood that the quality of the facial image in the target facial image is relatively high. For example, the target facial image may have suitable brightness and contrast, proper image proportion, no motion blur, high image quality, etc. During implementation, the target facial image can be degraded by reducing or increasing its brightness and contrast, adjusting its image proportion to make the image proportion unbalanced, etc., so as to obtain a first facial image, that is, a facial image with lower clarity.

[0082] In this way, training data of the target face image and the first face image can be obtained.

[0083] Step 102: Input the first facial image into a first network model to obtain a second facial image.

[0084] The first network model in this embodiment is a generative network model that takes a face image as input and takes a reconstructed image of the input face image as output.

[0085] In this embodiment, the first network model is used as a generator to process and reconstruct the input first face image. The first network model performs deblurring or resolution enhancement processing on the first face image, thereby obtaining a second face image from the first face image. In other words, the second face image is the reconstruction result of the first network model for the first face image.

[0086] Step 103: Input the target face image and the second face image into a second network model to obtain a discrimination result.

[0087] In this embodiment, the second network model is a discriminant network model that takes a face image as input and takes a result of determining the authenticity of the input face image as output.

[0088] The second network model in this embodiment is equivalent to the discriminator, and the first network model and the second network model together constitute a generative adversarial network for model training.

[0089] The discrimination result of the second network model includes the discrimination results of the overall authenticity and the authenticity of the local features of the input face image, wherein the overall authenticity refers to the discrimination result of the input face image from the global perspective of the face image, and the authenticity of the local features refers to the discrimination result of the local detail features of the face image.

[0090] Generally speaking, the discrimination result of the output of the second network model as a discriminator is a value between 0 and 1, wherein the closer the discrimination result is to 1, the higher the authenticity of the discrimination of the second network model. Conversely, if the discrimination result is closer to 0, the lower the authenticity of the discrimination of the second network model.

[0091] Step 104: Obtain a first loss function corresponding to the first network model, and adjust parameters of the first network model according to the first loss function.

[0092] Step 105: Obtain a second loss function corresponding to the second network model, and adjust parameters of the second network model according to the second loss function.

[0093] Step 106: Perform the above steps alternately to perform model training on the first network model and the second network model in turn.

[0094] Furthermore, a first loss function corresponding to the first network model and a second loss function corresponding to the second network model are established respectively, and the parameters of the corresponding first network model or the second network model are adjusted according to the established loss function to perform model training on the first network model and the second network model.

[0095] In this embodiment, the process of alternating training of the first network model and the second network model can be adjusted. For example, the first network model can be trained once, the second network model can be trained once, and then the first network model can be trained once, and so on; the first network model can also be trained multiple times, and then the second network model can be trained once, and then the first network model can be trained multiple times, and so on. Obviously, the training method for the first network model and the second network model in this embodiment is not limited to this.

[0096] Step 107: Use the trained first network model as a face reconstruction model.

[0097] In this embodiment, when the values ​​of the first loss function and the second loss function both reach the corresponding target thresholds, or when the first loss function and the second loss function both converge, it is considered that the first network model has been trained, and the trained first network model is a face reconstruction model that meets the needs of face reconstruction. It should be noted that the target threshold here can be set according to actual conditions, for example, it can be the minimum value or maximum value that the first loss function or the second loss function can reach.

[0098] The disclosed embodiment trains the first network model and the second network model by establishing a generative adversarial network including the first network model and the second network model, wherein the second network model includes the discrimination results of the overall authenticity and the authenticity of the local features, which helps to improve the accuracy of the judgment of the output result of the first network model, thereby improving the accuracy of the reconstruction model completed by the training for reconstructing the face image, and also helps to increase the iteration speed, thereby improving the model training efficiency.

[0099] In some embodiments, the first loss function includes a first sub-loss and a second sub-loss, and step 104 includes:

[0100] Acquire a first face part map and a second face part map corresponding to the target face image;

[0101] parsing the second facial image to obtain a third facial part map and a fourth facial part map corresponding to the second facial image;

[0102] Obtaining the first sub-loss according to a difference between the first face part map and the third face part map;

[0103] The second sub-loss is obtained according to a difference between the second face part map and the fourth face part map.

[0104] In this embodiment, the first face part map and the second face part map corresponding to the target face image can be directly provided by training data, or can be obtained by analyzing the target face image. The third face part map and the fourth face part map corresponding to the second face image are obtained by analyzing the second face image.

[0105] The facial image analysis to obtain the corresponding facial part map can be achieved by using a pre-trained face analysis model. The face analysis model can select an existing or improved face analysis model such as RoI Tanh (Face Parsing with RoI Tanh-Warping), which is not further limited here.

[0106] In this embodiment, the first face part map and the second face part map correspond to different areas of the same face image, and the areas corresponding to the first face part map and the third face part map are the same. In other words, the first face part map corresponds to a certain area of ​​the target face image, such as the eye area, and the third face part map corresponds to the eye area of ​​the second face image. Similarly, the second face part map and the fourth face part map correspond to the same area of ​​the face image.

[0107] In some embodiments, the first face part map includes a facial feature image of the face image, and the second face part map includes a skin image of the face image.

[0108] That is to say, the first face part map and the third face part map correspond to the facial features in the face image, which are also referred to as organ maps in this embodiment, and organs may refer to the facial features such as mouth, nose, eyes, eyebrows, ears, etc. The second face part map and the fourth face part map correspond to the skin area other than the facial features.

[0109] By respectively obtaining the first sub-loss corresponding to the organ map and the second sub-loss corresponding to the skin map, the reconstruction results of the first network model for the organ area and the skin area can be determined respectively, thereby improving the adjustment accuracy of the first network model and helping to improve the model training efficiency.

[0110] It should be understood that each sub-loss in this embodiment can be calculated in different ways. For example, the L1 loss of the first face part map and the third face part map can be used as the first sub-loss, or the L2 loss of the first face part map and the third face part map can be used as the first sub-loss. Obviously, when implemented, this embodiment is not further limited. Among them, the L1 loss refers to the minimum absolute deviation (LAD), and the L2 loss refers to the minimum square error (LSE). The specific calculation method can refer to the relevant technology and will not be repeated here.

[0111] In this embodiment, the first sub-loss is taken as the L2 loss between the organ map of the target face image generated by the face parsing model and the organ map of the second face image output by the first network model as an example. In this embodiment, the first sub-loss is recorded as L2_feat.

[0112] In this embodiment, the second sub-loss is taken as the L2 loss between the skin map of the target face image generated by the face parsing model and the skin map of the second face image output by the first network model as an example. In this embodiment, the second sub-loss is recorded as L2_skin.

[0113] It should be understood that the facial part map and skin map of the facial image are images, which can reflect the feeling from a human visual and subjective perspective, that is, they reflect the feeling of the similarity between the output result of the first network model and the target facial image from a human visual observation perspective.

[0114] In some embodiments, the first loss function includes a third sub-loss, and step 104 further includes:

[0115] Acquire first feature point data corresponding to the target face image;

[0116] Analyze the second facial image to obtain second feature point data corresponding to the second facial image;

[0117] The third sub-loss is obtained according to the difference between the first feature point data and the second feature point data.

[0118] In this embodiment, face alignment analysis is performed based on feature points in the target face image and the second face image. The process of the face alignment analysis can be understood as extracting the first feature point data of the target face image through the face alignment model, then extracting the second feature point data of the second face image, comparing the extracted first feature point data with the second feature point data, and determining the third sub-loss based on the difference value.

[0119] The analysis of feature point data can be understood as comparing the similarity between the output result of the first network model and the real face corresponding to the input image from a numerical perspective.

[0120] In some embodiments, feature point data of a facial image may be extracted by coordinate regression, which is fast and requires less computation.

[0121] In some other embodiments, the feature point data includes a heat map of a face image, and the heat map of the face image includes one or more of a left eye heat map, a right eye heat map, a nose heat map, a mouth heat map, and a facial contour heat map of the face image. For example, a left eye heat map refers to a heat map composed of key points in the area corresponding to the left eye, and a facial contour heat map refers to a heat map composed of key points corresponding to areas outside the organs, and so on, to generate multiple local heat maps constituting the face image. By generating multiple local heat maps constituting the face image, it is helpful to further improve the accuracy of the calculation of the feature point data of the face image.

[0122] In this embodiment, the key points are first determined, and the number of key points can be set as needed, for example, a 68-point heat map is selected; next, n heat maps that are the same as the number of key points n are output, which is 68 heat maps in this embodiment; further, the point with the highest peak in the heat map is found as the key point, or the contribution value of each pixel in the heat map is weightedly calculated to obtain the key point coordinates.

[0123] By obtaining feature point data of face images based on heat map regression, the calculation accuracy can be further improved.

[0124] During implementation, a pre-trained face alignment model is first provided. The face alignment model can specifically be a face alignment model such as AWing ([ICCV 2019] Adaptive Wing Loss for Robust Face Alignment via HeatmapRegression). For details, please refer to related technologies.

[0125] Next, the face alignment model is used to obtain the first feature point data of the target face image, that is, the heat map of the target face image, and the face alignment model is used to obtain the second feature point data of the second face image, that is, the heat map of the second face image.

[0126] Finally, the third sub-loss is obtained according to the difference between the first feature point data and the second feature point data. In this embodiment, the third sub-loss is the L2 loss of the heat map of the target face image and the heat map of the second face image, which is recorded as L2_heatmap.

[0127] In some embodiments, the first loss function includes a fourth sub-loss, and step 104 further includes:

[0128] Obtaining a first feature vector corresponding to the target face image;

[0129] Obtaining a second feature vector corresponding to the second facial image;

[0130] The fourth sub-loss is obtained according to the difference between the first eigenvector and the second eigenvector.

[0131] In this embodiment, feature analysis is also performed on the target face image and the second face image. Specifically, the feature vector of the target face image is first calculated, and then the feature vector of the second face image is calculated. Finally, the sub-loss is determined based on the difference between the two feature vectors.

[0132] In this embodiment, the cosine similarity of the two feature vectors is calculated, and then 1 minus the cosine similarity is used as the loss function corresponding to the feature analysis. In this embodiment, the fourth sub-loss is recorded as LCosSimilarity.

[0133] In some embodiments, the first loss function includes a fifth sub-loss, and step 104 further includes:

[0134] The fifth sub-loss is obtained according to the difference between the target face image and the second face image.

[0135] In this embodiment, the L2 loss of the target face image and the face reconstruction image output by the first network model is further introduced as the fifth sub-loss. During implementation, the difference value between the target face image and the second face image can be determined by a pre-trained face recognition model, and the face recognition model can be an existing or improved face recognition model such as ArcFace "ArcFace: Additive Angular MarginLoss for Deep Face Recognition". In the embodiment, the fifth sub-loss is recorded as L20.

[0136] In some embodiments, the first loss function includes one or more of the sixth sub-loss and the seventh sub-loss, and the step 104 further includes:

[0137] using a perceptual loss according to a difference between the eye region image of the target face image and the eye region image of the second face image as the sixth sub-loss; and / or

[0138] A perceptual loss according to the difference between the mouth area image of the target face image and the mouth area image of the second face image is used as the seventh sub-loss.

[0139] In this embodiment, the eye area image and the mouth area image are further analyzed respectively, and the perceptual loss in the eye area image of the target face image and the second face image is determined as the sixth sub-loss, denoted as L2_eye; the perceptual loss in the mouth area image of the target face image and the second face image is determined as the seventh sub-loss, denoted as L2_mouth.

[0140] In some embodiments, the first loss function includes an eighth sub-loss, and step 104 further includes:

[0141] The eighth sub-loss is obtained according to the generative adversarial loss between the first network model and the second network model.

[0142] During implementation, the second face image output by the first network model is first marked as true, specifically, for example, marked as 1, and then the second face image is input into one or more of the global discriminant subnetwork, the eye discriminant subnetwork, and the mouth discriminant subnetwork to obtain a corresponding discrimination result. The obtained discrimination result is a value between 0 and 1. According to the difference between the discrimination result and 1, the generative adversarial loss between the first network model and the second network model is obtained, which is recorded as the eighth sub-loss LG.

[0143] In this embodiment, the second network model includes one or more of a global discrimination subnetwork, an eye discrimination subnetwork, and a mouth discrimination subnetwork.

[0144] When the second face image marked as true is input into the global discriminant subnetwork, the global adversarial loss can be determined according to the judgment result of the global discriminant subnetwork, which is recorded as LG_all; when the second face image marked as true is input into the eye discriminant subnetwork, the eye adversarial loss can be determined according to the judgment result of the eye discriminant subnetwork, which is recorded as LG_eye; when the second face image marked as true is input into the mouth discriminant subnetwork, the mouth adversarial loss can be determined according to the judgment result of the mouth discriminant subnetwork, which is recorded as LG_mouth.

[0145] After determining the first to eighth sub-losses, the first loss function can be obtained. In this embodiment, the first loss function is recorded as L, then:

[0146] L=w1*L2_feat+w2*L2_skin+w3*L2_heatmap+w4*LCosSimilarity+w5*L20+w6*L2_eye+w7*L2_mouth+LG.

[0147] Among them, LG=w8*LG_all+w9*LG_eye+w10*LG_mouth.

[0148] In the above formula, w1 to w10 are weight coefficients corresponding to each loss value, which can be set as needed. For example, they can all be set to 1, or the coefficients corresponding to loss values ​​with greater importance can be set relatively larger according to the importance of different loss values, so as to obtain the first loss function corresponding to the first network model.

[0149] In some embodiments, the second loss function includes a first discriminative adversarial loss, and step 106 further includes:

[0150] Marking the second face image as false, marking the target face image as true, inputting the second face image and the target face image into the global discriminant subnetwork respectively, and obtaining a first discriminant result and a second discriminant result respectively;

[0151] The first discrimination adversarial loss is obtained according to the first discrimination result and the second discrimination result.

[0152] In this embodiment, the second network model includes a global discriminant subnetwork. When implemented, the second face image output by the first network model is first marked as false, for example, it can be marked as 0, and the target face image is marked as true, for example, it can be marked as 0. Then the second face image and the target face image are respectively input into the global discriminant subnetwork to obtain a discrimination result, which is a value between 0 and 1, wherein the discrimination result corresponding to the second face image is the first discrimination result, and the discrimination result corresponding to the second face image is the second discrimination result.

[0153] Next, the first discriminant adversarial loss corresponding to the first network model and the global discriminant subnetwork determined according to the obtained first discriminant result and the second discriminant result is recorded as LD_all.

[0154] In some embodiments, the second loss function includes a second discriminative adversarial loss and a third discriminative adversarial loss, and the step 106 further includes:

[0155] Obtaining a corresponding first eye image and a first mouth image according to the second face image;

[0156] Obtaining a corresponding second eye image and a second mouth image according to the target face image;

[0157] Mark the first eye image and the first mouth image as false, mark the second eye image and the second mouth image as true, input the first eye image and the second eye image into the eye discrimination subnetwork respectively, and output a third discrimination result and a fourth discrimination result respectively; input the first mouth image and the second mouth image into the mouth discrimination subnetwork respectively, and output a fifth discrimination result and a sixth discrimination result respectively;

[0158] Obtaining a second discrimination adversarial loss according to the third discrimination result and the fourth discrimination result;

[0159] The third discrimination adversarial loss is obtained according to the fifth discrimination result and the sixth discrimination result.

[0160] When determining the second discriminant adversarial loss and the third discriminant adversarial loss, it is necessary to extract the eye image and the mouth image of the second face image. In this embodiment, the eye image of the second face image is recorded as the first eye image, and the mouth image of the second face image is recorded as the first mouth image.

[0161] When determining the second discriminant adversarial loss and the third discriminant adversarial loss, it is also necessary to extract the eye image and mouth image of the target face image. In this embodiment, the eye image of the target face image is recorded as the second eye image, and the mouth image of the target face image is recorded as the second mouth image.

[0162] The extracted first eye image and the first mouth image are both marked as false, for example, both marked as 0, and the second eye image and the second mouth image are both marked as true, for example, both marked as 1.

[0163] The first eye image is input into the eye discrimination subnetwork to obtain a third discrimination result; the second eye image is input into the eye discrimination subnetwork to obtain a fourth discrimination result; the first mouth image is input into the mouth discrimination subnetwork to obtain a fifth discrimination result; the second mouth image is input into the mouth discrimination subnetwork to obtain a sixth discrimination result.

[0164] Finally, the second discrimination adversarial loss is obtained according to the difference between the third discrimination result and the fourth discrimination result, denoted as LD_eye, and the third discrimination adversarial loss is obtained according to the difference between the fifth discrimination result and the sixth discrimination result, denoted as LD_mouth.

[0165] After determining the first discriminant adversarial loss, the second discriminant adversarial loss and the third discriminant adversarial loss, the second loss function can be obtained, which is recorded as LD=w11*LD_all+w12*LD_mouth+w13*LD_mouth, where w11 to w13 are weight coefficients corresponding to each loss value.

[0166] The disclosed embodiment also provides a face reconstruction method.

[0167] like Figure 2 As shown, the face reconstruction method includes the following steps:

[0168] Step 201: Obtain an input image;

[0169] Step 202: Input the input image into a face reconstruction model to obtain face reconstruction data.

[0170] In this embodiment, the face reconstruction model is obtained by performing model training through any of the above face reconstruction model training methods.

[0171] In this embodiment, the face reconstruction model used is a face reconstruction model trained by the above-mentioned face reconstruction model training method. By inputting the input image into the face reconstruction model, a face reconstruction result with a high degree of consistency with the real face image can be output.

[0172] The present disclosure provides a training device for a face reconstruction model.

[0173] In one embodiment, Figure 3 As shown, the training device 300 for the face reconstruction model includes:

[0174] A training data acquisition module 301 is used to acquire training data, wherein the training data includes a target facial image and a first facial image corresponding to the target facial image, wherein the definition of the first facial image is smaller than the definition of the target facial image;

[0175] A first input module 302, configured to input the first face image into a first network model to obtain a second face image, wherein the first network model is a generative network model that takes a face image as input and takes a reconstructed image of the input face image as output;

[0176] A second input module 302 is used to input the target face image and the second face image into a second network model to obtain a discrimination result, wherein the second network model is a discrimination network model that takes the face image as input and takes the discrimination result of the authenticity of the input face image as output, and the discrimination result includes the discrimination result of the authenticity of the whole face image and the authenticity of the local features of the input face image;

[0177] A first loss function acquisition module 304 is used to acquire a first loss function corresponding to the first network model, and adjust the parameters of the first network model according to the first loss function;

[0178] A second loss function acquisition module 305, used to acquire a second loss function corresponding to the second network model, and adjust the parameters of the second network model according to the second loss function;

[0179] A training module 306, configured to perform model training on the first network model and the second network model in turn;

[0180] The face reconstruction model confirmation module 307 is used to use the trained first network model as the face reconstruction model, wherein, when the training is completed, the values ​​of the first loss function and the second loss function both reach corresponding target thresholds.

[0181] In some embodiments, the second loss function includes a first discriminative adversarial loss, and the second network model includes a global discriminative subnetwork;

[0182] The second loss function acquisition module 305 includes:

[0183] A first discrimination result acquisition submodule is used to mark the second face image as false, mark the target face image as true, input the second face image and the target face image into the global discrimination subnetwork respectively, and obtain a first discrimination result and a second discrimination result respectively;

[0184] The first discriminant adversarial loss acquisition submodule is used to obtain the first discriminant adversarial loss according to the first discriminant result and the second discriminant result.

[0185] In some embodiments, the second loss function includes a second discriminant adversarial loss and a third discriminant adversarial loss, and the second network model further includes an eye discriminant subnetwork and a mouth discriminant subnetwork;

[0186] The second loss function acquisition module 305 further includes:

[0187] The second discriminative adversarial loss acquisition submodule is used to

[0188] A first image acquisition submodule, configured to obtain a corresponding first eye image and a first mouth image according to the second face image;

[0189] A second image acquisition submodule, used to obtain a corresponding second eye image and a second mouth image according to the target face image;

[0190] a marking submodule, used for marking the first eye image and the first mouth image as false, marking the second eye image and the second mouth image as true, inputting the first eye image and the second eye image into the eye discrimination subnetwork respectively, and outputting a third discrimination result and a fourth discrimination result respectively; inputting the first mouth image and the second mouth image into the mouth discrimination subnetwork respectively, and outputting a fifth discrimination result and a sixth discrimination result respectively;

[0191] A second discriminant adversarial loss acquisition submodule, used to obtain a second discriminant adversarial loss according to the third discriminant result and the fourth discriminant result;

[0192] The third discriminant adversarial loss acquisition submodule is used to obtain the third discriminant adversarial loss according to the fifth discriminant result and the sixth discriminant result.

[0193] In some embodiments, the first loss function includes a first sub-loss and a second sub-loss;

[0194] The first loss function acquisition module 304 includes:

[0195] A face part map acquisition submodule, used to acquire a first face part map and a second face part map corresponding to the target face image;

[0196] a first parsing submodule, configured to parse the second facial image to obtain a third facial part map and a fourth facial part map corresponding to the second facial image, wherein the first facial part map and the second facial part map correspond to different regions of the same facial image, the first facial part map and the third facial part map correspond to the same region of different facial images, and the second facial part map and the fourth facial part map correspond to the same region of different facial images;

[0197] A first sub-loss acquisition submodule, configured to obtain the first sub-loss according to a difference between the first face part map and the third face part map;

[0198] The second sub-loss acquisition submodule is used to obtain the second sub-loss according to the difference between the second face part map and the fourth face part map.

[0199] In some embodiments, the first face part map includes a facial feature image of a face image, and the second face part map includes a skin image of the face image.

[0200] In some embodiments, the first loss function includes a third sub-loss;

[0201] The first loss function acquisition module 304 further includes:

[0202] A feature point data acquisition submodule, used to acquire first feature point data corresponding to the target face image;

[0203] A second parsing submodule, used to parse the second facial image to obtain second feature point data corresponding to the second facial image;

[0204] The second sub-loss acquisition submodule is used to obtain the third sub-loss according to the difference between the first feature point data and the second feature point data.

[0205] In some embodiments, the first feature point data includes a heat map of the target facial image, and the second feature point data includes a heat map of the second facial image, wherein the heat map includes one or more of a left eye heat map, a right eye heat map, a nose heat map, a mouth heat map, and a facial contour heat map of the facial image.

[0206] In some embodiments, the first loss function includes a fourth sub-loss;

[0207] The first loss function acquisition module 304 further includes:

[0208] A feature vector acquisition submodule, used to acquire a first feature vector corresponding to the target face image;

[0209] The feature vector acquisition submodule is further used to acquire a second feature vector corresponding to the second facial image;

[0210] The fourth sub-loss acquisition submodule is used to obtain the fourth sub-loss according to the difference between the first feature vector and the second feature vector.

[0211] In some embodiments, the first loss function includes a fifth sub-loss;

[0212] The first loss function acquisition module 304 further includes:

[0213] The fifth sub-loss acquisition submodule is used to obtain the fifth sub-loss according to the difference between the target face image and the second face image.

[0214] In some embodiments, the first loss function includes one or more of a sixth sub-loss and a seventh sub-loss;

[0215] The first loss function acquisition module 304 further includes:

[0216] a sixth sub-loss acquisition submodule, configured to use a perceptual loss of a difference between the eye area image of the target face image and the eye area image of the second face image as the sixth sub-loss; and / or

[0217] The seventh sub-loss acquisition submodule is used to use the perceptual loss of the difference between the mouth area image of the target face image and the mouth area image of the second face image as the seventh sub-loss.

[0218] In some embodiments, the first loss function includes an eighth sub-loss;

[0219] The first loss function acquisition module 304 further includes:

[0220] An eighth sub-loss acquisition submodule is used to obtain the eighth sub-loss according to the generative adversarial loss between the first network model and the second network model, wherein the second network model includes one or more of a global discriminant subnetwork, an eye discriminant subnetwork, and a mouth discriminant subnetwork, and the generative adversarial loss is to mark the second face image output by the first network model as true, and then input the second face image into one or more of the global discriminant subnetwork, the eye discriminant subnetwork, and the mouth discriminant subnetwork to obtain a discrimination result, and determine it according to the obtained discrimination result.

[0221] The training device for face reconstruction model of the embodiment of the present disclosure can implement each step of the training method for face reconstruction model of the embodiment described above, and can at least achieve the same or similar technical effects, which will not be repeated here.

[0222] The embodiment of the present disclosure provides a face reconstruction device.

[0223] like Figure 4 As shown, in one embodiment, the face reconstruction device 400 includes:

[0224] An input image acquisition module 401 is used to acquire an input image;

[0225] The input module 402 is used to input the input image into a face reconstruction model to obtain a face reconstruction image, wherein the face reconstruction model is obtained by model training using any of the above-mentioned face reconstruction model training methods.

[0226] The face reconstruction device of the embodiment of the present disclosure implements each step of the above-mentioned face reconstruction method and can at least achieve the same or similar technical effects, which will not be repeated here.

[0227] An embodiment of the present disclosure provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the training method for a face reconstruction model as described in any one of the above items, or implements the steps of the face reconstruction method as described above.

[0228] An embodiment of the present disclosure provides a readable storage medium having a computer program stored thereon. When the computer program is executed by the processor, the computer program implements the steps of the training method for a face reconstruction model as described in any one of the above items, or implements the steps of the face reconstruction method as described above.

[0229] The electronic device and readable storage medium of this embodiment can implement the above-mentioned training method of the face reconstruction model and the steps of the face reconstruction method, and can at least achieve the same or similar technical effects, which will not be repeated here.

[0230] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A training method for a face reconstruction model, comprising the following steps: Acquire training data, the training data including a target facial image and a first facial image corresponding to the target facial image, wherein the definition of the first facial image is smaller than the definition of the target facial image; Inputting the first face image into a first network model to obtain a second face image, wherein the first network model is a generative network model that takes the face image as input and takes a reconstructed image of the input face image as output; Inputting the target face image and the second face image into a second network model to obtain a discrimination result, wherein the second network model is a discrimination network model that takes the face image as input and takes a discrimination result of the authenticity of the input face image as output, and the discrimination result includes a discrimination result of the authenticity of the whole face image and the authenticity of the local features of the input face image; Obtaining a first loss function corresponding to the first network model, and adjusting parameters of the first network model according to the first loss function; Obtaining a second loss function corresponding to the second network model, and adjusting parameters of the second network model according to the second loss function; Perform the above steps alternately to perform model training on the first network model and the second network model in turn; Using the trained first network model as a face reconstruction model, wherein when the training is completed, the values ​​of the first loss function and the second loss function both reach corresponding target thresholds; The first loss function includes a first sub-loss and a second sub-loss; The obtaining a first loss function corresponding to the first network model includes: Acquire a first face part map and a second face part map corresponding to the target face image; parsing the second facial image to obtain a third facial part map and a fourth facial part map corresponding to the second facial image, wherein the first facial part map and the second facial part map correspond to different regions of the same facial image, the first facial part map and the third facial part map correspond to the same region of different facial images, and the second facial part map and the fourth facial part map correspond to the same region of different facial images; Obtaining a first sub-loss according to a difference between the first face part map and the third face part map, wherein the first sub-loss corresponds to an organ map in the face image; Obtaining the second sub-loss according to a difference between the second face part map and the fourth face part map, wherein the second sub-loss corresponds to a skin map in the face image; The second loss function includes a first discriminant adversarial loss, and the second network model includes a global discriminant subnetwork; The obtaining a second loss function corresponding to the second network model includes: Marking the second face image as false, marking the target face image as true, inputting the second face image and the target face image into the global discriminant subnetwork respectively, and obtaining a first discriminant result and a second discriminant result respectively; Obtaining the first discrimination adversarial loss according to the first discrimination result and the second discrimination result; And / or, the second loss function includes a second discriminant adversarial loss and a third discriminant adversarial loss, and the second network model further includes an eye discriminant subnetwork and a mouth discriminant subnetwork; The obtaining a second loss function corresponding to the second network model includes: Obtaining a corresponding first eye image and a first mouth image according to the second face image; Obtaining a corresponding second eye image and a second mouth image according to the target face image; Mark the first eye image and the first mouth image as false, mark the second eye image and the second mouth image as true, input the first eye image and the second eye image into the eye discrimination subnetwork respectively, and output a third discrimination result and a fourth discrimination result respectively; input the first mouth image and the second mouth image into the mouth discrimination subnetwork respectively, and output a fifth discrimination result and a sixth discrimination result respectively; Obtaining a second discrimination adversarial loss according to the third discrimination result and the fourth discrimination result; The third discrimination adversarial loss is obtained according to the fifth discrimination result and the sixth discrimination result.

2. The method according to claim 1, wherein: The first face part map includes a facial feature image of a face image, and the second face part map includes a skin image of the face image.

3. The method according to claim 1, wherein: The first loss function includes a third sub-loss; The obtaining of the first loss function corresponding to the first network model further includes: Acquire first feature point data corresponding to the target face image; Analyze the second facial image to obtain second feature point data corresponding to the second facial image; The third sub-loss is obtained according to the difference between the first feature point data and the second feature point data.

4. The method according to claim 3, wherein: The first feature point data includes a heat map of the target facial image, and the second feature point data includes a heat map of the second facial image, wherein the heat map includes one or more of a left eye heat map, a right eye heat map, a nose heat map, a mouth heat map, and a facial contour heat map of the facial image.

5. The method according to claim 1, wherein: The first loss function includes a fourth sub-loss; The obtaining of the first loss function corresponding to the first network model further includes: Obtaining a first feature vector corresponding to the target face image; Obtaining a second feature vector corresponding to the second facial image; The fourth sub-loss is obtained according to the difference between the first eigenvector and the second eigenvector.

6. The method according to claim 1, wherein: The first loss function includes a fifth sub-loss; The obtaining of the first loss function corresponding to the first network model further includes: The fifth sub-loss is obtained according to the difference between the target face image and the second face image.

7. The method according to claim 1, wherein: The first loss function includes one or more of a sixth sub-loss and a seventh sub-loss; The obtaining of the first loss function corresponding to the first network model further includes: using a perceptual loss according to a difference between the eye region image of the target face image and the eye region image of the second face image as the sixth sub-loss; and / or A perceptual loss according to the difference between the mouth area image of the target face image and the mouth area image of the second face image is used as the seventh sub-loss.

8. The method according to claim 1, wherein: The first loss function includes an eighth sub-loss; The obtaining of the first loss function corresponding to the first network model further includes: The eighth sub-loss is obtained according to the generative adversarial loss between the first network model and the second network model, wherein the second network model includes one or more of a global discriminant subnetwork, an eye discriminant subnetwork, and a mouth discriminant subnetwork, and the generative adversarial loss is obtained by marking the second face image output by the first network model as true, and then inputting the second face image into one or more of the global discriminant subnetwork, the eye discriminant subnetwork, and the mouth discriminant subnetwork to obtain a discrimination result, and the loss is determined according to the obtained discrimination result.

9. A face reconstruction method, comprising the following steps: Get the input image; The input image is input into a face reconstruction model to obtain a face reconstruction image, wherein the face reconstruction model is obtained by model training using the face reconstruction model training method according to any one of claims 1 to 8.

10. A training device for a face reconstruction model, comprising: A training data acquisition module, used to acquire training data, wherein the training data includes a target facial image and a first facial image corresponding to the target facial image, wherein the definition of the first facial image is smaller than the definition of the target facial image; A first input module, used for inputting the first face image into a first network model to obtain a second face image, wherein the first network model is a generative network model that takes the face image as input and takes a reconstructed image of the input face image as output; A second input module, used to input the target face image and the second face image into a second network model to obtain a discrimination result, wherein the second network model is a discrimination network model that takes the face image as input and takes the discrimination result of the authenticity of the input face image as output, and the discrimination result includes the discrimination result of the authenticity of the whole face image and the authenticity of the local features of the input face image; A first loss function acquisition module, used to acquire a first loss function corresponding to the first network model, and adjust parameters of the first network model according to the first loss function; A second loss function acquisition module, used to acquire a second loss function corresponding to the second network model, and adjust the parameters of the second network model according to the second loss function; A training module, used for performing model training on the first network model and the second network model in turn; A face reconstruction model confirmation module, used to use the trained first network model as the face reconstruction model, wherein, when the training is completed, the values ​​of the first loss function and the second loss function both reach corresponding target thresholds; The first loss function includes a first sub-loss and a second sub-loss; The obtaining a first loss function corresponding to the first network model includes: Acquire a first face part map and a second face part map corresponding to the target face image; parsing the second facial image to obtain a third facial part map and a fourth facial part map corresponding to the second facial image, wherein the first facial part map and the second facial part map correspond to different regions of the same facial image, the first facial part map and the third facial part map correspond to the same region of different facial images, and the second facial part map and the fourth facial part map correspond to the same region of different facial images; Obtaining a first sub-loss according to a difference between the first face part map and the third face part map, wherein the first sub-loss corresponds to an organ map in the face image; Obtaining the second sub-loss according to a difference between the second face part map and the fourth face part map, wherein the second sub-loss corresponds to a skin map in the face image; The second loss function includes a first discriminant adversarial loss, and the second network model includes a global discriminant subnetwork; The obtaining a second loss function corresponding to the second network model includes: Marking the second face image as false, marking the target face image as true, inputting the second face image and the target face image into the global discriminant subnetwork respectively, and obtaining a first discriminant result and a second discriminant result respectively; Obtaining the first discrimination adversarial loss according to the first discrimination result and the second discrimination result; , the second loss function includes a second discriminant adversarial loss and a third discriminant adversarial loss, and the second network model also includes an eye discriminant subnetwork and a mouth discriminant subnetwork; The obtaining a second loss function corresponding to the second network model includes: Obtaining a corresponding first eye image and a first mouth image according to the second face image; Obtaining a corresponding second eye image and a second mouth image according to the target face image; Mark the first eye image and the first mouth image as false, mark the second eye image and the second mouth image as true, input the first eye image and the second eye image into the eye discrimination subnetwork respectively, and output a third discrimination result and a fourth discrimination result respectively; input the first mouth image and the second mouth image into the mouth discrimination subnetwork respectively, and output a fifth discrimination result and a sixth discrimination result respectively; Obtaining a second discrimination adversarial loss according to the third discrimination result and the fourth discrimination result; The third discrimination adversarial loss is obtained according to the fifth discrimination result and the sixth discrimination result.

11. A face reconstruction device, comprising: An input image acquisition module, used for acquiring an input image; An input module is used to input the input image into a face reconstruction model to obtain a face reconstruction image, wherein the face reconstruction model is obtained by model training using the face reconstruction model training method of any one of claims 1 to 8.

12. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the training method for a face reconstruction model as described in any one of claims 1 to 8, or implements the steps of the face reconstruction method as described in claim 9.

13. A readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the steps of the training method for a face reconstruction model as described in any one of claims 1 to 8, or implements the steps of the face reconstruction method as described in claim 9.

Images