Model training, face image processing, face model processing method and device, electronic equipment and readable storage medium

Abstract

The present application relates to a kind of model training method and device, face image processing method and device, face model processing method, electronic equipment and readable storage medium, wherein one kind of model training method is by obtaining the synthetic data of face image, real data and target illumination condition, and based on synthetic data, first model is trained to convergence, so that the model at this time learns the distribution characteristics of synthetic data, then join real data and synthetic data in turn to model training, wherein the training of synthetic data is the distribution characteristics of model learning real data It plays the role of guidance and supervision;Since the distribution characteristics of real data are considered in the process of model training, the processing capacity of the finally trained model to real data is greatly improved, so that the face image re-illumination model trained can more realistically re-illuminate target illumination condition to the face image to be processed, and the illumination features of target illumination condition are maximized in the processed face image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning models, and in particular to a model training method and device, a face image processing method and device, a face model processing method, an electronic device, and a readable storage medium. BACKGROUND

[0002] With the development of digital image processing technology, in the processing of various different images, the change of image lighting conditions plays an increasingly important role in the image imaging process. Especially in the field of face images, lighting conditions are the most important factor affecting the imaging effect of face images. In order to realize the change of lighting conditions in complex backgrounds, it is necessary to perform relighting operations on face images in various scenes.

[0003] Face image relighting operation is to adjust the target face image according to the change of lighting or external environment, and generate a face image consistent with the specified target lighting condition. Face image relighting operation has very wide applications in the fields of face recognition, image-based rendering, and film post-production. For example, changing the lighting in a green screen scene to the lighting in a red screen background to produce natural shooting segments in other scenes in a photo studio.

[0004] In the prior art, face image relighting methods rely on the training of face image relighting models. Since it is difficult to shoot the face images of the same person in different scenes from the same angle, it is very difficult to obtain supervised data for model training. Most existing technologies are based on synthetic data for model training. The distribution of synthetic data and real data often differs greatly, so the model trained based on synthetic data often performs poorly when tested on real data.

[0005] In addition, the current metaverse is receiving unprecedented attention, and 3D digital people are the cornerstone of the metaverse, which is the avatar of real humans in the metaverse. However, the creation of 3D digital people is a time-consuming and laborious process, especially the rendering of light.

[0006] Light rendering of 3D face models is a very complex engineering, and only light sources include point light sources, spotlights, sunlight, and many other types, and also need to consider the coupling of materials and light sources, etc. For a 3D face model, it often takes designers several days or even weeks to design a good light, and the generalization of the light of a model is often poor, and the effect will be poor when copied to other models. Therefore, the current 3D face model relighting is a time-consuming and laborious process. SUMMARY

[0007] To at least partially overcome the problems existing in the related art, the present application provides a model training method and device, a face image processing method and device, a face model processing method, an electronic device and a readable storage medium, to solve the problems of poor testing effect of a face image relighting model trained based on synthetic data on real data in the prior art and time-consuming and laborious and low efficiency of 3D face model relighting.

[0008] According to a first aspect of an embodiment of the present application, a model training method is provided, comprising:

[0009] obtaining synthetic data, real data and a target lighting condition of a face image;

[0010] inputting the synthetic data and the target lighting condition into a pre-constructed first network model for training until the first network model converges, to obtain a first model;

[0011] inputting the real data and the target lighting condition, and the synthetic data and the target lighting condition into the first model alternately for training until the first model converges, to obtain a face image relighting model;

[0012] The face image relighting model is used to relight a specified target lighting condition into a face image to obtain a face image under the target lighting condition.

[0013] According to a second aspect of an embodiment of the present application, a face image processing method is provided, comprising:

[0014] obtaining a face image to be processed and a target lighting condition, the target lighting condition comprising an HDRI environment map or a lighting condition extracted from a specified face image;

[0015] inputting the face image and the target lighting condition into a face image relighting model, so that the face image relighting model relights the target lighting condition into the face image to obtain a face image under the target lighting condition; the face image relighting model is obtained by training the above-mentioned model training method.

[0016] According to a third aspect of an embodiment of the present application, a model training method is provided, comprising:

[0017] obtaining a face image under a target lighting condition and a ground truth of the face image under the target lighting condition, the face image under the target lighting condition being obtained by the above-mentioned face image processing method, and having a first resolution;

[0018] input the face image under the target light condition and the ground truth of the face image under the target light condition into the second network model to be trained to obtain a face image of a second resolution, the second resolution being greater than the first resolution;

[0019] After each round of training, the model parameters of the second network model are updated;

[0020] It is determined whether the updated second network model meets a preset second network model convergence condition, if yes, it is determined that the current model converges, and the second network model at this time is determined as the face image super-resolution reconstruction model; if not, the above training steps are repeatedly executed until the preset second network model convergence condition is met.

[0021] According to a fourth aspect of the embodiment of the present application, a face image processing method is provided, comprising:

[0022] A face image to be processed is obtained, the resolution of the face image being a first resolution;

[0023] The face image is input into a face image super-resolution reconstruction model to obtain a face image of a second resolution, the second resolution being greater than the first resolution;

[0024] The face image super-resolution reconstruction model is obtained through the above model training method.

[0025] According to a fifth aspect of the embodiment of the present application, a face model processing method is provided, comprising:

[0026] A 3D face model to be processed is converted into a 2D face image;

[0027] The 2D face image and a specified target light condition are input into a face image relighting model to enable the face image relighting model to relight the target light condition to the 2D face image to generate a face image of a first resolution under the target light condition;

[0028] The face image of the first resolution under the target light condition is input into a face image super-resolution reconstruction model to obtain a face image of a second resolution under the target light condition, the second resolution being greater than the first resolution;

[0029] The face image relighting model is obtained through the above model training method; and the face image super-resolution reconstruction model is obtained through the above model training method.

[0030] According to a sixth aspect of the embodiment of the present application, a model training device is provided, comprising:

[0031] An acquisition module is configured to acquire synthetic data, real data, and a target lighting condition of a face image.

[0032] A first training module is configured to input the synthetic data and the target lighting condition into a pre-constructed first network model for training until the first network model converges, thereby obtaining a first model.

[0033] A second training module is configured to input the real data and the target lighting condition and the synthetic data and the target lighting condition into the first model alternately for training until the first model converges, thereby obtaining a face image relighting model.

[0034] The face image relighting model is configured to relight a specified target lighting condition into an input face image, thereby obtaining a face image under the target lighting condition.

[0035] According to a seventh aspect of an embodiment of the present application, a face image processing device is provided, which comprises:

[0036] An acquisition module is configured to acquire a face image to be processed and a target lighting condition, wherein the target lighting condition comprises an HDRI environment map or a lighting condition extracted from a specified face image.

[0037] A relighting module is configured to input the face image and the target lighting condition into a face image relighting model, so that the face image relighting model relights the target lighting condition into the face image, thereby obtaining a face image under the target lighting condition; the face image relighting model is obtained by training according to the above-mentioned model training method.

[0038] According to an eighth aspect of an embodiment of the present application, a model training device is provided, which comprises:

[0039] An acquisition module is configured to acquire a face image under a target lighting condition and a ground truth of the face image under the target lighting condition, wherein the resolution of the face image under the target lighting condition is a first resolution, and the face image under the target lighting condition is obtained by using the above-mentioned face image processing method.

[0040] A training module is configured to input the face image under the target lighting condition and the ground truth of the face image under the target lighting condition into a pre-constructed second network model for training, thereby obtaining a face image with a second resolution, wherein the second resolution is greater than the first resolution.

[0041] An updating module is configured to update the model parameters of the second network model after each round of training is completed.

[0042] A judging module is configured to judge whether the updated second network model meets a preset second network model convergence condition, and if yes, determine that the current model converges, and determine the second network model at this time as the face image super-resolution reconstruction model; if no, repeatedly execute the training steps until the preset second network model convergence condition is met.

[0043] According to a ninth aspect of an embodiment of the present application, a face image processing apparatus is provided, comprising:

[0044] A obtaining module is configured to obtain a face image to be processed, the resolution of the face image being a first resolution.

[0045] A reconstruction module is configured to input the face image into a face image super-resolution reconstruction model to obtain a face image with a second resolution, the second resolution being greater than the first resolution.

[0046] The face image super-resolution reconstruction model is trained by the face image super-resolution reconstruction model training method.

[0047] According to a tenth aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein,

[0048] The memory stores a computer program that can be invoked by the processor.

[0049] The processor executes the method by invoking the computer program stored in the memory.

[0050] According to an eleventh aspect of an embodiment of the present application, a computer readable storage medium is provided, which stores a computer program that can be erased.

[0051] When the computer program runs on the computer device, the computer device executes the method.

[0052] The technical solution provided by the embodiments of the present application can have the following beneficial effects:

[0053] During model training, the synthetic data, real data and target lighting conditions of the face image are obtained, and the model is first trained to convergence based on the synthetic data, so that the model learns the distribution characteristics of the synthetic data at this time. Then, the real data and the synthetic data are alternately added to the model for training, wherein the training of the synthetic data plays a guiding and supervising role for the model to learn the distribution characteristics of the real data. Since the distribution characteristics of the real data are considered during model training, the processing capability of the finally trained model for the real data is greatly improved, so that the face image relighting model trained can more realistically relight the target lighting conditions to the face image to be processed, and the lighting features of the target lighting conditions are maximized in the processed face image.

[0054] By converting the 3D face model to be processed into a 2D face image, the heavy light operation of the 3D model is transferred to the 2D face image, compared with the prior art of directly using the target light condition to re-illuminate the 3D face model, and then converting the re-illuminated 3D face model into a 2D face image that can be received by the human eye. The technical scheme provided by the present application greatly reduces the technical difficulty of re-illuminating the face image on the 2D face image, and the face image after re-illumination is 2D and can be directly received by the human eye, so the processing speed is faster and the efficiency is higher.

[0055] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings, which are incorporated into the specification and constitute part of it, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0057] Figure 1 is a flowchart of a model training method according to an exemplary embodiment;

[0058] Figure 2 is a structural schematic diagram of a first network model according to an exemplary embodiment;

[0059] Figure 3 is an internal signal flow diagram of an LFM module according to an exemplary embodiment;

[0060] Figure 4 is a flowchart of a face image processing method according to an exemplary embodiment;

[0061] Figure 5 is a schematic block diagram of a model training device according to an exemplary embodiment;

[0062] Figure 6 is a schematic block diagram of a face image processing device according to an exemplary embodiment;

[0063] Figure 7 is a flowchart of a model training method according to another exemplary embodiment;

[0064] Figure 8 is a flowchart of a face image processing method according to another exemplary embodiment;

[0065] Figure 9 is a schematic block diagram of a model training device according to another exemplary embodiment;

[0066] Figure 10 is a schematic block diagram of a face image processing device according to another exemplary embodiment;

[0067] Figure 11 is a flowchart of a face model processing method according to another exemplary embodiment. DETAILED DESCRIPTION

[0068] The exemplary embodiments will be described in detail herein below with reference to the accompanying drawings. In the following description, the same drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0069] As described in the foregoing background, the prior art is based on synthetic data to train the face image re-illumination model. The distribution of synthetic data and real data often has a large difference, so the model trained based on synthetic data often has poor effect when tested on real data. In order to solve this technical problem, the present disclosure provides the following several exemplary embodiments.

[0070] It can be understood that a face image is stored in a digital matrix form in a computer. Taking a black and white face image as an example, the size of this digital matrix = the number of pixel values on the height of the face image x the number of pixel values on the width; if it is a color face image, there are three such digital matrices, i.e. three-channel digital matrices, which are used to describe the pixel values of RGB (red, green and blue) three colors respectively. Therefore, the "true value" mentioned in each embodiment of the present disclosure refers to the digital matrix of the face image without model processing. The pixel values corresponding to the face image are stored in the digital matrix. When the "true value" is input into any model mentioned in the present disclosure, any face image obtained is a "predicted value" relative to the "true value", for example, the "predicted value" mentioned in the following embodiments. and ".

[0071] Figure 1 is a flowchart of a model training method according to an exemplary embodiment, as shown in Figure 1 , the method comprises:

[0072] Step S11, obtaining synthetic data, real data and target illumination condition of a face image;

[0073] Step S12, inputting the synthetic data and the target illumination condition into a pre-constructed first network model for training until the first network model converges, obtaining a first model;

[0074] Step S13, input the real data and the target lighting condition, and the synthetic data and the target lighting condition into the first model alternately for training until the first model converges, to obtain a face image re-illumination model; the face image re-illumination model is used to re-illuminate a specified target lighting condition into a face image to obtain a face image under the target lighting condition.

[0075] It should be noted that the technical solutions provided in the embodiments are applicable to computer terminals and / or mobile terminals, and applicable scenarios include but are not limited to:

[0076] 1. Perform a lighting exchange task of a face image, for example, replace the lighting condition M in the to-be-processed face image A with the lighting condition N in the reference face image B.

[0077] 2. Add a target lighting condition selected by a user to a to-be-processed face image, for example, add a high dynamic range imaging (HDRI) environment map selected by the user to the to-be-processed face image.

[0078] In step S11, the "obtaining of synthetic data, real data, and a target lighting condition of a face image" includes:

[0079] Obtain a publicly available 3D face model (for example, FaceScape is a large-scale high-quality publicly available free 3D face dataset, including 18760 high-quality 3D face models. In specific implementation, the 3D face model can be obtained from FaceScape), render (for example, rendering can be performed in a modeling software Blender) each 3D face model using different high dynamic range imaging (HDRI) environment maps (the HDRI environment map is a kind of map way to simulate the environment, and its essence is a lighting condition image. There are many publicly available free HDRI environment maps on the network), and convert the rendered 3D face model into a 2D face image (the rendered 2D face image can obtain the ground truth I relit , the ground truth I a , and the ground truth I n of the albedo map of each 2D face image), and save the converted 2D face image as synthetic data.

[0080] Save the HDRI environment map corresponding to each synthetic data as a target lighting condition (the ground truth I t , the ground truth I sh , and the ground truth I sp of the target lighting condition are known), and store each synthetic data and the target lighting condition corresponding to each synthetic data in a database in pairs.

[0081] Obtain publicly available 2D high-definition face images (obtain public FFHQ and CelebA-HQ face image datasets from the Internet), save the face area in each 2D high-definition face image as real data (in order to exclude the interference of the background, use a face segmentation algorithm to segment the face and the background, and only keep the face area).

[0082] FFHQ stands for Flickr-Faces-Hight-Quality, which means high-definition face image dataset in Chinese. It was created as a benchmark for generative adversarial networks (GAN) and is also used in the training data set of Style GAN. It was open sourced in 2019. FFHQ is a high-quality face dataset containing 70,000 high-definition face images in PNG format with a resolution of 1024x1024.

[0083] CelebA is the abbreviation of CelebFaces Attribute, which means celebrity face attribute dataset. It contains 202,599 face images of 10,177 celebrities. Each image is labeled with features, including face bbox, 5 face feature point coordinates, and 40 attribute labels. CelebA is provided by the Chinese University of Hong Kong.

[0084] It can be understood that, since the synthetic data is pre-known to have true value I relit , and the real data lacks the true value of the same face under different target lighting conditions, the technical solution provided by the embodiment uses a semi-supervised training method to train the model to obtain a face image relighting model.

[0085] The step S12 of "inputting the synthetic data and the target lighting condition into the pre-constructed first network model for training" includes:

[0086] 1. Construct a first network model based on a deep learning framework (including but not limited to: Pytorch deep learning framework, TensorFlow deep learning framework, Keras deep learning framework); in specific practice, refer to Figure 2 , the first network model includes a light removal model, a light shadow feature model, and an image synthesis model.

[0087] In specific practice, the light removal model, the light and shadow feature model, and the image synthesis model all adopt the U-net type coding-decoding structure, which is composed of a compression channel and an expansion channel. The compression channel is an encoder for extracting the features of the face image layer by layer, and the expansion channel is a decoder for restoring the position information of the face image. Each hidden layer has a large number of feature dimensions, which is conducive to the model learning more diverse and comprehensive features.

[0088] The compression channel is a typical convolutional neural network structure, which repeatedly uses two convolutional layers, wherein the step size of the second convolutional layer is set to 2 to reduce the resolution of the feature map by one. In the expansion channel, first, perform a deconvolution operation to increase the resolution of the feature map by one, then splice the feature map obtained by cropping the corresponding compression channel, and then use a convolutional layer to extract features and repeat this structure. In the last output layer, use a convolutional layer to map the 64-channel feature map to a 3-channel output map.

[0089] Further, the light and shadow feature model and the image synthesis model both add an LFM (LFM, Lighting guided Feature Modulation, light guided feature modulation) module. Referring to Figure 3 When the LFM module is added to the light and shadow feature model, the input of the LFM module is the output of the last layer of the light and shadow feature model (such as Figure 3 a matrix of size h x w x c) and the lighting conditions input to the light and shadow feature model (such as Figure 3 a matrix of size 16 x 32 x 3), and its output is the input of the next convolutional layer.

[0090] The LFM module is used to extract lighting features (including at least lighting direction and lighting intensity) for re-illumination from the input lighting conditions. In specific practice, the internal signal flow diagram of the LFM module is as shown in Figure 3 Referring to Figure 3 , the 16 x 32 x 3 size lighting condition input to the LFM module is first adjusted to a 1536*1 size vector, then passes through the fully connected layer FC1, and outputs a 128*1 size one-dimensional lighting matrix; the 128*1 size one-dimensional lighting matrix respectively passes through the fully connected layer FC2 and the fully connected layer FC3, and respectively outputs a c*1 size feature vector, these two vectors are copied and expanded into a h*w*c size lighting feature matrix: the first lighting matrix γ and the second lighting matrix β; the first lighting matrix γ is calculated by point multiplication with the input matrix, and the point multiplication result is calculated by summation with the second lighting matrix β, and finally the output result is sent to the light and shadow feature model for subsequent calculation.

[0091] 2, the synthesized data I input (I input is Irelit any one of the person face image data in the synthetic data) into the first network model to obtain a first person face image I restore , comprising:

[0092] input the synthetic data I input into the de-light model to obtain an albedo map a normal map and a first predicted light condition The first predicted light condition is an estimated value of the light condition I s carried by the synthetic data;

[0093] input the first predicted light condition and the normal map into the light-shadow feature model to obtain a highlight map under the first predicted light condition and a shadow map under the first predicted light condition;

[0094] input the albedo map the normal map the highlight map under the first predicted light condition, the shadow map under the first predicted light condition, and the first predicted light condition into the image synthesis model to obtain a first person face image I restore under the first predicted light condition.

[0095] 3, input the target light condition I t into the first network model to obtain a second person face image under the target light condition comprising:

[0096] input the target light condition I t (I t is the light condition specified by the user in I s and the normal map into the light-shadow feature model to obtain a highlight map under the target light condition and a shadow map under the target light condition

[0097] input the albedo map the normal map the highlight map under the target light condition the shadow map under the target light condition and the target light condition I t into the image synthesis model to obtain a second person face image under the target light condition

[0098] 4. According to the first preset loss function, the first loss function value is calculated, and the gradient of each layer of the first network model is calculated using the chain rule; according to the gradient, the network parameters of the first network model are updated using the gradient descent algorithm and the back propagation algorithm.

[0099] In specific practice, the first preset loss function includes one or more combinations of the following items, including:

[0100] Albedo map and, the true value I of the albedo map a , the content loss function of the two

[0101] Normal map and, the true value I of the normal map n , the content loss function of the two

[0102] Specular map under the target lighting condition and, the true value I of the specular map under the target lighting condition sp , the content loss function of the two

[0103] Shadow map under the target lighting condition and, the true value I of the shadow map under the target lighting condition sh , the content loss function of the two

[0104] First face image I restore and, the true value I of the synthetic data input into the de-lighting model input , the content loss function L of the two c (I input , I restore );

[0105] Second face image and, the true value I of the synthetic data under the target lighting condition relit , the content loss function of the two

[0106] Predicted lighting condition and, the true value I of the predicted lighting condition s , the lighting loss function L of the two l ; the true value is the pixel value corresponding to the face image;

[0107] wherein, ω refers to a solid angle, which is a known value calculated according to a preset formula.

[0108] The definition of the content loss function is L c(x, y) = 1 - SSIM(x, y);

[0109] wherein, SSIM refers to a structural similarity, and is defined as follows:

[0110] wherein, C1 = 0.01 2 , C2 = 0.03 2 .

[0111] Specifically, in order to ensure the training effect of the first model, the first preset loss function can be set as:

[0112]

[0113] 5. Determine whether the updated first network model meets the preset first network model convergence condition, if yes, determine that the current model converges, and determine the first network model at this time as the first model; if not, repeat the above training steps until the preset first model convergence condition is met.

[0114] In specific practice, the preset first network model convergence condition includes:

[0115] The number of update iterations of the first network model reaches a preset number, and the first loss function value no longer decreases, and the structural similarity measured on the test set no longer improves;

[0116] The test set stores a plurality of face images and a true value corresponding to each face image; the structural similarity refers to that a face image in the test set is input into the first network model to generate a re-illumination face image, and the similarity between the predicted value of the re-illumination face image and the corresponding true value;

[0117] The predicted value refers to the pixel value of the face image after re-illumination under the target illumination condition, and the true value refers to the pixel value of the original face image under the target illumination condition.

[0118] The step S13 of “alternately inputting the real data and the target illumination condition, and the synthetic data and the target illumination condition into the first model for training” includes:

[0119] 1. Input the real data I′ input into the first model to obtain a third face image I′ restore under the illumination condition of the real data itself, including:

[0120] Input the real data I′ input into the de-illumination model to obtain the albedo map normal map and the second predicted illumination condition The second predicted illumination condition The estimated value is based on the lighting conditions inherent in the real data;

[0121] The second predicted lighting conditions and normal map The data is input into the light and shadow feature model to obtain the specular map under the second predicted lighting condition and the shadow map under the second predicted lighting condition.

[0122] Apply the albedo map Normal map Specular map under the second predicted lighting condition, shadow map under the second predicted lighting condition, and the second predicted lighting condition The image is input into the image synthesis model to obtain a third face image I′ under the second predicted lighting condition. restore .

[0123] 2. Set the target lighting conditions I′ t The image is input into the first model to obtain a fourth face image under the target lighting conditions. include:

[0124] Set the target lighting condition I′ t and normal map The data is input into the light and shadow feature model to obtain the specular map under the target lighting conditions. Shadow mapping under target lighting conditions

[0125] Apply the albedo map Normal map Specular map under target lighting conditions Shadow mapping under target lighting conditions and target illumination conditions I′ t The image is input into the image synthesis model to obtain a fourth face image under the target lighting conditions.

[0126] 3. Input the synthesized data into the first model to obtain the fifth face image under the lighting conditions provided by the synthesized data, including;

[0127] The synthetic data is input into the de-illuminated model to obtain an albedo map, a normal map, and a third predicted lighting condition; the third predicted lighting condition is an estimated value of the lighting condition provided by the synthetic data.

[0128] The third predicted lighting conditions and normal map are input into the lighting feature model to obtain the specular map and shadow map under the third predicted lighting conditions.

[0129] inputting the albedo map, the normal map, the highlight map under the third predicted light condition, the shadow map under the third predicted light condition, and the third predicted light condition into the image synthesis model to obtain a fifth face image under the third predicted light condition.

[0130] 4. inputting the target light condition into the first module to obtain a sixth face image under the target light condition, comprising:

[0131] inputting the target light condition and the normal map into the light and shadow feature model to obtain a highlight map under the target light condition and a shadow map under the target light condition;

[0132] inputting the albedo map, the normal map, the highlight map under the target light condition, the shadow map under the target light condition, and the target light condition into the image synthesis model to obtain the sixth face image under the target light condition.

[0133] According to the second preset loss function, a second loss function value is calculated, and the gradient of each layer of the first model is calculated using the chain rule; according to the gradient, the network parameters of the first model are updated using the gradient descent algorithm and the back propagation algorithm (in specific practice, the network parameters of the constructed network model can be optimized using the Adam optimizer, the basic learning rate (learning rate, lr) is set to 0.002, and the default parameters of the selected deep learning framework are used for the remaining parameters).

[0134] In specific practice, the second preset loss function includes one or more combinations of the following items, including:

[0135] The third face image I' restore , the real data I' input input into the light removal model c , the content loss function L input of both restore ;

[0136] The albedo map and the verification albedo map , the content loss function

[0137] The normal map and the verification normal map , the content loss function

[0138] The verification albedo map is obtained by inputting the fourth face image into the light removal model; the verification normal map by inputting the fourth face image to the de-light model.

[0139] Specifically, in order to ensure the training effect of the face image re-illumination model, the second preset loss function can be set as:

[0140]

[0141] 6. Determine whether the updated first model meets the preset first model convergence condition. If yes, it is determined that the current model converges, and the first model at this time is determined as the face image re-illumination model. If no, repeat the above training steps until the preset first model convergence condition is met.

[0142] In specific practice, the preset first model convergence condition includes:

[0143] The number of update iterations of the first model reaches a preset number (in specific practice, the preset number is set according to user needs, or is set according to experimental data, or is set according to historical experience value, for example, is set to 200,000 times), and the second loss function value no longer decreases, and the structural similarity measured on the test set no longer improves;

[0144] The test set stores a plurality of face images and a true value corresponding to each face image. The structural similarity refers to the similarity between the predicted value of the re-illumination face image generated by inputting the face image in the test set to the first model and the corresponding true value.

[0145] The predicted value refers to the pixel value of the face image re-illuminated under the target illumination condition. The true value refers to the pixel value of the original face image under the target illumination condition.

[0146] It can be understood that the technical scheme provided by the embodiment, during model training, the synthetic data, the real data and the target illumination condition of the face image are obtained, and the model is trained to convergence based on the synthetic data, so that the model learns the distribution characteristics of the synthetic data at this time. Then, the real data and the synthetic data are alternately trained on the model, wherein the training of the synthetic data plays a guiding and supervising role in learning the distribution characteristics of the real data. Since the distribution characteristics of the real data are considered in the model training process, the processing capability of the finally trained model on the real data is greatly improved, so that the face image re-illumination model trained can re-illuminate the target illumination condition to the face image to be processed more realistically, and the illumination features of the target illumination condition are maximized in the processed face image.

[0147] Figure 4is a flow chart of a face image processing method according to an exemplary embodiment, as shown in Figure 4 The method comprises:

[0148] Step S21, obtaining a face image to be processed and a target lighting condition, the target lighting condition comprising: an HDRI environment map or a lighting condition extracted from a specified face image;

[0149] Step S22, inputting the face image and the target lighting condition into a face image relighting model, so that the face image relighting model relights the target lighting condition into the face image to obtain a face image under the target lighting condition; the face image relighting model is trained by the model training method described above.

[0150] It should be noted that the technical scheme provided in this embodiment is applicable to a computer terminal and / or a mobile terminal, and applicable scenarios include but are not limited to:

[0151] 1. Perform a lighting exchange task of a face image, for example, replace the lighting condition M in the face image A to be processed with the lighting condition N in the reference face image B;

[0152] 2. Add the target lighting condition selected by the user to the face image to be processed, for example, add the HDRI environment map selected by the user to the face image to be processed.

[0153] It can be understood that the technical scheme provided in this embodiment is realized based on a face image relighting model, and the face image relighting model is trained by obtaining synthetic data, real data and a target lighting condition of a face image, and based on the synthetic data, the model is first trained to convergence, so that the model learns the distribution characteristics of the synthetic data at this time. Then, the real data and the synthetic data are alternately added to the model for training, wherein the training of the synthetic data plays a guiding and supervising role in learning the distribution characteristics of the real data by the model. Since the distribution characteristics of the real data are considered in the model training process, the processing capability of the finally trained model for the real data is greatly improved, so that the trained face image relighting model can more realistically relight the target lighting condition to the face image to be processed, and the lighting features of the target lighting condition are maximized in the processed face image.

[0154] Figure 5 is a schematic block diagram of a model training device 100 according to an exemplary embodiment, as shown in Figure 5 The device 100 comprises:

[0155] The obtaining module 101 is configured to obtain synthetic data, real data and a target lighting condition of a face image.

[0156] The first training module 102 is configured to input the synthetic data and the target lighting condition into a pre-constructed first network model for training until the first network model converges, so as to obtain a first model.

[0157] The second training module 103 is configured to input the real data and the target lighting condition and the synthetic data and the target lighting condition into the first model alternately for training until the first model converges, so as to obtain a face image relighting model.

[0158] The face image relighting model is configured to relight a specified target lighting condition into an input face image, so as to obtain a face image under the target lighting condition.

[0159] It should be noted that the implementation manners and beneficial effects of the above modules can refer to the detailed description of the corresponding steps in the foregoing embodiments, and the present embodiment will not be described herein.

[0160] It can be understood that the technical scheme provided by the present embodiment is that, during model training, the synthetic data, the real data and the target lighting condition of the face image are acquired, and the model is trained to convergence based on the synthetic data, so that the model learns the distribution characteristics of the synthetic data at this time, and then the real data and the synthetic data are alternately input to the model for training, wherein the training of the synthetic data plays a guiding and supervising role for the model to learn the distribution characteristics of the real data. Since the distribution characteristics of the real data are considered in the model training process, the processing capability of the finally trained model for the real data is greatly improved, so that the face image relighting model trained by the model can more realistically relight the target lighting condition to the face image to be processed, and the lighting features of the target lighting condition are maximized in the processed face image.

[0161] Figure 6 is a schematic block diagram of a face image processing apparatus 200 according to an example embodiment, as shown in Figure 6 The apparatus 200 comprises:

[0162] The acquisition module 201 is configured to acquire a face image to be processed and a target lighting condition, wherein the target lighting condition comprises an HDRI environment map or a lighting condition extracted from a specified face image.

[0163] The relighting module 202 is configured to input the face image and the target lighting condition into a face image relighting model, so that the face image relighting model relights the target lighting condition into the face image, so as to obtain a face image under the target lighting condition; and the face image relighting model is trained by the above-mentioned model training method.

[0164] It should be noted that the implementation and benefits of each module described above can refer to the detailed description of the corresponding steps in the foregoing embodiments, and the present embodiment will not be described again.

[0165] It can be understood that the technical solution provided by the present embodiment is realized based on the face image re-illumination model. The face image re-illumination model obtains the synthetic data, the real data and the target illumination condition of the face image, and first trains the model to convergence based on the synthetic data, so that the model learns the distribution characteristics of the synthetic data at this time. Then, the real data and the synthetic data are alternately trained to the model, wherein the training of the synthetic data plays a guiding and supervising role for the model to learn the distribution characteristics of the real data. Since the distribution characteristics of the real data are considered in the model training process, the processing capability of the finally trained model for the real data is greatly improved, so that the face image re-illumination model trained can more realistically re-illuminate the target illumination condition to the face image to be processed, and the illumination features of the target illumination condition are maximized in the processed face image.

[0166] Figure 7 is a flow chart of a model training method according to an exemplary embodiment, as shown in Figure 7 , the method comprises:

[0167] Step S31, obtaining a face image under a target illumination condition and a ground truth of the face image under the target illumination condition, wherein the face image under the target illumination condition is obtained by using the face image processing method described above, and the resolution thereof is a first resolution;

[0168] Step S32, inputting the face image under the target illumination condition and the ground truth of the face image under the target illumination condition into a pre-constructed second network model for training, to obtain a face image with a second resolution, wherein the second resolution is greater than the first resolution;

[0169] Step S33, updating the model parameters of the second network model after each training round is completed;

[0170] Step S34, determining whether the updated second network model meets a preset second network model convergence condition, if yes, determining that the current model converges, and determining the second network model at this time as a face image super-resolution reconstruction model; if not, repeating the training steps described above until the preset second network model convergence condition is met.

[0171] It should be noted that the technical solution provided by the present embodiment is applicable to a computer terminal and / or a mobile terminal, and is applicable to an application scenario in which the resolution of an input face image needs to be improved.

[0172] The "updating the model parameters of the second network model after each training round is completed" in step S33 comprises:

[0173] According to the preset loss function, a loss function value is calculated, and the gradient of each layer of the second network model is calculated using the chain rule;

[0174] According to the gradient, the network parameters of the second network model are updated using a gradient descent algorithm and a back propagation algorithm.

[0175] The "preset second network model convergence condition" in step S34 includes:

[0176] The number of update iterations of the second network model reaches a preset number, and the loss function value no longer decreases, and the peak signal-to-noise ratio between the output second resolution face image and the input true value measured on the test set no longer rises.

[0177] It can be understood that the face image relighting model mainly focuses on whether the relighting result of the face image is natural or not, and does not require clarity, so in order to save computing resources, it is generally performed at a lower resolution, for example, 256x256. Since the relighting image output by the face image relighting model is a low-resolution image, it is necessary to enhance the resolution of the low-resolution relighting image output by the face image relighting model, and the face image super-resolution reconstruction model provided in this embodiment solves this problem.

[0178] It should be noted that in addition to the face image super-resolution reconstruction model provided in this embodiment, which can reconstruct the input low-resolution face image into a high-resolution face image, the SR (Super-Resolution, super-resolution) model disclosed in the prior art can also cooperate with the above-mentioned face image relighting model to improve the image quality and improve the clarity and visual effect of the relighting face. However, since the face image super-resolution reconstruction model mentioned in this embodiment is trained based on the output image and the input image of the face image relighting model, it can ensure that the data distribution characteristics learned by the trained model are consistent with the data distribution characteristics learned by the above-mentioned face image relighting model, and can improve the robustness and accuracy of the trained model.

[0179] It can be understood that the technical solution provided in this embodiment is that the face image super-resolution reconstruction model is trained based on the input and output face images of the face image relighting model, and the trained model can stably improve the resolution of the face image after relighting by the face image relighting model, and improve the visual effect of the user.

[0180] Figure 8 is a flowchart of a face image processing method according to an example embodiment, as shown in Figure 8 The method comprises:

[0181] Step S41, obtaining a face image to be processed, the resolution of the face image being a first resolution;

[0182] Step S42, inputting the face image into a face image super-resolution reconstruction model to obtain a face image with a second resolution, the second resolution being greater than the first resolution;

[0183] The face image super-resolution reconstruction model is obtained by the model training method described above.

[0184] It should be noted that the technical solution provided in this embodiment is applicable to a computer terminal and / or a mobile terminal, and is applicable to an application scenario in which the resolution of an input face image needs to be improved.

[0185] It can be understood that the technical solution provided in this embodiment is realized based on a face image super-resolution reconstruction model, and the face image super-resolution reconstruction model is obtained based on a face image relighting model. The model obtained by training can stably improve the resolution of a face image after relighting by the face image relighting model, and improve the visual effect of a user.

[0186] Figure 9 is a schematic block diagram of a model training device 300 according to an example embodiment, as shown in Figure 9 The device 300 includes:

[0187] The obtaining module 301 is configured to obtain a face image under a target lighting condition and a ground truth of the face image under the target lighting condition, the resolution of the face image under the target lighting condition being a first resolution, and the face image under the target lighting condition being obtained by using the face image processing method described above.

[0188] The training module 302 is configured to input the face image under the target lighting condition and the ground truth of the face image under the target lighting condition into a pre-constructed second network model to perform training, and obtain a face image with a second resolution, the second resolution being greater than the first resolution.

[0189] The updating module 303 is configured to update the model parameters of the second network model after each round of training is completed.

[0190] The determining module 304 is configured to determine whether the updated second network model meets a preset second network model convergence condition, if yes, determine that the current model converges, and determine the second network model at this time as a face image super-resolution reconstruction model; if no, repeat the training step described above until the preset second network model convergence condition is met.

[0191] It should be noted that the implementation manner and beneficial effects of each module described above can be referred to the detailed description of the corresponding steps in the foregoing embodiments, and the present embodiment will not be described herein.

[0192] It can be understood that the technical scheme provided in the embodiment is that the face image super-resolution reconstruction model is trained based on the face image output by the face image relighting model, and the trained model can stably improve the resolution of the face image after relighting by the face image relighting model and improve the visual effect of the user.

[0193] Figure 10 is a schematic block diagram of a face image processing apparatus 400 according to an example embodiment, as shown in Figure 10 The apparatus 400 comprises:

[0194] The acquisition module 401 is configured to acquire a face image to be processed, and the resolution of the face image is a first resolution.

[0195] The reconstruction module 402 is configured to input the face image into a face image super-resolution reconstruction model to obtain a face image with a second resolution, and the second resolution is greater than the first resolution.

[0196] The face image super-resolution reconstruction model is trained by the face image super-resolution reconstruction model training method.

[0197] It should be noted that the implementation manner and beneficial effects of each module described above can refer to the detailed description of the corresponding steps in the foregoing embodiments, and the present embodiment will not be described again.

[0198] It can be understood that the technical scheme provided in the embodiment is based on the face image super-resolution reconstruction model, and the face image super-resolution reconstruction model is trained based on the face image output by the face image relighting model, and the trained model can stably improve the resolution of the face image after relighting by the face image relighting model and improve the visual effect of the user.

[0199] Further, as known from the foregoing background art, since the human eye can only receive a 2D face image, when using a 3D face model, it is generally converted into a 2D face image and output to a display device. The prior art is to directly relight the 3D face model using the target lighting condition, and then convert the relighted 3D face model into a 2D face image that can be received by the human eye, which is time-consuming and laborious. To solve this technical problem, the present application proposes the following several example embodiments.

[0200] Figure 11 is a flowchart of a face model processing method according to an example embodiment, as shown in Figure 11 The method comprises:

[0201] Step S51, converting the 3D face model to be processed into a 2D face image (in specific practice, the 3D face model to be processed can be first adjusted in 3D software to the expected effect in terms of posture, action, camera, and other parameters, and then converted into a 2D face image);

[0202] Step S52, inputting the 2D face image and the specified target lighting condition into the face image relighting model, so that the face image relighting model relights the target lighting condition into the 2D face image, to generate a first-resolution face image under the target lighting condition;

[0203] Step S53, inputting the first-resolution face image under the target lighting condition into the face image super-resolution reconstruction model, to obtain a second-resolution face image under the target lighting condition, the second resolution being greater than the first resolution;

[0204] The face image relighting model is trained by the above-mentioned model training method, and the face image super-resolution reconstruction model is trained by the above-mentioned model training method.

[0205] It should be noted that the 3D face model includes a dynamic 3D face model and a static 3D face model, the static 3D face model can be converted into a 2D face image, and the dynamic 3D face model can be converted into a video. If the 3D face model is a dynamic 3D face model, the video can be decomposed into an image sequence containing multiple 2D face images after being converted into a video. If it is an image sequence, multiple relit face images can be obtained by the method provided in this embodiment, and the multiple relit face images are combined into a video again, which is fused into a background video, to obtain a video under a target lighting condition.

[0206] Therefore, the technical solution provided in this embodiment is suitable for computer terminals (including servers) and / or mobile terminals (including but not limited to smart phones, tablet computers, VR headsets, VR glasses, etc.), and is suitable for scenarios including but not limited to:

[0207] 1. Re-rendering of lighting environment in 3D games, 3D movies, and 3D videos;

[0208] 2. Rendering of lighting environment in VR games;

[0209] 3. Replacement of lighting environment in electronic posters (a single relit face image is fused back into a background poster to obtain an electronic poster under a target lighting condition).

[0210] It can be understood that the technical scheme provided by the embodiment converts the 3D face model to be processed into a 2D face image, and transfers the relighting operation of the 3D model to the 2D face image, compared with the prior art of directly using the target lighting condition to relight the 3D face model, and then converting the relighted 3D face model into a 2D face image that can be received by the human eye. The technical scheme provided by the present application greatly reduces the technical difficulty of relighting the face image on the 2D face image. The face image after relighting can be directly received by the human eye because it is 2D. The processing speed is faster and the efficiency is higher.

[0211] According to an exemplary embodiment, an electronic device is shown, comprising a processor and a memory,

[0212] The memory stores a computer program that can be called by the processor.

[0213] The processor executes the above method by calling the computer program stored in the memory.

[0214] It should be noted that the electronic device includes but is not limited to: smart terminals (such as mobile phones, tablets, smart watches, etc.) and computer devices.

[0215] The processor includes but is not limited to: CPU, single-chip microcomputer, PLC controller, FPGA controller, etc.

[0216] The memory can include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory; it can also include other removable / non-removable, volatile / non-volatile computer system storage media. The memory can include at least one program product having a set of (for example, at least one) program modules configured to perform the functions of the embodiments of the present application.

[0217] It can be understood that the technical scheme provided by the embodiment acquires the synthetic data, real data and target lighting condition of the face image, and first trains the model to convergence based on the synthetic data, so that the model learns the distribution characteristics of the synthetic data at this time. Then the real data and the synthetic data are alternately trained to the model, wherein the training of the synthetic data plays a guiding and supervising role in learning the distribution characteristics of the real data. Since the distribution characteristics of the real data are considered in the model training process, the processing capability of the finally trained model for the real data is greatly improved, so that the face image relighting model trained can more realistically relight the target lighting condition to the face image to be processed. The lighting features of the target lighting condition are maximized in the processed face image.

[0218] Further, by converting the 3D face model to be processed into a 2D face image, the heavy light operation of the 3D model is transferred to the 2D face image, compared with the prior art of directly using the target light condition to re-light the 3D face model, and then converting the re-lit 3D face model into a 2D face image that can be received by the human eye, the technical scheme provided by the application greatly reduces the technical difficulty of re-lighting the face image on the 2D face image, and the face image after the re-lighting is 2D and can be directly received by the human eye, so that the processing speed is faster and the efficiency is higher.

[0219] According to an exemplary embodiment, a computer readable storage medium is shown, and the computer readable storage medium stores a computer program which can be erased;

[0220] When the computer program runs on the computer device, the computer device executes the above method.

[0221] The computer readable storage medium disclosed in the embodiment includes but is not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0222] It can be understood that the technical scheme provided by the embodiment acquires the synthetic data, real data and target light condition of the face image, and based on the synthetic data, the model is trained to convergence first, so that the model learns the distribution characteristics of the synthetic data at this time, and then the real data and the synthetic data are alternately trained to the model, wherein the training of the synthetic data plays a guiding and supervising role for the model to learn the distribution characteristics of the real data. Since the distribution characteristics of the real data are considered in the model training process, the processing capability of the finally trained model for the real data is greatly improved, so that the face image re-lighting model trained can re-light the target light condition to the face image to be processed more realistically, and the light features of the target light condition are maximized in the processed face image.

[0223] Further, by converting the 3D face model to be processed into a 2D face image, the heavy light operation of the 3D model is transferred to the 2D face image, compared with the prior art of directly using the target light condition to re-illuminate the 3D face model, and then converting the 3D face model after the heavy light operation into a 2D face image that can be received by the human eye, the technical scheme provided by the present application greatly reduces the technical difficulty of the heavy light operation of the face image on the 2D face image, and the face image after the heavy light operation can be directly received by the human eye because it is 2D, so that the processing speed is faster and the efficiency is higher.

[0224] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0225] It should be noted that, in the description of the present application, the terms "first", "second", etc. are only for the purpose of description, and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is at least two.

[0226] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for performing specific logic functions or steps in the process, and that the various systems discussed herein can include one or more circuits that can implement these code modules, segments, or portions of code. The various circuits can comprise one or more processors (shared, dedicated, or group) and / or memories (shared, dedicated, or group) that execute one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functionality.

[0227] It should be understood that each part of the present application can be realized by hardware, software, firmware or their combination. In the above embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized by hardware, and as in another embodiment, it can be realized by any one or their combination of the following technologies known in the art: discrete logic circuit with logic gate circuit for implementing logic function on data signal, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0228] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium, and the program includes one or a combination of steps of the method embodiment when executed.

[0229] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0230] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0231] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0232] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A model training method, characterized in that, The method comprises the following steps: obtaining synthetic data, real data and a target lighting condition of a face image; inputting the synthetic data and the target lighting condition into a pre-constructed first network model for training until the first network model converges, to obtain a first model; wherein the first network model comprises a light removal model, a light and shadow feature model and an image synthesis model, and each of the light removal model, the light and shadow feature model and the image synthesis model is composed of a compression channel and an expansion channel, wherein each compression channel is used for extracting features of the face image layer by layer, and each expansion channel is used for restoring position information of the face image; inputting the real data and the target lighting condition and the synthetic data and the target lighting condition into the first model alternately for training until the first model converges, to obtain a face image relighting model, comprising: inputting the real data into the first model to obtain a third face image under a lighting condition of the real data; inputting the target lighting condition into the first model to obtain a fourth face image under the target lighting condition; inputting the synthetic data into the first model to obtain a fifth face image under a lighting condition of the synthetic data; inputting the target lighting condition into the first model to obtain a sixth face image under the target lighting condition; calculating a second loss function value according to a second preset loss function, and calculating gradients of each layer of the first model using the chain rule; updating network parameters of the first model using a gradient descent algorithm and a back propagation algorithm according to the gradients; determining whether the updated first model meets a preset first model convergence condition, if yes, determining that the current model converges, and determining the first model at this time as the face image relighting model; if not, repeating the above training steps until the preset first model convergence condition is met; the face image relighting model is used for relighting a specified target lighting condition into a face image to obtain a face image under the target lighting condition.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining synthetic data, real data and a target lighting condition of a face image; obtaining publicly available 3D face models, rendering each 3D face model using different high dynamic range imaging (HDRI) environment maps, converting the rendered 3D face models into 2D face images, and saving the converted 2D face images as synthetic data; saving the HDRI environment map corresponding to each synthetic data as a target lighting condition, and storing each synthetic data and the target lighting condition corresponding to each synthetic data in a database in pairs; 3. The method of claim 1, wherein, obtaining publicly available 2D high-definition face images, and saving the face region in each 2D high-definition face image as real data. The method comprises the following steps: The synthetic data is input into the first network model to obtain a first face image under a light condition self-brought by the synthetic data, including: inputting the synthetic data into the de-light model to obtain an albedo map, a normal map and a first predicted light condition; the first predicted light condition is an estimated value of the light condition self-brought by the synthetic data; inputting the first predicted light condition and the normal map into the light-shadow feature model to obtain a highlight map under the first predicted light condition and a shadow map under the first predicted light condition; inputting the albedo map, the normal map, the highlight map under the first predicted light condition, the shadow map under the first predicted light condition and the first predicted light condition into the image synthesis model to obtain the first face image under the first predicted light condition; The target light condition is input into the first network model to obtain a second face image under the target light condition, including: inputting the target light condition and the normal map into the light-shadow feature model to obtain a highlight map under the target light condition and a shadow map under the target light condition; inputting the albedo map, the normal map, the highlight map under the target light condition, the shadow map under the target light condition and the target light condition into the image synthesis model to obtain the second face image under the target light condition; According to a first preset loss function, a first loss function value is calculated, and the gradient of each layer of the first network model is calculated using the chain rule; according to the gradient, the network parameters of the first network model are updated using a gradient descent algorithm and a back propagation algorithm; It is judged whether the updated first network model meets a preset first network model convergence condition, if yes, it is judged that the current model converges, and the first network model at this time is determined as a first model; if not, the above training steps are repeatedly executed until the preset first model convergence condition is met; The first preset loss function includes one or more combinations of the following items, including: The highlight map under the target light condition, and a true value of the highlight map under the target light condition, a content loss function of the two; The shadow map under the target light condition, and a true value of the shadow map under the target light condition, a content loss function of the two; The second face image, and a true value of the synthetic data under the target light condition, a content loss function of the two; The first preset loss function further includes one or more of the following items: The albedo map, and a true value of the albedo map, a content loss function of the two; The normal map, and a true value of the normal map, a content loss function of the two; The first face image, and a true value of the synthetic data input into the de-light model, a content loss function of the two; The predicted light condition, and a true value of the predicted light condition, a light loss function of the two; the true value is a pixel value corresponding to a face image.

4. The method of claim 3, wherein, The preset first network model convergence condition includes: The number of update iterations of the first network model reaches a preset number, and the first loss function value no longer decreases, and the structural similarity measured on a test set no longer improves; The test set stores a plurality of face images and corresponding true values of each face image; the structural similarity refers to a similarity between a predicted value of a re-illumination face image and a corresponding true value after the face image in the test set is input into the first network model; The predicted value refers to a pixel value of a face image after re-illumination under a target illumination condition, and the true value refers to a pixel value of an original face image under the target illumination condition.

5. The method of claim 1, wherein, The first model comprises a de-light model, a light-shadow feature model and an image synthesis model; the third face image under the illumination condition of the real data is obtained by inputting the real data into the first model, comprising: The inverse albedo map, the normal map and the second predicted illumination condition are obtained by inputting the real data into the de-light model; the second predicted illumination condition is an estimated value of the illumination condition of the real data; The second predicted illumination condition and the normal map are input into the light-shadow feature model to obtain a highlight map under the second predicted illumination condition and a shadow map under the second predicted illumination condition; The inverse albedo map, the normal map, the highlight map under the second predicted illumination condition, the shadow map under the second predicted illumination condition and the second predicted illumination condition are input into the image synthesis model to obtain the third face image under the second predicted illumination condition.

6. The method of claim 5, wherein, The fourth face image under the target illumination condition is obtained by inputting the target illumination condition into the first model, comprising: The target illumination condition and the normal map are input into the light-shadow feature model to obtain a highlight map under the target illumination condition and a shadow map under the target illumination condition; The inverse albedo map, the normal map, the highlight map under the target illumination condition, the shadow map under the target illumination condition and the target illumination condition are input into the image synthesis model to obtain the fourth face image under the target illumination condition.

7. The method of claim 6, wherein, The fifth face image under the illumination condition of the synthetic data is obtained by inputting the synthetic data into the first model, comprising: The inverse albedo map, the normal map and the third predicted illumination condition are obtained by inputting the synthetic data into the de-light model; the third predicted illumination condition is an estimated value of the illumination condition of the synthetic data; The third predicted illumination condition and the normal map are input into the light-shadow feature model to obtain a highlight map under the third predicted illumination condition and a shadow map under the third predicted illumination condition; The inverse albedo map, the normal map, the highlight map under the third predicted illumination condition, the shadow map under the third predicted illumination condition and the third predicted illumination condition are input into the image synthesis model to obtain the fifth face image under the third predicted illumination condition; The sixth face image under the target illumination condition is obtained by inputting the target illumination condition into the first model, comprising: The target illumination condition and the normal map are input into the light-shadow feature model to obtain a highlight map under the target illumination condition and a shadow map under the target illumination condition; input the albedo map, the normal map, the high-light map under the target light condition, the shadow map under the target light condition and the target light condition into the image synthesis model to obtain a sixth face image under the target light condition.

8. The method of claim 3, wherein, The second preset loss function comprises a combination of one or more of the following: The albedo map, and a content loss function of the verification albedo map; The normal map, and a content loss function of the verification normal map; The second preset loss function further comprises a combination of one or more of the following: The third face image, and a content loss function of the real data input into the de-light model; The verification albedo map is obtained by inputting the fourth face image into the de-light model; and the verification normal map is obtained by inputting the fourth face image into the de-light model.

9. The method of claim 4, wherein, The preset first model convergence condition comprises: The number of update iterations of the first model reaches a preset number, and the second loss function value no longer decreases, and the structural similarity measured on the test set no longer improves; The test set stores a plurality of face images and a corresponding true value of each face image; the structural similarity refers to the similarity between the predicted value of the face image under the target light condition and the corresponding true value after the face image in the test set is input into the first model to generate a face image under the heavy light condition; The predicted value refers to the pixel value of the face image under the target light condition; and the true value refers to the pixel value of the original face image under the target light condition.

10. A face image processing method, characterized by, The method comprises: obtaining a face image to be processed and a target light condition, wherein the target light condition comprises an HDRI environment map or a light condition extracted from a specified face image; inputting the face image and the target light condition into a face image heavy light model to enable the face image heavy light model to re-illuminate the target light condition into the face image to obtain a face image under the target light condition; and the face image heavy light model is trained by the model training method of any one of claims 1-9.

11. A model training method, comprising: The method comprises: obtaining a face image under a target light condition and a true value of the face image under the target light condition, wherein the face image under the target light condition is obtained by the face image processing method of claim 10, and the resolution of the face image under the target light condition is a first resolution; inputting the face image under the target light condition and the true value of the face image under the target light condition into a pre-constructed second network model to train the face image under the target light condition to obtain a face image under a second resolution, wherein the second resolution is greater than the first resolution; updating the model parameters of the second network model after each training round is completed; determining whether the updated second network model meets a preset second network model convergence condition, if yes, determining that the current model converges, and determining the second network model at this time as a face image super-resolution reconstruction model; and if no, repeatedly executing the training steps until the preset second network model convergence condition is met.

12. The method of claim 11, wherein, The updating of the model parameters of the second network model after each training round is completed comprises: According to a preset loss function, a loss function value is calculated, and gradients of each layer of the second network model are calculated using a chain rule; According to the gradients, network parameters of the second network model are updated using a gradient descent algorithm and a back propagation algorithm.

13. The method of claim 11, wherein, The preset second network model convergence condition includes: The number of update iterations of the second network model reaches a preset number, the loss function value no longer decreases, and the peak signal-to-noise ratio between the output second-resolution face image and the input true value measured on the test set no longer increases.

14. A face image processing method, characterized by, Including: Obtaining a face image to be processed, the resolution of the face image being a first resolution; Inputting the face image into a face image super-resolution reconstruction model to obtain a second-resolution face image, the second resolution being greater than the first resolution; The face image super-resolution reconstruction model is trained by the model training method of any one of claims 11-13.

15. A face model processing method, comprising: Including: Converting a 3D face model to be processed into a 2D face image; Inputting the 2D face image and a specified target lighting condition into a face image relighting model to make the face image relighting model relight the target lighting condition into the 2D face image to generate a first-resolution face image under the target lighting condition; Inputting the first-resolution face image under the target lighting condition into a face image super-resolution reconstruction model to obtain a second-resolution face image under the target lighting condition, the second resolution being greater than the first resolution; The face image relighting model is trained by the model training method of any one of claims 1-9, and the face image super-resolution reconstruction model is trained by the model training method of any one of claims 11-13.

16. A model training apparatus, comprising: Including: An acquisition module is configured to acquire synthetic data, real data, and a target lighting condition of a face image; A first training module is configured to input the synthetic data and the target lighting condition into a pre-constructed first network model for training until the first network model converges to obtain a first model; wherein the first network model includes a light removal model, a light shadow feature model, and an image synthesis model, and each of the light removal model, the light shadow feature model, and the image synthesis model is composed of a compression channel and an expansion channel, wherein each compression channel is configured to extract features of a face image layer by layer, and each expansion channel is configured to restore position information of the face image. The second training module is configured to input real data and target lighting conditions and input synthetic data and target lighting conditions alternately into the first model for training until the first model converges, so as to obtain a face image relighting model; specifically, the real data is input into the first model to obtain a third face image under a lighting condition self-brought by the real data; the target lighting condition is input into the first model to obtain a fourth face image under the target lighting condition; the synthetic data is input into the first model to obtain a fifth face image under a lighting condition self-brought by the synthetic data; the target lighting condition is input into the first model to obtain a sixth face image under the target lighting condition; a second loss function value is calculated according to a second preset loss function, and a gradient of each layer of the first model is calculated using a chain rule; the network parameters of the first model are updated using a gradient descent algorithm and a back propagation algorithm according to the gradient; and it is determined whether the updated first model meets a preset first model convergence condition; if yes, it is determined that the current model converges, and the first model at this time is determined as the face image relighting model; and if no, the above training step is repeatedly executed until the preset first model convergence condition is met. The face image relighting model is configured to relight a specified target lighting condition into an input face image to obtain a face image under the target lighting condition.

17. A face image processing apparatus, comprising: The method comprises the following steps: The acquisition module is configured to acquire a face image to be processed and a target lighting condition, wherein the target lighting condition comprises an HDRI environment map or a lighting condition extracted from a specified face image. The relighting module is configured to input the face image and the target lighting condition into the face image relighting model, so that the face image relighting model relights the target lighting condition into the face image to obtain a face image under the target lighting condition; and the face image relighting model is trained by the model training method in any one of claims 1 to 9.

18. A model training apparatus, comprising: The method comprises the following steps: The acquisition module is configured to acquire a face image under a target lighting condition and a ground truth of the face image under the target lighting condition, wherein the resolution of the face image under the target lighting condition is a first resolution, and the face image under the target lighting condition is acquired by the face image processing method in claim 10. The training module is configured to input the face image under the target lighting condition and the ground truth of the face image under the target lighting condition into a second network model constructed in advance for training, so as to obtain a face image with a second resolution, wherein the second resolution is greater than the first resolution. The updating module is configured to update the model parameters of the second network model after each round of training is completed. The judgment module is configured to determine whether the updated second network model meets a preset second network model convergence condition; if yes, it is determined that the current model converges, and the second network model at this time is determined as a face image super-resolution reconstruction model; and if no, the above training step is repeatedly executed until the preset second network model convergence condition is met.

19. A face image processing apparatus, comprising: The acquisition module is configured to acquire a face image to be processed, wherein the resolution of the face image is a first resolution. ​ a reconstruction module configured to input the face image into a face image super-resolution reconstruction model to obtain a face image of a second resolution, the second resolution being greater than the first resolution; the face image super-resolution reconstruction model is trained by the face image super-resolution reconstruction model training method of any one of claims 11-13.

20. An electronic device, comprising: comprising: a processor and a memory, the memory stores a computer program that can be invoked by the processor; the processor executes the method of any one of claims 1-12 by invoking the computer program stored in the memory.

21. A computer-readable storage medium, characterized in that, a computer program that can be written to the computer readable storage medium; when the computer program runs on the computer device, the computer device executes the method of any one of claims 1-12.

Images