A model training method and system, a face image de-lighting method and system

Abstract

The present invention relates to a model training method and system, a facial image de-lighting method and system, an electronic device and a readable storage medium. The method obtains a high-definition facial image data set, selects facial images with uniform illumination from the high-definition facial image data set, and re-illuminates the selected facial images with uniform illumination to generate a facial re-illuminated image set, thereby realizing the self-generation of model training data, getting rid of the dependence on expensive data acquisition equipment, and having low cost and more popular algorithm. Furthermore, compared with the prior art, the technical solution provided by the present invention does not require 2D to 3D conversion, and can achieve de-lighting of facial images only through 2D facial images, greatly improving the de-lighting effect of facial images.

Description

Technical Field

[0001] The present invention relates to the field of deep learning model technology, and in particular to a model training method and system, a facial image delighting method and system, an electronic device and a readable storage medium. Background Art

[0002] With the development of hardware and the Internet, taking photos has become more and more convenient. However, due to the limitations of the shooting environment, the photos taken are often easily affected by light and shadow, such as overexposure caused by too strong light, or inappropriate shadow position, etc. Photo retouchers can use a series of professional tools to solve this problem, but the cost of doing so is relatively high.

[0003] Currently, there have been some progress in facial image delighting methods (including delighting and / or deshadowing), but the following problems still exist: 1) Facial image delighting methods rely on the training of facial image delighting models, and there is no large public dataset available for the training of facial image delighting models. The data of methods with better results are often collected based on expensive light stages, resulting in the low popularity of the algorithms; 2) The existing technology estimates the 3D shape of the face from the 2D face image, then delights the estimated 3D face shape, and finally renders it back to a 2D image to obtain a facial image with uniform lighting. Since there will be large errors in the conversion process from 2D to 3D, the final delighting effect of the facial image is affected. Summary of the invention

[0004] In order to overcome the problems existing in the related art to at least a certain extent, the present invention provides a model training method and system, a face image delighting method and system, an electronic device and a readable storage medium, so as to solve the problem that the face image delighting method in the prior art relies on the training of the face image delighting model, and the training of the face image delighting model requires the use of expensive equipment for data collection, resulting in high cost and low algorithm popularity.

[0005] According to a first aspect of an embodiment of the present invention, a model training method is provided, comprising:

[0006] Get a high-definition face image dataset;

[0007] Screening out facial images with uniform illumination from the high-definition facial image dataset, and storing the screened facial images with uniform illumination in a real facial image set;

[0008] Re-illuminating the screened face images with uniform illumination to obtain face images with shadows and highlights, and generating a set of face re-illumination images;

[0009] The face images in the face re-illuminated image set and the face images in the real face image set are input into a pre-built network model for training until the model converges, thereby obtaining a face image de-lighting model. The face image de-lighting model is used to perform de-lighting on the input face images to obtain face images with uniform illumination.

[0010] Furthermore, the step of screening out facial images with uniform illumination from the high-definition facial image dataset includes:

[0011] Performing face segmentation on each high-definition face image in the high-definition face image data set to obtain a face region in the face image;

[0012] For each segmented face region, it is determined whether a face image corresponding to the face region is a face image with uniform illumination according to the ratio of highlight pixels to shadow pixels in the face region.

[0013] Further, judging whether the face image corresponding to the face area is a face image with uniform illumination according to the ratio of highlight pixels to shadow pixels in the face area includes:

[0014] In the face area, pixels with pixel values ​​higher than a first preset pixel value are marked as highlight pixels, and pixels with pixel values ​​lower than a second preset pixel value are marked as shadow pixels; the second preset pixel value is lower than the first preset pixel value;

[0015] Calculate the ratio of the sum of the number of shadow pixels and the number of highlight pixels to the total number of pixels in the face area;

[0016] If the proportion is lower than a threshold, the facial image corresponding to the facial region is marked as a facial image with uniform illumination.

[0017] Furthermore, the re-illumination of the screened facial image with uniform illumination includes:

[0018] Randomly selecting spherical harmonic function coefficients, adding the illumination direction and illumination intensity corresponding to the spherical harmonic function coefficients to the screened face image with uniform illumination, and obtaining a face image with shadows and highlights by adjusting the scaling scale of the spherical harmonic function coefficients;

[0019] The silhouette data is added to the face image with shadows and highlights as the outline of an external shadow or by adding Perlin noise to imitate the shadow caused by an external object with no fixed shape.

[0020] Furthermore, the step of inputting the face images in the face re-illumination image set and the face images in the real face image set into a pre-built network model for training includes:

[0021] Defining a loss function of the network model;

[0022] Updating the generator and discriminator of the network model until the model converges;

[0023] The generator is used to receive the re-illuminated face image and perform light removal on the re-illuminated face image to generate a face image with uniform illumination;

[0024] The discriminator is used to receive a face image generated by a generator, or a real face image with uniform lighting, and output a confidence level that the face image belongs to a real face image with uniform lighting.

[0025] Furthermore, the loss function of the network model at least includes:

[0026] A standard deviation-based regularization term for optimizing the generator;

[0027] The regularization term is used to constrain the lighting effect of the face image generated by the generator.

[0028] Furthermore, the updating of the generator and the discriminator of the network model includes:

[0029] Updated generators, including:

[0030] The network parameters of the discriminator are fixed, and the face images in the face re-illuminated image set are sent to the generator in batches to output face images with uniform illumination;

[0031] According to the confidence of the uniformly illuminated face image output by the discriminator, the loss function value is calculated, and the model parameters of each layer of the generator are derived to obtain the gradient of each layer of the generator;

[0032] According to the gradient, the network parameters of the generator are updated using the gradient descent algorithm and the back propagation algorithm;

[0033] Update the discriminator, including:

[0034] Inputting the face image generated by the generator and the face image in the real face image set into the discriminator in turn, and obtaining the first confidence and the second confidence that the input face image belongs to the real face image with uniform illumination respectively;

[0035] Calculate the loss function value according to the first confidence and the second confidence, and use the chain rule to calculate the gradient of each layer of the discriminator;

[0036] According to the gradient, the network parameters of the discriminator are updated using the gradient descent algorithm and the back propagation algorithm.

[0037] Furthermore, the method further comprises:

[0038] Setting the number of optimization iterations of the generator and the discriminator, when the number of optimization iterations reaches a preset number, the loss function value no longer decreases, and the structural similarity measured on the test set no longer increases, the model is judged to be converged, and the generator at this time is determined as a face image de-lighting model;

[0039] The test set stores multiple face images and the true value corresponding to each face image; the true value refers to the pixel value of the face image when it is not re-illuminated;

[0040] The structural similarity refers to the similarity between the predicted value of the uniformly illuminated face image and the corresponding true value after the face image in the test set is input into the generator;

[0041] The predicted value refers to the pixel value of the face image with uniform illumination.

[0042] According to a second aspect of an embodiment of the present invention, a method for removing light from a facial image is provided, comprising:

[0043] Perform face alignment on the face image collected in real time to identify the face area, and align the identified face area to the specified position;

[0044] The face image after face alignment is input into the face image delighting model for delighting processing to obtain a face image with uniform illumination; the face image delighting model is obtained by using the model training method as described above.

[0045] According to a third aspect of an embodiment of the present invention, a model training system is provided, including:

[0046] An acquisition module is used to acquire a high-definition face image dataset;

[0047] A screening module, used to screen out facial images with uniform illumination from the high-definition facial image data set, and store the screened facial images with uniform illumination in a real facial image set;

[0048] A re-illumination module, used for re-illuminating the screened face images with uniform illumination to generate a set of re-illumination face images;

[0049] The training module is used to input the face images in the face re-illuminated image set and the face images in the real face image set into a pre-built network model for training until the model converges, thereby obtaining a face image de-lighting model. The face image de-lighting model is used to perform de-lighting processing on the input face images to obtain face images with uniform illumination.

[0050] Furthermore, the network model includes:

[0051] A generator, used for receiving a re-illuminated face image, and performing a de-lighting process on the re-illuminated face image to generate a face image with uniform illumination;

[0052] The discriminator is used to receive the face image generated by the generator, or the real face image with uniform illumination, and output the confidence that the face image is a real face image with uniform illumination.

[0053] Furthermore, the generator comprises:

[0054] The encoder is used to extract features from the face image after re-illumination to obtain the face features without the influence of illumination;

[0055] The decoder is used to restore the facial features without the influence of lighting to a facial image with uniform lighting.

[0056] Furthermore, the discriminator comprises:

[0057] The convolution layer is used to extract features from the input face image with uniform illumination.

[0058] The residual module is used to perform secondary feature extraction and downsampling on the features extracted by the convolutional layer;

[0059] The fully connected layer is used to calculate the confidence that the input uniformly illuminated face image belongs to a true uniformly illuminated face image based on the features output by the residual module, and output the confidence.

[0060] According to a fourth aspect of an embodiment of the present invention, a facial image light removal system is provided, comprising:

[0061] An alignment module is used to perform face alignment on the face image collected in real time to identify the face area and align the identified face area to a specified position;

[0062] The delighting module is used to input the face image after face alignment into the face image delighting model for delighting processing to obtain a face image with uniform illumination; the face image delighting model is obtained using the model training method as described above.

[0063] According to a fifth aspect of an embodiment of the present invention, there is provided an electronic device, including: a processor and a memory, wherein:

[0064] The memory stores a computer program that can be called by the processor;

[0065] The processor executes the above-mentioned facial image delighting model training method and / or facial image delighting method by calling the computer program stored in the memory.

[0066] According to a sixth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a rewritable computer program stored thereon;

[0067] When the computer program is executed on a computer device, the computer device executes the above-mentioned training method of the facial image delighting model and / or the facial image delighting method.

[0068] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0069] By acquiring a high-definition face image dataset, screening out face images with uniform illumination from the high-definition face image dataset, and re-illuminating the screened face images with uniform illumination to generate a face re-illumination image set, self-generation of model training data is achieved, getting rid of dependence on expensive data acquisition equipment, with low cost and more popular algorithm.

[0070] Furthermore, the technical solution provided by the present invention does not require 2D to 3D conversion and can achieve deglazing of facial images only through 2D facial images, which greatly improves the deglazing effect of facial images.

[0071] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0073] Figure 1 is a flow chart of a model training method according to an exemplary embodiment;

[0074] Figure 2 is a schematic diagram of the structure of a generator according to an exemplary embodiment;

[0075] Figure 3 is a schematic diagram of the structure of a GAN block according to an exemplary embodiment;

[0076] Figure 4 is a schematic diagram of the structure of a discriminator according to an exemplary embodiment;

[0077] Figure 5 is a schematic structural diagram of a residual module according to an exemplary embodiment;

[0078] Figure 6 is a schematic block diagram of a model training system according to an exemplary embodiment;

[0079] Figure 7The present invention is a schematic block diagram of a facial image delighting system according to an exemplary embodiment. DETAILED DESCRIPTION

[0080] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0081] As mentioned in the previous background technology, the existing facial image delighting method relies on the training of the facial image delighting model, and the training of the facial image delighting model requires the use of expensive equipment for data collection, resulting in high costs and low algorithm popularity. In order to solve this technical problem, the present invention provides the following exemplary embodiments.

[0082] It should be noted that the “delighting” mentioned in the various embodiments of the present invention refers to removing highlights (overexposure) and / or shadows in the current face image; “relighting” refers to adding highlights (overexposure) and / or shadows to the current face image; and “real face image with uniform lighting” refers to a face image in a set of real face images.

[0083] Figure 1 is a flowchart of a method for training a face image de-lighting model according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0084] Step S11, obtaining a high-definition face image dataset;

[0085] Step S12, screening out facial images with uniform illumination from the high-definition facial image data set, and storing the screened facial images with uniform illumination in a real facial image set;

[0086] Step S13, re-illuminating the screened facial images with uniform illumination to obtain facial images with shadows and highlights, and generating a set of re-illumination facial images;

[0087] Step S14: input the face images in the face re-illuminated image set and the face images in the real face image set into a pre-constructed network model for training until the model converges, thereby obtaining a face image de-lighting model, wherein the face image de-lighting model is used to perform de-lighting on the input face images to obtain face images with uniform illumination.

[0088] It should be noted that the technical solution provided in this embodiment is applicable to computers and / or mobile terminals, and applicable scenarios include but are not limited to: highlights (overexposure) and / or shadows in the current face image.

[0089] In step S11, “obtaining a high-definition face image dataset” is specifically as follows:

[0090] Obtain the publicly available FFHQ and CelebA-HQ face image datasets from the Internet.

[0091] Among them, FFHQ, which stands for Flickr-Faces-Hight-Quality (Flickr-Faces-HQ), was created as a benchmark for generative adversarial networks (GANs) and is also used in the training dataset of Style GAN. It was open sourced in 2019. FFHQ is a high-quality face dataset containing 70,000 high-definition face images in PNG format with a resolution of 1024x1024.

[0092] The high definition refers to a face image with a resolution greater than 1024x1024.

[0093] CelebA is the abbreviation of CelebFaces Attribute, which is a celebrity face attribute dataset. It contains 202,599 face images of 10,177 celebrities. Each image is marked with features, including face bbox annotation box, 5 facial feature point coordinates and 40 attribute tags. CelebA is openly provided by the Chinese University of Hong Kong.

[0094] In step S12, “selecting facial images with uniform illumination from the high-definition facial image dataset” includes:

[0095] Performing face segmentation on each high-definition face image in the high-definition face image data set to obtain a face region in the face image;

[0096] In the face area, pixels with pixel values ​​higher than a first preset pixel value are marked as highlight pixels, and pixels with pixel values ​​lower than a second preset pixel value are marked as shadow pixels; the second preset pixel value is lower than the first preset pixel value;

[0097] Calculate the ratio of the sum of the number of shadow pixels and the number of highlight pixels to the total number of pixels in the face area;

[0098] If the proportion is lower than a threshold, the facial image corresponding to the facial region is marked as a facial image with uniform illumination.

[0099] In specific practice, the first preset pixel value, the second preset pixel value, and the threshold are set according to user needs, or according to experimental data, or according to historical experience values. For example, the first preset pixel value is set to 200, and the second preset pixel value is set to 114.

[0100] Step S13 of “re-illuminating the screened face images with uniform illumination to generate a set of re-illumination face images” includes:

[0101] Randomly selecting spherical harmonic function coefficients, adding the illumination direction and illumination intensity corresponding to the spherical harmonic function coefficients to the screened face image with uniform illumination, and obtaining a face image with shadows and highlights by adjusting the scaling scale of the spherical harmonic function coefficients;

[0102] The silhouette data is added to the face image with shadows and highlights as the outline of an external shadow or by adding Perlin noise to imitate the shadow caused by an external object with no fixed shape.

[0103] It should be noted that spherical harmonic lighting is widely used in the field of modern graphics rendering and can quickly simulate complex real-time lighting. Spherical harmonic lighting is a lighting algorithm whose core is a special function defined on a sphere: spherical harmonic function, which can capture lighting and re-light it later and can display global illumination style area light sources and soft shadows in real time.

[0104] The spherical harmonic function has 9 coefficients, and the sizes of these 9 coefficients jointly determine the current lighting direction and lighting intensity. These 9 coefficients can be multiplied by the scaling scale to adjust the lighting intensity (the lighting direction remains unchanged, and the lighting direction will only change if each coefficient is adjusted separately). If the scaling scale is greater than 1, it means that the light intensity increases, and if it is less than 1, it means that the light intensity decreases. By adjusting the scaling scale of the spherical harmonic function coefficients, a face image with shadows and highlights can be obtained.

[0105] Since the face image selected in this application is a color image, each of the three RGB channels corresponds to 9 coefficients, so a 3*9 coefficient matrix is ​​formed.

[0106] This 3*9 coefficient matrix and the screened face images with uniform lighting are input into the learning model to learn the lighting direction and intensity corresponding to this 3*9 coefficient matrix, and obtain face images with shadows and highlights.

[0107] It is understandable that the re-illumination of the screened facial images with uniform illumination must not only simulate the shadows cast by natural light on the facial images (achieved by adjusting the spherical harmonic function coefficients as mentioned above), but also simulate the shadows cast by external objects in the environment on the facial images.

[0108] There are two ways to simulate the shadows caused by external objects in the environment on the face image:

[0109] 1. Randomly select one or more silhouettes from the publicly available silhouette dataset on the Internet, imitate the shadows caused by external objects of fixed shapes, and add them to the face image with shadows and highlights (by calling the fusion function to fuse the silhouette data and the face image data).

[0110] 2. Generate Perlin noise through a noise function to imitate the shadow caused by an external object with no fixed shape, and add it to the face image with shadows and highlights (by calling a fusion function to fuse the silhouette data and the face image data).

[0111] Step S14 of “inputting the face images in the face re-illumination image set and the face images in the real face image set into a pre-built network model for training” includes:

[0112] Defining a loss function of the network model;

[0113] Updating the generator and discriminator of the network model until the model converges;

[0114] In specific practice, the network model is constructed based on a deep learning framework (including but not limited to: Pytorch deep learning framework, TensorFlow deep learning framework), and the network model includes:

[0115] A generator, used for receiving a re-illuminated face image, and performing a de-lighting process on the re-illuminated face image to generate a face image with uniform illumination;

[0116] The discriminator is used to receive the face image generated by the generator, or the real face image with uniform illumination, and output the confidence that the face image is a real face image with uniform illumination.

[0117] See also Figure 2 , the generator comprises:

[0118] The encoder is used to extract features from the face image after re-illumination to obtain the face features without the influence of illumination;

[0119] The decoder is used to restore the facial features without the influence of lighting to a facial image with uniform lighting.

[0120] See also Figure 2 , the encoder includes multiple convolutional layers Conv and a fully connected layer FC, where the parameters of each convolutional layer Conv can be set as shown in the following Table 1 (the step size in Table 1 refers to the step size of each convolution kernel movement, and the output size refers to the vector dimension output after convolution):

[0121]

[0122]

[0123] Table 1

[0124] It can be understood that the convolution layer Conv in the encoder is mainly used for feature extraction, and the fully connected layer FC is mainly used for feature dimensionality reduction, which reduces the dimension of the features extracted by the convolution layer Conv to a length of 512.

[0125] See also Figure 2 The final output of the encoder is a feature vector Z of length 512. The output of each convolutional layer Conv will also be saved and provided to the decoder through cross-layer connections.

[0126] The feature vector Z with a length of 512 output by the encoder is passed through a mapping network consisting of 8 fully connected layers to generate a feature vector W with a length of 512.

[0127] The encoder includes a Const matrix and multiple GAN blocks, where the fixed matrix is ​​the Const matrix, which is a fixed matrix of size 4×4×512. All GAN ​​blocks have the same structure, as follows Figure 3 shown.

[0128] Figure 3 In the figure, A represents the feature vector W of length 512, and B represents the intermediate result output by each convolutional layer Conv of the encoder.

[0129] Further, see Figure 4 , the discriminator comprises:

[0130] A convolutional layer Conv, used to extract features from the input face image with uniform illumination;

[0131] Multiple residual blocks are used to perform secondary feature extraction and downsampling on the features extracted by the convolutional layer;

[0132] The fully connected layer (also known as the linear layer) is used to calculate the confidence that the input uniformly illuminated face image belongs to a true uniformly illuminated face image based on the features output by the residual module, and output the confidence.

[0133] Among them, all residual blocks have the same structure, such as Figure 5 As shown in Figure 1, each residual module consists of two convolutions and one bypass convolution, and finally the two convolution inputs are summed. Figure 5 In , the activation function type can be leakyrelu.

[0134] Furthermore, the loss function of defining the network model at least includes:

[0135] 1) Reconstruction loss function: This loss function only participates in optimizing the generator;

[0136] Among them, λ l1 are manually specified hyperparameters, refers to the face images in the face re-illumination image set, y refers to the real face image with uniform illumination, and ||·||1 refers to the L1 norm.

[0137] 2) Generate adversarial loss functions, including the loss function used to optimize the generator:

[0138]

[0139] The loss function used to optimize the discriminator is:

[0140] Where D(y) refers to the confidence that the image y input to the discriminator is a true face image with uniform illumination; refers to the image input to the discriminator The confidence level of a real face image with uniform illumination. The higher the value, the greater the possibility that it is a real face image with uniform illumination.

[0141] 3) Perceptual loss function: This loss function only participates in optimizing the generator;

[0142] Here, φ refers to the extracted perceptual features. In this embodiment, the convolutional layers Conv1 to Conv5 of the VGG-19 network are used to extract features.

[0143] 4) Regularization term based on standard deviation: This loss function only participates in optimizing the generator;

[0144] Among them, λ reg is a specified hyperparameter, std(·) refers to the calculated standard deviation, and ReLU(x) = max(0, x).

[0145] In summary, the total loss function is: L = L recon +L GAN +L per +L reg ;

[0146] Among them, the regularization term is used to constrain the lighting effect of the face image generated by the generator.

[0147] It is understandable that for a high-quality face image with uniform illumination, the pixel values ​​are between 0 and 255, and the means are different, but the standard deviations are between 20 and 30. Therefore, this embodiment proposes a constraint item based on the standard deviation to constrain the standard deviation of the face image output by the generator, thereby ultimately improving the de-lighting effect and significantly suppressing the highlights.

[0148] In step S14, “updating the generator and discriminator of the network model” includes:

[0149] Updated generators, including:

[0150] The network parameters of the discriminator are fixed, and the face images in the face re-illuminated image set are sent to the generator in batches to output face images with uniform illumination;

[0151] According to the confidence of the uniformly illuminated face image output by the discriminator, the loss function value is calculated, and the gradient of each layer of the generator is calculated using the chain rule (the chain rule is to derive the model parameters of each layer of the generator to obtain the gradient of each layer of the generator);

[0152] According to the gradient, the network parameters of the generator are updated using the gradient descent algorithm and the back propagation algorithm;

[0153] Update the discriminator, including:

[0154] Inputting the face image generated by the generator and the face image in the real face image set into the discriminator in turn, and obtaining the first confidence and the second confidence that the input face image belongs to the real face image with uniform illumination respectively;

[0155] Calculate the loss function value according to the first confidence and the second confidence, and use the chain rule to calculate the gradient of each layer of the discriminator;

[0156] According to the gradient, the network parameters of the discriminator are updated using the gradient descent algorithm and the back propagation algorithm.

[0157] Furthermore, the number of optimization iterations of the generator and the discriminator is set. When the number of optimization iterations reaches a preset number, the loss function value no longer decreases, and the structural similarity measured on the test set no longer increases, the model is judged to be converged, and the generator at this time is determined to be a face image delighting model.

[0158] The test set stores multiple face images and the true value corresponding to each face image; the true value refers to the pixel value of the face image when it is not re-illuminated;

[0159] The structural similarity refers to the similarity between the predicted value of the uniformly illuminated face image and the corresponding true value after the face image in the test set is input into the generator;

[0160] The predicted value refers to the pixel value of the face image with uniform illumination.

[0161] In specific practice, the number of optimization iterations is set according to user needs, or according to experimental data, or according to historical experience values, for example, set to 100,000 times.

[0162] It can be understood that the technical solution provided in this embodiment, by acquiring a high-definition face image dataset, screens out face images with uniform lighting from the high-definition face image dataset, and re-illuminates the screened face images with uniform lighting to generate a re-illuminated face image set, thereby achieving self-generation of model training data, getting rid of dependence on expensive data acquisition equipment, and having low cost and more popular algorithms.

[0163] Furthermore, the technical solution provided in this embodiment does not require 2D to 3D conversion and can achieve deglazing of facial images only through 2D facial images, which greatly improves the deglazing effect of facial images.

[0164] In specific practice, the face image delighting model trained by the above model training method can be used for face image delighting. A face image delighting method according to an exemplary embodiment includes:

[0165] Perform face alignment on the face image collected in real time to identify the face area, and align the identified face area to the specified position;

[0166] The face image after face alignment is input into the face image delighting model for delighting processing to obtain a face image with uniform illumination; the face image delighting model is obtained by using the above-mentioned model training method.

[0167] It can be understood that the technical solution provided by this embodiment requires obtaining the facial image delighting model through the training method of the facial image delighting model described in Example 1, and the training method of the facial image delighting model described in Example 1 obtains a high-definition facial image data set, screens out facial images with uniform illumination from the high-definition facial image data set, and re-illuminates the screened facial images with uniform illumination to generate a re-illuminated facial image set, thereby realizing self-generation of model training data, getting rid of dependence on expensive data acquisition equipment, and having low cost and stronger algorithm popularization.

[0168] Furthermore, the technical solution provided in this embodiment does not require 2D to 3D conversion and can achieve deglazing of facial images only through 2D facial images, which greatly improves the deglazing effect of facial images.

[0169] Figure 6is a schematic block diagram of a model training system 100 according to an exemplary embodiment. Figure 6 As shown, the system 100 includes:

[0170] An acquisition module 101 is used to acquire a high-definition face image dataset;

[0171] A screening module 102 is used to screen out facial images with uniform illumination from the high-definition facial image data set, and store the screened facial images with uniform illumination in a real facial image set;

[0172] A re-illumination module 103 is used to re-illuminate the screened facial images with uniform illumination to obtain facial images with shadows and highlights, and generate a set of re-illumination facial images;

[0173] The training module 104 is used to input the face images in the face re-illuminated image set and the face images in the real face image set into a pre-constructed network model for training until the model converges, thereby obtaining a face image de-lighting model. The face image de-lighting model is used to perform de-lighting on the input face images to obtain face images with uniform illumination.

[0174] It should be noted that, since the implementation methods and beneficial effects of the above modules can be referred to the detailed description of the corresponding steps in the above embodiments, they will not be repeated in this embodiment.

[0175] It can be understood that the technical solution provided in this embodiment, by acquiring a high-definition face image dataset, screens out face images with uniform lighting from the high-definition face image dataset, and re-illuminates the screened face images with uniform lighting to generate a re-illuminated face image set, thereby achieving self-generation of model training data, getting rid of dependence on expensive data acquisition equipment, and having low cost and more popular algorithms.

[0176] Furthermore, the technical solution provided in this embodiment does not require 2D to 3D conversion and can achieve deglazing of facial images only through 2D facial images, which greatly improves the deglazing effect of facial images.

[0177] Figure 7 is a schematic block diagram of a facial image de-lighting system 200 according to an exemplary embodiment. Figure 7 As shown, the system 200 includes:

[0178] The alignment module 201 is used to perform face alignment on the face image collected in real time to identify the face area and align the identified face area to a specified position;

[0179] The delighting module 202 is used to input the face image after face alignment into the face image delighting model for delighting processing to obtain a face image with uniform illumination; the face image delighting model is obtained using the above-mentioned model training method.

[0180] It can be understood that the technical solution provided in this embodiment, by acquiring a high-definition face image dataset, screens out face images with uniform lighting from the high-definition face image dataset, and re-illuminates the screened face images with uniform lighting to generate a re-illuminated face image set, thereby achieving self-generation of model training data, getting rid of dependence on expensive data acquisition equipment, and having low cost and more popular algorithms.

[0181] Furthermore, the technical solution provided in this embodiment does not require 2D to 3D conversion and can achieve deglazing of facial images only through 2D facial images, which greatly improves the deglazing effect of facial images.

[0182] According to an exemplary embodiment, an electronic device includes: a processor and a memory.

[0183] Wherein, the memory stores a computer program that can be called by the processor;

[0184] The processor executes the above-mentioned facial image delighting model training method and / or facial image delighting method by calling the computer program stored in the memory.

[0185] It should be noted that electronic devices include but are not limited to: smart terminals (for example, mobile phones, tablet computers, smart watches, etc.) and computer devices.

[0186] The processor includes but is not limited to: CPU, single chip microcomputer, PLC controller, FPGA controller, etc.

[0187] The memory may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory; it may also include other removable / non-removable, volatile / non-volatile computer system storage media. The memory may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0188] It can be understood that the technical solution provided in this embodiment, by acquiring a high-definition face image dataset, screens out face images with uniform lighting from the high-definition face image dataset, and re-illuminates the screened face images with uniform lighting to generate a re-illuminated face image set, thereby achieving self-generation of model training data, getting rid of dependence on expensive data acquisition equipment, and having low cost and more popular algorithms.

[0189] Furthermore, the technical solution provided in this embodiment does not require 2D to 3D conversion and can achieve deglazing of facial images only through 2D facial images, which greatly improves the deglazing effect of facial images.

[0190] According to an exemplary embodiment, a computer-readable storage medium is shown, on which a rewritable computer program is stored;

[0191] When the computer program runs on a computer device, the computer device executes the above-mentioned training method of the face image delighting model and / or the face image delighting method.

[0192] The computer-readable storage medium disclosed in this embodiment includes, but is not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0193] It can be understood that the technical solution provided in this embodiment, by acquiring a high-definition face image dataset, screens out face images with uniform lighting from the high-definition face image dataset, and re-illuminates the screened face images with uniform lighting to generate a re-illuminated face image set, thereby achieving self-generation of model training data, getting rid of dependence on expensive data acquisition equipment, and having low cost and more popular algorithms.

[0194] Furthermore, the technical solution provided in this embodiment does not require 2D to 3D conversion and can achieve deglazing of facial images only through 2D facial images, which greatly improves the deglazing effect of facial images.

[0195] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0196] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0197] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0198] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0199] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0200] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0201] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0202] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0203] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A model training method, characterized in that: include: Get a high-definition face image dataset; Screening out facial images with uniform illumination from the high-definition facial image dataset, and storing the screened facial images with uniform illumination in a real facial image set; Re-illuminating the screened face images with uniform illumination to obtain face images with shadows and highlights, and generating a set of face re-illumination images; The re-illumination of the screened uniformly illuminated face image comprises: randomly selecting spherical harmonic function coefficients, adding the illumination direction and illumination intensity corresponding to the spherical harmonic function coefficients to the screened uniformly illuminated face image, and obtaining the face image with shadows and highlights by adjusting the scaling scale of the spherical harmonic function coefficients; adding the silhouette data as the outline of the external shadow or adding Perlin noise to imitate the shadow caused by an external object with no fixed shape to the face image with shadows and highlights; The face images in the face re-illuminated image set and the face images in the real face image set are input into a pre-built network model for training until the model converges, thereby obtaining a face image de-lighting model. The face image de-lighting model is used to perform de-lighting on the input face images to obtain face images with uniform illumination.

2. The method according to claim 1, characterized in that: The step of screening out facial images with uniform illumination from the high-definition facial image dataset comprises: Performing face segmentation on each high-definition face image in the high-definition face image data set to obtain a face region in the face image; For each segmented face region, it is determined whether a face image corresponding to the face region is a face image with uniform illumination according to the ratio of highlight pixels to shadow pixels in the face region.

3. The method according to claim 2, characterized in that The step of judging whether the face image corresponding to the face area is a face image with uniform illumination according to the ratio of highlight pixels to shadow pixels in the face area includes: In the face area, pixels with pixel values ​​higher than a first preset pixel value are marked as highlight pixels, and pixels with pixel values ​​lower than a second preset pixel value are marked as shadow pixels; the second preset pixel value is lower than the first preset pixel value; Calculate the ratio of the sum of the number of shadow pixels and the number of highlight pixels to the total number of pixels in the face area; If the proportion is lower than a threshold, the facial image corresponding to the facial region is marked as a facial image with uniform illumination.

4. The method according to claim 1, characterized in that The step of inputting the face images in the face re-illumination image set and the face images in the real face image set into a pre-built network model for training comprises: Defining a loss function of the network model; Updating the generator and discriminator of the network model until the model converges; The generator is used to receive the re-illuminated face image and perform light removal on the re-illuminated face image to generate a face image with uniform illumination; The discriminator is used to receive a face image generated by a generator, or a real face image with uniform illumination, and output a confidence level that the face image belongs to a real face image with uniform illumination.

5. The method according to claim 4, characterized in that The loss function of the network model at least includes: A standard deviation-based regularization term for optimizing the generator; The regularization term is used to constrain the lighting effect of the face image generated by the generator.

6. The method according to claim 4, characterized in that The updating of the generator and the discriminator of the network model comprises: Updated generators, including: The network parameters of the discriminator are fixed, and the face images in the face re-illuminated image set are sent to the generator in batches to output face images with uniform illumination; According to the confidence of the uniformly illuminated face image output by the discriminator, the loss function value is calculated, and the model parameters of each layer of the generator are derived to obtain the gradient of each layer of the generator; According to the gradient, the network parameters of the generator are updated using the gradient descent algorithm and the back propagation algorithm; Update the discriminator, including: Inputting the face image generated by the generator and the face image in the real face image set into the discriminator in turn, and obtaining the first confidence and the second confidence that the input face image belongs to the real face image with uniform illumination respectively; Calculate the loss function value according to the first confidence and the second confidence, and use the chain rule to calculate the gradient of each layer of the discriminator; According to the gradient, the network parameters of the discriminator are updated using the gradient descent algorithm and the back propagation algorithm.

7. The method according to claim 6, characterized in that Also includes: Setting the number of optimization iterations of the generator and the discriminator, when the number of optimization iterations reaches a preset number, the loss function value no longer decreases, and the structural similarity measured on the test set no longer increases, the model is judged to be converged, and the generator at this time is determined as a face image de-lighting model; The test set stores multiple face images and the true value corresponding to each face image; the true value refers to the pixel value of the face image when it is not re-illuminated; The structural similarity refers to the similarity between the predicted value of the uniformly illuminated face image and the corresponding true value after the face image in the test set is input into the generator; The predicted value refers to the pixel value of the face image with uniform illumination.

8. A method for removing light from a face image, characterized in that: include: Perform face alignment on the face image collected in real time to identify the face area, and align the identified face area to the specified position; The face image after face alignment is input into the face image delighting model for delighting to obtain a face image with uniform illumination; the face image delighting model is obtained by using the model training method described in any one of claims 1 to 7.

9. A model training system, characterized in that: include: An acquisition module is used to acquire a high-definition face image dataset; A screening module, used to screen out facial images with uniform illumination from the high-definition facial image data set, and store the screened facial images with uniform illumination in a real facial image set; A re-illumination module, used for re-illuminating the screened face images with uniform illumination to generate a set of re-illumination face images; The re-illumination of the screened uniformly illuminated face image comprises: randomly selecting spherical harmonic function coefficients, adding the illumination direction and illumination intensity corresponding to the spherical harmonic function coefficients to the screened uniformly illuminated face image, and obtaining the face image with shadows and highlights by adjusting the scaling scale of the spherical harmonic function coefficients; adding the silhouette data as the outline of the external shadow or adding Perlin noise to imitate the shadow caused by an external object with no fixed shape to the face image with shadows and highlights; The training module is used to input the face images in the face re-illuminated image set and the face images in the real face image set into a pre-built network model for training until the model converges, thereby obtaining a face image de-lighting model. The face image de-lighting model is used to perform de-lighting processing on the input face images to obtain face images with uniform illumination.

10. The system according to claim 9, characterized in that The network model includes: A generator, used for receiving a re-illuminated face image, and performing a de-lighting process on the re-illuminated face image to generate a face image with uniform illumination; The discriminator is used to receive the face image generated by the generator, or the real face image with uniform illumination, and output the confidence that the face image is a real face image with uniform illumination.

11. The system according to claim 10, characterized in that The generator comprises: The encoder is used to extract features from the face image after re-illumination to obtain the face features without the influence of illumination; The decoder is used to restore the facial features without the influence of lighting to a facial image with uniform lighting.

12. The system according to claim 10, characterized in that The discriminator comprises: The convolution layer is used to extract features from the input face image with uniform illumination. The residual module is used to perform secondary feature extraction and downsampling on the features extracted by the convolutional layer; The fully connected layer is used to calculate the confidence that the input uniformly illuminated face image belongs to a true uniformly illuminated face image based on the features output by the residual module, and output the confidence.

13. A facial image delighting system, characterized in that: include: An alignment module is used to perform face alignment on the face image collected in real time to identify the face area and align the identified face area to a specified position; A delighting module is used to input the face image after face alignment into the face image delighting model for delighting to obtain a face image with uniform illumination; the face image delighting model is obtained by using the model training method described in any one of claims 1 to 7.

14. An electronic device, characterized in that: include: processor and memory, wherein The memory stores a computer program that can be called by the processor; The processor executes the model training method described in any one of claims 1 to 7 and / or the facial image delighting method described in claim 8 by calling the computer program stored in the memory.

15. A computer-readable storage medium, characterized in that: A rewritable computer program is stored thereon; When the computer program runs on a computer device, the computer device executes the model training method described in any one of 1 to 7, and / or the facial image delighting method described in claim 8.

Images