A method for face recognition after makeup
By adopting a GAN-based deep learning method in facial recognition, using style transfer AdaIn and multiple loss functions to generate facial pictures after makeup, the problem of low accuracy of facial recognition after makeup is solved and higher recognition accuracy is achieved.
Patent Information
- Application Number
- CN202210765753.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-30
- Publication Date
- 2025-06-20
- Estimated Expiration
- 2042-06-30
AI Technical Summary
The prior art has low recognition accuracy in facial recognition after makeup, especially in the absence of a large amount of comparison data before and after makeup, it is difficult to improve the accuracy of the model.
Using a deep learning method based on GAN, we extract the identity features in the input image and introduce independent control vectors, and use style transfer AdaIn to generate face pictures after makeup, and adjust the loss function to converge the training and improve the recognition accuracy.
By introducing multiple related loss functions, including ordinary GAN loss, makeup loss and cyclic consistency loss, the accuracy of the face recognition model after makeup was successfully improved, and the problem of low recognition accuracy was solved.
Smart Images
Figure CN115294626B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of face recognition, and more specifically, relates to a face recognition method after makeup. Background Art
[0002] Due to the rapid development of deep learning, face recognition has also developed rapidly, and both the accuracy and speed have been greatly improved. However, in the application of face recognition, due to makeup, the facial features change significantly, especially when there is a large deviation between the actual collected photo and the ID photo, resulting in poor performance of the general face recognition model. Existing face recognition technologies are mainly divided into two categories. The first category is based on traditional manually designed facial features to obtain the face recognition result; the second category of methods is to automatically extract features based on deep learning, and train the features automatically extracted from the training data to obtain a face recognition model. These two categories of methods are generally used for general face recognition.
[0003] For existing face recognition technologies, whether it is the early traditional methods or the latest deep learning methods for extracting facial features, due to being aimed at general scenarios, when there are significant changes between the collected photo and the ID photo, such as after heavy makeup, the recognition effect is not good; in addition, since it is difficult to collect a large number of one-to-one photos before and after makeup, it is difficult to collect one-to-one data before and after makeup and add it to the training to improve the accuracy of the model. Summary of the Invention
[0004] The purpose of the embodiments of this application is to provide a face recognition method after makeup to solve the technical problem of low recognition accuracy of the face recognition model in the process of face recognition after makeup in the existing technology.
[0005] To achieve the above purpose, the technical solution adopted in this application is: to provide a face recognition method after makeup, including:
[0006] Collect a number of pictures after makeup and without makeup as input pictures input;
[0007] Extract the identity feature fid from the input picture input;
[0008] Input a random vector, and output multiple independent control vectors w by extracting style information j , where each independent control vector w j Controls a facial feature respectively;
[0009] Input the identity feature fid and the weighted independent control vector w j Into the style transfer AdaIn, and output a new identity feature y i ;
[0010] Input the new identity feature y i Output the final post - makeup face image output through multiple de - convolution operations;
[0011] Compare the input image input with the post - makeup face image output, and reconstruct a loss function L total , where the loss function L tota has the following formula:
[0012] L total =λ GAN L GAN +λ makeup L makeup +λ cycle L cycle ,
[0013] L GAN is the ordinary GAN loss, L makeup is the makeup loss, L cycle is the loss generated by unpaired data in unsupervised learning, and λ GAN , λ makeup and λ cycle are the weights of L GAN , L makeup and L cycle respectively;
[0014] Feed the loss function L total back to the style transfer AdaIn and adjust the weights of each independent control vector w j so that the loss function L total converges.
[0015] Preferably, the method for outputting multiple independent control vectors w j by extracting style information includes the following steps:
[0016] Input an arbitrary 1 * N - dimensional random vector into a multi - layer convolution;
[0017] Input the output of the multi - layer convolution into a fully - connected layer;
[0018] Map and decouple it into three independent control vectors w j , where each independent control vector w j controls a face feature respectively.
[0019] Preferably, the three independent control vectors w j control the generation styles of the corresponding eyes, mouth, and skin respectively.
[0020] Preferably, the formula for calculating the style transfer AdaIn is:
[0021]
[0022] where x i is the identity feature fid of the i-th input, μ(x i ) is the mean of x i , σ(x i ) is the standard deviation of x i , w j is an independent control vector, w s,j and w b,j are the weight values of the independent control vector w j ;
[0023] The formula for outputting the new identity feature y i is:
[0024]
[0025] where ε j is the random vector noise.
[0026] Preferably, the method for generating the new identity feature y i through multiple deconvolution operations includes:
[0027] Input the new identity feature y i into a neural network composed of a deconvolution layer and an activation layer, and finally the deconvolution layer outputs the final made-up face picture output.
[0028] Preferably, the loss L cycle generated for unsupervised learning of unpaired data is calculated by the formula:
[0029]
[0030] where G and F are different generators, x is the input of G and can generate a fake y graph, y is the input of F and can generate a fake x graph, E represents expectation, x ∼ Pdata(x) means the variable x follows the distribution Pdata(x), and y ∼ Pdata(y) means the variable y follows the distribution Pdata(y).
[0031] Preferably, the formula for the ordinary GAN loss L GAN is:
[0032] L GAN (G, D Y , X, Y) = E y～pdata(y) [logD Y (y)] + E x～pdata(x) [log(1 - D Y (G(x)))]
[0033] where D Y is the discriminator.
[0034] Preferably, the makeup loss L makeup has the following formula:
[0035] L makeup (x, y) = L eye (x, y) + L mouth (x, y) + L skin (x, y),
[0036] wherein, L eye represents the loss generated by the independent control vector generation data for controlling the eyes, L mouth represents the loss generated by the independent control vector generation data for controlling the mouth, L skin represents the loss generated by the independent control vector generation data for controlling the skin.
[0037] Preferably, λ GAN 、λ makeup and λ cycle are the weights of L GAN 、L makeup and L cycle respectively, where:
[0038] λ GAN = 0.5, λ makeup = 0.2, λ cycle = 0.3.
[0039] Compared with the prior art, the post - makeup face recognition method provided in this application uses GAN and different parts of the face to generate post - makeup data when performing face recognition on post - makeup faces. By making corresponding adjustments to the original loss function and introducing multiple related loss functions, the training can finally converge, improving the accuracy of the post - makeup face recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0041] Figure 1 is a schematic flowchart of the post - makeup face recognition method provided in the embodiments of this application;
[0042] Figure 2 is a flowchart of extracting the identity feature fid from the input picture input provided in the embodiments of this application;
[0043] Figure 3Flowchart for outputting multiple independent control vectors by extracting style information provided by embodiments of the present application;
[0044] Figure 4 Schematic diagram of the state comparison of the face model before and after makeup provided by embodiments of the present application;
[0045] Figure 5 Flowchart for using the identity feature fid and the independent control vector as the input of AdaIn provided by embodiments of the present application;
[0046] Figure 6 Schematic diagram of the process of performing multiple deconvolution operations on the new identity feature provided by embodiments of the present application. Detailed implementation manners
[0047] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0048] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.
[0049] It should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present application.
[0050] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, "a plurality" means two or more, unless otherwise specifically defined.
[0051] The present invention adopts a feature extraction method based on deep learning. At the same time, in order to improve the consistency of features before and after makeup, the loss function is redefined and a loss related to makeup is introduced to address the new challenges brought by new applications. Therefore, the purpose of the present invention is to solve the problem of face recognition after makeup by data generation and introducing makeup-related losses without adding a large amount of comparison data before and after makeup to the training.
[0052] Please refer to Figure 1 together, and now the face recognition method after makeup provided by the embodiments of the present application will be described. The face recognition method after makeup includes:
[0053] Step S01, collect a number of pictures after makeup and without makeup as input pictures input;
[0054] Step S02, extract the identity feature fid from the input picture input;
[0055] Step S03, input a random vector, and output multiple independent control vectors w by extracting style information j , where each independent control vector w j controls a face feature respectively;
[0056] Step S04, input the identity feature fid and the weighted independent control vector w j into the style transfer AdaIn, and output a new identity feature y i ;
[0057] Step S05, output the final face picture after makeup output through multiple deconvolution operations for the new identity feature y i ;
[0058] Step S06, compare the input picture input with the face picture after makeup output, and reconstruct a loss function L total , where the formula of the loss function L tota is:
[0059] L total = λ GAN L GAN + λ makeup L makeup + λ cycle L cycle ,
[0060] L GAN is the ordinary GAN loss, L makeup is the makeup loss, L cycle is the loss generated by unpaired data in unsupervised learning, λ GAN , λ makeup and λcycle are L GAN , L makeup and L cycle respectively;
[0061] Feed the loss function L total back to the style transfer AdaIn and adjust the weights of each independent control vector w j so that the loss function L total converges.
[0062] It can be understood that considering that it is difficult to obtain the data after makeup, in step S01, several pictures after makeup and without makeup are collected as the input picture input. The pictures after makeup and without makeup do not need to correspond one by one, and only a small amount of data is required, which reduces the difficulty of data acquisition. Among them, the input picture input only needs to be labeled with identity information (ID) and whether it is made up or not.
[0063] In step S02, please refer to Figure 2 , extract the identity feature fid from the input picture input. The extraction method can use the CNN convolutional neural network. For example, input the input picture input into the convolutional layer, residual layer, convolutional layer, residual layer and fully connected layer. It should be noted that the fully connected layer is a special convolutional layer, and its function is to extract features and finally output the identity feature fid. For example, the identity feature fid is a 1*1024-dimensional feature vector.
[0064] In step S03, input a random vector, where the random vector can be any 1*N-dimensional random vector, aiming to output multiple independent control vectors w j through a random vector. Since each independent control vector w j controls a face feature respectively, multiple face features can be extracted simultaneously.
[0065] In step S04, use the identity feature fid obtained in step S2 and the independent control vector w j obtained in step S3 as the input of the style transfer AdaIn. Each independent control vector w j needs to set an initial weight value, and use GAN and different parts of the face to generate the data after makeup to complete the style transfer and retain the initial style. Finally, the final made-up face picture output is output through step S05.
[0066] In step S06, when performing face recognition on a made-up face, when calculating the loss for the first time, due to the lack of one-to-one corresponding data between the made-up and non-made-up faces, the cycle loss is used. Considering the lack of a large amount of data for training, data after makeup is generated based on GAN and different parts of the face. By making corresponding adjustments to the original loss function, the loss function is redesigned, and the ordinary GAN loss L GAN , the makeup loss L makeup , and the loss L cycle generated by unpaired data in unsupervised learning are introduced. The loss function L total is fed back to the style transfer AdaIn and the weight of each independent control vector w j的 is adjusted, so that the training can finally converge. The new made-up face recognition model can achieve high accuracy, and finally solves the problem of low accuracy in face recognition after makeup.
[0067] It should be added that since λ GAN , λ makeup , and λ cycle are the weights of L GAN , L makeup , and L cycle respectively, so λ GAN + λ cycle + λ makeup = 1.
[0068] The made-up face recognition method provided by this application, compared with the prior art, when performing face recognition on a made-up face, uses GAN and different parts of the face to generate data after makeup. By making corresponding adjustments to the original loss function and introducing multiple related loss functions, the training can finally converge and the accuracy of the made-up face recognition model is improved.
[0069] In another embodiment of this application, please also refer to Figure 3 , in step S03, the method of outputting multiple independent control vectors w j by extracting style information includes the following steps:
[0070] Input any 1*N-dimensional random vector into the multi-layer convolution;
[0071] Input the output of the multi-layer convolution into the fully connected layer;
[0072] Map and decouple it into three independent control vectors w j , where each independent control vector w j controls a face feature respectively.
[0073] Furthermore, please also refer to Figure 3 , the three independent control vectors w jControl the generation styles corresponding to the eyes, mouth, and skin respectively.
[0074] It can be understood that the fully connected layer is a special convolutional layer, which is used to extract features and finally output three independent control vectors w j . Please refer to Figure 4 , Figure 4 In the left face model in, the state before makeup is shown, and the state after makeup is shown in the right face model. During the makeup process, people mainly decorate the eyes, mouth, and skin, that is, the changes in these three parts are the largest. Therefore, the three independent control vectors wj respectively control the generation styles corresponding to the eyes, mouth, and skin, and the detailed effects of the makeup can be accurately extracted.
[0075] In another embodiment of the present application, please refer to Figure 5 and Figure 6 , in step S04, the formula for calculating the style transfer AdaIn is:
[0076]
[0077] where x i is the identity feature fid of the i-th input, μ(x i ) is the mean of x i , σ(x i ) is the standard deviation of x i , w j is an independent control vector, w s,j and w b,j are the weight values of the independent control vector w j ;
[0078] The formula for outputting the new identity feature y i is:
[0079]
[0080] where ε j is a random vector noise.
[0081] Furthermore, in step S05, the method of passing the new identity feature y i through multiple deconvolution operations includes:
[0082] Input the new identity feature y i into a neural network composed of a deconvolution layer and an activation layer, and finally the deconvolution layer outputs the final post-makeup face image output.
[0083] It can be understood that in step S04, by combining AdaIn with a random vector noise, the effect of makeup control has more details. Set w s,jand w b,j is to independently control the weight value of vector w j In this way, every time the identity feature fid and the independent control vector w are input j , the loss function L total will feedback to the style transfer AdaIn once, and adjust the weight of each independent control vector w j , that is, adjust w s,j and w b,j .
[0084] In another embodiment of the present application, in step S06, the loss L generated for unsupervised learning of unpaired data cycle has the formula:
[0085]
[0086] where G and F are different generators, x is the input of G and can generate a fake y graph, y is the input of F and can generate a fake x graph, E represents expectation, x~Pdata(x) means that the variable x follows the distribution Pdata(x), and y~Pdata(y) means that the variable y follows the distribution Pdata(y).
[0087] It can be understood that for the loss L generated for unsupervised learning of unpaired data cycle , since there is no one-to-one corresponding data, but we hope that the images between the corresponding domains are one-to-one, that is, A-B-A can be transferred back, and a cyclic consistency loss needs to be used. Through the above formula, when we send x into the generator G, the obtained is a fake y graph, and then send this fake y graph into F, a more fake x graph is obtained. Ideally, the more fake x graph at this time should be almost the same as the original x graph. This also constitutes a cycle, so it is called cyclic consistency loss.
[0088] Furthermore, in step S06, the formula of the ordinary GAN loss L GAN is:
[0089] L GAN (G, D Y , X, Y) = E y～pdata(y) [logD Y (y)] + E x～pdata(x) [log(1 - D Y (G(x)))]
[0090] where D Y is the discriminator.
[0091] It can be understood that the generator G is used to generate the picture Y from the random variable X. Therefore, during training, G(x) is made as close to Y as possible; and the discriminator D_Y is used to distinguish between real and fake samples.
[0092] Furthermore, in step S06, the makeup loss L makeup has the following formula:
[0093] L makeup (x, y) = L eye (x, y) + L mouth (x, y) + L skin (x, y),
[0094] wherein, L eye represents the loss generated by the independent control vector for controlling the eyes to generate data, L mouth represents the loss generated by the independent control vector for controlling the mouth to generate data, L skin represents the loss generated by the independent control vector for controlling the skin to generate data.
[0095] It can be understood that by adding the makeup loss, that is, the loss of generating data by the vector of the original three parts, to the original loss function, the problem of low accuracy of face recognition after makeup is further solved.
[0096] In another embodiment of the present application, in step S06, λ GAN , λ makeup and λ cycle are respectively the weights of L GAN , L makeup and L cycle , where:
[0097] λ GAN = 0.5, λ makeup = 0.2, λ cycle = 0.3.
[0098] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present application shall be included within the protection scope of the present application.
Claims
1. A post - makeup face recognition method, characterized in that, Including: Collect a number of pictures with and without makeup as input pictures input; Extract the identity feature fid from the input picture input; Input a random vector and output multiple independent control vectors w by extracting style information j , where each independent control vector w j controls a facial feature respectively; The method of outputting multiple independent control vectors w by extracting style information j comprises the following steps: Input an arbitrary 1*N-dimensional random vector into multiple convolutional layers; Input the output of the multiple convolutional layers into the fully connected layer; Map and decouple into three independent control vectors w j , where each independent control vector w j controls a face feature respectively; the three independent control vectors w j control the generation styles corresponding to eyes, mouth and skin respectively; Input the identity feature fid and the independent control vector w with the set weight j into the style transfer AdaIn, and output the new identity feature y i ; The formula for calculating the style transfer AdaIn is: where x i is the identity feature fid of the i-th input, μ(x i ) is the mean of x i , and σ(x i ) is the standard deviation of x i . w j is an independent control vector, and w s,j and w b,j are the weight values of the independent control vector w j ; Output the new identity feature y i The formula for which is: where ε j is the random vector noise; The new identity feature y i Output the final made-up face image output through multiple deconvolution operations; The new identity feature y i The method of performing multiple deconvolution operations includes: Input the new identity feature y i into a neural network composed of a deconvolution layer and an activation layer, and finally output the final post-makeup face image output by the deconvolution layer; Compare the input image input with the post-makeup face image output, and reconstruct a loss function L total , where the loss function L tota has the following formula: L total = λ GAN L GAN + λ makeup L makeup + λ cycle L cycle , L GAN is the ordinary GAN loss, L makeup is the makeup loss, L cycle is the loss generated by unpaired data in unsupervised learning, λ GAN 、λ makeup and λ cycle are the weights of L GAN 、L makeup and L cycle respectively; the formula for the loss L cycle generated by unpaired data in unsupervised learning is: Where G and F are different generators, x is the input of G, which can generate a fake y map, y is the input of F, which can generate a fake x map, E represents the expectation, x~Pdata(x) means that the variable x follows the distribution Pdata(x), and y~Pdata(y) means that the variable y follows the distribution Pdata(y); Feed the loss function L total back to the style transfer AdaIn and adjust the weights of each independent control vector w j such that the loss function L total converges.
2. The post - makeup face recognition method according to claim 1, characterized in that, Ordinary GAN loss L GAN The formula is as follows: L GAN (G, D Y , X, Y) = E y～pdata(y) [log D Y (y)] + E x～pdata(x) [log(1 - D Y (G(x)))] Among them, D Y is the discriminator.
3. The post - makeup face recognition method according to claim 2, characterized in that, Makeup loss L makeup The formula is as follows: L makeup (x, y) = L eye (x, y) + L mouth (x, y) + L skin (x, y), Among them, L eye represents the loss generated by the independent control vector generation data for controlling the eyes, L mouth represents the loss generated by the independent control vector generation data for controlling the mouth, L skin represents the loss generated by the independent control vector generation data for controlling the skin.
4. The post - makeup face recognition method according to claim 1, characterized in that, λ GAN , makeup and cycle L GAN , L makeup and L cycle The weight of , where: λ GAN + λ cycle + λ makeup = 1 5. The post - makeup face recognition method according to claim 4, characterized in that, λ GAN = 0.5, λ makeup = 0.2, λ cycle = 0.3。
Citation Information
Patent Citations
Image processing method and device, storage medium and electronic equipment
CN113822793A