Face makeup recommendation method based on neural network expansion face decomposition

Abstract

This invention discloses a face makeup recommendation method based on neural network-based face decomposition, comprising the following steps: establishing a makeup template library; selecting a face makeup image from the makeup template library, performing color space conversion on the input image and the face makeup image, and decomposing the face into layers to obtain their respective detail layers, structural layers, and color layers; using a neural network to extract features from the detail layers, structural layers, and color layers to obtain detail features, structural features, and color features respectively; calculating the similarity of detail features, structural features, and color features between the input image and the face makeup image; traversing the makeup template library to select the next face makeup image, and calculating the similarity of detail features, structural features, and color features with the input image; and selecting the face makeup image in the makeup template library with the highest similarity to the input image in terms of detail features, structural features, and color features as the recommendation result.

Description

Technical Field

[0001] This invention relates to the field of image processing and recommendation technology, and more particularly to the field of facial makeup recommendation technology, specifically to a facial makeup recommendation method based on neural network-based facial decomposition. Background Technology

[0002] Makeup can transform one's appearance, making women more beautiful and charming. Makeup has become an important way for many women to express themselves and pursue beauty. It has also become an indispensable part of many women's lives, and their demand for makeup is rising accordingly. However, the market is now flooded with a wide variety of cosmetics, and with the development of the internet, all sorts of exquisite and beautiful makeup looks are emerging. How to choose the right cosmetics and makeup looks for oneself has become a problem that women inevitably face. This is where an intelligent makeup recommendation system comes in, assisting with makeup application and recommending suitable looks.

[0003] Makeup recommendation systems can recommend different makeup looks based on the user's appearance and personal preferences, combined with facial details, structure, and skin tone information. This not only highlights the user's personality but also saves the user time spent on matching and selecting makeup looks, eliminating the hassle of finding the right look for themselves and providing convenience in the pursuit of beauty. Among existing makeup recommendation methods, Liu et al. (Liu L, Xing J, Liu S, et al. Wow! You are so beautiful today! [J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2014, 11(1s): 1-22.) used a multi-tree model to explore the complex relationships between high-level beauty attributes, mid-level beauty-related attributes, and low-level image attributes of facial images, selecting the makeup look that best matches the user as the recommendation result. Alaskar et al. (Alashkar T, Jiang S, Fu Y. Rule-based facial makeup recommendation system[C]. 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition, 2017:325-330.) constructed a rule-based makeup recommendation system by establishing a knowledge base and simulating the relationship between facial style attributes and makeup style attributes. This system can also recommend different makeup looks based on different application scenarios and user preferences. Liu et al. (Liu S, Ou X, Qian R, et al. Makeup like a superstar: Deep localized makeup transfer network[J]. arXiv, 2016:1604.07102.) used a neural network to extract facial depth features and determined the recommended makeup look based on the minimum Euclidean distance between the depth features of the input face and the depth features of the makeup template. Alaskar et al. (Alashkar T, Jiang S, Wang S, et al. Examples-rules guided deep neural network for makeup recommendation[C]. Proceedings of the AAAI Conference on Artificial Intelligence.2017,31(1).) used deep networks to combine facial features, makeup examples, and makeup rules to establish a case-rules guided makeup recommendation model.However, existing methods do not comprehensively consider the details, structure, and color information of the face, resulting in unsuitable makeup recommendations. Summary of the Invention

[0004] The purpose of this invention is to propose a facial makeup recommendation method based on neural network-based face decomposition. This method utilizes a neural network unrolling method (hereinafter referred to as Unrolling) to decompose the face layer and recommend makeup that matches facial features, thereby obtaining better recommendation results that are more in line with human preferences.

[0005] The present invention is achieved by at least one of the following technical solutions.

[0006] A face makeup recommendation method based on neural network-based face decomposition includes the following steps:

[0007] Collect various facial makeup images and build a makeup template library;

[0008] Select a facial makeup image from the makeup template library, and perform color space conversion between the input image and the facial makeup image;

[0009] Based on the input image after color space conversion and the face makeup image, a layer decomposition based on an iterative process is performed to obtain the respective detail layer, structure layer and color layer;

[0010] Based on the detail layer, structure layer, and color layer of the input image and the facial makeup image, a neural network is used to extract features from the detail layer, structure layer, and color layer, respectively, to obtain detail features, structure features, and color features.

[0011] Calculate the similarity of detail features, structural features, and color features between the input image and the facial makeup image based on detail features, structural features, and color features.

[0012] The next facial makeup image is selected by traversing the makeup template library, and the similarity between the image and the detailed features, structural features and color features of the input image is calculated.

[0013] The recommended result is the facial makeup image from the makeup template library that has the highest similarity to the input image in terms of detail features, structural features, and color features.

[0014] Furthermore, the facial makeup images in the makeup template library are divided into girlish makeup, ancient style makeup, European and American style makeup, smoky makeup, and heavy makeup.

[0015] Furthermore, the color space is converted from RGB space to CIELAB space to obtain L color space containing detailed information in CIELAB space. * Channel, containing color information * Channel and b* aisle.

[0016] Furthermore, the layer decomposition is a layer decomposition based on neural network to expand face decomposition, specifically by performing image decomposition on the CIELAB color space to decompose a detail layer d, a structure layer s, and a color layer c.

[0017] Among them, L in the CIELAB color space * After the channels are decomposed by a neural network, the structural layer s of the face image is separated, and the detail layer d is separated by L... * Subtracting the structural layers of the face image from the channels yields:

[0018] s = UnrollingFace(L * )

[0019] d = L * -s

[0020] Where s is the structural layer of the face image, d is the detail layer of the face image, and L * L represents the CIELAB space * The channel, UnrollingFace(), represents the neural network unrolling decomposition module;

[0021] Color layer c is where a is... * and b * The channels combined yield:

[0022] c = a * +b *

[0023] Where c represents the color layer of the image, a * a represents the CIELAB space * Channel, b * b represents the CIELAB space * aisle.

[0024] Furthermore, the neural network adopts a CNN neural network, which is used to extract features from the detail layer to obtain the depth features corresponding to the detail layer d of the input image, thereby obtaining the detail features of the input image.

[0025] Furthermore, the neural network adopts a CNN neural network, which is used to extract features from the structural layer to obtain the depth features corresponding to the structural layer s of the input image, thereby obtaining the structural features of the input image.

[0026] Furthermore, the neural network adopts a CNN neural network, which is used to extract features from the color layer to obtain the depth features corresponding to the color layer c of the input image, thereby obtaining the color features of the input image.

[0027] Furthermore, the neural network adopts a CNN neural network, which extracts features from the detail layer to obtain the depth features corresponding to the detail layer d, structure layer s, and color layer c of the facial makeup image, thereby obtaining the detail features, structure features, and color features of the facial makeup image.

[0028] Furthermore, the process of calculating the similarity of detail features, structural features, and color features between the input image and the facial makeup image is as follows:

[0029] The similarity between the input image and the nth makeup image is calculated using the following formula:

[0030]

[0031]

[0032]

[0033] Where ||·||2 represents calculating the L2 norm, and n represents the nth makeup image in the makeup template library. and Let x represent the similarity of detail features, structural features, and color features between the input image and the i-th makeup image, respectively. d x s and x c These represent the detail features, structural features, and color features of the input image, respectively. and Let each represent the detailed features, structural features, and color features of the nth makeup image;

[0034] The final quality score is obtained through training based on the similarity of detailed features, structural features, and color features. The formula is as follows:

[0035]

[0036] Among them, S n w represents the similarity between the input image and the nth makeup image. d w s and w c These are the weights of the similarity between detail features, structural features, and color features.

[0037] Furthermore, the facial makeup image in the makeup template library that has the highest similarity to the input image in terms of detail features, structural features, and color features is used as the recommendation result, specifically:

[0038]

[0039] Among them, Y n S represents the most similar facial makeup image in the makeup template library. n This represents the similarity between the input image and the nth makeup image.

[0040] The present invention has the following advantages and effects compared with the prior art:

[0041] (1) This invention is aimed at recommending facial makeup. It uses Unrolling face decomposition to obtain the structural information, detail information and color information of the image. Then, it uses the structural information, detail information and color information to judge and compare the makeup image, which can achieve better makeup recommendation results.

[0042] (2) Compared with other face makeup recommendation methods, this invention utilizes the CIELAB image space and makes full use of the image's lighting information, detail information and structural information to recommend makeup. Lighting information, detail information and structural information are very important and sensitive for face comparison and makeup recommendation problems. Making full use of lighting information, detail information and structural information can better recommend face makeup. Attached Figure Description

[0043] Figure 1 This is a flowchart of the facial makeup recommendation method based on neural network-based facial decomposition according to the present invention;

[0044] Figure 2 This is an algorithm flowchart of the neural network unfolding face decomposition module of the present invention;

[0045] Figure 3 This is a neural network structure diagram of the H() function in the neural network unfolding face decomposition module of the present invention;

[0046] Figure 4 This is a diagram showing the decomposition result of the neural network unfolding face decomposition according to the present invention. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0048] Example 1

[0049] like Figure 1The face makeup recommendation method based on neural network-based face decomposition, as shown, has the following specific steps:

[0050] 1. Establish a makeup template library

[0051] Collect various facial makeup images, establish a makeup template library, and classify the facial makeup images into five categories: girlish makeup, ancient style makeup, European and American style makeup, smoky makeup, and heavy makeup.

[0052] 2. Color space conversion

[0053] For the input image X respectively in And the nth reference makeup image Y in the makeup template library n Convert from RGB color space to CIELAB color space.

[0054] 3. Unrolling Face Decomposition

[0055] The input image and reference makeup image, in their L channels of the CIELAB color space, are fed into the neural network unrolling (hereinafter referred to as Unrolling) decomposition module. Unrolling decomposition employs an iterative solution method, with the following constraints:

[0056]

[0057] Where I is the input image of the Unrolling decomposition, u is the output image, w(Z) represents the constraint term associated with the output image u, and λ is the constraint term coefficient.

[0058] The complex constraints are integrated into a single constraint term related to the output image u. Solving this constraint yields the Lagrange augmented form:

[0059]

[0060] Where μ and α represent the constraint coefficients of the Lagrange augmentation.

[0061] Solving using the ADMM method, we can obtain...

[0062]

[0063] Differentiating the above equation and setting the derivative to 0, we get...

[0064]

[0065] The solution is obtained through iterative solving. In each iteration, the output image u, the intermediate variable Z, and the constraint coefficients μ and α are updated sequentially. The expression can then be derived:

[0066]

[0067] Z k μ k and α k Let Z represent the intermediate variable Z and the constraint coefficients μ and α in the k-th iteration, respectively. k+1 Z k+1 μ k+1 and α k+1 Let u represent the output image u, the intermediate variable Z, and the constraint coefficients μ and α in the (k+1)th iteration, respectively. H() is a shrink function. The decomposition result of face decomposition in this embodiment is shown in the figure below. Figure 4 As shown in Figures a, b, and c, these represent the input image, the output facial structure information, and the obtained facial detail information, respectively.

[0068] The above iterative update process can be expanded using the Unrolling method, as shown in the appendix. Figure 2 As shown, by combining neural network training technology, the iterative optimization process is transformed into an end-to-end training process, which encompasses the prior knowledge of traditional optimization iterative algorithms and the advantages of neural network technology.

[0069] To achieve better processing results, the function H() uses a neural network to compute the output. The neural network structure employs a fully convolutional network, as shown in the attached diagram. Figure 3 As shown. The number of iterations for Unrolling is 5.

[0070] Input the L channel of the input image and the makeup reference image into the Unrolling decomposition module to obtain the decomposed structure map.

[0071] u s =UnrollingFace(I L )

[0072] Wherein, UnrollingFace() represents the Unrolling decomposition module, I L This represents the input L-channel image, u s This represents the structural information obtained after decomposition;

[0073] Input L-channel image I L Subtract the structural information u obtained after decomposition s To obtain detailed information, the expression is as follows:

[0074] u d =I L -u struc

[0075] Among them, I L This represents the input L-channel image, u dThis indicates the detailed information obtained after decomposition.

[0076] Color layer c is the color layer a in the CIELAB color space. * and b * The channels combined yield:

[0077] c = a * +b *

[0078] Where c represents the color layer of the image, a * a represents the CIELAB space * Channel, b * b represents the CIELAB space * aisle.

[0079] 4. Feature Extraction

[0080] Feature extraction was performed using a VGGFace CNN neural network (Parkhi OM, Vedaldi A, Zisserman A. Deep face recognition[J].2015.). The output of the VGGFace neural network was taken as the features of the input image, and the features were applied to the input image X. in And makeup image Y n Feature extraction is performed on each of the detail layer d, structure layer s, and color layer c to obtain their respective detail features, structure features, and color features.

[0081] The similarity between the input image and the nth makeup image is calculated using the following formula:

[0082]

[0083]

[0084]

[0085] Where ||·||2 represents calculating the L2 norm, and n represents the nth makeup image in the makeup template library. and Let x represent the similarity of detail features, structural features, and color features between the input image and the nth makeup image, respectively. d x s and x c These represent the detail features, structural features, and color features of the input image, respectively. and Let each represent the detailed features, structural features, and color features of the i-th makeup image.

[0086] 5. Evaluation Score

[0087] The final quality score is obtained through training based on the similarity of detailed features, structural features, and color features, using the following formula:

[0088]

[0089] Among them, S n w represents the similarity between the input image and the nth makeup image. d w s and w c These are the weights of the similarity between detail features, structural features, and color features.

[0090] The final output is expressed as follows:

[0091]

[0092] Among them, Y n This represents the most similar facial makeup image in the makeup template library.

[0093] Example 2

[0094] This embodiment provides a facial makeup recommendation system based on neural network-based facial decomposition, including:

[0095] The color space conversion module converts images from RGB space to CIELAB space, making it easier to extract features.

[0096] The face decomposition module is used to decompose face images. It uses a neural network unfolding face decomposition method to obtain the corresponding structural information, detail information and color information.

[0097] The feature extraction module is used to extract features from the structural, detail, and color information obtained by the face decomposition module. It uses the VGGNet neural network and takes the output of the hidden layer of the neural network as the corresponding structural, detail, and color features.

[0098] The evaluation module is used to evaluate facial makeup. It calculates the feature similarity between the corresponding features of the input face image and the makeup image by using the structural features, detail features and color features obtained by the feature extraction module, and obtains an evaluation score. The best recommended makeup is obtained based on the evaluation score.

[0099] This embodiment of the facial makeup recommendation system based on neural network-based facial decomposition can execute the facial makeup recommendation method based on neural network-based facial decomposition provided in embodiment 1 of the present invention, and has the corresponding functions and beneficial effects of the method.

[0100] Example 3

[0101] This embodiment also provides a facial makeup recommendation system based on neural network-based facial decomposition, including:

[0102] At least one processor;

[0103] At least one memory for storing at least one program;

[0104] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0105] The facial makeup recommendation system based on neural network-based face decomposition in this embodiment can execute the facial makeup recommendation method based on neural network-based face decomposition provided in embodiment 1 of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0106] In summary, this embodiment, for facial makeup recommendation, utilizes a neural network to decompose the face, obtaining structural, detail, and color information of the image. Then, it uses these structural, detail, and color information to judge and compare makeup images, achieving better makeup recommendation results. Compared to other facial makeup recommendation methods, this embodiment utilizes the CIELAB image space, fully leveraging the image's lighting, detail, and structural information for makeup recommendation. Lighting, detail, and structural information are crucial and sensitive for face comparison and makeup recommendation; fully utilizing these information enables better facial makeup recommendation.

[0107] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A facial makeup recommendation method based on neural network-based facial decomposition, characterized in that, Includes the following steps: Collect various facial makeup images and build a makeup template library; Select a facial makeup image from the makeup template library, and perform color space conversion between the input image and the facial makeup image; Based on the input image after color space conversion and the face makeup image, a layer decomposition based on an iterative process is performed to obtain the respective detail layer, structure layer and color layer; Based on the detail layer, structure layer, and color layer of the input image and the facial makeup image, a neural network is used to extract features from the detail layer, structure layer, and color layer, respectively, to obtain detail features, structure features, and color features, as follows: The input image and the reference makeup image, in their L channels of the CIELAB color space, are fed into the Unrolling module of the neural network. The Unrolling decomposition uses an iterative solution method, with the following constraints. in, The input image is the one used for unrolling decomposition. It outputs an image. Representation and output image Related constraints, These are constraint coefficients; The complex constraints are integrated into a single constraint term related to the output image u. This constraint is solved, and the Lagrange augmented form of the constraint is written: in, and Denotes the constraint coefficients of the Lagrange augmented form; Solving using the ADMM method, we can obtain... Differentiating the above equation and setting the derivative to 0, we get... The solution is obtained through iterative steps, and the output image is updated sequentially in each iteration. intermediate variables and constraint coefficients and The expression obtained is: in , and They represent the first time. Intermediate variables in the next iteration and constraint coefficients and ; , , and They represent the first time. Output image in +1 iteration intermediate variables and constraint coefficients and , It is a shrink function; Based on detail features, structural features, and color features, the similarity of detail features, structural features, and color features between the input image and the facial makeup image is calculated as follows: Calculate the input image and the first The similarity of the detailed features, structural features, and color features of the Zhang makeup image is calculated using the following formula: in, This indicates that we are looking for the L2 norm. This indicates the first makeup template in the makeup template library. Zhang's makeup image, , and Represent the input image and the first... The similarity of the detailed features, structural features, and color features of the Zhang makeup images. , and These represent the detail features, structural features, and color features of the input image, respectively. , and Other means the first The detailed, structural, and color features of Zhang's makeup image; The final quality score is obtained through training based on the similarity of detailed features, structural features, and color features. The formula is as follows: in, Indicates the input image and the first Similarity between the images of Zhang's makeup , and These are the weights of the similarity between detail features, structural features, and color features, respectively. The next facial makeup image is selected by traversing the makeup template library, and the similarity between the image and the detailed features, structural features and color features of the input image is calculated. The recommended result is the facial makeup image from the makeup template library that has the highest similarity to the input image in terms of detail features, structural features, and color features. Specifically: in, This represents the most similar facial makeup image in the makeup template library. Indicates the input image and the first Similarity between the images of the makeup looks.

2. The facial makeup recommendation method based on neural network-based facial decomposition according to claim 1, characterized in that, The makeup template library contains facial makeup images categorized into girlish makeup, traditional Chinese makeup, Western-style makeup, smoky makeup, and heavy makeup.

3. The facial makeup recommendation method based on neural network-based facial decomposition according to claim 1, characterized in that, The color space is converted from RGB space to CIELAB space, resulting in a CIELAB space containing detailed information. Channels containing color information Channels and aisle.

4. The facial makeup recommendation method based on neural network-based facial decomposition according to claim 1, characterized in that, The neural network described is a CNN (Neural Network), which is used to extract features from the detail layer of the input image. The corresponding depth features are used to obtain the detailed features of the input image.

5. The facial makeup recommendation method based on neural network-based facial decomposition according to claim 1, characterized in that, The neural network described is a CNN (Neural Network), which is used to extract features from the structural layers of the input image. The corresponding depth features are used to obtain the structural features of the input image.

6. The facial makeup recommendation method based on neural network-based facial decomposition according to claim 1, characterized in that, The neural network described is a CNN (Neural Network), which is used to extract features from the color layer of the input image. The corresponding depth features are used to obtain the color features of the input image.

7. The facial makeup recommendation method based on neural network-based face decomposition according to claim 1, characterized in that, The neural network used is a CNN (Neural Network), which extracts features from the detail layer to obtain the detail layer of the facial makeup image. Structural layer Color layer The corresponding depth features are used to obtain the detailed features, structural features, and color features of the facial makeup image.

Images