Intelligent makeup scheme recommendation generation method based on face recognition

Abstract

The invention relates to the field of makeup scheme recommendation, and discloses an intelligent makeup scheme recommendation generation method based on face recognition, and the method comprises the steps: obtaining a front eye opening image, a front eye closing image and a side face tone-up image through a camera, extracting skin color information, skeleton contour information and a facial feature vector through an image recognition model, and generating a base makeup and make-up recommendation scheme according to the skin color information, the skeleton contour information and the facial feature vector. And generating a complete scheme through the fusion function and displaying the effect. And the makeup image of the user can be acquired for comparative analysis to generate optimization suggestions. According to the method, through multi-view image acquisition and preprocessing, facial feature multi-dimensional extraction, personalized makeup scheme generation, scheme fusion and visualization and optimization iteration, the problem of insufficient facial region division and feature extraction precision in the prior art is solved, and accurate personalized makeup scheme recommendation is realized; and the recommendation efficiency, the fitting degree and the user experience are improved.

Description

Technical Field

[0001] This invention relates to the field of makeup scheme recommendation, and more particularly to an intelligent makeup scheme recommendation generation method based on facial recognition. Background Technology

[0002] Currently, with the integration of artificial intelligence and the beauty industry, image analysis-based intelligent makeup recommendation technology has gradually become a research hotspot. Existing makeup recommendation methods mainly rely on human experience or fixed templates; for example, beauty consultants recommend cosmetics based on subjective observation of users' skin tone, face shape, and other features. These methods have significant technical shortcomings:

[0003] The core defects of traditional technology

[0004] Traditional methods rely solely on visual observation or simple two-dimensional image analysis to determine skin tone and face shape, failing to accurately extract three-dimensional features such as facial bone contours and skin tone distribution. For example, in skin tone classification, manual judgment struggles to accurately distinguish the subtle differences between "warm-toned yellow-white" and "cool-toned pink-white," making precise differentiation difficult and leading to biased foundation shade recommendations. In contour analysis, the inability to quantify parameters such as jaw angle and cheekbone prominence results in poor compatibility between makeup solutions (such as contouring and blush placement) and the user's facial structure.

[0005] Although the Chinese invention patent with announcement number CN115577183A proposes an image-based skin color matching method, it only involves simple comparison of RGB values ​​and does not combine skeletal feature analysis, making it difficult to achieve personalized recommendations for each individual.

[0006] Based on this, we propose a method for generating intelligent makeup scheme recommendations based on face recognition. Summary of the Invention

[0007] To address the technical problem of insufficient accuracy in facial region segmentation and feature extraction in existing technologies, this invention provides a method for generating intelligent makeup scheme recommendations based on face recognition.

[0008] This invention is achieved using the following technical solution: 1. Multi-view image acquisition and preprocessing

[0009] A collection of unedited facial images of the user obtained through the camera {I open ,I close ,I side};in:

[0010] I open A frontal image with eyes open, used to capture eye details;

[0011] I close The image is a frontal view with eyes closed, which is helpful for analyzing eyelid texture;

[0012] I 3DThis method uses 3D structured light scans to directly generate facial depth maps, resolving errors in skeletal contours. Multi-view images are used to comprehensively capture facial features (such as eye details, eyelid texture, and skeletal contours), providing a data foundation for subsequent analysis.

[0013] 2. Multidimensional extraction of facial features

[0014] Extracting three main categories of features using image recognition models:

[0015] Skin color information C: obtained through region segmentation and Bayesian inference;

[0016] Skeletal contour information S b Extracted via Graph Convolutional Network (GCN);

[0017] Facial feature vector F f It includes global features such as texture and symmetry.

[0018] 2.1 Skin color information extraction and analysis

[0019] Parametric division of facial regions

[0020] Forehead region boundary equation: This equation is used to define the boundary of the forehead region. Where p hairline (t) represents the hairline parameter curve (the value of (t) ranges from 0 to 1), d h β represents the horizontal distance from the eyebrow tail to the facial contour, used to shift the boundary line downwards from the hairline; β is the gradient weight of the eyebrow boundary. This is the gradient vector for the eyebrow boundary points, used to fine-tune the boundary to fit the eyebrow shape. It accurately delineates the forehead region, providing boundary criteria for skin tone extraction.

[0021] Closed curve of cheek area: R cheek =∫(LF) eye (s)⊕L laugh (s)⊕p eye-contour (s) where L eye (s) represents the eyeliner, L laugh (s) represents the law line, (p) eye-contour (s) is the curve of the intersection of the eye position and the contour, and the boundary of the cheek area is determined by closed integral.

[0022] In this formula, "⊕" is a curve blending operator that functions to blend the eye lines using the LF method. eye (s), nasolabial folds L laugh (s) and the curve p at the intersection of eye position and contour eye-contour (s) These curves are combined according to specific rules to form a closed curve R in the cheek area. cheekIt is similar to piecing together different curve segments to form a complete boundary line.

[0023] "(s)" is a representation of a parameter, where "s" is a parameter variable. LF eye (s), L laugh (s), p eye-contour (s) represent the curves of the eye line, nasolabial fold, and the intersection of the eye position and the facial contour, all of which are functions of the parameter "s". As the parameter "s" varies within a certain range, these curves will exhibit corresponding shape changes. By setting and calculating the parameter "s", the specific shape and position of the curves can be determined.

[0024] Determine the area of ​​your cheeks, ensuring that key areas such as the apples of your cheeks are covered.

[0025] Skin color category Bayesian inference assumes that the skin color database follows a mixture Gaussian distribution:

[0026] For the RGB value C of the region region Skin color category is matched using maximum posterior probability: Among them, {π k μ k , Σ k Let} represent the distribution parameters, and N(·) be the Gaussian probability density function. This model treats the skin color database as a mixture of K Gaussian distributions, and determines the best-matching category by calculating the probability that a sample belongs to each distribution, thus achieving accurate skin color classification.

[0027] Because human skin color exhibits clustered distribution characteristics in the RGB space;

[0028] π k Let N(C; μ) be the prior probability of skin color class k. k ,Σ k ) describes the probability density of this skin color type, where μ k The mean (representing a typical skin tone value) Σ k The covariance (reflecting the range of skin color distribution) is calculated using Bayesian posterior probability, and the observed skin color C is... region Classifying it to the most likely distribution falls under generative model inference.

[0029] Quantify skin tone characteristics to achieve accurate classification (e.g., warm-toned yellow 1, cool-toned pink 2, etc.).

[0030] 2.2 Deep Learning Extraction of Skeletal Contour Features

[0031] Key skeletal point localization: Extracting feature point sets S from the brow bone, bridge of the nose, cheekbone, and jaw. b ={p brow ,p nose,p cheek ,p jaw}

[0032] Graph Convolutional Network (GCN) Feature Processing: X (L+1) =σ(AX) (L) W (L) )in, X is a normalized adjacency matrix describing the topological relationships of skeletal points; (L) Let W be the feature matrix of the l-th layer. (L) Let be the weight matrix, and σ be the ReLU activation function. This network extracts global shape features of the contour through multi-layer feature transformation, effectively capturing the facial skeletal structure.

[0033] Treat facial bone points as nodes in a graph structure, and the connections between points (such as brow bone-bridge of nose) as edges, construct an adjacency matrix A; The Laplacian matrix normalization operation ensures smooth feature propagation between nodes; X (L) W (L) For the linear transformation of features, σ(·) introduces nonlinear activation, enabling the network to learn the topological features of skeletal points (such as facial contours and skeletal convexity); the multilayer GCN gradually extracts abstract features from local point coordinates to global contour morphology through the "neighbor feature aggregation" mechanism.

[0034] Deep features are extracted from skeletal point coordinates to capture the overall shape of the facial contour.

[0035] Triple-state loss function contour matching: L=max(d(S,M) p )-d(S,M n )+margin,0) where d(X,X) is the Euclidean distance, M p For positive sample models (similar contours), M n The model uses negative samples (dissimilar contours), and the margin is the interval threshold. By optimizing this loss, the model learns to distinguish between similar and dissimilar contours, ultimately matching the optimal model M. best This improves the accuracy of contour matching.

[0036] By training and optimizing this loss, the model learns to distinguish between similar and dissimilar contours, thereby improving matching accuracy.

[0037] 2.3 Mixed facial contour feature data

[0038] Generative Adversarial Networks (GANs) are introduced to synthesize hybrid facial contour data, combined with skeletal contour information S from the file. b The extraction logic (including feature point sets such as brow bone and bridge of nose) has the following core equation:

[0039] Using random noise vector Z and real skeletal contour features (Taken from the skeletal point set of typical facial features in the database {p brow ,p nose ,p cheek ,p jaw} as input, generate blended facial contour features

[0040]

[0041] Where Z~N(0,1) is a random noise vector, introducing diversity;

[0042] For real skeletal contour features (such as the coordinates of feature points for oval or diamond-shaped faces);

[0043] The generator G learns the correlation between different facial features through a multi-layer neural network (including convolutional layers and the ReLU activation function) and outputs a blended contour feature (such as the transitional shape of "oval face brow bone + diamond face jaw").

[0044] With contour feature S b (reality or generated Given the input, output the probability that it is a true contour:

[0045] P real =D(S) b The discriminator D extracts contour topological features through graph convolutional layers (GCN processing logic in the reference file), and outputs the probability P after passing through a fully connected layer and a sigmoid activation function. real ∈(0,1);

[0046] P real ≈1 indicates that it is determined to be a true contour, P fake ≈0 indicates a synthesized contour. The discriminator D maximizes its ability to distinguish between real and fake contours. The generator learns the correlation of skeletal features of different face shapes (such as oval and diamond faces) to synthesize mixed contours with intermediate transitional forms (such as "oval face + high cheekbones" or "diamond face + rounded jaw"). After manual verification, the model was added to the makeup category database, expanding the database to cover the face shapes of 98% of the population and improving the generalization of the GCN contour matching model.

[0047] The GAN model described above can automatically generate diverse mixed facial contour data, solving the problem of insufficient rare facial sample data in the original database, and enabling intelligent makeup recommendations to be adapted to a wider range of users.

[0048] 3. Personalized makeup plan generation

[0049] Foundation makeup recommendation model: based on skin tone category Cclass Color code matching is optimized using a matrix factorization model:

[0050] in;

[0051] B: The relationship matrix between shade and skin tone, where rows represent skin tone categories, columns represent foundation shades, and element values ​​represent matching degree.

[0052] U and V: Low-rank matrices, where U represents the latent features of skin tone (such as "warm" or "cool") and V represents the latent features of color number. The implicit relationship between the two is extracted through matrix decomposition.

[0053] E: Sparse error matrix, used to correct data noise (such as abnormal color code matching records).

[0054] Norm squared measures the error in matrix reconstruction.

[0055] The ||E||0 norm constrains the number of non-zero elements in the error matrix to avoid overfitting.

[0056] λ: Regularization parameter, balancing reconstruction error and sparsity.

[0057] It is the original matrix B and the approximate matrix UV T Measurement of differences between them:

[0058] Where ||·||F is the Frobenius norm, which is calculated by taking the square root of the sum of the squares of all elements in the matrix;

[0059] After squaring, that is Directly representing the sum of squares of all elements, the smaller the value, the better the UV. T The better the approximation effect for B.

[0060] This model extracts the potential correlation between shade number and skin tone through dimensionality reduction and recommends the best shade number B. rec And the effect is superimposed through a linear mixture model: Among them, B e The image represents the base makeup effect, where α∈[0,1] is the transparency parameter, achieving a realistic rendering of the base makeup effect.

[0061] The "shade-skin tone" correlation matrix B is decomposed into the product of low-rank matrices U and V, where U represents the latent features of skin tone and V represents the latent features of shade. To reconstruct the error term and ensure that the decomposed matrix approximates the original matrix; ‖E‖0 is a sparse regularization term that constrains the error matrix E to have the fewest non-zero elements, thus avoiding overfitting; this model achieves personalized recommendations by mining the implicit correlation between color codes and skin tones (such as warm skin tones being suitable for a certain type of color code), and belongs to a matrix factorization variant of collaborative filtering.

[0062] Makeup Recommendation Model: Retrieve M best The corresponding makeup categories are layered using a linear blending model to create the following effects: Among them, C e This is the image for the makeup effect. β is the transparency parameter, which supports dynamic adjustment based on the intensity of the makeup to ensure a natural fit between the makeup and the facial contours.

[0063] It recommends the foundation shade that best matches the user's skin tone from a vast array of shades, improving recommendation efficiency and accuracy.

[0064] 4. Solution Integration and Visualization

[0065] Merge the base makeup and color makeup schemes using a weighted fusion function: S = F base (C)⊕F color (S b ,F f ), where ⊕ is the weighted fusion operator, and the weights are based on the contour matching degree sim(S,M). best The weight is dynamically adjusted (e.g., when the matching degree is ≥0.8, the weight W = 0.6), and the final effect is rendered in real time on the 3D interface to achieve an intuitive display of the solution.

[0066] Foundation Makeup Solution F base (C) Makeup solutions based on skin tone characteristics F color (S b ,F f It relies on contours and facial features, and merges them through the weighted operator ⊕; the dynamic weight adjustment mechanism is based on the logic that "the higher the contour matching degree, the greater the weight of the makeup scheme", ensuring that the makeup and facial structure are compatible, which is a rule-based decision fusion.

[0067] 5. Solution Optimization and Iteration Mechanism

[0068] Obtain images of users after applying makeup, and optimize the solution through multi-scale similarity evaluation:

[0069] A facial recognition-based intelligent makeup solution generation system includes:

[0070] Image acquisition module: dual-lens camera and ring light to acquire multi-view images of bare faces;

[0071] Feature Analysis Module:

[0072] Skin color extraction unit: performs region segmentation and Bayesian inference;

[0073] Contour extraction unit: runs the GCN network and matches it with the triplet loss;

[0074] Solution generation module: Generates base makeup / color makeup solutions based on matrix factorization and linear mixture model;

[0075] Optimization module: Multi-scale similarity evaluation and energy function optimization;

[0076] Visualization module: 3D rendering and interactive effect preview.

[0077] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0078] 1. This invention acquires multi-view unedited images of faces through a camera, and combines region segmentation algorithms and image recognition models to accurately extract multi-dimensional features such as skin color and bone contours. This solves the problem that traditional methods have difficulty in quantifying facial parameters, and realizes personalized basic data collection and analysis for each individual, laying a high-precision data foundation for subsequent solution generation.

[0079] 2. By leveraging technologies such as Bayesian inference, graph convolutional networks, and matrix factorization, this invention achieves accurate skin color classification, skeletal contour depth matching, and intelligent generation of base / makeup schemes. Compared with traditional methods such as SVM and CNN, the classification and matching accuracy is significantly improved, and the scheme generation time is greatly reduced, thus improving recommendation efficiency and real-time performance.

[0080] 3. Through a weighted fusion function and a multi-scale similarity evaluation mechanism, this invention achieves the natural integration and effect visualization of base makeup and color makeup solutions. It can also be iteratively optimized based on the user's actual makeup effect, while supporting multi-scene and skin type adaptation, thus enhancing the practicality of the solution and user experience. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0082] Figure 2 This describes the working principle of the facial feature extraction module of the present invention.

[0083] Figure 3 This is a diagram of the base makeup recommendation model architecture of the present invention;

[0084] Figure 4 This invention relates to a makeup contour matching process.

[0085] Explanation of key symbols:

[0086] I: Collection of unedited images; C: Facial skin tone information; S b : Skeletal contour information; F f Facial feature vector; C class Skin tone category; M best : The contour model with the highest matching degree; F fusion : Scheme fusion function. Detailed Implementation

[0087] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0088] Example 1:

[0089] Please combine Figures 1-4 This embodiment proposes a main process for generating intelligent makeup solutions based on facial recognition.

[0090] Application scenario: Beauty consultants use smart terminals equipped with this system (such as tablets or makeup mirrors) to provide users with personalized makeup solutions.

[0091] Step 1: Image Acquisition and Preprocessing

[0092] Staff operation:

[0093] The system guides users to sit in a fixed position. The system has a built-in posture detection module that uses key skeletal points (such as the shoulders, neck, and top of the head) to determine the user's sitting posture. When the head tilt angle is greater than 5° or the left and right tilt angle is greater than 10°, the interface will prompt "Please adjust your head position" in real time and display the standard facial midline in the preview screen using AR lines.

[0094] Adjust the seat height so that your face is level with the center of the camera (resolution ≥ 5 million pixels, supports multi-angle shooting) to ensure uniform lighting, and use a professional ring light for supplemental lighting;

[0095] Click the "Start Data Acquisition" button on the system interface and take photos sequentially:

[0096] Frontal open-eye photo: The user looks straight ahead at the camera with both eyes naturally open; the frontal open-eye photo captures details of the eyes, such as the shape of the double eyelids and the color of the pupils;

[0097] Frontal closed-eye photo: The user closes their eyes and keeps their facial muscles relaxed; frontal closed-eye photo analysis of eyelid texture (applicable to eyeliner / eyeshadow design);

[0098] 3D structured light scanning: This is a 3D structured light scan (such as the iPhone Face ID module) that directly generates a facial depth map. It accurately extracts the skeletal contour (such as the three-dimensional coordinates of the mandibular angle and the height of the bridge of the nose) through three-dimensional data, solving the error problem in skeletal contour extraction in traditional two-dimensional side profile photos.

[0099] The system automatically stores the image set as I = {I} open ,I close ,I 3D} and perform brightness equalization preprocessing.

[0100] Step 2: Facial Feature Extraction

[0101] Skin color information extraction:

[0102] Staff operation:

[0103] Click the "Feature Analysis" button on the system interface, and the software will automatically perform region division:

[0104] Divide the face into regions, including the forehead, cheeks, and neck.

[0105] The parameterized boundary equation for the forehead region is as follows:

[0106]

[0107] Where, p hairline (t) represents the hairline parameter curve, with values ​​ranging from 0 to 1; β represents the gradient weight of the eyebrow boundary. The gradient vector of the eyebrow boundary point;

[0108] p hairline (t) is the hairline parameter curve, which varies between 0 and 1 and can be understood as the position parameter along the hairline. h It is the horizontal distance from the tail of the eyebrow to the contour of the face. This means fine-tuning the boundary line according to the sloping direction (gradient) of the eyebrow boundary to ensure that the boundary fits the shape of the eyebrow;

[0109] Skin color information for each region is extracted and compared with skin color cards in the skin color database to determine the skin color category. The Bayesian inference formula for skin color category is as follows:

[0110]

[0111] RGB values ​​C extracted from each region region By inferring the matching skin color category using Bayesian methods:

[0112]

[0113] Among them, {π k μ k , Σ k} represents the parameters of the mixture Gaussian distribution of the skin color database;

[0114] p(C|θ): The probability that skin color C belongs to a certain category under the model parameter θ;

[0115] π k : The prior probability of the kth skin color class (e.g., the proportion of "cool-toned pinkish-white" in the population);

[0116] N(C;μ k ,∑ k): Gaussian probability density function, describing the distribution characteristics of the k-th skin color class;

[0117] in:

[0118] μ k : The mean of the k-th skin color (representing the typical RGB value of this skin color).

[0119] Σ k : The covariance matrix of the k-th skin color class (reflecting the distribution range of RGB values ​​for that skin color class).

[0120] K: The total number of skin tone categories (e.g., warm yellow-white, cool pink-white, etc.).

[0121] d h β is predicted using a CNN (inputting eyebrow tail coordinates and eyebrow gradient); the CNN prediction equation is defined as follows: the input vector X (containing p) is constructed using features such as eyebrow tail coordinates and eyebrow gradient vector. brow coordinates (direction and magnitude), after processing by the CNN model, the predicted parameter values ​​are directly output:

[0122]

[0123] in:

[0124] X is the input feature vector, focusing on the key features of the eyebrow tail and eyebrow boundary;

[0125] The horizontal distance from the eyebrow tail to the facial contour predicted by CNN;

[0126] The gradient weights for the eyebrow boundary predicted by the CNN;

[0127] The model internally uses convolutional layers, activation functions (such as ReLU), and fully connected layers to map input features to parameter outputs, making region segmentation more efficient and reducing processing time to less than 50ms.

[0128] The system is based on the hairline parameter curve p hairline (t)(t∈[0,1]) and the gradient vector of the eyebrow boundary The system automatically marks the hairline apex and eyebrow tail boundary points, allowing staff to manually fine-tune the boundary line parameter d. h And β to adapt to special hairstyles;

[0129] Cheek area: Software recognizes eye lines L eye (s), nasolabial folds L laugh (s) and the intersection of the eye contour p eye-contour (s), generate a closed curve, and staff confirm whether key areas such as the cheekbones are covered;

[0130] Neck area: The system automatically cuts out a 5cm area below the jawline, and staff can adjust the cutting frame.

[0131] The system extracts the RGB values ​​of each region and compares them with the built-in skin tone database (which includes 200+ standard skin tone cards (including subsets of Asian, European, and African skin tones), and supports importing third-party brand color cards (such as the Pantone skin tone database)). The parameter {π} k μ k , Σ k Perform Bayesian inference;

[0132] calculate Generate skin tone categories (e.g., warm-toned yellow 1, cool-toned pink 2, etc.).

[0133] C class The final skin tone category (e.g., "warm-toned yellow 1 light" or "cool-toned pink 2 light");

[0134] L(C region μ k ,Σ k Likelihood function, i.e., the RGB values ​​of the region C. region The probability of belonging to the kth skin color class.

[0135] The formula calculates the posterior probability, and then... region It categorizes skin to the most likely skin color category, achieving accurate classification.

[0136] Skeletal contour and feature vector extraction:

[0137] Automatic software processing:

[0138] The feature point set S of the brow bone, bridge of the nose, cheekbone, and jaw is located using a deep learning model (such as MTCNN). b ={p brow ,p nose ,p cheek ,p jaw};

[0139] Graph Convolutional Networks (GCNs) are used to process contour point sets, and the adjacency matrix is ​​normalized. Calculate the feature transformation: X (L+1) =σ(AX) (L) W (L) Extract the frontal facial contours (including the hairline);

[0140] This formula is used to extract deep features of the skeletal contour, where X (L) It is the feature matrix of the Lth layer (each row represents the feature of a skeletal point); It is a matrix constructed based on the connection relationships between skeletal points (such as the connection between the brow bone point and the bridge of the nose point), used to convey the feature information of adjacent points; W (L) σ represents the network weights, used to learn feature transformations, while σ is an activation function (such as ReLU) used to introduce non-linearity. Through multi-layer computation, the overall shape features of the skeletal contour can be extracted.

[0141] The extracted contours are compared with 100+ standard contour models in the makeup category database using a triplet loss function: L=max(d(S,M) p )-d(S,M n +margin, matching the contour model M with the highest similarity. best ;

[0142] in;

[0143] L: Loss value, used to optimize the contour matching model.

[0144] d(S,Mp): Feature distance (Euclidean distance) between user contour S and positive sample model Mp (similar contours, such as face shapes with ±10% difference).

[0145] d(S,Mn): Feature distance between user contour S and negative sample model Mn (dissimilar contours, such as heterogeneous face shapes).

[0146] margin: The interval threshold (default 0.3) is used to ensure the distinction between similar and dissimilar contours.

[0147] The training set contains 10,000+ contour models (covering 12 face shapes including oval and diamond shapes), with M positive samples. p For the same type of face shape, ±10% contour difference, negative sample M n For irregular face shapes, the margin is set to 0.3 (Euclidean distance threshold); the Adam optimizer is used with a learning rate of 0.001, and the validation set accuracy reaches 96.3% after 200 training rounds.

[0148] Facial feature extraction experimental data

[0149]

[0150] Step 3: Generate base makeup and color makeup solutions

[0151] Base makeup recommendations:

[0152] System operation:

[0153] According to skin color category C class Matching a database of foundation makeup products (containing 50+ brands and 300+ shades), and using a low-rank matrix factorization model B=UV... T+E optimizes shade recommendations, outputting 3-5 candidate shades (such as YSL Feather Foundation B20, Estée Lauder DW1C1, etc.);

[0154] When staff click on a candidate color number on the interface, the system applies the color number effect through a linear blending model. (β = 0.7 is the default transparency) Overlay onto I open On the screen, you can preview the base makeup effect in real time.

[0155] Makeup Recommendations:

[0156] Automatically generated by the system:

[0157] Retrieve and M best Corresponding makeup categories, such as "contouring solutions for oval faces" and "blush application techniques for diamond-shaped faces";

[0158] Products (such as eyebrow pencil shades and eyeshadow palette colors) are recommended for different areas such as eyebrows, eyeshadow, blush, and lipstick. Makeup effects are then layered using a linear blending model, and staff can adjust the β value (0.3-0.9) to control the intensity of the makeup.

[0159] For example, when matching the "heart-shaped face" contour model M best At that time, the system will automatically recommend:

[0160] Base makeup: Estée Lauder Double Wear Foundation 1C1 (matches warm-toned fair skin through matrix decomposition), with a β=0.6 when layered, preserving skin texture;

[0161] Makeup: Apply contour powder 2mm below the cheekbone (according to S...) b (Calculation of the coordinates of the mid-cheekbone feature point), the blush position is the midpoint of the line connecting the outer corner of the eye and the corner of the mouth, and a coral blush C is superimposed through a linear blending model. e The preset blush template is β = 0.4.

[0162] Comparison of data on the generated effects of base makeup and color makeup solutions:

[0163]

[0164] Step 4: Solution Integration and Visualization

[0165] Staff operation:

[0166] Clicking the "Fusion Scheme" button will activate the fusion function S=F. base (C)⊕F color (S b ,F f (⊕ is a weighted fusion operator; when the contour matching degree is ≥0.8, the weight w = 0.6) merges the base makeup and color makeup schemes;

[0167] In the 3D preview interface, staff can drag the slider to switch between "natural face - light makeup - heavy makeup" effects. The system renders the blended makeup effect in real time and supports zooming in to view details such as eyes and lips.

[0168] Step 5: Solution Optimization and Iteration

[0169] User feedback after applying makeup:

[0170] Staff members photographed the user's makeup application after completing the plan and imported the images into the system.

[0171] The system converts the image after makeup and the expected effect into grayscale images and calculates the similarity between the makeup and the face fit (SSIM) (with a threshold set to 0.7).

[0172] If the similarity is insufficient, the algorithm analyzes the differences (such as the position of the blush being off or the thickness of the eyeliner being uneven) using a local facial recognition algorithm, generates optimization suggestions (such as "the blush should be blended 2mm upwards towards the apple cheek"), and automatically updates the scheme.

[0173] Example 2: Scheme Optimization Based on Multimodal Data

[0174] Improvements: Based on Example 1, add user skin type data (such as oily or dry) and makeup scene (daily, commuting, evening) input.

[0175] Suitable for oily skin:

[0176] When skin type is determined to be oily, the recommended foundation application includes an oil-control factor screening option. A matte finish foundation (such as MakeUp For Ever HD foundation) is preferred. A "T-zone setting step" should be added to the plan, suggesting using loose powder to press and set the forehead and bridge of the nose. The setting area should be identified via S... b The T-zone outline is automatically annotated.

[0177] Suitable for dry skin:

[0178] In the context of an evening event, the system recommends a moisturizing foundation (such as Guerlain L'Or Radiance Foundation) based on the "dry skin + heavy makeup" tag, and inserts a "moisturizing primer" step before the base makeup routine. The amount of primer used is calculated based on the area of ​​the face (0.05ml is recommended per square centimeter for the cheek area).

[0179] Commuting Makeup Optimization: For commuting scenarios, makeup recommendations lean towards natural colors (such as earth-toned eyeshadow and nude lipstick), with a default makeup effect transparency β of 0.5. A "5-minute quick makeup" step guide is also generated, prioritizing "foundation makeup → eyebrows → lipstick" and ignoring complex contouring steps.

[0180] Evening Makeup Optimization:

[0181] Using the example of "cool-toned pinkish-white + oval face" characteristics, a high-coverage foundation (such as NARS Luminous Silk Foundation) is recommended for the base makeup. When matching the shade using a matrix factorization model, prioritize shades with fine shimmer. For the color makeup, use the "evening highlighting scheme," layering highlighter C on the brow bone, bridge of the nose, and apple cheeks. e For the gold highlight template, β = 0.4, the highlight area is determined according to S. b The brightness is automatically adjusted based on the degree of bone protrusion.

[0182] Multi-scenario adaptability experiment

[0183]

[0184] Staff operation:

[0185] Before the solution is generated, guide users to fill out a skin questionnaire or obtain water and oil data through a skin analyzer;

[0186] Select a makeup look / scene, and the system will determine the appropriate look based on the "skin type-scene-shade" correlation matrix (using the matrix factorization model B=UV). T (+E training) Optimize foundation recommendations, such as recommending moisturizing foundation for dry skin in evening events.

[0187] System hardware and software configuration

[0188] Hardware modules:

[0189] Image acquisition module: binocular camera (infrared + RGB), ring light, touch screen;

[0190] Processing unit: NVIDIA Jetson AGX Orin (200 TOPS computing power), supporting real-time GCN computing.

[0191] Software architecture:

[0192] Operating system: Ubuntu 20.04; Deep learning framework: PyTorch 1.12;

[0193] Database: MySQL stores skin tone cards, makeup models, and user history schemes.

[0194] Implementation principle:

[0195] This solution captures multi-view, bare-faced images using a camera, extracts facial features using a region segmentation algorithm and a GCN network, generates accurate recommendations based on Bayesian inference and matrix factorization models, and finally visualizes the results through a fusion function. Staff guide users through data collection and solution adjustments via a human-computer interaction interface. The system iterates through multi-scale similarity evaluation to improve the practicality and personalization of makeup suggestions.

[0196] The above implementation methods can be adjusted in terms of step order or modules can be added or removed according to actual needs. For example, automatic fill light and posture prompts can be integrated into the beauty mirror device to further simplify the operation process for staff.

[0197] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method for generating intelligent makeup scheme recommendations based on facial recognition, characterized in that, include: Step S1: Acquire a bare face image of the user through a camera, including a frontal photo with eyes open, a frontal photo with eyes closed, and a 3D structured light scan photo; The set of unedited facial images I = {I open ,I close ,I 3D }; in: I open This is a frontal image with eyes open, used to clearly show the eye features; I close The image is a frontal view with eyes closed, which facilitates analysis of the condition of the eyelids and the skin around the eyes. I 3D It generates facial depth maps directly from 3D structured light scanning images; Step S2: Analyze the bare-faced image using an image recognition model to extract facial skin tone information C and skeletal contour information S. b and facial feature vector F f ; Step S3: Generate a foundation makeup recommendation scheme based on the extracted skin tone information C, and based on the bone contour information S... b and facial features F f Generate makeup recommendation schemes; Step S4: Step S4: Through the fusion function F fusion (·) Integrate the base makeup recommendation scheme with the color makeup recommendation scheme to generate a complete intelligent makeup scheme and display the effect on the user's facial image.

2. The intelligent makeup scheme recommendation generation method based on face recognition as described in claim 1, characterized in that, The step of extracting facial skin color information in step S2 includes: Step S201: Divide the facial area into the forehead area, cheek area and neck area; The parameterized boundary equation for the forehead region is as follows: Where, p hairline (t) represents the hairline parameter curve, with the value of t ranging from 0 to 1; β represents the gradient weight of the eyebrow boundary. The gradient vector of the eyebrow boundary point; p hairline (t) represents the hairline parameter curve, where t varies between 0 and 1, and can be understood as the positional parameter along the hairline. h It is the horizontal distance from the tail of the eyebrow to the facial contour, used to shift the boundary line downwards from the hairline. This means fine-tuning the boundary line according to the sloping direction (gradient) of the eyebrow boundary to ensure that the boundary fits the shape of the eyebrow; Step S202: Extract skin color information for each region and compare it with skin color cards in the skin color database to determine the skin color category. The Bayesian inference formula for skin color category is as follows: RGB values ​​C extracted from each region region By inferring the matching skin color category using Bayesian methods: Among them, {π k μ k , Σ k } represents the parameters of the mixture Gaussian distribution of the skin color database; p(C|θ): The probability that skin color C belongs to a certain category under the model parameter θ; π k : The prior probability of skin color class k; N(C;μ k ,∑ k ): Gaussian probability density function, describing the distribution characteristics of the k-th skin color class; in: μ k : The mean of the k-th skin color; Σ k : The covariance matrix of the k-th skin color class; K: Total number of skin color categories; Step S203: The forehead area is divided by marking the hairline and eyebrow boundary points, and setting the area boundary line by combining the horizontal distance between the eyebrow tail and the facial contour; Step S204: The cheek area is divided by extracting the eye lines and nasolabial folds, and combining the intersection of the eye position markers and the facial contour to determine the closed area. The closed curve is defined by the following equation: Among them, L eye (s) represents the eyeliner, L laugh (s) represents the law line, p eye-contour (s) is the curve at the intersection of the eye position and the contour.

3. The intelligent makeup scheme recommendation generation method based on face recognition as described in claim 1, characterized in that, The steps for generating a base makeup recommendation scheme include: Based on skin tone category, match the foundation makeup information database to generate corresponding foundation shade recommendations; attach the image effect of the foundation shade recommendations to the user's face in a front-facing photo with eyes open; The color code matching is optimized using a matrix factorization model, which is as follows: in; B: The relationship matrix between shade and skin tone, where rows represent skin tone categories, columns represent foundation shades, and element values ​​are the matching degree; U and V: Low-rank matrices, where U represents the latent features of skin tone (such as "warm" or "cool"), and V represents the latent features of color number. The implicit relationship between the two is extracted through matrix decomposition. E: Sparse error matrix, used to correct data noise (such as abnormal color code matching records); Norm squared measures the error in matrix reconstruction. The ||E||0 norm constrains the number of non-zero elements in the error matrix to avoid overfitting. λ: Regularization parameter, balancing reconstruction error and sparsity; It is the original matrix B and the approximate matrix UV T Measurement of differences between them: Where ||·||F is the Frobenius norm, which is calculated by taking the square root of the sum of the squares of all elements in the matrix; After squaring, that is Directly representing the sum of squares of all elements, the smaller the value, the better the UV. T The better the approximation effect for B.

4. The intelligent makeup scheme recommendation generation method based on face recognition as described in claim 1, characterized in that, The steps in step S2 for extracting skeletal contour information and facial features include: Step S21: Extraction of key skeletal feature points: Extract the skeletal contour information of the user's brow bone, bridge of the nose, cheekbone, and jaw; extract the feature point set S of the brow bone, bridge of the nose, cheekbone, and jaw. b ={p brow ,p nose ,p cheek ,p jaw }; Step S22: Contour feature processing of graph convolutional network: Extract facial contour information from the user's front view, including the overall facial contour and hairline; process the contour point set using a graph convolutional network (GCN): X (L+1) =σ(AX) (L) W (L) ),in This is the normalized adjacency matrix; Step S23: Triplet loss function contour matching: The extracted contour information is compared with models in the makeup category database to obtain the contour model with the highest matching degree; contour similarity is calculated using the triplet loss function. L=max(d(S,M p )-d(S,M n ()+margin,0), and match the best contour model M best ; This formula is used to calculate the similarity between the current user profile S and the model in the database, where d(X,X) is the feature distance; M p It is a "positive sample" model, with a contour similar to S; M n It is a "negative sample" model, a contour that is not similar to S.

5. The intelligent makeup scheme recommendation generation method based on face recognition as described in claim 4, characterized in that, The steps for generating a makeup recommendation scheme include: retrieving the contour model M with the highest matching degree. best Based on the corresponding makeup category, generate makeup recommendations; then, apply the makeup effects using a linear blending model. Among them, C e This is a makeup effect image, where β is the transparency parameter; the effect is overlaid through linear blending.

6. The intelligent makeup scheme recommendation generation method based on face recognition as described in claim 1, characterized in that, Also includes: Obtain the image of the user after applying makeup according to the intelligent makeup solution; Compare and analyze the post-makeup images with the expected results of the intelligent makeup solution: Convert to grayscale image and calculate similarity; When the similarity does not reach the preset threshold, the matching differences of each region are analyzed by a local facial recognition algorithm. Based on the makeup details that need improvement, optimization suggestions are generated, and the intelligent makeup solution is updated.

7. The intelligent makeup scheme recommendation generation method based on face recognition as described in claim 1, characterized in that, In step S4, the fusion function F fusion (·)satisfy: in, This is a weighted scheme fusion operator, where the weights are dynamically adjusted based on the contour matching degree.

8. A smart makeup scheme generation system based on facial recognition, characterized in that, include: The image acquisition module is used to acquire a set of unedited images I. The feature analysis module includes: A skin color extraction unit is used to perform the region segmentation and Bayesian inference as described in claim 2; A contour extraction unit is used to perform the GCN feature extraction and contour matching as described in claim 4; The solution generation module is used to perform the base makeup / color makeup recommendation and effect overlay as described in claims 3 and 5; The optimization module is used to perform the multi-scale similarity evaluation and optimization suggestion generation as described in claim 6; the visualization module is used to fuse the schemes and render the display effect.

Images