A makeup mirror personalized makeup assisting method and system based on facial features

Abstract

The application provides a makeup mirror personalized makeup assisting method and system based on facial features, and belongs to the technical field of image display. The method comprises the following steps: constructing a three-dimensional face grid through facial feature points in a user face image; generating a static face grid representing the inherent asymmetric contour of the user face based on the three-dimensional coordinates of the facial feature points in the three-dimensional face grid; continuously monitoring the displacement of the facial feature points in the three-dimensional face grid, generating a dynamic elastic deformation field associated with the current expression change, superimposing the static face grid as a deformation base with the dynamic elastic deformation field, and generating a dynamic adaptive grid for guiding the user to make up; constructing a personalized makeup layer according to the dynamic adaptive grid combined with a standard makeup map selected by the user, and guiding the user to make up based on the personalized makeup layer. The technical scheme provided by the application can realize the coupling mapping between the makeup texture and the micro-expression of the user in the makeup assisting.

Description

Technical Field

[0001] This application relates to the field of image display technology, and more specifically, to a personalized makeup assistance method and system based on facial features using a makeup mirror. Background Technology

[0002] With the rapid development of augmented reality and computer vision technologies, image display has evolved from simple pixel presentation to digital interfaces with intelligent interactive capabilities. In the fields of facial feature analysis and personalized makeup assistance, high-precision facial key point detection, skin color semantic segmentation, and facial mesh reconstruction technologies provide a sub-millimeter-level spatial positioning foundation for virtual makeup try-on. This technology not only reconstructs the reflection imaging logic of traditional makeup mirrors, but also gives rise to data-driven personalized makeup recommendation engines, making image display an aesthetic computing bridge connecting digital aesthetics and physical entities.

[0003] In existing face recognition-based beauty image display assistance, face detection is first performed using real-time images captured by a camera, extracting hundreds of sparse and dense key feature points, and reconstructing a 3D facial deformation model based on these. Then, multiple editable virtual makeup layers are generated according to preset makeup optical properties. Finally, a differentiable renderer projects the makeup layers onto a 2D image frame using perspective projection for display. During the display process, the color response of the synthesized image is adjusted, resulting in a simulated makeup effect image with real-time facial micro-expression deformation output on the screen. However, in real makeup scenarios, facial expressions are continuous and non-rigid. Dynamic evolution characteristics, such as lip stretching caused by smiling, eyelid creases caused by squinting, and cheekbone skin shearing deformation caused by jaw rotation during speech, all cause spatial offset of texture coordinates under static base mesh. This mapping mismatch between dynamic geometry and static texture makes it impossible for lipstick boundaries, eyeliner curvature, and blush gradients in the preset makeup texture to be adjusted synchronously with the facial elastic deformation field. This results in visual artifacts of makeup texture patterns relative to facial anatomy in the image display. Therefore, how to achieve coupling mapping between makeup textures and user facial micro-expressions in beauty assistance has become a challenge for the industry. Summary of the Invention

[0004] This application provides a personalized makeup assistance method and system based on facial features, which can realize the coupling mapping between makeup texture and user facial micro-expressions in makeup assistance.

[0005] In a first aspect, this application provides a personalized makeup assistance method using a makeup mirror based on facial features, comprising the following steps: Acquire user facial images captured by a makeup mirror, and construct a three-dimensional facial mesh reflecting the user's current expression based on facial feature points in the user facial images; Based on the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh, contour feature recognition is performed on the user's face, thereby generating a static facial mesh that represents the inherent asymmetrical contour of the user's face. The displacement of facial feature points in the three-dimensional facial mesh is continuously monitored, and a dynamic elastic deformation field associated with the current expression change is generated. The static facial mesh is used as a deformation base and superimposed with the dynamic elastic deformation field to generate a dynamic adaptation mesh that guides the user's makeup application. Obtain the standard makeup texture selected by the user, construct the user's current personalized makeup layer based on the current geometry of the dynamically adapting mesh and the standard makeup texture, and overlay the personalized makeup layer on the user's facial image to guide the user in makeup operations.

[0006] In some embodiments, facial feature points in the user's facial image are determined using the following steps: The user's facial image is subjected to multi-scale Gaussian difference filtering to obtain a facial scale space response map; On the facial scale spatial response map, surface fitting is used to locate sub-pixel level extreme points as candidate feature points; Based on the gradient direction distribution of the neighborhood of the candidate feature points, a main direction is assigned to each candidate feature point and a multi-dimensional feature description vector is generated. The multidimensional feature description vector is graph matched with a pre-constructed facial topology model to obtain facial feature points with anatomical semantics.

[0007] In some embodiments, constructing a three-dimensional facial mesh reflecting the user's current expression based on facial feature points in the user's facial image specifically includes: The standard 3D mesh in the preset parameterized facial template is spatially matched and mapped to the facial feature points in the user's facial image to generate an initial personalized 3D facial mesh. Based on the consistency of local neighborhood image textures of each vertex in the initial personalized facial 3D mesh, the spatial position of the vertex is optimized by photometric stereo to obtain a refined facial 3D mesh. The Laplacian coordinates of the refined 3D facial mesh are extracted, and the expression key points among all facial feature points are used as anchor points. The facial 3D mesh is driven to perform expression transfer by minimizing the Laplacian deformation energy function, thereby generating a facial 3D mesh that reflects the user's current expression.

[0008] In some embodiments, the process of recognizing contour features of the user's face based on the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh, and then generating a static facial mesh representing the inherent asymmetric contour of the user's face, specifically includes: Extract the three-dimensional coordinates of facial feature points from the facial three-dimensional mesh, calculate the spatial vector difference between symmetrical facial feature points on both sides of the facial midline, and generate a facial asymmetry deviation field. The facial asymmetry deviation field is decoupled from facial motion by expression decomposition, and the non-rigid deformation components caused by the current expression are eliminated by projection decomposition to obtain the inherent facial asymmetry deviation field that is only related to the morphology of bones and soft tissues. Using the inherent asymmetric deviation field of the face as a vertex displacement constraint, a rigidity-preserving asymmetric deformation is performed on the reference symmetric facial mesh to generate a static facial mesh that characterizes the inherent asymmetric contour of the user's face.

[0009] In some embodiments, continuously monitoring the displacement of facial feature points in the facial 3D mesh to generate a dynamic elastic deformation field associated with the current expression change specifically includes: For continuously acquired facial 3D mesh sequences, the spatiotemporal trajectories of corresponding facial feature points are identified, and the 3D displacement vector of each facial feature point from the initial static pose to the current frame is calculated. Using the three-dimensional displacement vector as the boundary condition, and the static facial mesh as the solution domain of the elasticity model, the static equilibrium equation of the non-uniform anisotropic elastic body is solved by applying the finite element method, and the continuous displacement distribution at any position within the solution domain is obtained. The continuous displacement distribution is expressed as a grid vertex-level displacement vector field, generating a dynamic elastic deformation field associated with the current facial expression change.

[0010] In some embodiments, superimposing the static facial mesh as a deformation base with the dynamic elastic deformation field to generate a dynamically adaptable mesh to guide the user's makeup application specifically includes: The dynamic elastic deformation field is decomposed into spinors to obtain the rotation tensors and translation vectors acting on the vertices of the static facial mesh. The rotation tensor is applied to each vertex of the static facial mesh in a local rigid transformation manner, and the translation vector is accumulated according to the corresponding vertex to obtain a combined deformable mesh that integrates the inherent asymmetric contour of the face and the dynamic changes of the current expression. A volume-preserving constraint based on anatomical muscle attachment points is applied to local regions in the combined deformable mesh, and vertex displacement is corrected to eliminate non-physiological mesh distortion, thereby generating a dynamically adapted mesh that guides the user in applying makeup.

[0011] In some embodiments, facial images of the user are captured by a camera integrated into the makeup mirror.

[0012] Secondly, this application provides a personalized makeup assistance system for a makeup mirror based on facial features, used to execute a personalized makeup assistance method for a makeup mirror based on facial features. The system includes: The acquisition module is used to acquire user facial images captured by the makeup mirror, and construct a three-dimensional facial mesh reflecting the user's current expression based on the facial feature points in the user facial images. The processing module is used to perform contour feature recognition on the user's face based on the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh, and then generate a static facial mesh that represents the inherent asymmetrical contour of the user's face. The processing module is also used to continuously monitor the displacement of facial feature points in the facial 3D mesh, generate a dynamic elastic deformation field associated with the current expression change, and superimpose the static facial mesh as a deformation base with the dynamic elastic deformation field to generate a dynamic adaptation mesh to guide the user's makeup application. The execution module is used to obtain the standard makeup texture selected by the user, construct the user's current personalized makeup layer based on the current geometry of the dynamic adaptation mesh and the standard makeup texture, and overlay the personalized makeup layer on the user's facial image to guide the user to perform makeup operations.

[0013] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, the processor being configured to acquire the code and execute the above-described personalized makeup assistance method based on facial features using a makeup mirror.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described personalized makeup assistance method based on facial features using a makeup mirror.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The personalized makeup assistance method and system based on facial features provided in this application first acquires a user's facial image captured by the makeup mirror, and constructs a three-dimensional facial mesh reflecting the user's current expression based on the facial feature points in the user's facial image. Second, based on the three-dimensional coordinates of the facial feature points in the three-dimensional facial mesh, contour feature recognition is performed on the user's face, thereby generating a static facial mesh representing the user's inherent asymmetrical facial contour. Then, the displacement of the facial feature points in the three-dimensional facial mesh is continuously monitored, and a dynamic elastic deformation field associated with the current expression change is generated. The static facial mesh is used as a deformation base and superimposed on the dynamic elastic deformation field to generate a dynamically adapted mesh to guide the user's makeup application. Finally, a standard makeup texture selected by the user is acquired, and a personalized makeup layer for the user is constructed based on the current geometric shape of the dynamically adapted mesh and the standard makeup texture. The personalized makeup layer is then superimposed on the user's facial image to guide the user in makeup application.

[0016] Therefore, this application can achieve coupling mapping between makeup texture and user facial micro-expressions in beauty assistance. First, by constructing a 3D facial mesh reflecting the user's current expression, makeup guidance can be elevated from a 2D plane to a 3D geometric level, allowing subsequent analysis to be based on the real facial spatial morphology, overcoming the technical deficiency of traditional 2D image methods that cannot perceive facial depth and surface changes. Second, based on the 3D coordinates of facial feature points, contour feature recognition is performed to generate a static facial mesh representing the user's inherent asymmetric facial contour. This can decouple the inherent asymmetric information of the face from the current expression, avoiding the problem in existing technologies of confusing facial changes with innate contours, leading to distortion in asymmetric analysis. Then, by continuously monitoring the displacement of feature points to generate a dynamic elastic deformation field, and using the static facial mesh as the deformation base to generate a dynamically adaptable mesh, a dynamic elastic deformation field is achieved. This invention achieves a coordinated expression of the inherent facial contours and real-time facial expression changes, enabling the makeup guidance platform to conform to the user's individualized asymmetrical facial structure and update dynamically with facial expressions. This avoids the visual artifacts caused by the inability of preset makeup texture maps to adjust synchronously with the facial elastic deformation field, resulting in the appearance of makeup texture patterns relative to facial anatomy in the image display. Finally, a personalized makeup layer is constructed and overlaid based on the current geometry of the dynamically adapting mesh, allowing standard makeup texture maps to adaptively deform and seamlessly blend according to the user's real-time three-dimensional facial shape. This achieves a coupled mapping between makeup textures and the user's facial micro-expressions, significantly improving the accuracy and convenience of users performing makeup operations according to the guidance. In summary, the technical solution provided in this application can achieve a coupled mapping between makeup textures and the user's facial micro-expressions in beauty assistance. Attached Figure Description

[0017] Figure 1 This is an exemplary flowchart of a personalized makeup assistance method based on facial features using a makeup mirror, as shown in some embodiments of this application. Figure 2 This is a schematic diagram showing the distribution of facial feature points in a user's facial image according to some embodiments of this application; Figure 3 This is an exemplary flowchart illustrating the determination of a static facial mesh according to some embodiments of this application; Figure 4 This is a structural schematic diagram of a personalized makeup assistance system based on facial features, as shown in some embodiments of this application. Figure 5 This is a schematic diagram of the structure of a computer device that implements a personalized makeup assistance method based on facial features using a makeup mirror, according to some embodiments of this application. Detailed Implementation

[0018] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] refer to Figure 1 This figure is an exemplary flowchart of a personalized makeup assistance method based on facial features using a makeup mirror, according to some embodiments of this application. The figure mainly includes the following steps: In step S101, a user's facial image captured by a makeup mirror is obtained, and a three-dimensional facial mesh reflecting the user's current expression is constructed based on the facial feature points in the user's facial image.

[0020] It should be noted that the makeup mirror in this application refers to an enhanced near-eye reflective display device that integrates a camera, an embedded computing unit, and an intelligent display interaction interface. In terms of its physical form, it has the dual functions of traditional mirror reflection and digital screen display. When the auxiliary mode is not activated, it presents the user's real face with a high-fidelity reflective mirror. After activating the beauty assistance function, the display screen embedded in the mirror can overlay and render virtual makeup layers and provide real-time guidance. Its design goal is to upgrade the passive reflection observation function of the traditional makeup mirror into an active intelligent beauty interaction terminal.

[0021] Specifically, the user's facial image is captured by a camera integrated into the makeup mirror. The user's facial image refers to digital image data containing the user's complete facial structure and local skin texture details. This user's facial image is an authorized image and is used to provide the original pixel input for face detection and feature point localization algorithms. The temporal continuity of this user's facial image ensures that the dynamic deformation field can capture the user's facial expression changes in real time, thereby ensuring the synchronous locking of virtual makeup and real face in spatial position and deformation trend. It is the core information carrier that connects the physical face and digital makeup for accurate mapping.

[0022] In some embodiments, facial feature points in the user's facial image are determined using the following steps: The user's facial image is subjected to multi-scale Gaussian difference filtering to obtain a facial scale space response map; On the facial scale spatial response map, surface fitting is used to locate sub-pixel level extreme points as candidate feature points; Based on the gradient direction distribution of the neighborhood of the candidate feature points, a main direction is assigned to each candidate feature point and a multi-dimensional feature description vector is generated. The multidimensional feature description vector is graph matched with a pre-constructed facial topology model to obtain facial feature points with anatomical semantics.

[0023] In specific implementation, firstly, multi-scale Gaussian difference filtering is performed on the acquired user facial images. Specifically, a Gaussian pyramid of the user facial images is constructed, and the difference between adjacent layers within the same group of the pyramid is calculated to generate facial scale spatial response maps that highlight the speckled structure at different scales. The facial scale spatial response map is an image that records the local extreme value response of each pixel in the scale space. Secondly, surface fitting is used on the facial scale spatial response map to locate sub-pixel level extreme points as candidate feature points. Specifically, the response values ​​of each pixel on the facial scale spatial response map are first compared with the response values ​​of its 26 three-dimensional neighborhood pixels to detect extreme points in the discrete space. Then, a three-dimensional quadratic Taylor expansion is performed on the neighborhood of the extreme point, and the vertex position of the fitted surface is calculated by differentiation, improving the accuracy of the extreme point coordinates from the pixel level to the sub-pixel level, thereby filtering out low contrast and unstable edges. The process involves defining a fixed point and selecting the remaining extreme points after filtering as candidate feature points. These candidate feature points are scale-invariant interest points with sub-pixel accuracy. Then, based on the gradient direction distribution of the neighborhood of each candidate feature point, a principal direction is assigned to each candidate feature point, and a multi-dimensional feature description vector is generated. Specifically, a local neighborhood is defined on the Gaussian image corresponding to the scale of the candidate feature point. The local neighborhood can be set according to actual needs and is not limited here. The histograms of the gradient magnitude and direction of each pixel in the neighborhood are statistically analyzed. The direction corresponding to the peak value of the histogram is taken as the principal direction of the candidate feature point. The local neighborhood is then rotated to the principal direction and divided into 4×4 sub-regions. The gradient accumulation values ​​in 8 directions are calculated for each sub-region, and the results are connected to form a 128-dimensional feature description vector. The feature description vector is a local texture representation that is invariant to changes in illumination and viewing angle.Finally, the multidimensional feature description vectors are graph-matched with the pre-constructed facial topology model to obtain facial feature points with anatomical semantics. The pre-constructed facial topology model is a graph-structured data, where nodes are key points of the eyes, eyebrows, nose, mouth, etc., with standard feature description vectors, and edges are the spatial constraints between key points. The facial topology model can be set up as follows: multiple face sample images are collected, and anatomically consistent facial key points are manually annotated on each sample image by professionals. Using key points as nodes and the spatial connections between key points as edges, a facial attribute map for each face is constructed. Then, the facial attribute maps of all samples are subjected to Pluck analysis in a normalized coordinate system to eliminate rotation, translation, and scale differences. The arithmetic mean of the aligned node coordinates is calculated to obtain the average facial feature map. The graph model is used, and then, with the nodes of the average facial graph model as the center, the distribution of feature description vectors of all sample corresponding nodes is statistically analyzed. The mean of this vector distribution is used as the standard feature description vector of the model node. Finally, a facial topology model is formed with the average spatial position as the geometric anchor point, the standard feature description vector as the matching basis, and the node connection edges encoding spatial constraints. During matching, the multi-dimensional feature description vector is used as the source node, and the Euclidean distance is calculated with the standard feature description vector of the model node in the facial topology model to establish an initial correspondence. Then, the random sampling consensus algorithm is used to iteratively eliminate mismatches, and the distance and angle constraints of the edges in the graph model are introduced for global optimization. Finally, the successfully matched candidate feature points are assigned anatomical labels to their corresponding model nodes, such as the inner canthus of the left eye, the tip of the nose, and the corner of the right lip, forming facial feature points with anatomical semantics.

[0024] refer to Figure 2 This figure is a schematic diagram of the distribution of facial feature points in a user's facial image according to some embodiments of this application. In this figure, the black solid dots represent facial feature points extracted after the makeup mirror collects the user's facial image. The dots are arranged in sections along the outer contour of the face, eyebrows, eyes, nose, and lips. Solid lines and dashed lines connect the feature points to form a triangular topological mesh structure. This mesh can generate a three-dimensional facial mesh that reflects the user's real-time expressions based on the two-dimensional coordinate mapping of each feature point. The feature points of the outer jaw, cheekbone, and forehead contour are used to identify the inherent asymmetric contours of the face to construct a static facial mesh. The dense feature points at the eyes, eyebrows, nose, and lips are used to track the point displacement caused by changes in expression in real time and generate a dynamic elastic deformation field. The overall point mesh system serves as a base to carry the fitting mapping of the standard makeup texture, ultimately realizing the personalized virtual makeup layer overlay display that adapts to the static asymmetry and dynamic expressions of the face.

[0025] It should be noted that, in this application, facial feature points refer to iconic coordinate points used to characterize the spatial location of key anatomical structures of the human face, such as the brow peak, the corner of the eye, the tip of the nose, and the inflection point of the lip line. Determining facial feature points can provide geometric references and deformation anchor points for the subsequent construction of a facial 3D mesh that reflects the user's current expression. This allows the parametric facial template to accurately adapt the standard mesh shape to the user's individual facial shape based on the correspondence of these feature points, while driving the mesh to generate personalized expression reconstruction that matches the current muscle movement.

[0026] In some embodiments, constructing a three-dimensional facial mesh reflecting the user's current expression based on facial feature points in the user's facial image is achieved through the following steps: The standard 3D mesh in the preset parameterized facial template is spatially matched and mapped to the facial feature points in the user's facial image to generate an initial personalized 3D facial mesh. Based on the consistency of local neighborhood image textures of each vertex in the initial personalized facial 3D mesh, the spatial position of the vertex is optimized by photometric stereo to obtain a refined facial 3D mesh. The Laplacian coordinates of the refined 3D facial mesh are extracted, and the expression key points among all facial feature points are used as anchor points. The facial 3D mesh is driven to perform expression transfer by minimizing the Laplacian deformation energy function, thereby generating a facial 3D mesh that reflects the user's current expression.

[0027] In practice, the first step is to call a preset parametric facial template. This preset parametric facial template is a set of pre-defined standard 3D face meshes containing vertex positions, triangular facet topology, and skin weights. The process of setting up the parametric facial template is as follows: collect 3D facial scan data of multiple people with neutral expressions, perform registration and non-rigid iterative nearest-neighbor algorithm alignment on all scan data to establish dense correspondences between vertices, perform principal component analysis on each aligned sample mesh, retain the first few principal components and the average face shape, control the facial morphology changes by adjusting the coefficients of each principal component, and form a set of variable mesh models driven by low-dimensional mixed shape parameters. The average mesh of this model is used as the standard 3D mesh, and the anatomical features such as the corners of the eyes and the tip of the nose marked on each sample mesh are used as the reference. Using the mean coordinates of the learned feature points as preset feature points, a parameterized facial template is set. The preset parameterized facial template is then spatially matched and mapped with the facial feature points. Specifically, a similarity transformation that minimizes the sum of squared Euclidean distances between the template feature points and the user's facial feature points is solved to obtain rotation, translation, and global scale parameters. The parameterized facial template mesh is then aligned to the user's face. Using the aligned template feature points as control points, thin-plate spline interpolation is used to perform radial basis function-driven nonlinear deformation on each non-feature point vertex, generating an initial personalized 3D facial mesh whose shape matches the user's individual facial contour. This initial personalized 3D facial mesh refers to an initial 3D model representing the individual facial geometry of the user under neutral facial expressions. Then, the initial... The process begins by optimizing the local neighborhood image texture consistency of each vertex in the personalized 3D facial mesh using photometric stereo optimization. Specifically, local texture blocks are extracted from the user's facial image centered on each mesh vertex. Based on the point light source model and the Lambertian reflection assumption, the pixel intensity changes of these local texture blocks under multiple frames of different known lighting directions are analyzed. Iterative optimization using least-squares solutions to the vertex normal vector and surface reflectivity is employed to adjust the depth value of each vertex along its normal, minimizing the sum of reprojection texture errors under all lighting conditions. This results in a more detailed, refined 3D facial mesh. This refined 3D facial mesh is a facial model with enhanced geometric fidelity achieved by incorporating the principle of luminosity restoration. Finally, the Laplacian coordinates of the refined 3D facial mesh are extracted. The Laplacian coordinates represent the coordinates of each mesh vertex in the Cartesian plane. The difference vector between the Cartesian coordinates and the center position of the vertex in its topological neighborhood is used to encode the local geometric details of the surface. Then, the expression key points among all facial feature points are used as spatial anchor points. The expression key points are semantic points such as the corners of the mouth, the corners of the eyes, and the endpoints of the eyebrows that are significantly displaced with the expression. A Laplacian deformation energy function is constructed. This function is composed of a weighted sum of the anchor point position constraint term and the Laplacian coordinate preservation term. By solving the sparse linear system, the energy function is minimized, driving the facial 3D refinement mesh to move from the current neutral position to the new position reflecting the expression while keeping the Laplacian coordinates of each vertex unchanged. This naturally preserves the local concavity and convex details of the mesh vertices after the expression migration, and finally generates a facial 3D mesh that reflects the user's current expression.

[0028] It should be noted that the facial 3D mesh in this application refers to a digital surface model used to characterize the user's current facial 3D geometric shape and real-time expression changes. Its purpose is to serve as the geometric input for subsequent contour feature recognition, provide the 3D coordinates of facial feature points to calculate the facial asymmetry deviation field, and provide a reference topology for continuous displacement monitoring of the dynamic elastic deformation field, so that the makeup guidance process can be accurately established on the user's real 3D shape and dynamic changes.

[0029] In step S102, the contour features of the user's face are identified based on the three-dimensional coordinates of the facial feature points in the facial three-dimensional mesh, thereby generating a static facial mesh that represents the inherent asymmetrical contour of the user's face.

[0030] In some embodiments, reference Figure 3 As shown in the figure, this is an exemplary flowchart of determining a static facial mesh according to some embodiments of this application. In this embodiment, the user's face contour features are identified based on the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh, and then a static facial mesh representing the inherent asymmetric contour of the user's face is generated by the following steps: In step S1021, the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh are extracted, the spatial vector difference between symmetrical facial feature points on both sides of the facial midline is calculated, and a facial asymmetry deviation field is generated. In step S1022, the facial asymmetry deviation field is decoupled from facial expression motion, and the non-rigid deformation components caused by the current expression are eliminated by projection decomposition to obtain the facial inherent asymmetry deviation field that is only related to the morphology of bones and soft tissues. In step S1023, the inherent asymmetric deviation field of the face is used as a vertex displacement constraint to perform rigidity-preserving asymmetric deformation on the reference symmetric face mesh, thereby generating a static face mesh that characterizes the inherent asymmetric contour of the user's face.

[0031] In practice, firstly, the 3D coordinates of facial feature points labeled with anatomical semantics are extracted from the 3D facial mesh reflecting the user's current expression. The sagittal plane, defined by the line connecting the center of the eyebrows to the tip of the nose and the lines connecting the midpoints of the inner canthi on both sides, serves as the facial midline reference plane. Symmetrical facial feature point pairs are established on both sides of this reference plane using predefined semantic pairing rules, such as left and right outer canthi, and left and right lip corners. The spatial vector difference between each pair of symmetrical facial feature points is calculated, i.e., the 3D deviation vector is obtained by subtracting the left feature point coordinate from the right feature point coordinate. The deviation vectors of all symmetrical facial feature point pairs together constitute the original deviation field of facial asymmetry. This original deviation field is used for... The three-dimensional spatial offset distribution between corresponding anatomical feature points on the left and right sides of the user's face under the influence of facial expression movement is characterized. Then, the original facial asymmetry deviation field is decoupled from facial expression movement. Specifically, a three-dimensional facial mesh of the user in a neutral, expressionless state is collected as a personal neutral reference. The spatial vector difference between pairs of symmetrical facial feature points of the same name on the personal neutral reference is calculated as the personal neutral asymmetry deviation field. The three-dimensional vectors of the two fields on the same symmetrical point pair are dot-producted and projected onto the direction spanned by the personal neutral asymmetry deviation field. The projected components are related to the inherent structure of the skeletal soft tissue, while the orthogonal residual components perpendicular to the neutral direction are the current expression. The dynamic non-rigid deformation components caused by motion are eliminated by retaining the projection components and discarding the orthogonal components, thus removing the influence of facial expressions. This yields a facial inherent asymmetry deviation field that is only related to the morphology of bones and soft tissues. This facial inherent asymmetry deviation field is used to characterize the spatial offset pattern of the user's inherent left-right asymmetric contour in a neutral state. Finally, the facial inherent asymmetry deviation field is used as a vertex displacement constraint to perform rigid-preserving asymmetric deformation on a reference symmetrical facial mesh. In practice, a strictly left-right symmetrical reference symmetrical facial mesh is first constructed by mirror averaging the user's personal neutral reference mesh. The vertices on this reference symmetrical facial mesh are then subjected to geodesic distances to each facial feature point. The soft skin is bound, and half of the deviation vector at each symmetrical facial feature point in the inherent asymmetry deviation field of the face is used as the position offset of the corresponding facial feature point and its left and right neighboring vertices on the reference symmetrical facial mesh. Then, these facial feature points are used as constraints of known displacements, and the remaining region of the mesh is used as the solution domain. A rigid deformation energy function is introduced, that is, by minimizing the Frobenius norm difference between the transformation matrix of each triangular facet and the nearest rotation matrix, the spatial position of all unconstrained vertices is solved iteratively. Under the premise of satisfying the feature point offset, the relative shape distortion of each local facet is minimized, and finally, a static facial mesh representing the inherent asymmetric contour of the user's face is obtained by deformation.

[0032] It should be noted that, in this application, the static facial mesh refers to a three-dimensional mesh model that represents the personalized static contour of a user's face, determined solely by the asymmetry of bones and soft tissues after excluding facial expression interference. The reason for determining the static facial mesh is that subsequent makeup guidance needs to be performed on a stable facial base that is not affected by real-time facial expression changes. If makeup is directly superimposed on a mesh containing facial expression deformation, the makeup texture will lose its reference consistency due to repeated stretching with facial expressions. Its function is to serve as a deformation base, providing the user's inherent asymmetric contour reference for generating a dynamically adaptable mesh by superimposing a dynamic elastic deformation field, so that makeup guidance can be adjusted synchronously with facial expressions while conforming to the individual's facial structure.

[0033] In step S103, the displacement of facial feature points in the three-dimensional facial mesh is continuously monitored, a dynamic elastic deformation field associated with the current expression change is generated, and the static facial mesh is superimposed on the dynamic elastic deformation field as a deformation base to generate a dynamic adaptation mesh to guide the user's makeup application.

[0034] In some embodiments, continuously monitoring the displacement of facial feature points in the facial 3D mesh and generating a dynamic elastic deformation field associated with the current expression change is achieved through the following steps: For continuously acquired facial 3D mesh sequences, the spatiotemporal trajectories of corresponding facial feature points are identified, and the 3D displacement vector of each facial feature point from the initial static pose to the current frame is calculated. Using the three-dimensional displacement vector as the boundary condition, and the static facial mesh as the solution domain of the elasticity model, the static equilibrium equation of the non-uniform anisotropic elastic body is solved by applying the finite element method, and the continuous displacement distribution at any position within the solution domain is obtained. The continuous displacement distribution is expressed as a grid vertex-level displacement vector field, generating a dynamic elastic deformation field associated with the current facial expression change.

[0035] In specific implementation, firstly, for the continuous acquisition of a 3D facial mesh sequence reflecting the user's current expression by the makeup mirror, an optical flow tracing algorithm is used to identify the spatiotemporal trajectory of corresponding facial feature points in the mesh vertex space. Specifically, taking each facial feature point on the neutral expression mesh of the first frame as the initial position, the corresponding point position that minimizes the weighted error of the distance between local neighborhood vertices is searched on the mesh of subsequent frames. The coordinates of the corresponding points in each frame are concatenated to form a spatiotemporal trajectory. Then, taking the position of the feature point in the initial static posture as the reference origin, the 3D displacement vector of each facial feature point from the initial static posture to the current frame is calculated. The 3D displacement vector is the 3D motion of each feature point due to the change in expression. Then, using the 3D displacement vectors of all facial feature points as the boundary conditions of known displacements, the tetrahedral solid mesh obtained by pre-solidifying the static facial mesh using a tetrahedral generation algorithm is used as the solution domain of the elasticity model. This tetrahedral mesh is a set of finite element discrete elements filling the internal region of the facial epidermis. According to the muscle direction in the facial anatomy and the differences in tissue hardness in different regions, non-uniformity is assigned to each element in the tetrahedral mesh. Anisotropic elastic material properties are used, specifically, smaller Young's moduli along the muscle fiber direction are set for elements in the distribution area of ​​facial muscles such as the zygomaticus major, while larger isotropic Young's moduli are set for elements in the bone attachment area. A static equilibrium equation for a non-uniform anisotropic elastic body is established, taking into account the material anisotropy and spatial distribution differences. The equation is a sparse linear system of equations with the displacement of tetrahedral element nodes as unknowns. The conjugate gradient method is used to solve the equation to obtain the three-degree-of-freedom displacement solution of all nodes in the entire tetrahedral mesh under the conditions of internal force equilibrium and boundary displacement constraints. That is, the continuous displacement distribution at any position in the solution domain is solved. The continuous displacement distribution represents the spatial migration state of the facial tissue from the initial static posture to the current expression under the forced constraint of the three-dimensional displacement vector of the facial feature points. Finally, the displacement solutions of the tetrahedral mesh nodes are extracted according to their one-to-one correspondence with the vertices of the static facial mesh, and organized into a mesh vertex-level displacement vector field with mesh vertices as indices and displacement vectors as values. This vector field is the dynamic elastic deformation field corresponding to the expression of the current frame.

[0036] It should be noted that the dynamic elastic deformation field in this application refers to the displacement vector field obtained by expressing the continuous displacement distribution with the vertices of the facial 3D mesh as indices. It is used to characterize the 3D displacement distribution of each vertex of the static facial mesh from the inherent asymmetric contour base of the face to the current expression posture under the current expression change. Since facial expression movement is not a simple linear scaling, but a non-uniform continuous deformation driven by muscles and constrained by tissue elasticity, the displacement vector of sparse feature points alone cannot describe the precise offset of each vertex of the mesh globally. Therefore, by determining the dynamic elastic deformation field, it can be superimposed on the static facial mesh as the deformation base to generate a dynamic adaptation mesh that reflects the user's innate contour asymmetry and changes synchronously with real-time expressions, providing an accurate geometric deformation reference for subsequent makeup layer mapping.

[0037] In some embodiments, the static facial mesh is superimposed on the dynamic elastic deformation field as a deformation base to generate a dynamically adaptable mesh that guides the user's makeup application. This is achieved through the following steps: The dynamic elastic deformation field is decomposed into spinors to obtain the rotation tensors and translation vectors acting on the vertices of the static facial mesh. The rotation tensor is applied to each vertex of the static facial mesh in a local rigid transformation manner, and the translation vector is accumulated according to the corresponding vertex to obtain a combined deformable mesh that integrates the inherent asymmetric contour of the face and the dynamic changes of the current expression. A volume-preserving constraint based on anatomical muscle attachment points is applied to local regions in the combined deformable mesh, and vertex displacement is corrected to eliminate non-physiological mesh distortion, thereby generating a dynamically adapted mesh that guides the user in applying makeup.

[0038] In practical implementation, firstly, a spinor decomposition is performed on the dynamic elastic deformation field. This spinor decomposition uses the local neighborhood of each vertex of the static face mesh as the processing unit, taking the local small facet formed by the vertex and its ring of neighboring vertices as the analysis object. The Jacobian matrix between the vertex's own displacement vector and the displacement vectors of its neighboring vertices is calculated. This Jacobian matrix is ​​then subjected to extreme decomposition, decomposing it into the product of an orthogonal matrix and a symmetric positive definite matrix. The orthogonal matrix corresponds to the rotation tensor of the local facet, representing the rigid body rotation that occurs in the vertex's neighborhood without changing its shape. The remaining amount after subtracting the rotation from the displacement vector is the translation vector of the vertex. This extreme decomposition is performed vertex by vertex. The entire dynamic elastic deformation field is decomposed into a set of rotation tensors and a set of translation vectors covering all vertices. Then, the rotation tensors are applied to each vertex of the static face mesh using a local rigid transformation. Specifically, for each vertex of the static face mesh, its rotation tensor is extracted, and the edge vectors of the one-ring neighborhood of that vertex are multiplied one by one with the rotation tensor to achieve the rotation transformation of the edge vectors. The rotated edge vectors are connected to form a new configuration of the local neighborhood after rotation deformation. At the same time, the translation vectors are directly accumulated according to the coordinates of the corresponding vertices, and the positions of the rotated and deformed vertices are shifted along the direction of the translation vectors. After traversing all vertices, a preliminary deformation result is obtained. The overlapping areas of adjacent face patches in the result are then analyzed. The average displacement of vertices is used to eliminate adjacency inconsistencies caused by independent deformation of each vertex, ultimately forming a combined deformable mesh that integrates the inherent asymmetrical contour of the user's face represented by the static facial mesh with the dynamic elastic deformation site representing the current expression's dynamic changes. The combined deformable mesh refers to a mesh model that represents the three-dimensional geometric shape of the user's face, simultaneously superimposed with the real-time displacement caused by the current facial expression movement, based on the inherent asymmetrical contour of the skeletal and soft tissues. Finally, volume constraints based on anatomical muscle attachment points are applied to local regions in the combined deformable mesh. During implementation, the mesh is divided into areas such as the frontalis muscle, orbicularis oculi muscle, orbicularis oris muscle, according to the action unit coverage area defined by the facial motion coding system. The mesh partitions corresponding to facial muscles such as the zygomaticus major muscle are used. The pre-labeled mesh vertices of the muscle attachment points in each muscle partition are used as constraint references. For each muscle partition, the volume change of the partition relative to the static facial mesh in the combined deformation mesh is calculated. If the volume expansion or contraction of a partition exceeds the preset physiological threshold, such as the maximum volume change rate range of human facial tissue, a volume preservation penalty term is introduced in the partition. The vertices in the partition are iteratively adjusted in the opposite direction of the volume gradient by the Lagrange multiplier method. Under the premise of keeping the attachment point position unchanged, the partition is compressed or expanded until the volume deviation returns to the physiological range. After this correction, a dynamic adaptation mesh for guiding users to apply makeup without puncture, tearing, or abnormal bulging is obtained.

[0039] It should be noted that the dynamic adaptation mesh in this application refers to a three-dimensional mesh model used to characterize the user's face at the current moment, simultaneously reflecting the inherent asymmetrical contour and real-time expression changes, and conforming to the physiological deformation law of facial tissue. Because the combined deformation mesh may produce geometric defects such as patch puncture and abnormal bulges due to the incomplete coordination of the displacement field in local areas during the process of independently applying spin and translation per vertex, which do not conform to the actual physical behavior of facial muscle-skin linkage, if not corrected, it will cause texture tearing or misalignment in subsequent makeup texture mapping. Therefore, determining the dynamic adaptation mesh can serve as the geometric carrier for makeup mapping, providing a three-dimensional reference surface that is precisely synchronized with the user's current facial shape and expression for constructing personalized makeup layers, ensuring that the contours of the eyeliner, lip line, and other sections in the makeup layer remain in line with the dynamic changes of the face, without drifting or penetrating.

[0040] In step S104, the user-selected standard makeup texture is obtained, and the user's current personalized makeup layer is constructed based on the current geometry of the dynamic adaptation mesh and the standard makeup texture. The personalized makeup layer is then overlaid on the user's facial image to guide the user in performing makeup operations.

[0041] In some embodiments, obtaining the user-selected standard makeup texture and constructing the user's current personalized makeup layer based on the current geometry of the dynamically adapted mesh and the standard makeup texture is achieved through the following steps: Receive the standard makeup texture selected by the user, and map the standard makeup texture to the parameterized texture space of the dynamically adapting mesh through two-dimensional texture coordinates to generate the initial makeup texture mapping; Calculate the local differential properties of the current geometry of the dynamically adapting mesh relative to the reference plane, and use the local differential properties to perform texture gradient adaptive deformation of the initial makeup texture mapping oriented towards the surface curvature to obtain a geometrically adapted makeup texture. The geometrically adapted makeup texture is rendered in segments according to the topology of the dynamically adapted mesh to construct a personalized makeup layer that is synchronized with the user's current facial shape and expression.

[0042] In practice, the user first selects a standard makeup texture from a pre-set makeup library on the makeup mirror interface. This standard makeup texture is a two-dimensional image drawn in the UV space of a standard human face texture coordinate system, containing color information of various makeup elements such as eyeshadow outlines, blush areas, and lip gloss shapes. The standard makeup texture is then mapped to the parametric texture space of the dynamic adaptation mesh using two-dimensional texture coordinates. Specifically, during the construction of the dynamic adaptation mesh, the UV texture coordinates of each vertex of the parametric face template are used to sample the texture color value of each pixel in the standard makeup texture using bilinear interpolation, based on the correspondence between its UV coordinates and the UV coordinates of the mesh vertices. A pixel-by-pixel mapping is established between the texture pixel and the spatial position of the mesh surface, generating an initial makeup texture that is tiled and applied to the surface of the current dynamic adaptation mesh. The initial makeup texture is mapped to the correspondence between each sampleable point on the mesh surface and the texture color value. Then, the local differential properties of the current geometry of the dynamically adapted mesh relative to the reference plane are calculated. The reference plane is the two-dimensional manifold parameterized plane unfolded by the dynamically adapted mesh in the expressionless face-facing sampling state. The local differential properties specifically include the first and second fundamental form coefficients calculated from the first-order partial derivatives of the mesh vertex positions, namely E, F, G and L, M, N. E, F, and G are the first fundamental form coefficients of the surface, where E is the inner product of the tangent vector in the u direction of the surface parameter domain, F is the inner product of the tangent vectors in the u and v directions, and G is the inner product of the tangent vector in the v direction. The three together describe the geometric quantities related to the intrinsic metric of the surface, such as the arc length element, the area element, and the angle between the two directions.L, M, and N are the second fundamental form coefficients of the surface. L is the dot product of the second-order partial derivative in the u-direction of the surface parameter domain and the unit normal vector of the surface; M is the dot product of the mixed second-order partial derivative in the u and v directions and the unit normal vector of the surface; and N is the dot product of the second-order partial derivative in the v-direction and the unit normal vector of the surface. These three coefficients collectively encode the curvature of the surface in space. By combining the first and second fundamental form coefficients, the Gaussian curvature and mean curvature of each point can be calculated. The area stretching factor of the neighborhood surface of each vertex is calculated using the first fundamental form coefficients, and the Gaussian curvature and mean curvature at each vertex are calculated using the second fundamental form coefficients. These curvature values ​​constitute the local differential attributes of the facial surface encoded by the curvature of the surface. Then, these local differential attributes are used to adaptively deform the initial makeup texture map based on the surface curvature. For example, in concave areas with large absolute values ​​of mean curvature, such as the sides of the nose or the eye sockets, the RGB gradient vectors of each pixel in the initial makeup texture map are nonlinearly compressed along the gradient direction according to the normal curvature direction of that area to counteract the visual aggregation of textures caused by the surface curvature. In the prominent areas of the forehead or cheekbones, stretching is performed along the gradient direction to compensate for the sparse texture caused by surface expansion. The scaling factors for compression and stretching are determined by a preset curvature-scaling mapping function based on the Gaussian curvature sign and average curvature amplitude at that point. After this pixel-level gradient deformation, a geometrically adapted makeup texture is obtained. The geometrically adapted makeup texture refers to a texture that corrects visual deformation based on the geometric characteristics of the curved surface while maintaining the semantics of the makeup pattern. Finally, the geometrically adapted makeup texture is fragment-shaded and rendered according to the topology of the dynamically adapted mesh. That is, the triangular facets of the dynamically adapted mesh are used as rendering units. In the graphics rendering pipeline, rasterized fragment sampling is performed on the interior of each facet according to the vertex texture coordinates and the geometrically adapted makeup texture passed by the vertex shader. At the same time, a lighting model based on the dot product of the vertex normal of the dynamically adapted mesh and the ambient light direction is introduced to adjust the fragment brightness. Anisotropic filtering is used for texture sampling at the edges of the facets to eliminate texture blur under tilted viewpoints. The rendering result is written to the frame buffer to form a personalized makeup layer that is synchronized with the user's current facial shape and expression.

[0043] It should be noted that, in this application, the personalized makeup layer refers to the image layer on which the makeup colors and patterns are superimposed on the real-time shape of the user's face. Since standard makeup textures are planar images drawn according to universal texture coordinates, they cannot directly adapt to the differences in facial contours between different users and the dynamic deformation of curved surfaces caused by the same user's facial expressions. Without geometric adaptation and mesh binding rendering, the makeup will be stretched and deformed in the undulating areas of the face or the facial features will be misaligned. Therefore, the personalized makeup layer is determined to serve as the direct visual content to be displayed and superimposed. After being integrated with the user's facial image, it guides the user to perform makeup operations according to the real-time dynamically fitted partition contours, ensuring that the drawn eyebrows, eyeliner, lips, and other makeup maintain accurate position and natural shape when the face moves.

[0044] In some embodiments, the personalized makeup layer is overlaid on the user's facial image to guide the user in makeup application, which is achieved through the following steps: Based on the current geometry of the dynamically adapting mesh, the personalized makeup layer is rendered onto the imaging plane of the user's facial image through perspective projection transformation to generate an initial overlay image; The edge pixels of the personalized makeup layer in the initial overlay image are fused with the corresponding areas in the user's facial image using Poisson image editing to generate a makeup guide image; Extract the outline boundaries of each makeup section in the personalized makeup layer, overlay them onto the makeup guide image as guide lines, and display them on the makeup mirror to guide the user to perform makeup operations according to the section outlines.

[0045] In practice, firstly, based on the current geometry of the dynamically adapted mesh, the intrinsic parameter matrix and distortion coefficients of the makeup mirror camera are read. A perspective projection transformation matrix is ​​then constructed from the 3D world coordinate system to the 2D image pixel coordinate system. The coordinates of each vertex of the dynamically adapted mesh bound to the personalized makeup layer are then homogeneously mapped using this perspective projection transformation matrix. During mapping, the vertex 3D coordinates are multiplied by the view matrix to perform camera pose transformation, and then multiplied by the perspective projection transformation matrix again for perspective division. This projects the mesh vertices in 3D space onto the imaging plane of the user's face image. During this process, the depth buffer of the graphics rendering pipeline is used to detect the occlusion relationships of each facet, discarding those occluded by the face itself. The process involves: first, rendering the hidden facets of the user's face image using rasterization to output an initial overlay image with the same resolution as the user's facial image, ensuring correct perspective and occlusion; then, performing Poisson-based image editing to fuse the edge pixels of the personalized makeup layer in the initial overlay image with the corresponding regions in the user's facial image. Specifically, this involves extracting the rendering mask of the personalized makeup layer in the initial overlay image, expanding the mask boundary outward by several pixels to obtain a fusion zone region, using the pixel color values ​​at the outer boundary of the fusion zone in the user's facial image as Dirichlet boundary conditions, and using the pixel color gradient of the region inside the fusion zone in the initial overlay image as a guiding gradient field, and constructing a gradient field on this fusion zone region. A Poisson equation is constructed, in the form that the Laplacian operator of the fused image equals the divergence of the guiding gradient field. This equation is solved using the Gauss-Seidel iteration method or the conjugate gradient method to obtain the new color values ​​of each pixel within the fusion band. This ensures a smooth color transition at the makeup layer boundary to the real skin, eliminating boundary color differences and jagged edges caused by rendering accuracy and shooting noise, generating a makeup guiding image with a natural overall visual effect. Finally, the contour boundaries of each makeup section in the personalized makeup layer are extracted. This process first uses the semantic tags of each makeup element generated in the personalized makeup layer, such as the eyeliner area, eyebrow area, lip gloss area, and blush area, and then applies binary masks to the corresponding sections. The Canny edge detection operator extracts closed contour chains, which are then simplified using the Douglas-Puk algorithm to reduce jagged edges and jitter, resulting in smooth vector contour boundaries for each makeup area. These are then overlaid onto the makeup guidance image using a graphic overlay layer, following preset guide line styles (e.g., blue dashed lines for eyeliner, pink dotted lines for blush, and red solid lines for lips). The guide line width adaptively matches the pixel density of the makeup mirror screen to ensure visual clarity. Finally, the makeup guidance image with overlaid guide lines is displayed in real-time on the makeup mirror screen, guiding users to perform makeup operations such as drawing eyeliner, filling in eyebrows, and applying lip gloss according to the area contours.

[0046] In another aspect, in some embodiments, this application provides a personalized makeup assistance system based on facial features using a makeup mirror, see reference. Figure 4The figure is a schematic diagram of the structure of a personalized makeup assistance system based on facial features, according to some embodiments of this application. The personalized makeup assistance system based on facial features includes: an acquisition module 201, a processing module 202, and an execution module 203, which are described below: The acquisition module 201 in this application is mainly used to acquire the user's facial image captured by the makeup mirror, and to construct a three-dimensional facial mesh reflecting the user's current expression based on the facial feature points in the user's facial image. Processing module 202, in this application, is mainly used to perform contour feature recognition on the user's face based on the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh, and then generate a static facial mesh that represents the inherent asymmetrical contour of the user's face. The processing module 202 is also used to continuously monitor the displacement of facial feature points in the facial three-dimensional mesh, generate a dynamic elastic deformation field associated with the current expression change, and superimpose the static facial mesh as a deformation base with the dynamic elastic deformation field to generate a dynamic adaptation mesh to guide the user's makeup application. The execution module 203 in this application is mainly used to obtain the standard makeup texture selected by the user, construct the user's current personalized makeup layer based on the current geometric shape of the dynamic adaptation mesh and the standard makeup texture, and overlay the personalized makeup layer on the user's facial image to guide the user to perform makeup operations.

[0047] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, the processor being configured to acquire the code and execute the above-described personalized makeup assistance method based on facial features using a makeup mirror.

[0048] In some embodiments, reference Figure 5 This figure is a schematic diagram of the structure of a computer device implementing a personalized makeup assistance method based on facial features using a makeup mirror, according to some embodiments of this application. The personalized makeup assistance method based on facial features using a makeup mirror in the above embodiments can be implemented through... Figure 5 The computer device shown is used to implement this, and the computer device includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0049] The processor 301 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the personalized makeup assistance method based on facial features in this application.

[0050] The communication bus 302 can be used to transmit information between the aforementioned components.

[0051] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via the communication bus 302. The memory 303 may also be integrated with the processor 301.

[0052] The memory 303 stores program code for executing the solution of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the determination of the personalized makeup assistance method based on facial features using a makeup mirror can be achieved through the processor 301 and one or more software modules in the program code in the memory 303.

[0053] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0054] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0055] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0056] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described personalized makeup assistance method for a makeup mirror based on facial features.

[0057] Although preferred embodiments of this application have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0058] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application.

Claims

1. A personalized makeup assistance method using a makeup mirror based on facial features, characterized in that, Includes the following steps: Acquire user facial images captured by a makeup mirror, and construct a three-dimensional facial mesh reflecting the user's current expression based on facial feature points in the user facial images; Based on the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh, contour feature recognition is performed on the user's face, thereby generating a static facial mesh that represents the inherent asymmetrical contour of the user's face. The displacement of facial feature points in the three-dimensional facial mesh is continuously monitored, and a dynamic elastic deformation field associated with the current expression change is generated. The static facial mesh is used as a deformation base and superimposed with the dynamic elastic deformation field to generate a dynamic adaptation mesh that guides the user's makeup application. Obtain the standard makeup texture selected by the user, construct the user's current personalized makeup layer based on the current geometry of the dynamically adapting mesh and the standard makeup texture, and overlay the personalized makeup layer on the user's facial image to guide the user in makeup operations.

2. The method as described in claim 1, characterized in that, The facial feature points in the user's facial image are determined using the following steps: The user's facial image is subjected to multi-scale Gaussian difference filtering to obtain a facial scale space response map; On the facial scale spatial response map, surface fitting is used to locate sub-pixel level extreme points as candidate feature points; Based on the gradient direction distribution of the neighborhood of the candidate feature points, a main direction is assigned to each candidate feature point and a multi-dimensional feature description vector is generated. The multidimensional feature description vector is graph matched with a pre-constructed facial topology model to obtain facial feature points with anatomical semantics.

3. The method as described in claim 1, characterized in that, Constructing a 3D facial mesh reflecting the user's current expression based on facial feature points in the user's facial image specifically includes: The standard 3D mesh in the preset parameterized facial template is spatially matched and mapped to the facial feature points in the user's facial image to generate an initial personalized 3D facial mesh. Based on the consistency of local neighborhood image textures of each vertex in the initial personalized facial 3D mesh, the spatial position of the vertex is optimized by photometric stereo to obtain a refined facial 3D mesh. The Laplacian coordinates of the refined 3D facial mesh are extracted, and the expression key points among all facial feature points are used as anchor points. The facial 3D mesh is driven to perform expression transfer by minimizing the Laplacian deformation energy function, thereby generating a facial 3D mesh that reflects the user's current expression.

4. The method as described in claim 1, characterized in that, Based on the three-dimensional coordinates of facial feature points in the facial 3D mesh, contour feature recognition of the user's face is performed, and a static facial mesh representing the inherent asymmetric contour of the user's face is generated. Specifically, this includes: Extract the three-dimensional coordinates of facial feature points from the facial three-dimensional mesh, calculate the spatial vector difference between symmetrical facial feature points on both sides of the facial midline, and generate a facial asymmetry deviation field. The facial asymmetry deviation field is decoupled from facial motion by expression decomposition, and the non-rigid deformation components caused by the current expression are eliminated by projection decomposition to obtain the inherent facial asymmetry deviation field that is only related to the morphology of bones and soft tissues. Using the inherent asymmetric deviation field of the face as a vertex displacement constraint, a rigidity-preserving asymmetric deformation is performed on the reference symmetric facial mesh to generate a static facial mesh that characterizes the inherent asymmetric contour of the user's face.

5. The method as described in claim 1, characterized in that, Continuously monitoring the displacement of facial feature points in the facial 3D mesh to generate a dynamic elastic deformation field associated with the current expression change specifically includes: For continuously acquired facial 3D mesh sequences, the spatiotemporal trajectories of corresponding facial feature points are identified, and the 3D displacement vector of each facial feature point from the initial static pose to the current frame is calculated. Using the three-dimensional displacement vector as the boundary condition, and the static facial mesh as the solution domain of the elasticity model, the static equilibrium equation of the non-uniform anisotropic elastic body is solved by applying the finite element method, and the continuous displacement distribution at any position within the solution domain is obtained. The continuous displacement distribution is expressed as a grid vertex-level displacement vector field, generating a dynamic elastic deformation field associated with the current facial expression change.

6. The method as described in claim 1, characterized in that, The static facial mesh is superimposed on the dynamic elastic deformation field as a deformation base to generate a dynamically adapted mesh that guides the user's makeup application. Specifically, this includes: The dynamic elastic deformation field is decomposed into spinors to obtain the rotation tensors and translation vectors acting on the vertices of the static facial mesh. The rotation tensor is applied to each vertex of the static facial mesh in a local rigid transformation manner, and the translation vector is accumulated according to the corresponding vertex to obtain a combined deformable mesh that integrates the inherent asymmetric contour of the face and the dynamic changes of the current expression. A volume-preserving constraint based on anatomical muscle attachment points is applied to local regions in the combined deformable mesh, and vertex displacement is corrected to eliminate non-physiological mesh distortion, thereby generating a dynamically adapted mesh that guides the user in applying makeup.

7. The method as described in claim 1, characterized in that, The user's facial images are captured by a camera integrated into the makeup mirror.

8. A personalized makeup assistance system for a makeup mirror based on facial features, used to execute the personalized makeup assistance method for a makeup mirror based on facial features as described in any one of claims 1 to 7, characterized in that, The system includes: The acquisition module is used to acquire user facial images captured by the makeup mirror, and construct a three-dimensional facial mesh reflecting the user's current expression based on the facial feature points in the user facial images. The processing module is used to perform contour feature recognition on the user's face based on the three-dimensional coordinates of facial feature points in the facial three-dimensional mesh, and then generate a static facial mesh that represents the inherent asymmetrical contour of the user's face. The processing module is also used to continuously monitor the displacement of facial feature points in the facial 3D mesh, generate a dynamic elastic deformation field associated with the current expression change, and superimpose the static facial mesh as a deformation base with the dynamic elastic deformation field to generate a dynamic adaptation mesh to guide the user's makeup application. The execution module is used to obtain the standard makeup texture selected by the user, construct the user's current personalized makeup layer based on the current geometric shape of the dynamic adaptation mesh and the standard makeup texture, and overlay the personalized makeup layer on the user's facial image to guide the user to perform makeup operations.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing code, and the processor being configured to retrieve the code and execute the personalized makeup assistance method based on facial features as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the personalized makeup assistance method based on facial features as described in any one of claims 1 to 7.

Images