Virtual makeup method, system and device

Abstract

A virtual makeup method, system and device, comprising the following steps: marking 3D facial feature points in a 3DMM model, projecting the 3DMM model to obtain a UV expansion map, projecting the 3D facial feature points to obtain 3D feature texture points, extracting the texture coordinates of the 3D feature texture points; performing face detection and feature point detection on a 2D image to obtain 2D facial feature points and Euler angle parameters; correcting the texture coordinates of contour texture points in the 3D feature texture points to obtain corrected 3D feature texture points; performing extended interpolation on the corrected 3D feature texture points to obtain extended texture points; using normalized 2D facial feature points and 3D feature texture points as control points, transforming the 3D texture points to obtain the transformed coordinates of the 3D texture points; designing makeup on the UV expansion map; and applying makeup to the facial image on the 2D image. The present invention can improve the sense of reality, perspective and fit of virtual makeup.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual makeup, and in particular to a virtual makeup method, system and device. Background Art

[0002] As the times change, people, especially women, have higher requirements for appearance. Makeup is essential for women. The vast number of makeup products will cost time and money for makeup tryers. The advent of virtual reality technology gives makeup tryers a better makeup experience. Virtual makeup can show the makeup effect to the makeup tryer in a very short time.

[0003] The current virtual makeup trial technology mainly includes 2D facial feature point interpolation method and 3D facial model reconstruction method. The 2D facial feature point interpolation method first interpolates the 2D facial feature points to build a facial mesh model, and then uses the coordinate pairs of vertex coordinates and texture coordinates for texture mapping. This method maps the texture of the face except the facial features to the face, which will be stretched with the facial posture, that is, the texture has no perspective. The 3D facial model reconstruction method uses the 2D facial feature points and the corresponding 3DMM facial feature points as constraints to perform single-image face reconstruction, obtain a 3D model and projection parameters that can better represent the face in the 2D image, and use the vertex coordinates, texture coordinates and projection parameters of the 3D model to map the texture to the 2D face. Since this method requires single-image face reconstruction and parameter optimization, it has high computational complexity and poor stability compared to the 2D facial feature point interpolation method, resulting in the inability to fit the contour of the 3D model when it is mapped to the 2D face. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a virtual makeup method, system and device, which can effectively overcome the problems of texture stretching, weak perspective and poor fit in virtual makeup.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A virtual makeup method comprises the following steps: marking a plurality of 3D facial feature points in a 3DMM model, projecting the 3DMM model to obtain a UV unfolded image, projecting the 3D facial feature points to obtain 3D feature texture points, and extracting texture coordinates of the 3D feature texture points; performing face detection and feature point detection on a 2D image to obtain 2D facial feature points and Euler angle parameters of the same number as the 3D facial feature points; correcting the texture coordinates of contour texture points in the 3D feature texture points according to the 2D facial feature points and Euler angle parameters to obtain corrected contour texture points, and Correcting 3D feature texture points; performing extension interpolation on the corrected 3D feature texture points to obtain extended texture points, wherein the extended texture points and the corrected 3D feature texture points constitute 3D texture points; normalizing the 2D facial feature points, using the normalized 2D facial feature points and the corrected 3D feature texture points as control points, transforming the 3D texture points to obtain transformed coordinates of the 3D texture points; designing makeup on the UV unfolded image; applying makeup to the facial image on the 2D image according to the texture coordinates and transformed coordinates of the 3D texture points on the makeup.

[0007] Preferably, the method for correcting the texture coordinates of the contour texture points in the 3D feature texture points is: taking the 3D contour points in the 3D facial feature points of the front face at 0° as a reference, rotating the 3DMM model around the Z axis to a posture every 1°, and calculating the texture coordinates of the contour texture points under the posture; rotating the 2D facial feature points to a horizontal angle and normalizing them, selecting the contour texture points corresponding to the posture according to the Euler angle parameters, and correcting the texture coordinates of the contour texture points.

[0008] Preferably, the rotation range of the 3DMM model around the Z axis is from 0° for the front face to 50° for the side face.

[0009] Preferably, the method for calculating the texture coordinates of the contour texture points under the posture is: uniformly sampling the 5 3D contour points under the posture, fitting all the 3D contour points under the posture according to the B-spline curve, and calculating the texture coordinates of the contour texture points.

[0010] Preferably, the texture coordinates of the contour texture points are corrected based on the facial feature points in the 2D facial feature points rotated to a horizontal angle and normalized and the facial feature points in the 3D feature texture points in the corresponding state.

[0011] Preferably, the Lagrange interpolation algorithm, the Newton interpolation algorithm and the linear interpolation algorithm are used to perform extended interpolation on the modified 3D feature texture points.

[0012] Preferably, the 3D texture points are triangulated to obtain a triangular mesh of the 3D texture points, and the 3D texture points are transformed using a similarity-preserving deformation algorithm.

[0013] Preferably, a Z value is assigned to the transformed coordinates of the modified 3D texture point.

[0014] Based on the same inventive concept, the present invention also provides a virtual makeup system, including: a labeling module, used to label 3D facial feature points in a 3DMM model; a UV module, used to project the 3DMM model to obtain a UV expansion map, project the 3D facial feature points to obtain 3D feature texture points, and extract the texture coordinates of the 3D feature texture points; a detection module, used to perform face detection and feature point detection on a 2D image, and obtain 2D facial feature points and Euler angle parameters with the same number as the 3D facial feature points; a correction module, used to correct the texture coordinates of the contour texture points in the 3D feature texture points according to the 2D facial feature points and Euler angle parameters, and obtain the corrected contour texture points. and a modified 3D feature texture point; an interpolation module, used for extending and interpolating the modified 3D feature texture point to obtain the extended texture point and the texture coordinates of the extended texture point, wherein the extended texture point and the modified 3D feature texture point constitute a 3D texture point; a transformation module, used for normalizing the 2D face feature point, using the normalized 2D face feature point and the modified 3D feature texture point as control points, transforming the 3D texture point to obtain the transformed coordinates of the 3D texture point; a design module, used for designing makeup on a UV unfolded image; and a makeup application module, used for applying makeup to the face image on the 2D image according to the texture coordinates and the transformed coordinates of the 3D texture point on the makeup.

[0015] Based on the same inventive concept, the present invention also provides a virtual makeup device, which is used to run a program, wherein the above-mentioned virtual makeup method is executed when the program is running.

[0016] The beneficial effects of the present invention are:

[0017] (1) The texture coordinates of the contour texture points are corrected to improve the fit of the virtual makeup on the 2D face image;

[0018] (2) Extending and interpolating the modified 3D feature texture points to generate additional vertices for texture mapping, thereby improving the realism of virtual makeup on 2D facial images;

[0019] (3) Assigning Z values ​​to the transformed coordinates of the modified 3D feature texture points to improve the perspective of the virtual makeup on the 2D face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A schematic diagram comparing the makeup effects of a 2D facial feature point interpolation method and the method of the present invention;

[0021] Figure 2 Schematic diagram comparing the makeup effects of a 3D face model reconstruction method and the method described in the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] Embodiment 1

[0024] like Figures 1-2 As shown, this embodiment combines a 2D facial feature point interpolation method and a 3D facial reconstruction method to provide a 2.5D-based virtual makeup method for real-time virtual makeup and mapping.

[0025] In the invention, 2.5D technology is a technology that combines 2D image information and 3D graphic information. 2D images only contain one plane dimension information, while 3D graphics contain multi-dimensional spatial stereo information. 2.5D stores both 2D pixel information and 3D depth information, and can be converted to and from 2D image information or 3D graphic information.

[0026] The method comprises the following steps:

[0027] 118 3D facial feature points are marked in the 3DMM model, and the 3DMM model is cylindrically projected to obtain a UV unfolded image. Correspondingly, the 3D facial feature points are projected to obtain 3D feature texture points on the UV unfolded image, and the texture coordinates of the 3D feature texture points are extracted.

[0028] 3DMM (3D Morphable Face Model) is a basic 3D face model that uses a fixed number of points to represent a face. Its core idea is that faces can be matched one-to-one in 3D space and can be derived from the weighted linear addition of many other faces by orthogonal basis.

[0029] In other embodiments, more 3D facial feature points may be marked on the facial model; however, this requires more sophisticated algorithms and more powerful hardware computing capabilities.

[0030] Calculate the texture coordinates of the 3D contour points in the 3D facial feature points in different postures. The 3D contour points are projected cylindrically to obtain the contour texture points on the UV unfolded image, that is, the texture coordinates of the contour texture points in the 3D feature texture points in different postures are calculated. Taking the 3D contour points of the front face at 0° as the reference, the 3DMM model is rotated around the Z axis every 1° to a posture, and 5 3D contour points in the posture are uniformly sampled, and all 3D contour points in the posture are fitted according to the B-spline curve. Calculate the texture coordinates of the 3D contour points in the posture.

[0031] The texture coordinates of the facial texture points in the 3D feature texture points under different postures directly use the texture coordinates of the facial texture points at 0° on the front face.

[0032] The rotation range of the 3DMM model around the Z axis is from 0° for the front face to 50° for the side face. This rotation range basically covers the horizontal rotation area of ​​the human head.

[0033] By collecting a small number of 3D contour points and fitting all 3D contour points, the sampling work can be reduced while ensuring the accuracy of the fitted 3D contour points and improving the calculation efficiency of the texture coordinates of the contour texture points.

[0034] Input a video containing a face for detection. Perform face detection and 2D face feature point detection on each frame of 2D image, and obtain 118 2D face feature points and Euler angle parameters. That is, the 2D face feature points and 3D feature texture points form a one-to-one correspondence.

[0035] Rotate the 2D facial feature points to the horizontal angle and normalize them. According to the Euler angle corresponding to the Z axis, select the 3D feature texture points in the corresponding posture. Based on the facial features points in the 2D facial feature points, the 2D contour points and the facial features points in the 3D feature texture points, use the Euler angles corresponding to the X, Y, and Z axes to correct the texture coordinates of the contour texture points in this posture to obtain the corrected contour texture points, so that the shape of the corrected contour texture points is close to the shape of the 2D contour points. The corrected contour texture points and other 3D feature texture points constitute the corrected 3D feature texture points.

[0036] By correcting the shape of the contour texture points, the fit between the texture and the facial contour on the 2D image can be improved when texture mapping is performed for subsequent makeup.

[0037] The modified 3D feature texture points are extended and interpolated using the Lagrange interpolation algorithm, the Newton interpolation algorithm, and the linear interpolation algorithm to obtain the extended texture points and their texture coordinates. The extended texture points cover the entire face area. The extended texture points and the modified 3D feature texture points constitute the 3D texture points. The 3D texture points are triangulated to construct a triangular mesh of the 3D texture points.

[0038] 3D facial feature points only mark 3D facial features and 3D contour points, that is, 3D feature texture points only include facial features and contour texture points. Since the posture of the face in the video will change, it is necessary to extend and interpolate the facial features and contour texture points to obtain extended texture points that can dynamically cover the face area. Extending texture points can not only increase the vertices of texture mapping, but also these vertices can adapt to facial images with different positions and orientations, improve the accuracy of makeup texture projection onto 2D facial images, effectively avoid texture stretching, and improve the realism of makeup.

[0039] The 2D facial feature points are normalized, and the normalized 2D facial feature points and 3D texture points are used as control points. The texture coordinates of the 3D texture points are transformed using a similarity-preserving deformation algorithm to ensure the similarity between the relative positions of the 2D facial extension points and the relative positions of the 3D texture points, and to obtain the transformed coordinates of the 3D texture points.

[0040] The 2D face extension point is a point on the 2D image after the texture coordinates of the extended texture point are transformed.

[0041] The 2D face extension points and the 2D face feature points constitute the 2D face points. The 2D face points serve as the vertices of the 2D image of the current frame, covering the entire face area on the 2D image.

[0042] By performing texture mapping on a 2D image through the texture coordinates and transformation coordinates of 3D texture points, the realism of applying virtual makeup on a 2D face image can be improved.

[0043] Based on the self-developed real-time reconstruction technology of 3D facial points, the transformation coordinate Z value is assigned to each 3D texture point, that is, the 2D facial point is assigned a depth value, so that the virtual makeup mapped on the 2D facial image has a sense of perspective.

[0044] The designer designs the makeup on the UV expansion map, and the makeup corresponds to the position of the 3D texture points on the UV expansion map.

[0045] According to the texture coordinates and transformation coordinates of the 3D texture points on the makeup, the makeup texture is mapped onto the 2D face image, and OpenGL is used for texture mapping to complete the makeup process.

[0046] The method described in this embodiment utilizes face detection and feature point detection technology, interpolation technology, 2D mesh deformation technology and model expansion technology, which can effectively overcome the problems of texture stretching and weak perspective caused by the 2D face feature point interpolation method, and the problem of contour inability to fit caused by the 3D face reconstruction method.

[0047] The method described in this embodiment allows the user to experience the makeup effect with just one click, which is simple to operate and reduces the experience cost. It also allows the user to quickly experience the makeup effect in various states such as raising the head, lowering the head, and turning the face.

[0048] Comparison between 2D makeup effect and 2.5D effect Figure 1 As shown, the left side is a schematic diagram of the makeup effect of the 2D facial feature point interpolation method, and the right side is a schematic diagram of the makeup effect of the method described in this embodiment. 2D makeup produces texture stretching that distorts the makeup effect, and 2.5D makeup can greatly improve texture stretching and maintain the original base image shape.

[0049] Comparison of 3D makeup effect and 2.5D makeup effect Figure 2 As shown, the left side is a schematic diagram of the makeup effect of the 3D face reconstruction method, and the right side is a schematic diagram of the makeup effect of the method described in this embodiment. In the head-down posture, the texture of the 3D makeup at the chin will float outward and not fit the facial contour, while the texture of the 2.5D makeup fits the facial contour.

[0050] Those skilled in the art can understand that all or part of the steps in the above method embodiments can be completed by instructing related hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0051] Embodiment 2

[0052] This embodiment provides a 2.5D-based virtual makeup system, including:

[0053] The annotation module is used to annotate 3D facial feature points in the 3DMM model.

[0054] The UV module is used to cylindrically project the 3DMM model to obtain a UV unfolded image, cylindrically project the 3D facial feature points to obtain 3D feature texture points, and extract the texture coordinates of the 3D feature texture points.

[0055] The calculation module uses the 3D contour points in the 3D facial feature points at 0° of the front face as the reference, and rotates the 3DMM model around the Z axis from 0° of the front face to 50° of the side face, with every 1° as one posture. The calculation module is used to uniformly sample 5 3D contour points in each posture, fit all 3D contour points according to the B-spline curve, and calculate the texture coordinates of the 3D contour points. The cylindrical projection of the 3D contour points obtains the contour texture points on the UV unfolding map, that is, calculates the texture coordinates of the contour texture points in the 3D feature texture points.

[0056] The detection module is used to perform face detection and feature point detection on the 2D image to obtain 2D face feature points and Euler angle parameters with the same number as the 3D face feature points.

[0057] The correction module is used to correct the texture coordinates of the contour texture points based on the 2D face feature points and Euler angle parameters, taking the facial features points in the 2D face feature points, the 2D contour points and the facial features points in the 3D feature texture points as references, and obtain the corrected contour texture points.

[0058] The modified contour texture points and other 3D feature texture points constitute the modified 3D feature texture points.

[0059] The interpolation module is used to perform extended interpolation on the modified 3D feature texture points using the Lagrange interpolation algorithm, the Newton interpolation algorithm and the linear interpolation algorithm to obtain extended texture points and texture coordinates of the extended texture points.

[0060] The extended texture points cover the entire face area. The extended texture points and the modified 3D feature texture points constitute the 3D texture points.

[0061] The triangulation module is used to triangulate 3D texture points and construct a triangular mesh of 3D texture points.

[0062] The transformation module is used to normalize the 2D facial feature points, use the normalized 2D facial feature points and the modified 3D feature texture points as control points, transform the 3D texture points, obtain the transformed coordinates of the 3D texture points, and assign the transformed coordinate Z value to each 3D texture point.

[0063] The texture coordinates of the 3D texture points are transformed into 2D face points on the 2D image.

[0064] The 2D face points are used as vertices of the 2D image of the current frame, covering the entire face area on the 2D image.

[0065] Design module, used to design makeup on UV unfolded map.

[0066] The makeup module is used to apply makeup to the face image on the 2D image according to the texture coordinates and transformation coordinates of the 3D texture points on the makeup.

[0067] Embodiment 3

[0068] This embodiment provides a 2.5D-based virtual makeup device, which is used to run a program, wherein the virtual makeup method described in the first embodiment is executed when the program is run.

[0069] The device can be portable, so that users can use it at home to experience the effects of various makeup on their faces. Users can freely match the makeup until they find a satisfactory one. The device can update the makeup regularly, such as the color number of the latest beauty products, to improve the user experience.

[0070] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A virtual makeup method, characterized in that: The following steps are involved: Marking a plurality of 3D facial feature points in a 3DMM model, projecting the 3DMM model to obtain a UV expansion map, projecting the 3D facial feature points on the UV expansion map to obtain 3D feature texture points, extracting texture coordinates of the 3D feature texture points, wherein the texture coordinates are two-dimensional coordinates in UV space; Performing face detection and feature point detection on the 2D image to obtain 2D face feature points and Euler angle parameters that are the same in number as the 3D face feature points; According to the 2D face feature points and the Euler angle parameters, the texture coordinates of the contour texture points in the 3D feature texture points are corrected to obtain the corrected contour texture points and the corrected 3D feature texture points; wherein the method for correcting the texture coordinates of the contour texture points in the 3D feature texture points is: Taking the 3D contour points in the 3D facial feature points of the front face at 0° as the reference, the 3DMM model is rotated around the Z axis at intervals of 1° to a posture, and the texture coordinates of the contour texture points under the posture are calculated; The 2D facial feature points are rotated to a horizontal angle and normalized, the contour texture points corresponding to the posture are selected according to the Euler angle parameters, and the texture coordinates of the contour texture points are corrected; further, the facial features points in the 2D facial feature points rotated to a horizontal angle and normalized and the facial features points in the 3D feature texture points corresponding to the posture are used as references to correct the texture coordinates of the contour texture points; Extending and interpolating the modified 3D feature texture points to obtain extended texture points, wherein the extended texture points and the modified 3D feature texture points constitute 3D texture points; further, extending and interpolating the modified 3D feature texture points using a Lagrange interpolation algorithm, a Newton interpolation algorithm, and a linear interpolation algorithm; Normalizing the 2D facial feature points, using the normalized 2D facial feature points and the modified 3D feature texture points as control points, transforming the 3D texture points to obtain transformed coordinates of the 3D texture points; Designing makeup on the UV unfolded map; The human face image on the 2D image is made up according to the texture coordinates and the transformation coordinates of the 3D texture points on the make-up.

2. The virtual makeup method according to claim 1, characterized in that: The rotation range of the 3DMM model around the Z axis is from 0° for the front face to 50° for the side face.

3. The virtual makeup method according to claim 1, characterized in that: The method for calculating the texture coordinates of the contour texture points under the posture is: uniformly sampling the 5 3D contour points under the posture, fitting all the 3D contour points under the posture according to the B-spline curve, and calculating the texture coordinates of the contour texture points.

4. The virtual makeup method according to claim 1, characterized in that: The 3D texture points are triangulated to obtain a triangular mesh of the 3D texture points, and the 3D texture points are transformed using a similarity-preserving deformation algorithm.

5. The virtual makeup method according to claim 1, characterized in that: The transformed coordinates of the modified 3D texture point are assigned a Z value.

6. A virtual makeup system, characterized in that: include: Annotation module, used to annotate 3D facial feature points in the 3DMM model; A UV module is used to project the 3DMM model to obtain a UV expansion map, project the 3D facial feature points on the UV expansion map to obtain 3D feature texture points, and extract texture coordinates of the 3D feature texture points, where the texture coordinates are two-dimensional coordinates in UV space; A detection module, used to perform face detection and feature point detection on the 2D image, and obtain 2D face feature points and Euler angle parameters with the same number as the 3D face feature points; A correction module is used to correct the texture coordinates of the contour texture points in the 3D feature texture points according to the 2D face feature points and the Euler angle parameters to obtain the corrected contour texture points and the corrected 3D feature texture points; wherein the method for correcting the texture coordinates of the contour texture points in the 3D feature texture points is: Taking the 3D contour points in the 3D facial feature points of the front face at 0° as the reference, the 3DMM model is rotated around the Z axis at intervals of 1° to a posture, and the texture coordinates of the contour texture points under the posture are calculated; The 2D facial feature points are rotated to a horizontal angle and normalized, the contour texture points corresponding to the posture are selected according to the Euler angle parameters, and the texture coordinates of the contour texture points are corrected; further, the facial features points in the 2D facial feature points rotated to a horizontal angle and normalized and the facial features points in the 3D feature texture points corresponding to the posture are used as references to correct the texture coordinates of the contour texture points; An interpolation module is used to perform extended interpolation on the modified 3D feature texture points to obtain extended texture points and texture coordinates of the extended texture points, wherein the extended texture points and the modified 3D feature texture points constitute 3D texture points; further, Lagrange interpolation algorithm, Newton interpolation algorithm and linear interpolation algorithm are used to perform extended interpolation on the modified 3D feature texture points; A transformation module, used to normalize the 2D facial feature points, and transform the 3D texture points using the normalized 2D facial feature points and the modified 3D feature texture points as control points to obtain transformation coordinates of the 3D texture points; Design module, used to design makeup on UV unfolded images; The makeup module is used to apply makeup to the face image on the 2D image according to the texture coordinates and transformation coordinates of the 3D texture points on the makeup.

7. A virtual makeup device, characterized in that: The device is used to run a program, wherein the program, when running, executes the virtual makeup method according to any one of claims 1 to 5.

Images