An intelligent cosmetic mirror and a method for displaying an auxiliary makeup

Abstract

The application discloses an intelligent cosmetic mirror and a method for displaying an auxiliary makeup, and is used for providing a makeup auxiliary function for a user based on the intelligent cosmetic mirror. The intelligent cosmetic mirror comprises a camera, a plane mirror, a display screen and a processor. In response to a makeup auxiliary function instruction of the intelligent cosmetic mirror, a human face posture in a human face image captured by the intelligent cosmetic mirror is determined according to a judgment rule of a human face part corresponding to the makeup auxiliary function instruction. A makeup model corresponding to the human face part and the human face posture is determined according to the human face part and the human face posture corresponding to the makeup auxiliary function instruction. An auxiliary makeup area is determined according to the makeup model and a human face key point position in the human face image. The auxiliary makeup area is displayed at a makeup position of the human face in the intelligent cosmetic mirror, wherein the makeup position is determined based on the human face part corresponding to the makeup auxiliary function instruction and the human face key point position.

Description

Technical Field

[0001] This invention relates to the field of smart makeup mirror technology, and in particular to a smart makeup mirror and a method for displaying and assisting in makeup application. Background Technology

[0002] There are many types of smart makeup mirrors on the market. They can usually provide users with makeup information based on the Internet while performing the functions of a makeup mirror. Some smart makeup mirrors can also provide personalized makeup recommendations based on the user's facial image to meet different makeup needs.

[0003] However, based on the smart makeup mirror scenario, there is currently no makeup assistance function that can convert the makeup selected by the user into a corresponding auxiliary planar makeup area and display the planar makeup area on the smart makeup mirror, so that the user can apply makeup in the planar auxiliary area by looking at their own face displayed on the smart makeup mirror and the planar makeup area that fits the face. Summary of the Invention

[0004] This invention provides a smart makeup mirror and a method for displaying auxiliary makeup application. It provides users with makeup assistance based on the smart makeup mirror, ensuring that users can quickly and accurately align the auxiliary makeup area with the position of the user's face displayed on the smart makeup mirror during the makeup process, thereby improving the user experience.

[0005] In a first aspect, an embodiment of the present invention provides a smart makeup mirror, comprising a camera, a plane mirror, a display screen, and a processor, wherein:

[0006] The camera is used to capture faces;

[0007] The plane mirror is used to display the face of the user using the smart makeup mirror using imaging principles;

[0008] The display screen is used to display the makeup assistance area;

[0009] The processor is configured to execute:

[0010] In response to the makeup assistance function command of the smart makeup mirror, the facial pose in the facial image captured by the smart makeup mirror is determined according to the facial feature judgment rule corresponding to the makeup assistance function command.

[0011] Based on the facial features and facial pose corresponding to the makeup assistance function command, determine the makeup model corresponding to the facial features and facial pose;

[0012] Based on the makeup model and the positions of facial key points in the facial image, determine the makeup auxiliary area;

[0013] The makeup assistance area is displayed on the face in the smart makeup mirror, wherein the makeup position is determined based on the face part corresponding to the makeup assistance function command and the position of the face key points.

[0014] In some embodiments, the processor is specifically configured to perform:

[0015] The facial pose in the face image is determined based on a head pose estimation algorithm; or,

[0016] The facial pose in the facial image is determined based on the positional relationship between the contour key points that characterize the facial contour in the facial key points.

[0017] In some embodiments, the processor is specifically configured to perform:

[0018] Based on facial symmetry, multiple pairs of key points are selected from the contour key points, wherein each pair of key points is symmetrical about the facial midline, which divides the face into two equal parts.

[0019] The facial pose is determined based on a first distance between keypoints on one side of the face in a plurality of keypoint pairs, and a second distance between keypoints on the other side of the face in a plurality of keypoint pairs.

[0020] In some embodiments, the processor is specifically configured to perform:

[0021] If the facial feature corresponding to the makeup assistance function instruction is the eyebrows, and the ratio of the first distance to the second distance is less than or equal to a first threshold, then the facial pose is determined to be a frontal view; otherwise, the facial pose is determined to be a side view, wherein the first distance is greater than or equal to the second distance; or,

[0022] If the facial feature corresponding to the makeup assistance function instruction is the cheek, and the ratio of the first distance to the second distance is less than or equal to the second threshold, then the facial posture is determined to be a frontal face. If the ratio is greater than the second threshold and less than or equal to the third threshold, then the facial posture is determined to be a small profile. If the ratio is greater than the third threshold, then the facial posture is determined to be a large profile, wherein the first distance is greater than or equal to the second distance.

[0023] In some embodiments, the processor is specifically configured to perform:

[0024] If the facial feature is the eyebrows and the facial pose is a frontal view, then the makeup model is determined to be a frontal eyebrow makeup model, wherein the auxiliary eyebrow areas on both sides of the frontal eyebrow makeup model are symmetrical; or,

[0025] If the facial feature is the eyebrows and the facial pose is a side profile, then the makeup model is determined to be a side profile eyebrow makeup model. In this side profile eyebrow makeup model, one side of the eyebrow auxiliary area is a half-eyebrow, and the arch of the other side's eyebrow auxiliary area is less pronounced than the arch of the eyebrow auxiliary area in the front-face eyebrow makeup model. The tail of the other side's eyebrow auxiliary area is longer than the tail of the eyebrow auxiliary area in the front-face eyebrow makeup model; or...

[0026] If the facial feature is the cheek and the facial pose is a frontal view, then the makeup model is determined to be a frontal face makeup model, wherein the auxiliary makeup areas on both sides of the frontal face makeup model are symmetrical; or,

[0027] If the facial feature is the cheek and the facial pose is a small profile, then the makeup model is determined to be a small profile makeup model, wherein the auxiliary makeup area on one side of the small profile makeup model is smaller than the auxiliary makeup area on the other side; or,

[0028] If the face part is the cheek and the face posture is a large side profile, then the makeup model is determined to be a large side profile makeup model, wherein the large side profile makeup model only has one side profile makeup auxiliary area.

[0029] In some embodiments, the processor is specifically configured to perform:

[0030] Based on the area determined by the location of the facial key points, the makeup model is scaled proportionally to determine the makeup auxiliary area.

[0031] In some embodiments, the processor is specifically configured to perform:

[0032] The makeup auxiliary area is rotated according to the rotation angle of the makeup position relative to the horizontal line;

[0033] The rotated makeup assistance area is displayed at the makeup application position.

[0034] Secondly, an embodiment of the present invention provides a smart makeup mirror, comprising a camera, a plane mirror, a display screen, and a processor, wherein:

[0035] The camera is used to capture faces;

[0036] The plane mirror is used to display the face of the user using the smart makeup mirror using imaging principles;

[0037] The display screen is used to display the makeup assistance area;

[0038] The processor is configured to execute:

[0039] In response to the makeup assistance function command of the smart makeup mirror, a makeup assistance area corresponding to the facial posture of the person is displayed at the makeup position on the face in the smart makeup mirror in the following manner:

[0040] If the face is in a frontal position, a makeup assistance area is displayed at the makeup application position on the frontal face. The makeup assistance area is used to indicate the makeup application position on the frontal face and is attached to the makeup application position on the frontal face.

[0041] If the face is in profile, a makeup assistance area is displayed at the makeup application position on the profile, wherein the makeup assistance area is used to indicate the makeup application position on the profile and fits onto the makeup application position on the profile.

[0042] The makeup application location is determined based on the facial features and key facial points corresponding to the makeup assistance function command.

[0043] Thirdly, an embodiment of the present invention provides a method for displaying auxiliary makeup, comprising:

[0044] In response to the makeup assistance function command of the smart makeup mirror, the facial pose in the facial image captured by the smart makeup mirror is determined according to the facial feature judgment rule corresponding to the makeup assistance function command.

[0045] Based on the facial features and facial pose corresponding to the makeup assistance function command, determine the makeup model corresponding to the facial features and facial pose;

[0046] Based on the makeup model and the positions of facial key points in the facial image, determine the makeup auxiliary area;

[0047] The makeup assistance area is displayed on the face in the smart makeup mirror, wherein the makeup position is determined based on the face part corresponding to the makeup assistance function command and the position of the face key points.

[0048] In some embodiments, determining the facial pose in a facial image captured by a smart makeup mirror includes:

[0049] The facial pose in the face image is determined based on a head pose estimation algorithm; or,

[0050] The facial pose in the facial image is determined based on the positional relationship between the contour key points that characterize the facial contour in the facial key points.

[0051] In some embodiments, determining the facial pose in the facial image based on the positional relationship between contour key points representing the facial contour in the facial key points includes:

[0052] Based on facial symmetry, multiple pairs of key points are selected from the contour key points, wherein each pair of key points is symmetrical about the facial midline, which divides the face into two equal parts.

[0053] The facial pose is determined based on a first distance between keypoints on one side of the face in a plurality of keypoint pairs, and a second distance between keypoints on the other side of the face in a plurality of keypoint pairs.

[0054] In some embodiments, determining the facial pose in the facial image captured by the smart makeup mirror according to the facial feature determination rule corresponding to the makeup assistance function instruction includes:

[0055] If the facial feature corresponding to the makeup assistance function instruction is the eyebrows, and the ratio of the first distance to the second distance is less than or equal to a first threshold, then the facial pose is determined to be a frontal view; otherwise, the facial pose is determined to be a side view, wherein the first distance is greater than or equal to the second distance; or,

[0056] If the facial feature corresponding to the makeup assistance function instruction is the cheek, and the ratio of the first distance to the second distance is less than or equal to the second threshold, then the facial posture is determined to be a frontal face. If the ratio is greater than the second threshold and less than or equal to the third threshold, then the facial posture is determined to be a small profile. If the ratio is greater than the third threshold, then the facial posture is determined to be a large profile, wherein the first distance is greater than or equal to the second distance.

[0057] In some embodiments, determining the makeup model corresponding to the facial features and the facial pose includes:

[0058] If the facial feature is the eyebrows and the facial pose is a frontal view, then the makeup model is determined to be a frontal eyebrow makeup model, wherein the auxiliary eyebrow areas on both sides of the frontal eyebrow makeup model are symmetrical; or,

[0059] If the facial feature is the eyebrows and the facial pose is a side profile, then the makeup model is determined to be a side profile eyebrow makeup model. In this side profile eyebrow makeup model, one side of the eyebrow auxiliary area is a half-eyebrow, and the arch of the other side's eyebrow auxiliary area is less pronounced than the arch of the eyebrow auxiliary area in the front-face eyebrow makeup model. The tail of the other side's eyebrow auxiliary area is longer than the tail of the eyebrow auxiliary area in the front-face eyebrow makeup model; or...

[0060] If the facial feature is the cheek and the facial pose is a frontal view, then the makeup model is determined to be a frontal face makeup model, wherein the auxiliary makeup areas on both sides of the frontal face makeup model are symmetrical; or,

[0061] If the facial feature is the cheek and the facial pose is a small profile, then the makeup model is determined to be a small profile makeup model, wherein the auxiliary makeup area on one side of the small profile makeup model is smaller than the auxiliary makeup area on the other side; or,

[0062] If the face part is the cheek and the face posture is a large side profile, then the makeup model is determined to be a large side profile makeup model, wherein the large side profile makeup model only has one side profile makeup auxiliary area.

[0063] In some embodiments, determining the makeup auxiliary region based on the makeup model and the facial key point positions in the facial image includes:

[0064] Based on the area determined by the location of the facial key points, the makeup model is scaled proportionally to determine the makeup auxiliary area.

[0065] In some embodiments, displaying the makeup assistance area at the makeup application position of the face in the smart makeup mirror includes:

[0066] The makeup auxiliary area is rotated according to the rotation angle of the makeup position relative to the horizontal line;

[0067] The rotated makeup assistance area is displayed at the makeup application position.

[0068] Fourthly, an embodiment of the present invention provides a method for displaying auxiliary makeup, comprising:

[0069] In response to the makeup assistance function command of the smart makeup mirror, a makeup assistance area corresponding to the facial posture of the person is displayed at the makeup position on the face in the smart makeup mirror in the following manner:

[0070] If the face is in a frontal position, a makeup assistance area is displayed at the makeup application position on the frontal face. The makeup assistance area is used to indicate the makeup application position on the frontal face and is attached to the makeup application position on the frontal face.

[0071] If the face is in profile, a makeup assistance area is displayed at the makeup application position on the profile, wherein the makeup assistance area is used to indicate the makeup application position on the profile and fits onto the makeup application position on the profile.

[0072] The makeup application location is determined based on the facial features and key facial points corresponding to the makeup assistance function command.

[0073] Fifthly, embodiments of the present invention also provide a device for displaying auxiliary makeup, comprising:

[0074] A pose determination unit is used to respond to the makeup assistance function command of the smart makeup mirror and determine the face pose in the face image captured by the smart makeup mirror according to the judgment rule of the face part corresponding to the makeup assistance function command.

[0075] A model determination unit is used to determine a makeup model corresponding to the face part and the face pose based on the face part corresponding to the makeup assistance function instruction.

[0076] A region determination unit is used to determine a makeup auxiliary region based on the makeup model and the positions of facial key points in the face image;

[0077] The display area unit is used to display the makeup assistance area at the makeup position of the face in the smart makeup mirror, wherein the makeup position is determined based on the face part corresponding to the makeup assistance function command and the position of the face key points.

[0078] In some embodiments, the attitude determination unit is specifically used for:

[0079] The facial pose in the face image is determined based on a head pose estimation algorithm; or,

[0080] The facial pose in the facial image is determined based on the positional relationship between the contour key points that characterize the facial contour in the facial key points.

[0081] In some embodiments, the attitude determination unit is specifically used for:

[0082] Based on facial symmetry, multiple pairs of key points are selected from the contour key points, wherein each pair of key points is symmetrical about the facial midline, which divides the face into two equal parts.

[0083] The facial pose is determined based on a first distance between keypoints on one side of the face in a plurality of keypoint pairs, and a second distance between keypoints on the other side of the face in a plurality of keypoint pairs.

[0084] In some embodiments, the attitude determination unit is specifically used for:

[0085] If the facial feature corresponding to the makeup assistance function instruction is the eyebrows, and the ratio of the first distance to the second distance is less than or equal to a first threshold, then the facial pose is determined to be a frontal view; otherwise, the facial pose is determined to be a side view, wherein the first distance is greater than or equal to the second distance; or,

[0086] If the facial feature corresponding to the makeup assistance function instruction is the cheek, and the ratio of the first distance to the second distance is less than or equal to the second threshold, then the facial posture is determined to be a frontal face. If the ratio is greater than the second threshold and less than or equal to the third threshold, then the facial posture is determined to be a small profile. If the ratio is greater than the third threshold, then the facial posture is determined to be a large profile, wherein the first distance is greater than or equal to the second distance.

[0087] In some embodiments, the determining model unit is specifically used for:

[0088] If the facial feature is the eyebrows and the facial pose is a frontal view, then the makeup model is determined to be a frontal eyebrow makeup model, wherein the auxiliary eyebrow areas on both sides of the frontal eyebrow makeup model are symmetrical; or,

[0089] If the facial feature is the eyebrows and the facial pose is a side profile, then the makeup model is determined to be a side profile eyebrow makeup model. In this side profile eyebrow makeup model, one side of the eyebrow auxiliary area is a half-eyebrow, and the arch of the other side's eyebrow auxiliary area is less pronounced than the arch of the eyebrow auxiliary area in the front-face eyebrow makeup model. The tail of the other side's eyebrow auxiliary area is longer than the tail of the eyebrow auxiliary area in the front-face eyebrow makeup model; or...

[0090] If the facial feature is the cheek and the facial pose is a frontal view, then the makeup model is determined to be a frontal face makeup model, wherein the auxiliary makeup areas on both sides of the frontal face makeup model are symmetrical; or,

[0091] If the facial feature is the cheek and the facial pose is a small profile, then the makeup model is determined to be a small profile makeup model, wherein the auxiliary makeup area on one side of the small profile makeup model is smaller than the auxiliary makeup area on the other side; or,

[0092] If the face part is the cheek and the face posture is a large side profile, then the makeup model is determined to be a large side profile makeup model, wherein the large side profile makeup model only has one side profile makeup auxiliary area.

[0093] In some embodiments, the determining region unit is specifically used for:

[0094] Based on the area determined by the location of the facial key points, the makeup model is scaled proportionally to determine the makeup auxiliary area.

[0095] In some embodiments, the display area unit is specifically used for:

[0096] The makeup auxiliary area is rotated according to the rotation angle of the makeup position relative to the horizontal line;

[0097] The rotated makeup assistance area is displayed at the makeup application position.

[0098] Fourthly, embodiments of the present invention also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in the first aspect above.

[0099] These or other aspects of this application will become more apparent in the following description of embodiments. Attached Figure Description

[0100] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0101] Figure 1 A schematic diagram of a smart makeup mirror provided in an embodiment of the present invention;

[0102] Figure 2 This is a schematic diagram of facial key point annotation provided in an embodiment of the present invention;

[0103] Figure 3A This is a schematic diagram of facial key points required to determine facial pose, provided by an embodiment of the present invention.

[0104] Figure 3B A 3D schematic diagram of head posture provided for an embodiment of the present invention;

[0105] Figure 4 This is a schematic diagram of facial key points required to determine facial pose, provided by an embodiment of the present invention.

[0106] Figure 5 A side profile diagram provided for an embodiment of the present invention;

[0107] Figure 6 This is a schematic diagram of a frontal eyebrow makeup model provided in an embodiment of the present invention;

[0108] Figure 7 This is a schematic diagram of a side profile eyebrow makeup model provided in an embodiment of the present invention;

[0109] Figure 8 This is a schematic diagram of a frontal facial makeup model provided in an embodiment of the present invention;

[0110] Figure 9 This is a schematic diagram of a small side profile makeup model provided in an embodiment of the present invention;

[0111] Figure 10This is a schematic diagram of a large side profile makeup model provided in an embodiment of the present invention;

[0112] Figure 11 A schematic diagram illustrating the construction process of a frontal eyebrow makeup model provided in an embodiment of the present invention;

[0113] Figure 12 This is a schematic diagram illustrating the construction of a half-eyebrow model according to an embodiment of the present invention;

[0114] Figure 13 A comparative schematic diagram of a front and side profile eyebrow makeup model provided in an embodiment of the present invention;

[0115] Figure 14 A schematic diagram illustrating the construction of a blush model provided in an embodiment of the present invention;

[0116] Figure 15 This is a schematic diagram of an eyebrow makeup model adjustment provided in an embodiment of the present invention;

[0117] Figure 16 A flowchart of an adaptive makeup assistance method provided in an embodiment of the present invention;

[0118] Figure 17 A schematic diagram of another smart makeup mirror provided in an embodiment of the present invention;

[0119] Figure 18 This is a flowchart illustrating an embodiment of a method for displaying auxiliary makeup according to an embodiment of the present invention;

[0120] Figure 19 This is a schematic diagram of a device for displaying auxiliary makeup according to an embodiment of the present invention. Detailed Implementation

[0121] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0122] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0123] The application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0124] It should be noted that the smart makeup mirror in this embodiment is equipped with a display screen and a processor, wherein the display screen is connected to the processor. In one possible implementation, the display screen in the smart makeup mirror can be a display screen with an integrated mirror (i.e., a mirror display screen). Specifically, an integrated mirror display screen has a mirror material disposed on the outer surface of the display panel. When the display screen is off, the smart makeup mirror functions similarly to a regular mirror and can be used for grooming. When the display screen is on, it can display content for intelligent interaction with the user. The user can also select the display mode, such as a full-screen display mode or a half-screen display mode. If the user selects a half-screen display mode, the upper half of the smart makeup mirror display can display content for intelligent interaction with the user, while the lower half of the display screen functions similarly to a regular mirror and can be used for grooming. Alternatively, the lower half of the smart makeup mirror display can display content for intelligent interaction with the user, while the upper half functions similarly to a regular mirror and can be used for grooming. If the user selects a full-screen display mode, the content for intelligent interaction with the user can be displayed in full screen. The makeup assistance display method provided in this embodiment is based on the aforementioned smart makeup mirror. In order to achieve display control of the smart makeup mirror, this embodiment provides a makeup assistance display method applied to the smart makeup mirror. It can display auxiliary makeup on the display screen of the smart makeup mirror according to the different postures of the user's face during the process of applying makeup using the mirror function of the smart makeup mirror, thereby providing auxiliary functions for the user's makeup application and improving the user experience.

[0125] In some embodiments, this invention provides a makeup application scenario using a smart makeup mirror. In this scenario, when a user applies makeup using the smart makeup mirror, they first select the area requiring makeup assistance, such as the eyebrows. The smart makeup mirror responds to the makeup assistance function command and determines to assist with makeup application on the eyebrows in the captured facial image. First, the user's facial posture is determined. Based on the eyebrows and facial posture, a makeup model corresponding to the eyebrows and facial posture is determined. For example, the makeup model corresponding to the eyebrows and side profile posture is determined to be a side profile eyebrow makeup model. Based on the makeup model and the key points of the face, an assistance area, namely the eyebrow makeup area, is determined and displayed on the eyebrow position of the face shown in the smart makeup mirror. The user only needs to apply makeup to the displayed eyebrow makeup area.

[0126] The core idea of ​​this embodiment is to first determine the current facial posture and select the makeup model corresponding to different postures for makeup assistance. This allows the displayed makeup assistance area to be adaptively adjusted according to the flexibility of the facial posture during makeup application, so that the flat makeup assistance area can better match the facial features presented in the smart makeup mirror, thus satisfying the user experience in multiple ways.

[0127] like Figure 1 As shown in the diagram, this embodiment of the invention provides a smart makeup mirror, including a camera 100, a plane mirror 101, a display screen 102, and a processor 103, wherein:

[0128] The camera 100 is used to capture human faces;

[0129] The plane mirror 101 is used to display the face of the user using the smart makeup mirror using the imaging principle; wherein, the plane mirror is a plane mirror made of mirror material;

[0130] The display screen 102 is used to display a makeup assistance area; in this embodiment, the makeup assistance area can be the outer outline of a makeup or a complete makeup. As long as it can provide makeup assistance to the user, any type of makeup assistance area is within the protection scope of this invention.

[0131] In this embodiment, the plane mirror can be integrated with the display screen to form a single mirror display, or they can be two separate components with the plane mirror positioned above the display screen. The plane mirror and the display screen are the same size and can be combined. When the display screen is off, the smart makeup mirror functions similarly to a regular mirror, providing ordinary mirror imaging. When the display screen is on, the smart makeup mirror can select part or all of the display screen to show content that allows for intelligent interaction with the user, such as displaying makeup assistance areas that the user needs.

[0132] The processor 103 is configured to execute:

[0133] 1) In response to the makeup assistance function command of the smart makeup mirror, determine the facial pose in the facial image captured by the smart makeup mirror according to the facial feature judgment rule corresponding to the makeup assistance function command;

[0134] In this embodiment, the facial pose includes, but is not limited to, frontal face and side face, and the side face can be further divided into small side face and large side face.

[0135] In some implementations, prior to responding to makeup assistance commands from a smart makeup mirror, the following also applies:

[0136] Based on a facial landmark detection algorithm, the location of facial landmarks is detected. Optionally, the Practical Facial Landmark Detector (PFLD) algorithm can be used. PFLD is a high-precision, fast, and small-model facial landmark detection model that achieves ultra-real-time performance on mobile devices (model size 2.1Mb, 140fps on a Qualcomm ARM 845 processor), capable of detecting the locations of 106 facial landmarks. Using the 106 landmark annotation data, details such as eye contours are located, for example... Figure 2 As shown.

[0137] 2) Based on the facial features and facial pose corresponding to the makeup assistance function instructions, determine the makeup model corresponding to the facial features and facial pose;

[0138] In practice, firstly, different facial features correspond to different makeup models. For example, the makeup auxiliary areas determined by the eyebrow makeup model and the face makeup model are necessarily different. Furthermore, the embodiments of the present invention can further distinguish the same makeup model based on different facial poses corresponding to the same makeup model. That is, the frontal face pose corresponds to the frontal face eyebrow makeup model, and the side face pose corresponds to the side face eyebrow makeup model.

[0139] 3) Determine the makeup auxiliary area based on the makeup model and the location of facial key points in the face image;

[0140] It should be noted that the makeup model in this embodiment is a standardized model, and it still needs to be adjusted according to different facial features. The adjustment method needs to be based on the key points of the face.

[0141] 4) The makeup assistance area is displayed at the makeup position of the face in the smart makeup mirror, wherein the makeup position is determined based on the face part corresponding to the makeup assistance function command and the position of the face key points.

[0142] In this embodiment, the makeup assistance area can change with the change of the face posture. Regardless of whether the face posture in the smart makeup mirror is a frontal or side view, the corresponding makeup assistance area can be displayed on the face of the smart makeup mirror, and its size and position can be matched with the makeup position of the face.

[0143] In practice, if the facial feature corresponding to the makeup assistance function command is the eyebrows, the makeup application position on the face is determined based on the key points of the eyebrows; that is, the makeup application position is the eyebrows. After determining the makeup assistance area, it is necessary to display the makeup assistance area at the makeup application position. In this embodiment, the makeup assistance area is initially determined based on the horizontal line, and then adjusted according to the rotation angle of the user's eyebrows relative to the horizontal line before displaying. This ensures that the makeup assistance area finally displayed at the makeup application position can perfectly match the user's makeup application position.

[0144] In some embodiments, the rules for determining the facial features corresponding to different makeup assistance function instructions can be the same or different, and the embodiments of the present invention do not impose excessive limitations. It is easy to understand that during the makeup process for a frontal and side profile, there are certain differences in the determination of the frontal and side profile for different makeup methods for eyebrows and cheeks. For example, regardless of the degree of side profile, the eyebrow makeup on one side can be set as half-eyebrow makeup, and the eyebrow makeup on the other side can be set as full-eyebrow makeup. However, if the side profile is very large, the cheek makeup on the side showing very little cheek area may not be assisted, and only the side showing more cheek area will be assisted. In actual makeup application, for example, when applying blush to one side of the face, there is no need to assist the other side. Therefore, the rules for determining the frontal or side profile can be different depending on the facial features, and different rules can be set to determine the facial posture corresponding to different facial features during makeup assistance.

[0145] In some embodiments, this embodiment provides the following method for determining facial pose:

[0146] Method 1: Determine the face pose in the face image based on a head pose estimation algorithm;

[0147] In implementation, the head pose estimation algorithm is used to obtain the head's pose angles from a facial image. In 3D space, the rotation of an object can be represented by three Euler angles: pitch (rotation around the X-axis), yaw (rotation around the Y-axis), and roll (rotation around the Z-axis), commonly referred to as head tilt, head shake, and head turn. The execution steps of the head pose estimation algorithm include:

[0148] For the calibration camera, the head pose angle is solved by the transformation relationship matrix between the coordinates of n points of the face in the 3D world coordinate system and the corresponding point set projected into the 2D image coordinate system.

[0149] During implementation, such as Figure 3AAs shown, five 2D keypoints (points 1, 11, 34, 46, and 47) from the facial keypoints are used. Then, the POSIT pose estimation algorithm in OpenCV projects the five 3D keypoints in the world coordinate system onto these five 2D keypoints through transformations such as rotation and translation. The transformation matrix is ​​then estimated, and finally, the head pose angles in the 2D plane are obtained. Figure 3B As shown, these are Yaw, Pitch, and Roll, respectively.

[0150] Based on the facial feature determination rules corresponding to the makeup assistance function commands, thresholds are set for the facial features corresponding to the makeup assistance function commands, namely the yaw threshold, pitch threshold, and roll threshold. The facial pose in the face image is determined based on the relationship between the determined head pose angle and the threshold values. The threshold values ​​can be different for different facial features.

[0151] Method 2: Determine the facial pose in the facial image based on the positional relationship between the contour key points that represent the facial contour in the facial key points.

[0152] This method determines facial pose based on the positional relationship between key contour points, and it is simple and effective.

[0153] In some embodiments, facial pose is determined based on the positional relationship between contour key points in the following manner:

[0154] 1) Based on facial symmetry, select multiple pairs of key points from the contour key points, wherein each pair of key points is symmetrical about the facial midline, which divides the face into two equal parts;

[0155] During implementation, two representative pairs of contour key points can be selected, such as... Figure 4 As shown, a pair of contour key points, 80 and 88, are used to represent the outermost contour of the face, and another pair of key points, 38 and 39, are used to represent the nose contour.

[0156] 2) Determine the face pose based on the first distance between key points on one side of the face in multiple key point pairs, and the second distance between key points on the other side of the face in multiple key point pairs.

[0157] If multiple keypoints exist on one side of the face, two of them can be selected according to preset rules to calculate the first distance. Similarly, if multiple keypoints exist on the other side of the face, two of them can be selected according to preset rules to calculate the second distance. Optionally, the two farthest keypoints on one side of the face can be selected to calculate the first distance, and the two farthest keypoints on the other side of the face can be selected to calculate the second distance.

[0158] In implementation, since the human face is symmetric, there is a preset relationship among the 80th point, 38th point, 39th point, and 88th point of the facial key points. In a frontal face scenario, the distance from point 80 to point 38 is set as D1, and the distance from point 39 to point 88 is set as D2. Based on the symmetry of the human face, when it is a frontal face, the ratio of D1 / D2 is close to 1. If the difference between the ratio of D1 / D2 and 1 is large, it is determined that a side face situation has occurred. The degree of the side face will affect the display effect of the makeup assistance area.

[0159] In some embodiments, a slight side face has little impact on cheek assistance and eyebrow makeup assistance. However, if a side face situation as shown in Figure 5 occurs, the frontal face makeup model has lost the meaning of makeup assistance, and it will also affect the aesthetic feeling visually. For the cheek makeup assistance situation, when the degree of the side face is large, the user mainly applies makeup to one side of the face, and on the other side, the face is very small or almost non-existent in the display screen of the intelligent makeup mirror. For this situation, a strategy of not performing makeup assistance can be adopted. In this embodiment, the frontal face and side face are judged and corresponding makeup assistance is performed, effectively optimizing the effect of makeup assistance and enhancing the reliability of makeup assistance.

[0160] In some embodiments, according to the different facial parts corresponding to the makeup assistance function instruction, different rules for judging the frontal face and side face are set according to the first distance and the second distance, as follows:

[0161] 21) The facial part corresponding to the makeup assistance function instruction is the eyebrow part:

[0162] If the ratio of the first distance to the second distance is less than or equal to the first threshold, it is determined that the facial pose is a frontal face;

[0163] If the ratio of the first distance to the second distance is greater than the first threshold, it is determined that the facial pose is a side face;

[0164] where the first distance is greater than or equal to the second distance;

[0165] In implementation, when D1 > D'2:

[0166] If D1 / D2 <= 1.3, it is determined that the facial pose is a frontal face.

[0167] If D1 / D2 > 1.3, it is determined that the facial pose is a side face.

[0168] When D1 < D2:

[0169] If D2 / D1 <= 1.3, it is determined that the facial pose is a frontal face.

[0170] If D2 / D1 > 1.3, it is determined that the facial pose is a side face.

[0171] 22) The facial part corresponding to the makeup assistance function instruction is the cheek:

[0172] If the ratio of the first distance to the second distance is less than or equal to the second threshold, determine that the facial pose is a frontal face;

[0173] If the ratio is greater than the second threshold and less than or equal to the third threshold, determine that the facial pose is a small side face;

[0174] If the ratio is greater than the third threshold, determine that the facial pose is a large side face;

[0175] Where the first distance is greater than or equal to the second distance.

[0176] In implementation, when D1 > D2:

[0177] If D1 / D2 <= 1.4, determine that the facial pose is a frontal face.

[0178] If D1 / D2 <= 1.6, determine that the facial pose is a small side face.

[0179] If D1 / D2 > 1.6, determine that the facial pose is a large side face. For the side face with very small or almost no facial information in the picture of the intelligent makeup mirror, no makeup assistance is performed, and only makeup assistance is performed on the other side face.

[0180] When D1 < D2:

[0181] D2 / D1 <= 1.4, and this state is determined to be a frontal face.

[0182] D2 / D1 <= 1.6, and this state is determined to be a small side face.

[0183] D2 / D1 > 1.6, and this state is determined to be a large side face. For the side face with very small or almost no facial information in the picture of the intelligent makeup mirror, no makeup assistance is performed, and only makeup assistance is performed on the other side face.

[0184] In some embodiments, according to the facial part corresponding to the makeup assistance function instruction and the facial pose, the makeup model corresponding to the facial part and the facial pose is determined by the following method:

[0185] Method 1: If the facial part is the eyebrow part and the facial pose is a frontal face, determine that the makeup model is a frontal face eyebrow makeup model;

[0186] As Figure 6 shown, the two eyebrow shape assistance areas in the frontal face eyebrow makeup model are symmetrical.

[0187] Method 2: If the face part is the eyebrows and the face posture is a side profile, then the makeup model is determined to be a side profile eyebrow makeup model.

[0188] like Figure 7 As shown, one side of the eyebrow auxiliary area in the side profile eyebrow makeup model is a half eyebrow, and the arch of the other side eyebrow auxiliary area is smaller than that of the eyebrow auxiliary area in the front profile eyebrow makeup model. The tail of the other side eyebrow auxiliary area is longer than that of the eyebrow auxiliary area in the front profile eyebrow makeup model.

[0189] Method 3: If the face part is the cheek and the face posture is a frontal view, then the makeup model is determined to be a frontal face makeup model;

[0190] like Figure 8 As shown, the auxiliary makeup areas on both sides of the face in the front face makeup model are symmetrical;

[0191] Method 4: If the face part is the cheek and the face posture is a small profile, then the makeup model is determined to be a small profile face makeup model.

[0192] like Figure 9 As shown, the auxiliary makeup area on one side of the face in the small side face makeup model is smaller than the auxiliary makeup area on the other side.

[0193] Method 5: If the face part is the cheek and the face posture is a large side profile, then the makeup model is determined to be a large side profile makeup model.

[0194] like Figure 10 As shown, the large side face makeup model only has one side face makeup auxiliary area.

[0195] In this embodiment, the construction process of the front-face eyebrow makeup model or the side-face eyebrow makeup model is similar. The construction process of the single-sided eyebrow makeup model is described in detail below:

[0196] In this embodiment, to more accurately depict the effects of different eyebrow shapes, a 42-point model is used to construct the eyebrow outline. For ease of explanation, the following uses a 28-point model as an example to illustrate the eyebrow makeup model construction process, specifically starting from the inner corner of the eyebrow to construct the eyebrow shape.

[0197] The following explanation uses the construction of a frontal eyebrow makeup model as an example:

[0198] like Figure 11As shown, this embodiment provides a construction process for a frontal eyebrow makeup model. Point A(X1, Y1) is the eyebrow initiation point, and point B(X2, Y2) is the eyebrow tail point. The horizontal line containing A and B is L2. Starting from point A, the upper eyebrow contour point C(X3, Y3) is constructed upwards. First, the area between points A and B is divided into 14 equal parts, each with a length of d. Then, the coordinates of point C(X3, Y3) are: X3 = X1 + d; Y3 = Y1 – tan(θ)d. The coordinates of point D(X4, Y4) can be further obtained based on the coordinates of point C, resulting in the coordinates of point D as: X4 = X3 + d; Y4 = Y3 – tan(θ')d. Here, θ and θ' are parameters adjusted based on eyebrow shape characteristics, representing the direction of the eyebrow shape. θ and θ' are determined based on the eyebrow shape features selected by the user, or based on eyebrow shape features recommended to the user. Similarly, the position information of other points can be obtained. The eyebrow shaping process on the other side is the same as described above, and will not be repeated here. The eyebrow makeup model provided in this embodiment is an eyebrow shape of a preset standard size, which can be adaptively adjusted based on the user's eyebrow size, that is, adaptive scaling adjustment can be achieved based on the length of points A and B.

[0199] The following explanation uses the construction of a side profile eyebrow makeup model as an example:

[0200] This embodiment provides a process for constructing a profile eyebrow makeup model. To ensure the eyebrow makeup model can adaptively adjust to changes in facial pose, this embodiment reconstructs the profile eyebrow makeup model, where one side of the eyebrow makeup uses a half-eyebrow model, such as... Figure 12 As shown, a 16-point construction method is used. The specific construction process can be found in the construction process of the aforementioned frontal eyebrow makeup model, and will not be repeated here. The effect of the eyebrow makeup on the other side differs from the effect of the same side eyebrow makeup in the frontal eyebrow makeup model. In implementation, the arch of the other eyebrow shape is reduced compared to the arch of the eyebrow shape in the frontal eyebrow makeup model, and the tail of the other eyebrow shape is extended compared to the tail of the eyebrow shape in the frontal eyebrow makeup model. It is easy to understand that, as... Figure 13 As shown, the tail of the eyebrow is compressed when facing forward, but it becomes longer when viewed from the side, and the arch of the eyebrow becomes less pronounced. In other words, when viewed from the side, the brow peak is moved forward, and the tail of the eyebrow naturally lengthens.

[0201] This embodiment also provides a facial makeup model construction process, taking blush makeup assistance as an example, that is, the construction of a blush model as an example for explanation. The construction process of the front-face blush model and the side-face blush model in this embodiment are similar, the difference being the selection of the blush starting point.

[0202] The following explanation uses the construction process of a front-facing blush model as an example:

[0203] In some embodiments, the blush model uses 42 points to construct the blush outline. For ease of explanation, an example using 16 points will be provided. Figure 14 As shown. The starting point A(X1, Y1) is the point between the wing of the nose and the contour points of the face. There is a certain proportional relationship between the starting point A, the wing of the nose, and the contour points. Points B(X2, Y2), C(X3, Y3), and D(X4, Y4) are three defined contour points. For the blush model, due to the diversity of blush types, A, B, C, and D are four points determined based on the blush type. To construct the blush model, points A and B are first on a horizontal line, the height from point C to AB is H1, and the height from point D to the line AB is H2.

[0204] The blush model for the upper half of AB is constructed as follows: First, the area between points A and B is divided into 8 equal parts, each with a length of d. The coordinates of point E (X5, Y5) are: X5 = X1 + d; Y5 = Y1 – tan(θ)d; where θ is a parameter determined based on the blush type, used to characterize the shape of the blush. Similarly, the blush model for the lower half of AB can be constructed. The constructed blush model is a preset standard size and can also be adaptively adjusted according to different users' face shapes. That is, the blush model can adaptively scale and adjust based on the length of points A and B, as well as the vertical width. Subsequent adjustments based on different users' face shapes can adaptively adjust the length and vertical width of the blush model.

[0205] The principle behind the above eyebrow makeup model construction process lies in determining the starting and ending points, and determining θ and θ' based on the eyebrow makeup type. The construction process of the side-profile blush model in this example follows the same principle as the eyebrow makeup model, and will not be repeated here. The difference between constructing the side-profile blush model and the eyebrow makeup model lies in the selection of the blush starting point A. The selection of the blush contour points is related to the shape and type of blush. Specifically, there is a certain proportional relationship between the selection of the starting point A and the distance from the wing of the nose (point 39) to the contour point (point 88), and this proportion needs to be adjusted according to the actual size of the human face. Figure 8-10 The image shows a schematic diagram of the blush model under different facial poses. Figure 9 A blush model for a small profile. Figure 10 A blush model for a large side profile. Figure 8 This is a model of blush on the front of the face.

[0206] After determining the makeup model corresponding to the face region and the face posture through the above process, the makeup auxiliary area determined by the makeup model is a standard area, which still needs to be adaptively adjusted according to the key points of the face.

[0207] In some embodiments, after determining the makeup model, before generating the makeup auxiliary area, it is necessary to first determine the corresponding area on the face, that is, determine the area based on the location of the key facial points. Then, the makeup model is scaled proportionally based on the area to determine the makeup auxiliary area. For example, if the makeup model is determined to be a front-face eyebrow makeup model, the area where the eyebrows are located is determined based on the location of the key facial points. After scaling the front-face eyebrow makeup model proportionally, the front-face eyebrow makeup model is made to fit the eyebrow shape of the face perfectly, thus determining the final eyebrow makeup auxiliary area.

[0208] In some embodiments, this embodiment also provides one or more ways to adaptively adjust the makeup auxiliary area:

[0209] 1) Based on the area determined by the location of the facial key points, the makeup model is scaled proportionally to determine the makeup auxiliary area;

[0210] 2) Rotate the makeup auxiliary area according to the rotation angle of the makeup position relative to the horizontal line; display the rotated makeup auxiliary area at the makeup position.

[0211] The following explanation uses the adjustment of the auxiliary area for eyebrow makeup determined by the front-face eyebrow makeup model as an example:

[0212] Step 1: Obtain the positions of the starting point P26(x26, y26) and the ending point P27(x27, y27) of the right eyebrow makeup auxiliary area in the front eyebrow makeup model, wherein the starting point and the ending point are used to characterize the starting point and the ending point of the eyebrow makeup.

[0213] Step 2: Divide the distance between the starting point P26 and the ending point P27 into 14 equal parts, each with a length of d'.

[0214] Step 3: Obtain the key point positions representing the right eyebrow from the facial key point positions;

[0215] Step 4: Adjust the front eyebrow makeup model based on the acquired key point information.

[0216] 41) When y26 = y27:

[0217] d'=(x26-x27) / 14; X3'=x26+d'; Y3'=y26-tan(θ)d';

[0218] Where θ represents the known parameters obtained during the construction of the frontal eyebrow makeup model.

[0219] 42) When y26 > y27, the angle between the straight line determined by the starting point P26 and the ending point P27 and the X-axis (horizontal axis) is α, as shown below. Figure 15As shown in the figure. First, taking the starting point P26 as the center and the distance between the starting point P26 and the end point P27 as the radius, rotate the end point P27 horizontally to the point P27'. Based on P26 and P27', adjust and adapt the frontal face eyebrow makeup model to the user's face shape. The calculation process is similar to that in step 41), and the specific process of obtaining the coordinates (X", Y") of the temporary point C" will not be elaborated here.

[0220] where d' = ((x26 - x27)^2 + (y26 - y27)^2)^(1 / 2) / 14; X" = x26 + d';

[0221] Y" = y26 - tan(θ)d';

[0222] After calculating all the points required for the eyebrow makeup auxiliary area determined by the frontal face eyebrow makeup model, rotate all the points. For example, for P27', with the distance from P26 to P27 as the radius r and P26 as the center, rotate it upward by an angle a to the position of P27. All other points are also rotated by an angle a, and rotate with P26 as the center and the distance from the points on the eyebrow makeup auxiliary area to P26 as the radius r, then the adaptive eyebrow makeup auxiliary area is obtained. Taking the rotation of the temporary point C"(X", Y") to the Figure 15 point C"(X3', Y3') in as an example for illustration:

[0223] r = ((X" - x26)^2 + (Y" - y26)^2)^(1 / 2); X3' = x26 + cos(θ + a) * r;

[0224] Y3' = y26 - sin(θ + a) * r;

[0225] 43) When y26 < y27, the angle formed by the line of P26 and P27 with the X-axis is -a. First, taking P26 as the center and the distance between P26 and P27 as the radius r, rotate P27 horizontally to the point 27'. Based on P26 and P27', adjust and adapt the frontal face eyebrow makeup model to the user's face shape. The calculation process is similar to that in step 41) and will not be elaborated here. The coordinates of the obtained temporary point C" are (X", Y").

[0226] d' = ((x26 - x27)^2 + (y26 - y27)^2)^(1 / 2) / 14; X" = x26 + d';

[0227] Y" = y26 - tan(θ)d'.

[0228] After calculating all points on the eyebrow makeup auxiliary area determined by the frontal eyebrow makeup model, all points are rotated. For example, P27' is rotated downwards by an angle 'a' with the distance from P26 to P27 as the radius 'r' and P26 as the center, to the position of P27. All other points are also rotated by an angle 'a', with P26 as the center and the distance from the point on the eyebrow makeup auxiliary area to P26 as the radius. This yields the adaptive eyebrow makeup auxiliary model. The temporary point C (X”, Y”) is rotated to... Figure 15 The calculation process of point C (X3', Y3') is illustrated using the following example:

[0229] Case 1: When θ > a:

[0230] r=((X”-x26)2+(Y”-y26)2)1 / 2;

[0231] X3' = x26 + cos(θ - a) * r;

[0232] Y3'=y26–sin(θ-a)*r.

[0233] Case 2, when θ <= a:

[0234] r=((X”-x26)2+(Y”-y26)2)1 / 2;

[0235] X3' = x26 + cos(a-θ)*r;

[0236] Y3'=y26+sin(a-θ)*r.

[0237] like Figure 16 As shown in the figure, this embodiment of the invention also provides an adaptive makeup assistance method applied to a smart makeup mirror. The specific implementation process of this method is as follows:

[0238] Step 1600: Take a picture of the face using the camera on the smart makeup mirror, and perform facial landmark detection based on the captured facial image to determine the facial landmarks;

[0239] Step 1601: In response to the makeup assistance function command of the smart makeup mirror, determine the facial pose in the facial image captured by the smart makeup mirror according to the facial feature judgment rule corresponding to the makeup assistance function command.

[0240] The facial poses include frontal and side profiles corresponding to the eyebrows; and frontal, small side profile, and large side profiles corresponding to the face.

[0241] Step 1602: Based on the facial features and facial pose corresponding to the makeup assistance function command, determine the makeup model corresponding to the facial features and facial pose.

[0242] Step 1603: Based on the area determined by the location of the facial key points, scale the makeup model proportionally to determine the makeup auxiliary area;

[0243] Step 1604: Rotate the makeup auxiliary area according to the rotation angle of the makeup position relative to the horizontal line, and display the rotated makeup auxiliary area at the makeup position.

[0244] In some examples, embodiments of the present invention are based on the same inventive concept, such as Figure 17 As shown, a smart makeup mirror is also provided, including a camera 1700, a plane mirror 1701, a display screen 1702, and a processor 1703, wherein:

[0245] The camera 1700 is used to capture faces;

[0246] The plane mirror 1701 is used to display the face of the user using the smart makeup mirror using the imaging principle;

[0247] The display screen 1702 is used to display the makeup assistance area;

[0248] The processor 1703 is configured to execute:

[0249] In response to the makeup assistance function command of the smart makeup mirror, a makeup assistance area corresponding to the facial posture of the person is displayed at the makeup position on the face in the smart makeup mirror in the following manner:

[0250] If the face is in a frontal position, a makeup assistance area is displayed at the makeup application position on the frontal face. The makeup assistance area is used to indicate the makeup application position on the frontal face and is attached to the makeup application position on the frontal face.

[0251] If the face is in profile, a makeup assistance area is displayed at the makeup application position on the profile, wherein the makeup assistance area is used to indicate the makeup application position on the profile and fits onto the makeup application position on the profile.

[0252] It should be noted that this embodiment can display different makeup assistance areas based on different facial postures. Since people often need to turn their faces during makeup application to see the makeup area more clearly, the makeup assistance areas required for frontal and side-facing makeup applications will inevitably be different. This embodiment can adjust the displayed makeup assistance area at any time according to the current facial posture. No matter how the user's facial posture changes, the displayed makeup assistance area can be perfectly aligned with the face in the smart makeup mirror, improving the user's makeup experience.

[0253] The makeup application location is determined based on the facial features and key facial points corresponding to the makeup assistance function command.

[0254] This embodiment displays different makeup assistance areas based on different facial postures. In practical applications, the fit of the makeup assistance areas can be adjusted in real time based on different facial postures. From the user's perspective, the displayed makeup assistance areas can adaptively correct themselves according to the user's different facial postures to fit the user's makeup application goals as closely as possible, greatly improving the user's makeup experience. The specific structure of the smart makeup mirror in this embodiment can refer to the structure of any of the smart makeup mirrors in the above embodiments, and will not be repeated here.

[0255] Based on the same inventive concept, this embodiment of the invention also provides a method for displaying auxiliary makeup. Since this method is the implementation method corresponding to the smart makeup mirror in this embodiment of the invention, and the principle of solving the problem by this method is similar to that of the smart makeup mirror, the implementation of this method can refer to the implementation of the smart makeup mirror, and the repeated parts will not be described again.

[0256] like Figure 18 As shown in the figure, the implementation flow of a method for displaying auxiliary makeup provided by an embodiment of the present invention is as follows:

[0257] Step 1800: In response to the makeup assistance function command of the smart makeup mirror, determine the facial pose in the facial image captured by the smart makeup mirror according to the facial feature judgment rule corresponding to the makeup assistance function command.

[0258] Step 1801: Determine the makeup model corresponding to the face part and the face posture based on the face part corresponding to the makeup assistance function instruction.

[0259] Step 1802: Determine the makeup auxiliary area based on the makeup model and the location of facial key points in the face image;

[0260] Step 1803: Display the makeup assistance area at the makeup position of the face in the smart makeup mirror, wherein the makeup position is determined based on the face part corresponding to the makeup assistance function command and the position of the face key points.

[0261] In some embodiments, determining the facial pose in a facial image captured by a smart makeup mirror includes:

[0262] The facial pose in the face image is determined based on a head pose estimation algorithm; or,

[0263] The facial pose in the facial image is determined based on the positional relationship between the contour key points that characterize the facial contour in the facial key points.

[0264] In some embodiments, determining the facial pose in the facial image based on the positional relationship between contour key points representing the facial contour in the facial key points includes:

[0265] Based on facial symmetry, multiple pairs of key points are selected from the contour key points, wherein each pair of key points is symmetrical about the facial midline, which divides the face into two equal parts.

[0266] The facial pose is determined based on a first distance between keypoints on one side of the face in a plurality of keypoint pairs, and a second distance between keypoints on the other side of the face in a plurality of keypoint pairs.

[0267] In some embodiments, determining the facial pose in the facial image captured by the smart makeup mirror according to the facial feature determination rule corresponding to the makeup assistance function instruction includes:

[0268] If the facial feature corresponding to the makeup assistance function instruction is the eyebrows, and the ratio of the first distance to the second distance is less than or equal to a first threshold, then the facial pose is determined to be a frontal view; otherwise, the facial pose is determined to be a side view, wherein the first distance is greater than or equal to the second distance; or,

[0269] If the facial feature corresponding to the makeup assistance function instruction is the cheek, and the ratio of the first distance to the second distance is less than or equal to the second threshold, then the facial posture is determined to be a frontal face. If the ratio is greater than the second threshold and less than or equal to the third threshold, then the facial posture is determined to be a small profile. If the ratio is greater than the third threshold, then the facial posture is determined to be a large profile, wherein the first distance is greater than or equal to the second distance.

[0270] In some embodiments, determining the makeup model corresponding to the facial features and the facial pose includes:

[0271] If the facial feature is the eyebrows and the facial pose is a frontal view, then the makeup model is determined to be a frontal eyebrow makeup model, wherein the auxiliary eyebrow areas on both sides of the frontal eyebrow makeup model are symmetrical; or,

[0272] If the facial feature is the eyebrows and the facial pose is a side profile, then the makeup model is determined to be a side profile eyebrow makeup model. In this side profile eyebrow makeup model, one side of the eyebrow auxiliary area is a half-eyebrow, and the arch of the other side's eyebrow auxiliary area is less pronounced than the arch of the eyebrow auxiliary area in the front-face eyebrow makeup model. The tail of the other side's eyebrow auxiliary area is longer than the tail of the eyebrow auxiliary area in the front-face eyebrow makeup model; or...

[0273] If the facial feature is the cheek and the facial pose is a frontal view, then the makeup model is determined to be a frontal face makeup model, wherein the auxiliary makeup areas on both sides of the frontal face makeup model are symmetrical; or,

[0274] If the facial feature is the cheek and the facial pose is a small profile, then the makeup model is determined to be a small profile makeup model, wherein the auxiliary makeup area on one side of the small profile makeup model is smaller than the auxiliary makeup area on the other side; or,

[0275] If the face part is the cheek and the face posture is a large side profile, then the makeup model is determined to be a large side profile makeup model, wherein the large side profile makeup model only has one side profile makeup auxiliary area.

[0276] In some embodiments, determining the makeup auxiliary region based on the makeup model and the facial key point positions in the facial image includes:

[0277] Based on the area determined by the location of the facial key points, the makeup model is scaled proportionally to determine the makeup auxiliary area.

[0278] In some embodiments, displaying the makeup assistance area at the makeup application position of the face in the smart makeup mirror includes:

[0279] The makeup auxiliary area is rotated according to the rotation angle of the makeup position relative to the horizontal line;

[0280] The rotated makeup assistance area is displayed at the makeup application position.

[0281] Based on the same inventive concept, this embodiment of the invention also provides a device for displaying auxiliary makeup. Since this device is the same as the device corresponding to the smart makeup mirror in this embodiment of the invention, and the principle of solving the problem by this device is similar to that of the smart makeup mirror, the implementation of this device can refer to the implementation of the smart makeup mirror, and the repeated parts will not be described again.

[0282] like Figure 19 As shown, the device includes:

[0283] The pose determination unit 1900 is used to respond to the makeup assistance function command of the smart makeup mirror and determine the face pose in the face image captured by the smart makeup mirror according to the judgment rule of the face part corresponding to the makeup assistance function command.

[0284] Model unit 1901 is used to determine a makeup model corresponding to the face part and the face pose based on the face part corresponding to the makeup assist function instruction.

[0285] Region determination unit 1902 is used to determine the makeup auxiliary region based on the makeup model and the position of facial key points in the face image;

[0286] Display area unit 1903 is used to display the makeup assistance area at the makeup position of a face in the smart makeup mirror, wherein the makeup position is determined based on the face part corresponding to the makeup assistance function command and the position of the face key points.

[0287] In some embodiments, the attitude determination unit is specifically used for:

[0288] The facial pose in the face image is determined based on a head pose estimation algorithm; or,

[0289] The facial pose in the facial image is determined based on the positional relationship between the contour key points that characterize the facial contour in the facial key points.

[0290] In some embodiments, the attitude determination unit is specifically used for:

[0291] Based on facial symmetry, multiple pairs of key points are selected from the contour key points, wherein each pair of key points is symmetrical about the facial midline, which divides the face into two equal parts.

[0292] The facial pose is determined based on a first distance between keypoints on one side of the face in a plurality of keypoint pairs, and a second distance between keypoints on the other side of the face in a plurality of keypoint pairs.

[0293] In some embodiments, the attitude determination unit is specifically used for:

[0294] If the facial feature corresponding to the makeup assistance function instruction is the eyebrows, and the ratio of the first distance to the second distance is less than or equal to a first threshold, then the facial pose is determined to be a frontal view; otherwise, the facial pose is determined to be a side view, wherein the first distance is greater than or equal to the second distance; or,

[0295] If the facial feature corresponding to the makeup assistance function instruction is the cheek, and the ratio of the first distance to the second distance is less than or equal to the second threshold, then the facial posture is determined to be a frontal face. If the ratio is greater than the second threshold and less than or equal to the third threshold, then the facial posture is determined to be a small profile. If the ratio is greater than the third threshold, then the facial posture is determined to be a large profile, wherein the first distance is greater than or equal to the second distance.

[0296] In some embodiments, the determining model unit is specifically used for:

[0297] If the facial feature is the eyebrows and the facial pose is a frontal view, then the makeup model is determined to be a frontal eyebrow makeup model, wherein the auxiliary eyebrow areas on both sides of the frontal eyebrow makeup model are symmetrical; or,

[0298] If the facial feature is the eyebrows and the facial pose is a side profile, then the makeup model is determined to be a side profile eyebrow makeup model. In this side profile eyebrow makeup model, one side of the eyebrow auxiliary area is a half-eyebrow, and the arch of the other side's eyebrow auxiliary area is less pronounced than the arch of the eyebrow auxiliary area in the front-face eyebrow makeup model. The tail of the other side's eyebrow auxiliary area is longer than the tail of the eyebrow auxiliary area in the front-face eyebrow makeup model; or...

[0299] If the facial feature is the cheek and the facial pose is a frontal view, then the makeup model is determined to be a frontal face makeup model, wherein the auxiliary makeup areas on both sides of the frontal face makeup model are symmetrical; or,

[0300] If the facial feature is the cheek and the facial pose is a small profile, then the makeup model is determined to be a small profile makeup model, wherein the auxiliary makeup area on one side of the small profile makeup model is smaller than the auxiliary makeup area on the other side; or,

[0301] If the face part is the cheek and the face posture is a large side profile, then the makeup model is determined to be a large side profile makeup model, wherein the large side profile makeup model only has one side profile makeup auxiliary area.

[0302] In some embodiments, the determining region unit is specifically used for:

[0303] Based on the area determined by the location of the facial key points, the makeup model is scaled proportionally to determine the makeup auxiliary area.

[0304] In some embodiments, the display area unit is specifically used for:

[0305] The makeup auxiliary area is rotated according to the rotation angle of the makeup position relative to the horizontal line;

[0306] The rotated makeup assistance area is displayed at the makeup application position.

[0307] Based on the same inventive concept, embodiments of the present invention also provide a computer storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0308] In response to the makeup assistance function command of the smart makeup mirror, the facial pose in the facial image captured by the smart makeup mirror is determined according to the facial feature judgment rule corresponding to the makeup assistance function command.

[0309] Based on the facial features and facial pose corresponding to the makeup assistance function command, determine the makeup model corresponding to the facial features and facial pose;

[0310] Based on the makeup model and the positions of facial key points in the facial image, determine the makeup auxiliary area;

[0311] The makeup assistance area is displayed on the face in the smart makeup mirror, wherein the makeup position is determined based on the face part corresponding to the makeup assistance function command and the position of the face key points.

[0312] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0313] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0314] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0315] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0316] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A smart makeup mirror, characterized in that, Includes a camera, a flat mirror, a display screen, and a processor, among which: The camera is used to capture faces; The plane mirror is used to display the face of the user using the smart makeup mirror using imaging principles; The display screen is used to display the makeup assistance area; The processor is configured to execute: In response to the makeup assistance function command of the smart makeup mirror, the facial pose in the facial image captured by the smart makeup mirror is determined according to the facial feature judgment rule corresponding to the makeup assistance function command. Based on the facial features and facial pose corresponding to the makeup assistance function command, determine the makeup model corresponding to the facial features and facial pose; If the starting and ending coordinates of the makeup auxiliary area in the makeup model of the face are different along the y-axis, the makeup auxiliary area is adjusted in the following way: The first step is to scale the makeup model proportionally based on the area determined by the location of facial key points to define the makeup auxiliary area: The key point positions in the makeup auxiliary area are determined by the following formula: d’ = ； X'' = x26 + d'; Y''=y26 - tan(θ)d'; Where (x26, y26) are the coordinates of the starting point, (x27, y27) are the coordinates of the ending point, n is the number of equal parts into which the distance between the starting and ending points is divided; d' is the length of each of the equal parts into which the distance between the starting and ending points is divided; (X'', Y'') are the coordinates of the key points in the makeup auxiliary area; θ is the known parameter obtained in the makeup model construction; The second step is to rotate the makeup auxiliary area according to the rotation angle of the makeup position relative to the horizontal line; The key point positions in the rotated makeup auxiliary area are determined by the following formula: ; If y26 > y27, then X3' = ;Y3'= ; If y26 < y27 and θ > a, then X3’ = ; Y3’ = ; If y26 < y27 and θ ; Y3’ = ; Where (X3', Y3') are the coordinates of the key points in the rotated makeup auxiliary area; r is the distance between the start and end points; The angle formed by the straight line defining the starting and ending points and the X-axis; The rotated makeup assistance area is displayed at the makeup application position, wherein the makeup application position is determined based on the facial features and key facial points corresponding to the makeup assistance function command.

2. The smart makeup mirror according to claim 1, characterized in that, The processor is specifically configured to execute: The facial pose in the face image is determined based on a head pose estimation algorithm; or, The facial pose in the facial image is determined based on the positional relationship between the contour key points that characterize the facial contour in the facial key points.

3. The smart makeup mirror according to claim 2, characterized in that, The processor is specifically configured to execute: Based on facial symmetry, multiple pairs of key points are selected from the contour key points, wherein each pair of key points is symmetrical about the facial midline, which divides the face into two equal parts. The facial pose is determined based on a first distance between keypoints on one side of the face in a plurality of keypoint pairs, and a second distance between keypoints on the other side of the face in a plurality of keypoint pairs.

4. The smart makeup mirror according to claim 3, characterized in that, The processor is specifically configured to execute: If the facial feature corresponding to the makeup assistance function instruction is the eyebrows, and the ratio of the first distance to the second distance is less than or equal to a first threshold, then the facial pose is determined to be a frontal view; otherwise, the facial pose is determined to be a side view, wherein the first distance is greater than or equal to the second distance; or, If the facial feature corresponding to the makeup assistance function instruction is the cheek, and the ratio of the first distance to the second distance is less than or equal to the second threshold, then the facial posture is determined to be a frontal face. If the ratio is greater than the second threshold and less than or equal to the third threshold, then the facial posture is determined to be a small profile. If the ratio is greater than the third threshold, then the facial posture is determined to be a large profile, wherein the first distance is greater than or equal to the second distance.

5. The smart makeup mirror according to claim 1, characterized in that, The processor is specifically configured to execute: If the facial feature is the eyebrows and the facial pose is a frontal view, then the makeup model is determined to be a frontal eyebrow makeup model, wherein the auxiliary eyebrow areas on both sides of the frontal eyebrow makeup model are symmetrical; or, If the face part is the eyebrows and the face posture is a side profile, then the makeup model is determined to be a side profile eyebrow makeup model. In the side profile eyebrow makeup model, one side of the eyebrow auxiliary area is a half eyebrow, the arch of the other side of the eyebrow auxiliary area is reduced compared to the arch of the eyebrow auxiliary area in the front face eyebrow makeup model, and the tail of the eyebrow of the other side of the eyebrow auxiliary area is extended compared to the tail of the eyebrow auxiliary area in the front face eyebrow makeup model. or, If the facial feature is the cheek and the facial pose is a frontal view, then the makeup model is determined to be a frontal face makeup model, wherein the auxiliary makeup areas on both sides of the frontal face makeup model are symmetrical; or, If the facial feature is the cheek and the facial pose is a small profile, then the makeup model is determined to be a small profile makeup model, wherein the auxiliary makeup area on one side of the small profile makeup model is smaller than the auxiliary makeup area on the other side; or, If the face part is the cheek and the face posture is a large side profile, then the makeup model is determined to be a large side profile makeup model, wherein the large side profile makeup model only has one side profile makeup auxiliary area.

6. A smart makeup mirror, characterized in that, Includes a camera, a flat mirror, a display screen, and a processor, among which: The camera is used to capture faces; The plane mirror is used to display the face of the user using the smart makeup mirror using imaging principles; The display screen is used to display the makeup assistance area; The processor is configured to execute: In response to the makeup assistance function command of the smart makeup mirror, a makeup assistance area corresponding to the facial posture of the person is displayed at the makeup position on the face in the smart makeup mirror in the following manner: If the face is in a frontal position, a makeup assistance area is displayed at the makeup application position on the frontal face. The makeup assistance area is used to indicate the makeup application position on the frontal face and is attached to the makeup application position on the frontal face. If the face is in profile, a makeup assistance area is displayed at the makeup application position on the profile, wherein the makeup assistance area is used to indicate the makeup application position on the profile and fits onto the makeup application position on the profile. The makeup application location is determined based on the facial features and key facial points corresponding to the makeup assistance function instructions. If the starting and ending coordinates of the makeup auxiliary area in the face makeup model are different along the y-axis, the makeup auxiliary area is adjusted in the following way: The first step is to scale the makeup model proportionally based on the area determined by the location of facial key points to define the makeup auxiliary area: The key point positions in the makeup auxiliary area are determined by the following formula: d’ = ； X'' = x26 + d'; Y''=y26 - tan(θ)d'; Where (x26, y26) are the coordinates of the starting point, (x27, y27) are the coordinates of the ending point, n is the number of equal parts into which the distance between the starting and ending points is divided; d' is the length of each of the equal parts into which the distance between the starting and ending points is divided; (X'', Y'') are the coordinates of the key points in the makeup auxiliary area; θ is the known parameter obtained in the makeup model construction; The second step is to rotate the makeup auxiliary area according to the rotation angle of the makeup position relative to the horizontal line; The key point positions in the rotated makeup auxiliary area are determined by the following formula: ; If y26 > y27, then X3' = ;Y3'= ; If y26 < y27 and θ > a, then X3’ = ; Y3’ = ; If y26 < y27 and θ ; Y3’ = ; Where (X3', Y3') are the coordinates of the key points in the rotated makeup auxiliary area; r is the distance between the start and end points; The angle formed by the straight line defining the starting and ending points and the X-axis; The rotated makeup assistance area is displayed at the makeup application position; the makeup model is determined based on the facial features and facial pose corresponding to the makeup assistance function command.

7. A method for displaying auxiliary makeup, characterized in that, The method includes: In response to the makeup assistance function command of the smart makeup mirror, the facial pose in the facial image captured by the smart makeup mirror is determined according to the facial feature judgment rule corresponding to the makeup assistance function command. Based on the facial features and facial pose corresponding to the makeup assistance function command, determine the makeup model corresponding to the facial features and facial pose; If the starting and ending coordinates of the makeup auxiliary area in the face makeup model are different along the y-axis, the makeup auxiliary area is adjusted in the following way: The first step is to scale the makeup model proportionally based on the area determined by the location of facial key points to define the makeup auxiliary area: The key point positions in the makeup auxiliary area are determined by the following formula: d’ = ； X'' = x26 + d'; Y''=y26 - tan(θ)d'; Where (x26, y26) are the coordinates of the starting point, (x27, y27) are the coordinates of the ending point, n is the number of equal parts into which the distance between the starting and ending points is divided; d' is the length of each of the equal parts into which the distance between the starting and ending points is divided; (X'', Y'') are the coordinates of the key points in the makeup auxiliary area; θ is the known parameter obtained in the makeup model construction; The second step is to rotate the makeup auxiliary area according to the rotation angle of the makeup position relative to the horizontal line; The key point positions in the rotated makeup auxiliary area are determined by the following formula: ; If y26 > y27, then X3' = ;Y3'= ; If y26 < y27 and θ > a, then X3’ = ; Y3’ = ; If y26 < y27 and θ ; Y3’ = ; Where (X3', Y3') are the coordinates of the key points in the rotated makeup auxiliary area; r is the distance between the start and end points; The angle formed between the straight line determining the starting and ending points and the X-axis; The rotated makeup assistance area is displayed at the makeup application position, wherein the makeup application position is determined based on the facial features and key facial points corresponding to the makeup assistance function command.

8. A method for displaying auxiliary makeup, characterized in that, The method includes: In response to the makeup assistance function command of the smart makeup mirror, a makeup assistance area corresponding to the facial posture of the person is displayed at the makeup position on the face in the smart makeup mirror in the following manner: If the face is in a frontal position, a makeup assistance area is displayed at the makeup application position on the frontal face. The makeup assistance area is used to indicate the makeup application position on the frontal face and is attached to the makeup application position on the frontal face. If the face is in profile, a makeup assistance area is displayed at the makeup application position on the profile, wherein the makeup assistance area is used to indicate the makeup application position on the profile and fits onto the makeup application position on the profile. The makeup application position is determined based on the facial features and key facial points corresponding to the makeup assistance function command. If the starting and ending points of the makeup assistance area in the facial makeup model have different y-axis coordinates, the makeup assistance area is adjusted in the following way: The first step is to scale the makeup model proportionally based on the area determined by the location of facial key points to define the makeup auxiliary area: The key point positions in the makeup auxiliary area are determined by the following formula: d’ = ； X'' = x26 + d'; Y''=y26 - tan(θ)d'; Where (x26, y26) are the coordinates of the starting point, (x27, y27) are the coordinates of the ending point, n is the number of equal parts into which the distance between the starting and ending points is divided; d' is the length of each of the equal parts into which the distance between the starting and ending points is divided; (X'', Y'') are the coordinates of the key points in the makeup auxiliary area; θ is the known parameter obtained in the makeup model construction; The second step is to rotate the makeup auxiliary area according to the rotation angle of the makeup position relative to the horizontal line; The key point positions in the rotated makeup auxiliary area are determined by the following formula: ; If y26 > y27, then X3' = ;Y3'= ; If y26 < y27 and θ > a, then X3’ = ; Y3’ = ; If y26 < y27, and θ < a, then X3’ = ; Y3’ = ; Where (X3', Y3') are the coordinates of the key points in the rotated makeup auxiliary area; r is the distance between the start and end points; The angle formed between the straight line determining the starting and ending points and the X-axis; The rotated makeup assistance area is displayed at the makeup application position; the makeup model is determined based on the facial features and facial pose corresponding to the makeup assistance function command.

Images