An intelligent system capable of realizing personalized makeup teaching and application thereof

Abstract

The application discloses an intelligent system capable of realizing individualized makeup teaching and application thereof, and the intelligent system comprises a video library provided with a plurality of classified makeup teaching video items with semantic classification labels, a classified makeup image library composed of a plurality of video frame images from the classified makeup teaching video items in the video library and with corresponding semantic classification labels, a face image acquisition module, a face feature analysis module, an adaptive makeup analysis module, a virtual makeup module and a video teaching module. The application can not only realize customization of individualized makeup, but also ensure that the customized individualized makeup has accurate corresponding teaching videos, and the customized individualized makeup can be effectively implemented, and the customized makeup effect is highly consistent with the learned makeup effect, so that the problem of difference between the actual makeup and the recommended makeup can be avoided, and people can easily and quickly obtain individualized makeup guidance suitable for themselves, and the application has a significant application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to an intelligent system capable of realizing personalized makeup teaching and application thereof, and belongs to the technical field of artificial intelligence. BACKGROUND

[0002] With the pursuit of fashion and beauty, more and more people, especially women, begin to pay attention to the improvement of personal image. In social occasions, appropriate makeup can show respect for the activity and attention to other participants. In some cultures, participants are required to be made up to meet the etiquette of certain social activities. Therefore, appropriate makeup has become an indispensable part of personal life and social activities, and is a direct way of cultural expression and identity recognition.

[0003] However, not everyone is good at makeup, and the makeup requirements are different for different occasions. To get the appropriate makeup, professional guidance is needed to master. Most people, especially beginners, often fail to draw makeup that is not suitable for themselves or is not coordinated due to insufficient understanding of their skin quality and facial features, insufficient knowledge of makeup, and lack of makeup skills, resulting in poor makeup effect and failure to achieve the ideal makeup effect.

[0004] Currently, people mainly learn makeup knowledge and skills by watching videos and articles provided by makeup bloggers on the network, some makeup APPs, or the spoken and written teachings of their friends and relatives. However, due to the different combinations of everyone's skin quality and facial features, the makeup teaching videos of network makeup bloggers are not precisely targeted, and people cannot judge which makeup blogger's makeup is suitable for themselves, so these video teachings are not truly practical. In addition, most existing makeup APPs lack personalized guidance, such as the "Follow Me to Learn Makeup" APP, the "Easy Makeup Learning" APP, and the "Makeup College" APP. These makeup APPs only provide comprehensive makeup knowledge and skill teaching, and people usually need to learn a lot and try multiple times for months or even longer to find the suitable makeup and accumulate makeup skills. People cannot quickly find the ideal makeup that fits themselves, and the repeated process of trying on and taking off makeup is not only time-consuming and labor-intensive, but also has the risk of damaging the facial skin. Although existing beauty camera APPs can virtually apply makeup to the photographed photos to achieve personalized makeup, the personalized makeup output by the beauty camera APP can only stay on the photos, and people cannot finally convert it into an actual makeup because they cannot accurately find the corresponding makeup teaching video.

[0005] In addition, although there are technologies in the prior art that recommend makeup information that matches the facial image of a user by obtaining the facial image of the user, such as the Chinese patent application with the application number CN201910357049.5, which discloses a makeup recommendation method, system and computer readable storage medium, the method comprising: S1, obtaining the individualized features of the user; S2, inputting the individualized features into a decision matrix to generate a recommendation result after operation; the decision matrix includes a set of decision functions of multiple adjustable dimensions of facial single elements, and the recommendation result includes a set of makeup means of multiple adjustable dimensions of facial single elements. This patent inputs the individualized features of the user into a decision matrix composed of a set of decision functions of multiple adjustable dimensions of facial single elements, thereby generating a recommendation result including a set of makeup means of multiple adjustable dimensions of facial single elements, which can generate more detailed makeup recommendations. The Chinese patent application with the application number CN202411540596.4 discloses a makeup recommendation method and device based on facial features, equipment and medium, the method comprising: obtaining the face image of the target user; inputting the face image into a face feature extraction model to obtain the facial feature information of the target user, recommending the target makeup to the target user according to the facial feature information through a comprehensive recommendation mechanism, the comprehensive recommendation mechanism including an algorithm recommendation mechanism, an expert recommendation mechanism and a popular makeup recommendation mechanism, finally, adjusting the weights of the algorithm recommendation mechanism, the expert recommendation mechanism and / or the popular makeup recommendation mechanism according to the feedback information of the target user on the target makeup to obtain the updated weights of the algorithm recommendation mechanism, the expert recommendation mechanism and / or the popular makeup recommendation mechanism, and the comprehensive recommendation mechanism recommends the target makeup to the target user through the updated weights; however, since makeup involves many steps and processing techniques, in the existing network makeup teaching videos and makeup APPs, there are either no precise targeting and lack of personalized guidance, or there are only single functions, or there are only theories and lack of practicality, so people, especially beginners who are not good at makeup, still have difficulty in effectively implementing and achieving the recommended makeup effect in reality, resulting in the recommended makeup information still lacking practicality and effectiveness.

[0006] As can be seen from the above, there is an urgent need for a product that can conveniently and efficiently implement personalized makeup teaching, but so far there has been no report on related products and technologies. SUMMARY

[0007] In view of the above problems and needs in the prior art, the purpose of the present application is to provide an intelligent system for implementing personalized makeup teaching and its application, in order to solve the defect that the prior art cannot provide personalized makeup teaching, which not only can realize the intelligent customization and virtual makeup of personalized makeup, but also can effectively implement the customized personalized makeup, so that everyone can easily obtain personalized makeup that fits themselves.

[0008] To achieve the above-mentioned object, the present application adopts the following technical solutions:

[0009] An intelligent system capable of realizing personalized makeup teaching, comprising:

[0010] A video library provided with a plurality of classified makeup teaching video items with semantic classification tags;

[0011] A classified makeup image library composed of a plurality of video frame images derived from the classified makeup teaching video items in the video library and having corresponding semantic classification tags;

[0012] A face image acquisition module for acquiring a user's bareheaded and unmade-up front face image;

[0013] A facial feature analysis module for identifying and analyzing the facial features and semantic classification of the user's image;

[0014] An adaptive makeup analysis module for analyzing the semantic classification makeup that is adapted to the user's facial feature analysis result;

[0015] A virtual makeup module for personalized customization of the classified makeup image based on the semantic classification tags and intuitive display of the customized classified makeup effect, so as to customize a personalized makeup that the user is satisfied with;

[0016] A video teaching module for playing the corresponding classified makeup teaching video according to the final customized classified makeup image and the makeup process step by step.

[0017] An embodiment, the classified makeup teaching video items in the video library are video entities (i.e. video files themselves) or / and video link addresses (i.e. URLs or paths pointing to videos).

[0018] An embodiment, the video library contains at least the classified makeup teaching video items of the primary semantic classification tags of foundation, eyebrow makeup, eye makeup, contouring, blush, and lip makeup.

[0019] As a preferred solution, each makeup teaching video item of the foundation category includes the teaching of each step of pre-makeup skin care, sunscreen and sunblock, concealer, and makeup fixing.

[0020] As a preferred solution, each makeup teaching video item of the foundation category is provided with a skin type (such as dry, oily, and mixed) secondary semantic classification tag.

[0021] As a preferred solution, each makeup teaching video item of the eyebrow makeup category is provided with a face shape (such as square face, round face, oval face, triangular face, diamond face, and oblong face) secondary semantic classification tag.

[0022] As a further preferred solution, each makeup teaching video item of the eyebrow makeup class is further provided with multi-level semantic classification tags of makeup style (such as: versatile, cute, gentle, Chinese style, ancient style, etc.) and eyebrow shape (such as: standard eyebrow, raised eyebrow, willow leaf eyebrow, crescent eyebrow, etc.).

[0023] As a preferred solution, each makeup teaching video item of the eye makeup class is provided with two-level semantic classification tags of eye shape (such as: apricot eye, round eye, Danfeng eye, peach blossom eye, drooping eye, dog eye, squint eye, etc.).

[0024] As a further preferred solution, each makeup teaching video item of the eye makeup class is further provided with multi-level semantic classification tags of makeup style (such as: cute puppy makeup, sweet and cool puppy makeup, pure desire big eye makeup, cool American style makeup, etc.) and applicable scene (such as: daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0025] As a preferred solution, each makeup teaching video item of the contouring class is provided with two-level semantic classification tags of face shape (such as: square face, round face, oval face, triangular face, diamond face, oblong face, etc.).

[0026] As a further preferred solution, each makeup teaching video item of the contouring class is further provided with three-level semantic classification tags of makeup style (such as: low saturation contouring, Korean style contouring, etc.).

[0027] As a preferred solution, each makeup teaching video item of the blush class is provided with two-level semantic classification tags of face shape (such as: square face, round face, oval face, triangular face, diamond face, oblong face, etc.).

[0028] As a further preferred solution, each makeup teaching video item of the blush class is further provided with multi-level semantic classification tags of makeup style (such as: sweet, clear, pure desire, cool, etc.) and applicable scene (such as: daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0029] As a preferred solution, each makeup teaching video item of the lip makeup class is provided with two-level semantic classification tags of lip shape (such as: cherry mouth, standard lip, thick lip, thin lip, smile lip, pout lip, etc.).

[0030] As a further preferred solution, each makeup teaching video item of the lip makeup class is further provided with multi-level semantic classification tags of makeup style (such as: 3D stereoscopic lip, Korean smile lip, cute pout lip, meaty cartoon lip, etc.) and applicable scene (such as: daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0031] In an embodiment, the video frame images in the classified makeup image library are derived from the eye makeup, eyebrow makeup, lip makeup, contouring, and blush classified makeup teaching video items in the video library.

[0032] In an embodiment, the construction of the classified makeup image library comprises the following steps:

[0033] a) Extracting the completed makeup frames of each category from the makeup teaching videos in the video library, respectively, corresponding to the eye makeup, eyebrow makeup, lip makeup, contour, and blush;

[0034] b) Standardizing the extracted makeup frames of each category, including size, color space, pixel value, and file format;

[0035] c) Labeling the standardized makeup frames of each category with the same semantic classification label as the semantic classification label of the corresponding makeup teaching video item.

[0036] An embodiment, the facial feature analysis module includes semantic classification of face shape, eye shape, and lip shape.

[0037] As a preferred solution, the facial feature analysis module further includes semantic classification of skin quality.

[0038] An embodiment, the facial feature analysis module implements a method for face shape classification, including the following steps:

[0039] A) Input the image into a facial feature detection model (such as a dlib model) for face key point positioning and detection;

[0040] B) Calculate the following parameters based on the detected face key points: forehead width, denoted as line1; length from cheekbone to chin line, denoted as line2; cheekbone width, denoted as line3; vertical length of face, denoted as line4; and angle of mandibular angle, denoted as θ;

[0041] C) Calculate the standard deviation (Standard Deviation) σ of the lengths of line1, line2, and line3, according to the following formula:

[0042] ;

[0043] D) Classify the face shape according to the following recognition conditions:

[0044]

[0045] An embodiment, the facial feature analysis module implements a method for eye shape classification, including the following steps:

[0046] E) Input the image into a facial feature detection model (such as a dlib model) to obtain eye key points;

[0047] F) Calculate the following parameters of the left or right eye based on the detected left or right eye key points:

[0048] ①The length between two eye corners l;

[0049] ②The maximum width of the eye (i.e. the distance between the highest point and the lowest point of the eye) h;

[0050] ③The eye shape ratio d = h / l;

[0051] ④The angle between the line connecting two eye corners and the horizontal line a;

[0052] G) The data provided by the face++ cloud service platform are defined as follows:

[0053]

[0054] H) The eye shape is classified according to the following identification conditions:

[0055]

[0056] One embodiment, the face feature analysis module implements the method for lip shape classification, comprising the following steps:

[0057] I) The image is input into a face feature detection model (such as a dlib model) to obtain lip key points;

[0058] J) The mouth length, mouth corner up angle, M shape obvious degree, upper lip thickness, and lower lip thickness are calculated according to the detected lip key points;

[0059] K) The lip shape is classified according to the following identification conditions:

[0060]

[0061] One embodiment, the makeup analysis module is adapted based on the semantic classification information of the face features of the user by the face feature analysis module, and keyword matching is achieved by searching a classified makeup image library using the semantic classification label.

[0062] A further embodiment, the makeup analysis module is adapted based on the semantic classification label of the eye shape of the user to search all eye makeup type makeup images containing the semantic classification label of the eye shape in the classified makeup image library.

[0063] A further embodiment, the makeup analysis module is adapted based on the semantic classification label of the lip shape of the user to search all lip makeup type makeup images containing the semantic classification label of the lip shape in the classified makeup image library.

[0064] A further embodiment, the makeup analysis module is adapted based on the semantic classification label of the face shape of the user to search all eyebrow makeup type makeup images, makeup correction type makeup images, and blush type makeup images containing the semantic classification label of the face shape in the classified makeup image library, respectively.

[0065] In an embodiment, the virtual makeup application module is configured to customize the various makeup images recommended by the adaptive makeup analysis module using automatic screening or / and manual selection in combination with secondary or multi-level keywords, and to generate real-time makeup images using a local makeup transfer technique.

[0066] In an embodiment, the virtual makeup application module is configured to generate segmentation masks for the customized classified makeup images and the user's bare face image without makeup using a semantic segmentation model (such as U-Net, DeepLab), and to generate the makeup image after local makeup transfer using an SSAT model and output the makeup image to the display module.

[0067] In an embodiment, the video teaching module is configured to sequentially play the corresponding classified makeup teaching videos in the order of foundation makeup, eyebrow makeup, eye makeup, contour makeup, blush makeup, and lip makeup.

[0068] As a preferred solution, the intelligent system is further provided with an intelligent makeup assistant module for providing the user with makeup-related information (such as: makeup-related tool and product information) and dressing and etiquette information services.

[0069] A method for implementing personalized makeup teaching using the intelligent system of the present application, comprising the following steps:

[0070] S1) Using the face image acquisition module to obtain the user's bare face image without makeup;

[0071] S2) Using the face feature analysis module to identify and analyze the face features and semantic classification of the user's image;

[0072] S3) Using the adaptive makeup analysis module, searching for eye makeup images containing the semantic classification label of the user's eye type in the classified makeup image database based on the identified semantic classification label of the user's eye type, searching for lip makeup images containing the semantic classification label of the user's lip type in the classified makeup image database based on the identified semantic classification label of the user's lip type, and searching for eyebrow makeup images, contour makeup images, and blush makeup images containing the semantic classification label of the user's face type in the classified makeup image database based on the identified semantic classification label of the user's face type, respectively;

[0073] S4) Using the virtual makeup application module, customizing the various makeup images recommended by the adaptive makeup analysis module using automatic screening or / and manual selection in combination with secondary or multi-level keywords, and generating real-time virtual makeup images and outputting the virtual makeup images to the display module;

[0074] S5) Using the video teaching module to sequentially play the corresponding classified makeup teaching videos in the order of foundation makeup, eyebrow makeup, eye makeup, contour makeup, blush makeup, and lip makeup.

[0075] A terminal device capable of realizing personalized makeup teaching, which is installed with the intelligent system of the present application and is provided with a camera and a display unit.

[0076] An embodiment, the terminal device includes but is not limited to a smart phone, a tablet, a computer, a smart TV, a smart mirror.

[0077] Compared with the prior art, the beneficial technical effects of the present application are that:

[0078] The present application sets up a video library composed of a plurality of classified makeup teaching video items with semantic classification labels and a classified makeup image library composed of a plurality of video frame images derived from the classified makeup teaching video items in the video library and having corresponding semantic classification labels, then customizes a local makeup from the classified makeup image library according to the semantic classification labels, thereby ensuring that the customized personalized makeup has a precise corresponding teaching video, solving the defect problem that the prior art cannot provide personalized makeup teaching, making the customized personalized makeup can be effectively implemented, and making what is seen can be learned, and because the customized classified makeup images are all from corresponding classified makeup teaching video frames, the customized makeup effect and the learned makeup effect can be kept highly consistent, which can effectively avoid the problem of difference between the actual makeup and the recommended makeup, making people can easily and quickly obtain personalized makeup guidance that fits themselves; therefore, the present application has significant progress and application prospect and market value compared with the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 The structure block diagram of the intelligent system capable of realizing personalized makeup teaching provided by the present application is shown.

[0080] Figure 2 The selection position diagram of eye key points when eye type recognition is shown.

[0081] Figure 3 The interface diagram when the intelligent system realizes the collection of uncrowned and unmade-up front face images provided by the embodiment is shown.

[0082] Figure 4 The interface diagram when the intelligent system realizes the recommendation of makeup images suitable for user eye makeup and lip makeup provided by the embodiment is shown. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical scheme and advantages of the present application more clear, the technical scheme of the present application will be described in detail below with the specific embodiments of the present application and the corresponding drawings.

[0084] Please refer to Figure 1 The intelligent system capable of realizing personalized makeup teaching provided by the present application is shown, which comprises:

[0085] 1) a video library provided with a plurality of classified makeup teaching video items with semantic classification labels;

[0086] 2) a classified makeup look image library composed of a plurality of video frame images derived from the classified makeup teaching video items in the video library and having corresponding semantic classification labels;

[0087] 3) a face image acquisition module for acquiring an uncrowned and unmade-up front face image of a user;

[0088] 4) a facial feature analysis module for identifying and analyzing the facial features and semantic classification of the user image;

[0089] 5) an adaptive makeup look analysis module for analyzing a semantic classification makeup look that is adapted to the results of the user's facial feature analysis;

[0090] 6) a virtual makeup module for customizing a classified makeup look based on the semantic classification label and visually displaying the customized classified makeup look effect, so as to customize a personalized makeup look that the user is satisfied with;

[0091] 7) a video teaching module for playing the corresponding classified makeup teaching video according to the final customized classified makeup look image and makeup process step-by-step.

[0092] Specifically, the classified makeup teaching video item is a video entity (i.e. a video file itself) or / and a video link address (i.e. a URL or path pointing to a video); the classified makeup teaching video can be recorded and customized by inviting makeup bloggers, makeup professionals, makeup volunteers, etc., or / and obtained by data processing of video data obtained through a network platform; and the video library can be continuously increased and updated.

[0093] In addition, the video library contains at least classified makeup teaching video items of each primary semantic classification label of foundation, eyebrow makeup, eye makeup, contouring, blush, and lip makeup, but it should be noted that the video library can also include classified makeup teaching video items of other semantic classification labels, such as skin type discrimination teaching video items, makeup removal and skin care teaching video items, makeup tool use teaching video items, makeup product selection teaching video items, etc., which are not listed here, in short, various classified makeup teaching video items related to makeup and having semantic classification labels can be in the video library.

[0094] As a preferred solution, each makeup teaching video item of the foundation category includes teaching of each step of pre-makeup skin care → sun protection and isolation → concealer → setting, and each makeup teaching video item of the foundation category is preferably provided with a skin type (such as dry, oily, and mixed) secondary semantic classification label, so as to accurately recommend corresponding foundation products according to the user's skin type.

[0095] As a preferred solution, each item of eyebrow makeup teaching video is provided with a face shape (such as square face, round face, oval face, triangular face, diamond face, oblong face, etc.) secondary semantic classification label; as a further preferred solution, each item of eyebrow makeup teaching video is also provided with a makeup style (such as versatile, cute, gentle, Chinese style, ancient style, etc.) and a eyebrow shape (such as standard eyebrow, raised eyebrow, willow leaf eyebrow, crescent eyebrow, etc.) multi-level semantic classification label.

[0096] As a preferred solution, each item of eye makeup teaching video is provided with an eye shape (such as apricot eye, round eye, Danfeng eye, peach blossom eye, drooping eye, dog eye, squint eye, etc.) secondary semantic classification label; as a further preferred solution, each item of eye makeup teaching video is also provided with a makeup style (such as cute dog makeup, sweet and cool dog makeup, pure desire big eye makeup, cold beauty makeup, etc.) and a suitable scene (such as daily makeup, banquet makeup, dating makeup, cos makeup, etc.) multi-level semantic classification label.

[0097] As a preferred solution, each item of makeup teaching video is provided with a face shape (such as square face, round face, oval face, triangular face, diamond face, oblong face, etc.) secondary semantic classification label; as a further preferred solution, each item of makeup teaching video is also provided with a makeup style (such as low saturation makeup, Korean style makeup, etc.) three-level semantic classification label.

[0098] As a preferred solution, each item of blush teaching video is provided with a face shape (such as square face, round face, oval face, triangular face, diamond face, oblong face, etc.) secondary semantic classification label; as a further preferred solution, each item of blush teaching video is also provided with a makeup style (such as sweet, clear, pure desire, cold, etc.) and a suitable scene (such as daily makeup, banquet makeup, dating makeup, cos makeup, etc.) multi-level semantic classification label.

[0099] As a preferred solution, each item of lip makeup teaching video is provided with a lip shape (such as cherry mouth, standard lip, thick lip, thin lip, smile lip, pout lip, etc.) secondary semantic classification label; as a further preferred solution, each item of lip makeup teaching video is also provided with a makeup style (such as 3D lip, Korean smile lip, cute pout lip, meat cartoon lip, etc.) and a suitable scene (such as daily makeup, banquet makeup, dating makeup, cos makeup, etc.) multi-level semantic classification label.

[0100] It should be noted that the definition of semantic classification label is not limited to the above description of the embodiment, and can be defined by itself based on the above inspiration; in addition, the semantic classification of face shape, eye shape, lip shape, makeup style and suitable scene can also be defined by itself, increased or decreased classification and changed the arrangement order of multi-level semantic label, and the application does not make special limitation to this.

[0101] Furthermore, in this embodiment, the video frame images in the categorized makeup image library are derived from makeup tutorial videos categorized as eye makeup, eyebrow makeup, lip makeup, contouring, and blush in the video library; the construction of the categorized makeup image library includes the following steps:

[0102] a) Extract clear video frames of each category of makeup tutorial videos, including eye makeup, eyebrow makeup, lip makeup, contouring, and blush, from the video library. Specifically, keyframe extraction based on frame differences can be performed using Python.

[0103] b) Standardize the extracted makeup video frames for each category, including size, color space, pixel value, and file format. Specifically, the OpenCV tool in the Python library can be used for standardization.

[0104] c) For each standardized makeup video frame image, assign a semantic classification label that matches the semantic classification label of its corresponding makeup tutorial video item. For example, if the extracted makeup video frame comes from a video file of an eyebrow makeup tutorial video item with the semantic classification label "eyebrow makeup / round face / willow leaf eyebrow", then the processed makeup video frame image will also be labeled as "eyebrow makeup / round face / willow leaf eyebrow". Semantic classification labeling of video frame images can be performed using LabelMe or VoTT tools. LabelMe is an image annotation tool based on Python and QT, supporting semantic segmentation and instance segmentation tasks. It can export annotation data in VOC and COCO formats, suitable for frame-by-frame annotation of video frames. VoTT (Visual Object Tagging Tool) is an open-source tool developed by Microsoft, supporting image and video annotation. It can export annotation data in multiple formats and supports integration with deep learning frameworks, suitable for semantic segmentation and object detection tasks.

[0105] Furthermore, in this embodiment, the facial feature analysis module includes semantic classification of face shape, eye shape, and lip shape. As a preferred embodiment, the facial feature analysis module also includes semantic classification of skin texture.

[0106] In this embodiment, the facial feature analysis module implements a face shape classification method, which includes the following steps:

[0107] A) Input the image into a facial feature detection model (such as the dlib open-source model) to locate and detect facial key points;

[0108] B) Calculate the following based on the detected facial key points: forehead width, denoted as line1; length from cheekbone to jawline, denoted as line2; cheekbone width, denoted as line3; vertical length of face, denoted as line4; and the curvature of the jaw angle, denoted as θ.

[0109] C) Calculate the standard deviation σ of the lengths of the three line segments line1, line2, and line3. The specific calculation formula is as follows:

[0110] ;

[0111] D) Classify face shapes based on the following recognition criteria:

[0112]

[0113] One implementation scheme, wherein the facial feature analysis module implements an eye shape classification method, includes the following steps:

[0114] E) Input the image into a facial feature detection model (such as the dlib open-source model) to obtain key points of the eyes;

[0115] F) Calculate the following parameters for the left or right eye based on the detected key points:

[0116] ① The length l between the outer corners of the eyes;

[0117] ② Maximum eye width (i.e., the distance between the highest and lowest points of the eye) h;

[0118] ③ The eye shape ratio d = h / l;

[0119] ④ The angle α between the line connecting the two corners of the eyes and the horizontal line;

[0120] Specifically, after obtaining facial feature points, the eyes can be defined as follows: Figure 2 The a1~a shown 10 With a total of 10 feature points, the above parameters can be calculated using the following formula:

[0121] ;

[0122] G) Define the data provided by the Face++ cloud service platform as follows:

[0123]

[0124] H) Classify eye shapes according to the following identification criteria:

[0125]

[0126] One embodiment, the face feature analysis module implements a method of lip shape classification, comprising the following steps:

[0127] I) input the image into a face feature detection model (such as a dlib model) to obtain lip key points;

[0128] J) calculate the mouth length, mouth corner up angle, M shape prominence, upper lip thickness, and lower lip thickness according to the detected lip key points;

[0129] K) perform lip shape classification according to the following recognition conditions:

[0130]

[0131] However, it should be noted that the classification method for face shape, eye shape, and lip shape is not limited to the method described in this embodiment, and other existing technologies can also be used, for example, the DeepFace tool, which is an open source face recognition project that supports multiple models (such as FaceNet, ArcFace, Dlib, etc.). In addition, the classification definition of the face shape, eye shape, and lip shape by this module is consistent with the corresponding classification in the video library. The recognition conditions for face shape, eye shape, and lip shape can be obtained through experience summary by analyzing the corresponding face shape, eye shape, and lip shape in the video library. In the later period, machine learning can also be used to introduce the features of the corresponding face shape, eye shape, and lip shape in the video library and introduce similarity to improve the more refined matching and classification of the user's face features.

[0132] In addition, in the embodiment, the adaptive makeup analysis module is based on the semantic classification information of the facial feature analysis module on the user's facial features, and uses the semantic classification label to search the classified makeup image library for keyword matching; further, the adaptive makeup analysis module is based on the semantic classification label of the user's eye type to search all eye makeup makeup images in the classified makeup image library containing the eye type semantic classification label, for example: if the semantic classification of the user's eye type is "dog eye", then use "eye makeup / dog eye" for semantic search in the classified makeup image library, as long as all eye makeup makeup images containing the eye type semantic classification label "dog eye" are analyzed by the module as adaptive eye makeup makeup; similarly, the adaptive makeup analysis module is based on the semantic classification label of the user's lip type to search all lip makeup makeup images in the classified makeup image library containing the lip type semantic classification label, for example: if the semantic classification of the user's lip type is "smile lip", then use "smile lip" for semantic search in the classified makeup image library, as long as all lip makeup makeup images containing the lip type semantic classification label "smile lip" are analyzed by the module as adaptive lip makeup makeup; similarly, the adaptive makeup analysis module is based on the semantic classification label of the user's face type to search all eyebrow makeup makeup images, makeup makeup images and blush makeup makeup images in the classified makeup image library containing the face type semantic classification label as adaptive eyebrow makeup makeup, makeup makeup and blush makeup, that is, the eyebrow makeup makeup image, the makeup makeup image and the blush makeup makeup image are searched in the classified makeup image library according to the semantic classification label of the user's face type to obtain the respective adaptive makeup image.

[0133] In addition, in this embodiment, the virtual makeup module customizes various makeup images by automatically screening or / and manually checking according to the various makeup images recommended by the adaptive makeup analysis module and in combination with secondary or multi-level keywords, and generates real-time makeup images by using a local makeup transfer technology; for example, if a user needs a "dog eye / daily" eye makeup, the system will preferentially display eye makeup images that contain the "dog eye / daily" semantic classification label, because the eye makeup suitable for "dog eye / daily" also has different makeup styles, so the user can further automatically screen by inputting keywords about makeup styles, such as "cool American style", or directly manually check the makeup styles of interest. After determining the eye makeup image that needs to be virtually applied, the virtual makeup module first uses a semantic segmentation model (such as U-Net, DeepLab) to generate segmentation masks for the customized classified makeup image and the user's makeup-free front face image, respectively, and then uses an SSAT (Symmetric Semantic-Aware Transformer Network) model to perform local makeup transfer and generate a makeup image after local makeup transfer, which is output to the display module. The user can replace other makeup images of different makeup styles displayed by the virtual makeup module until he is satisfied.

[0134] In the SSAT model, the mask of different regions of the face can be extracted based on the semantic segmentation network by the following steps, so as to realize local makeup transfer. The main implementation steps are as follows:

[0135] I. Introduction of semantic segmentation network

[0136] The SSAT model can extract masks of different regions of the face by introducing a pre-trained semantic segmentation network (such as DeepLab, U-Net or HRNet). These masks can segment the face into multiple regions, such as the eye, lip, face, etc. The specific implementation steps are as follows:

[0137] ① Select a semantic segmentation model: select a semantic segmentation model suitable for face segmentation, such as DeepLab or HRNet;

[0138] ② Train or load pre-trained model: train the semantic segmentation model using a public face dataset (such as CelebA or FFHQ), or directly load a pre-trained model;

[0139] ③ Extract mask: for the input face image, use the semantic segmentation model to generate a segmentation mask to segment the face into multiple regions.

[0140] II. Local makeup transfer

[0141] In the SSAT model, the specific steps for local makeup transfer using the extracted mask are as follows:

[0142] ① Feature extraction: use the feature extraction module of the SSAT model to extract the features of the source image (i.e. the makeup image) and the target image (i.e. the non-makeup image);

[0143] ② Feature alignment: use the semantic segmentation mask to align the makeup features of the source image (i.e. the makeup image) to the corresponding area of the target image (i.e. the non-makeup image), for example, only align the features of the eye area;

[0144] ③ Feature fusion: fuse the aligned makeup features with the features of the target image (i.e. the non-makeup image) to generate the result of local makeup transfer;

[0145] ④ Generate image: use the generator of the SSAT model to generate the final local makeup transfer image.

[0146] III. Loss function

[0147] In order to ensure the effect of local makeup transfer, a local loss term can be added to the loss function of the SSAT model, for example, a pixel-level histogram loss can be used to constrain the makeup transfer effect of local areas, specifically, the histogram loss can be calculated for local areas such as eyes and lips to ensure accurate transfer of makeup features.

[0148] In addition, in this embodiment, the video teaching module plays the corresponding classified makeup teaching video in the order of foundation makeup → eyebrow makeup → eye makeup → contour makeup → blush → lip makeup.

[0149] As a preferred solution, the intelligent system is also provided with an intelligent makeup assistant module for providing users with makeup-related information (such as makeup-related tool and product information) and dressing and etiquette information services.

[0150] The method for implementing personalized makeup teaching using the intelligent system of the present application comprises the following steps:

[0151] S1) Use the face image acquisition module to obtain the user's uncrowned non-makeup front face image; for example Figure 3 As shown, high-quality uncrowned non-makeup front face images can be obtained by embedding a front face shooting frame and providing semantic prompts such as "don't have bangs blocking" and the like;

[0152] S2) using a facial feature analysis module to identify and analyze the facial features and semantic classification of the user image; however, it should be noted that, because the eyebrow shape in the makeup look is usually determined by the face shape to determine the appropriate eyebrow makeup, and the original eyebrow shape has little to do with it, therefore, the analysis of the original eyebrow shape feature can also be omitted;

[0153] S3) using an adaptive makeup analysis module, searching for eye makeup category makeup images containing the semantic classification label of the identified user eye type in the classified makeup image library based on the semantic classification label of the identified user eye type, searching for lip makeup category makeup images containing the semantic classification label of the identified user lip type in the classified makeup image library based on the semantic classification label of the identified user lip type, searching for eyebrow makeup category makeup images, contour makeup category makeup images, and blush category makeup images containing the semantic classification label of the identified user face shape in the classified makeup image library based on the semantic classification label of the identified user face shape, respectively; for example, the analyzed adaptive eye makeup and lip makeup makeup images are as shown in Figure 4 In addition, more adaptive makeup images can also be viewed by clicking the "MORE" button below each category of makeup display column;

[0154] S4) using a virtual makeup module, customizing each category of makeup image based on the recommended category of makeup image by the adaptive makeup analysis module, and combining secondary or multi-level keywords to automatically filter or / and manually check, and generating a virtual makeup image in real time to output to the display module; for example, because the eye makeup makeup according to the "dog eye / daily" semantic classification label is also classified into different makeup style makeup images such as cute dog makeup, sweet and cool dog makeup, etc., therefore, the user can further automatically filter by inputting keywords related to the makeup style, such as "cute", or can directly manually check the "cute dog makeup" makeup style of interest, when the eye makeup makeup image is determined, the corresponding virtual makeup after eye makeup image can be generated through local makeup migration; similarly, personalized customization and real-time virtual makeup display of the corresponding makeup effect can be performed for eyebrow makeup, lip makeup, contour makeup, and blush category makeup images;

[0155] S5) using a video teaching module to sequentially play the corresponding classified makeup teaching video in the order of foundation→eyebrow makeup→eye makeup→contour→blush→lip makeup.

[0156] In addition, it should be noted that the skin quality analysis can also be tested and distinguished by the corresponding skin quality distinguishing teaching video item provided in the video library, and if there is difficulty in distinguishing, the ai assistant analysis function provided by the system can be used to assist in judgment.

[0157] In addition, the intelligent system described in the present application can also include a personalized data storage module and an intelligent makeup assistant, and the personalized data can include facial feature analysis data, collected makeup images, collected teaching videos, and collected cosmetics information.

[0158] A terminal device capable of realizing personalized makeup teaching, which is installed with the intelligent system and is provided with a camera and a display unit, and the terminal device includes but is not limited to a smart phone, a tablet, a computer, a smart television and a smart mirror.

[0159] Finally, it is necessary to point out here that the program modules described in the application can include one or more software components, including, for example, software objects, methods, data structures, etc., each of which can include computer executable instructions that, in response to execution, cause at least a part of the functions described in the application to be performed. The software components can be coded in any of a variety of programming languages. It should be understood that the embodiments in the application and the features in the embodiments can be combined with each other without conflict, and the protection scope of the application is not limited to the specific embodiments described above. Any changes or replacements within the technical scope disclosed in the application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the application.

Claims

1. An intelligent system capable of providing personalized makeup instruction, characterized in that: include: The video library contains several categories of beauty tutorial videos with semantic classification tags; A categorized makeup image library is constructed from video frame images sourced from a video library, each containing a corresponding semantic category label. The construction of this categorized makeup image library includes the following steps: a) Extract clear video frames of each category of makeup tutorial videos from the video library, showing the completed makeup look. b) Standardize the extracted makeup video frames for each category; c) Each standardized makeup video frame image is assigned a semantic classification label that is identical to the semantic classification label of its corresponding makeup tutorial video item; A face image acquisition module is used to acquire a frontal face image of the user without a hat or makeup. A facial feature analysis module is used to identify and analyze facial features and semantic classification of user images; The makeup analysis module is adapted to analyze semantically categorized makeup that matches the user's facial feature analysis results. Specifically, it is based on the semantic classification information of the user's facial features by the facial feature analysis module, and uses semantic classification tags to search the classification makeup image library for keyword matching. The virtual makeup module is used to personalize categorized makeup images based on semantic classification tags and intuitively display the customized categorized makeup effects, thereby creating a personalized makeup look that satisfies the user. The customization of various makeup images is based on various makeup images recommended by the makeup analysis module, combined with secondary or multi-level keywords, using automatic filtering and / or manual selection. The video tutorial module is used to play corresponding categorized makeup tutorial videos step by step based on the final customized categorized makeup images and makeup process.

2. The intelligent system according to claim 1, characterized in that: The category of beauty tutorial videos in the video library consists of video entities and / or video link addresses.

3. The intelligent system according to claim 1, characterized in that: The video library contains at least one category of beauty tutorial videos with primary semantic category tags for base makeup, eyebrow makeup, eye makeup, contouring, blush, and lip makeup.

4. The intelligent system according to claim 1, characterized in that: The video frame images in the categorized makeup image library are sourced from makeup tutorial videos categorized as eye makeup, eyebrow makeup, lip makeup, contouring, and blush.

5. The intelligent system according to claim 1, characterized in that: The facial feature analysis module includes semantic classification of face shape, eye shape, and lip shape.

6. The intelligent system according to claim 1, characterized in that: The virtual makeup module uses local makeup transfer technology to generate makeup images in real time.

7. The intelligent system according to claim 1, characterized in that: The video tutorial module plays corresponding makeup tutorial videos step by step in the following order: base makeup → eyebrow makeup → eye makeup → contouring → blush → lip makeup.

8. A method for implementing personalized makeup instruction using the intelligent system described in claim 1, characterized in that, The method includes the following steps: S1) Use the face image acquisition module to acquire a frontal face image of the user without a hat or makeup; S2) Use the facial feature analysis module to identify and analyze the facial features and semantic classification of user images; S3) Using the adaptive makeup analysis module, and based on the semantic classification tags of the identified user's eye shape, search the classification makeup image library for eye makeup images containing the semantic classification tags of that eye shape; based on the semantic classification tags of the identified user's lip shape, search the classification makeup image library for lip makeup images containing the semantic classification tags of that lip shape; and based on the semantic classification tags of the identified user's face shape, search the classification makeup image library for eyebrow makeup images, contouring makeup images, and blush makeup images containing the semantic classification tags of that face shape. S4) Using the virtual makeup module, based on the various makeup images recommended by the makeup analysis module, and combined with secondary or multi-level keywords, various makeup images are customized by automatic filtering and / or manual selection, and virtual makeup images are generated in real time and output to the display module. S5) Use the video tutorial module to play the corresponding makeup tutorial videos in the following order: base makeup → eyebrow makeup → eye makeup → contouring → blush → lip makeup.

9. A terminal device capable of providing personalized makeup instruction, characterized in that: The device is equipped with the intelligent system described in any one of claims 1 to 7 and includes a camera and a display unit.

Images