Intelligent system capable of realizing personalized beauty makeup teaching and application thereof

Abstract

The invention discloses an intelligent system capable of realizing personalized beauty makeup teaching and application thereof. The intelligent system comprises a video library provided with a plurality of classified makeup teaching video items with semantic classification labels, a classified makeup image library composed of a plurality of video frame images which come from the classified makeup teaching video items in the video library and have corresponding semantic classification labels, a face image acquisition module, a facial feature analysis module, a face recognition module and a face recognition module. According to the personalized makeup customization system and method, customization of personalized makeup can be achieved, it can be guaranteed that the customized personalized makeup has an accurate and corresponding teaching video, the customized personalized makeup can be practically and effectively implemented, and the personalized makeup can be more vivid and vivid. The customized makeup effect and the learned makeup effect can be kept highly consistent, the problem of difference between the actual makeup and the recommended makeup can be avoided, people can easily and quickly obtain personalized makeup guidance matched with themselves, and the method has remarkable application prospects.

Description

Technical Field

[0001] The present invention relates to an intelligent system capable of realizing personalized beauty makeup teaching and its application, belonging to the technical field of artificial intelligence. Background Art

[0002] With the pursuit of fashion and beauty by people, more and more people, especially the female group, begin to pay attention to the improvement of personal image. Because in social occasions, appropriate makeup can show respect for the event and attention to other participants. In some cultures, specific social activities also require participants to wear makeup to conform to etiquette. Therefore, appropriate makeup has become an indispensable part of personal life and social activities, and is a direct way of cultural expression and identity recognition.

[0003] However, not everyone is good at makeup, and the makeup requirements vary for different occasions. To obtain appropriate makeup, professional guidance is needed. Most people, especially beginners, often have poor makeup results because they do not have enough understanding of information such as their skin type and facial features, lack knowledge of makeup and makeup skills. As a result, the makeup they apply is not suitable for themselves or the makeup is uncoordinated, leading to a poor makeup effect and failing to achieve the ideal makeup effect.

[0004] Currently, people mainly learn makeup knowledge and skills by watching videos and articles provided by beauty makeup bloggers on the Internet, some beauty makeup APPs, or the personal instructions from relatives and friends around them. However, due to the different combinations of each person's skin type and facial features, the makeup teaching videos of online beauty makeup bloggers are not precisely targeted, and people cannot judge which beauty makeup blogger's makeup suits them, so these video teachings are not really practical. In addition, most of the existing beauty makeup APPs lack personalized guidance. For example, "Learn Makeup with Me" APP, "Easy Makeup Learning" APP, "Beauty Makeup Academy" APP, etc. These beauty makeup APPs only provide comprehensive teaching of makeup knowledge and skills. People usually need to learn a lot and then spend months or even longer time to try multiple times to explore the makeup that suits them and accumulate makeup skills. People cannot quickly find the ideal makeup that suits them. Moreover, the process of multiple attempts of applying and removing makeup on their own is not only time-consuming and costly, but also cumbersome in operation, and there is also a risk of damaging the facial skin. Although the existing beauty camera APPs can realize virtual makeup on the photographed photos to achieve personalized makeup looks, the personalized makeup looks output by the beauty camera APPs can only stay on the photos, and people cannot finally convert them into achievable actual makeup looks because they cannot accurately find the corresponding makeup teaching videos.

[0005] In addition, although there have been technical reports in the prior art on obtaining a user's facial image and intelligently recommending beauty makeup information that matches the facial image. For example, in a Chinese patent application with the application number CN201910357049.5, a beauty makeup recommendation method, system, and computer-readable storage medium are disclosed. The method includes: S1, obtaining the personalized characteristics of the user; S2, inputting the personalized characteristics into a decision matrix, and generating a recommendation result through calculation. The decision matrix includes a set of decision functions with multiple adjustable dimensions for single facial elements, and the recommendation result includes a set of beauty makeup means with multiple adjustable dimensions for single facial elements. This patent generates a recommendation result including a set of beauty makeup means with multiple adjustable dimensions for single facial elements by inputting the personalized characteristics of the user into a decision matrix composed of a set of decision functions with multiple adjustable dimensions for single facial elements, and can generate more refined beauty makeup recommendations. In a Chinese patent application with the application number CN202411540596.4, a beauty makeup recommendation method, device, equipment, and medium based on facial features are disclosed. The method includes: obtaining a face image of a target user; inputting the face image into a face feature extraction model to obtain the facial feature information of the target user, and recommending a target makeup look for the target user through a comprehensive recommendation mechanism based on the facial feature information. The comprehensive recommendation mechanism includes an algorithm recommendation mechanism, an expert recommendation mechanism, and a popular makeup look recommendation mechanism. Finally, according to the feedback information of the target user on the target makeup look, the weights corresponding to the algorithm recommendation mechanism, the expert recommendation mechanism, and / or the popular makeup look recommendation mechanism are adjusted respectively to obtain the updated weights of the algorithm recommendation mechanism, the expert recommendation mechanism, and / or the popular makeup look recommendation mechanism. The comprehensive recommendation mechanism recommends the target makeup look for the target user through the updated weights. However, since makeup itself involves numerous steps and processing techniques, under the premise that existing online beauty makeup teaching videos and beauty makeup APPs either lack precise pertinence and personalized guidance, or have a single function, or are too theoretical and lack practical operability, people, especially beginners who are not good at makeup, still find it difficult to effectively implement and achieve the effect of the recommended makeup look in reality, resulting in the lack of practicality and effectiveness of the recommended beauty makeup information.

[0006] In summary, it can be seen that there is an urgent need for a product that can achieve personalized beauty makeup teaching conveniently and efficiently, but there has been no report on related products and technologies so far. Summary of the Invention

[0007] Aiming at the above problems and requirements existing in the prior art, the purpose of the present invention is to provide an intelligent system capable of realizing personalized beauty makeup teaching and its application, so as to solve the defect problem that the prior art cannot provide personalized beauty makeup teaching. It can not only realize the intelligent customization and virtual makeup application of personalized makeup looks, but also, crucially, effectively implement the customized personalized makeup looks. It enables what is seen to be learnable, allowing people to easily and quickly obtain personalized makeup looks suitable for themselves.

[0008] To achieve the above-mentioned invention object, the present invention adopts the following technical solutions:

[0009] An intelligent system capable of realizing personalized beauty makeup teaching, comprising:

[0010] A video library, which is provided with a number of classified beauty makeup teaching video items with semantic classification labels;

[0011] A classified makeup image library, which is composed of a number of video frame images originating from the classified beauty makeup teaching video items in the video library and having corresponding semantic classification labels;

[0012] A face image acquisition module for obtaining a frontal face image of the user without a hat and without makeup;

[0013] A facial feature analysis module for identifying and analyzing the facial features and semantic classification of the user image;

[0014] A matching makeup analysis module for analyzing the semantic classification makeup that matches the result of the user's facial feature analysis;

[0015] A virtual makeup module for customizing the classified makeup image based on the semantic classification label and intuitively displaying the customized classified makeup effect, so as to customize the personalized makeup that the user is satisfied with;

[0016] A video teaching module for step-by-step playing the corresponding classified beauty makeup teaching video according to the finally customized classified makeup image and makeup process.

[0017] An implementation scheme, where the classified beauty makeup teaching video items in the video library are video entities (i.e., the video files themselves) or / and video link addresses (i.e., the URLs or paths pointing to the videos).

[0018] An implementation scheme, where the video library at least includes classified beauty makeup teaching video items with first-level semantic classification labels for base makeup, eyebrow makeup, eye makeup, contouring, blush, and lip makeup.

[0019] As a preferred scheme, each beauty makeup teaching video item in the base makeup category includes the teaching of each step from pre-makeup skin care → sunscreen isolation → concealer → setting powder.

[0020] As a preferred scheme, each beauty makeup teaching video item in the base makeup category is provided with a second-level semantic classification label for skin type (such as: dry, oily, combination oily, etc.).

[0021] As a preferred scheme, each beauty makeup teaching video item in the eyebrow makeup category is provided with a second-level semantic classification label for face shape (such as: square face, round face, oval face, triangular face, diamond face, rectangular face, etc.).

[0022] As a further preferred solution, each beauty makeup teaching video item in the eyebrow makeup category is also provided with multi-level semantic classification labels for makeup styles (such as: versatile, playful, gentle, Chinese style, ancient style, etc.) and eyebrow shapes (such as: standard eyebrows, arched eyebrows, willow leaf eyebrows, crescent eyebrows, etc.).

[0023] As a preferred solution, each beauty makeup teaching video item in the eye makeup category is provided with second-level semantic classification labels for eye shapes (such as: almond eyes, round eyes, phoenix eyes, peach blossom eyes, downturned eyes, puppy eyes, slanting eyes, etc.).

[0024] As a further preferred solution, each beauty makeup teaching video item in the eye makeup category is also provided with multi-level semantic classification labels for makeup styles (such as: cute puppy makeup, sweet and cool puppy makeup, pure and sexy big eye makeup, cool American makeup, etc.) and applicable scenarios (such as: daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0025] As a preferred solution, each beauty makeup teaching video item in the contour makeup category is provided with second-level semantic classification labels for face shapes (such as: square face, round face, oval face, triangular face, diamond face, rectangular face, etc.).

[0026] As a further preferred solution, each beauty makeup teaching video item in the contour makeup category is also provided with third-level semantic classification labels for makeup styles (such as: low-saturation contour, Korean-style contour, etc.).

[0027] As a preferred solution, each beauty makeup teaching video item in the blush makeup category is provided with second-level semantic classification labels for face shapes (such as: square face, round face, oval face, triangular face, diamond face, rectangular face, etc.).

[0028] As a further preferred solution, each beauty makeup teaching video item in the blush makeup category is also provided with multi-level semantic classification labels for makeup styles (such as: sweet, fresh, pure and sexy, cool, etc.) and applicable scenarios (such as: daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0029] As a preferred solution, each beauty makeup teaching video item in the lip makeup category is provided with second-level semantic classification labels for lip shapes (such as: cherry small lips, standard lips, thick lips, thin lips, smiling lips, pouting lips, etc.).

[0030] As a further preferred solution, each beauty makeup teaching video item in the lip makeup category is also provided with multi-level semantic classification labels for makeup styles (such as: 3D stereoscopic lips, Korean-style smiling lips, cute pouting lips, sensual manga lips, etc.) and applicable scenarios (such as: daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0031] An implementation solution is that the video frame images in the classified makeup image library are sourced from the beauty makeup teaching video items of each classification in the video library, including eye makeup, eyebrow makeup, lip makeup, contour makeup, and blush makeup.

[0032] An implementation solution is that the construction of the classified makeup image library includes the following steps:

[0033] a) Extract the makeup video frames of each classification that have completed the corresponding teaching makeup and are clear for the eye makeup, eyebrow makeup, lip makeup, contouring, and blush classification makeup teaching video items respectively from the video library;

[0034] b) Perform standardization processing on the extracted makeup video frames of each classification, including size, color space, pixel values, and file format;

[0035] c) Make semantic classification markings on the standardized makeup video frame images of each classification that are the same as the semantic classification labels of the corresponding classification makeup teaching video items.

[0036] An implementation scheme, the facial feature analysis module includes semantic classification of face shape, eye shape, and lip shape.

[0037] As a preferred scheme, the facial feature analysis module further includes semantic classification of skin type.

[0038] An implementation scheme, the method for the facial feature analysis module to achieve face shape classification includes the following steps:

[0039] A) Input the image into a facial feature detection model (such as the dlib model) for face key point location and detection;

[0040] B) Calculate according to the detected face key points: the width of the forehead, denoted as line1; the length from the cheekbone to the mandibular line, denoted as line2; the width of the cheekbone, denoted as line3; the vertical length of the face, denoted as line4; the radian of the mandibular angle, denoted as θ;

[0041] C) Calculate the standard deviation (Standard Deviation) σ of the lengths of the three line segments line1, line2, and line3. The specific calculation formula is as follows:

[0042] ;

[0043] D) Perform face shape classification according to the following recognition conditions:

[0044]

[0045] An implementation scheme, the method for the facial feature analysis module to achieve eye shape classification includes the following steps:

[0046] E) Input the image into a facial feature detection model (such as the dlib model) to obtain eye key points;

[0047] F) Calculate the following parameters of the left or right eye according to the detected left or right eye key points:

[0048] ① The length l between the inner corners of the two eyes;

[0049] ② The maximum width of the eyes (i.e., the distance between the highest point and the lowest point of the eyes) h;

[0050] ③ The eye shape ratio d = h / l;

[0051] ④ The angle α between the line connecting the inner corners of the two eyes and the horizontal line;

[0052] G) The following definitions are made based on the data provided by the face++ cloud service platform:

[0053]

[0054] H) Eye shape classification is carried out according to the following recognition conditions:

[0055]

[0056] An implementation scheme, the method for the facial feature analysis module to implement lip shape classification includes the following steps:

[0057] I) Input the image into a facial feature detection model (such as the dlib model) to obtain the key points of the lips;

[0058] J) Calculate the mouth length, the upward angle of the corners of the mouth, the obviousness of the M shape, the thickness of the upper lip, and the thickness of the lower lip according to the detected key points of the lips;

[0059] K) Lip shape classification is carried out according to the following recognition conditions:

[0060]

[0061] An implementation scheme, the adapted makeup analysis module is implemented by using the semantic classification information of the user's facial features by the facial feature analysis module and searching the classified makeup image library for keyword matching using the semantic classification labels.

[0062] A further implementation scheme, the adapted makeup analysis module searches the classified makeup image library for all eye makeup type makeup images containing the semantic classification label of the eye shape based on the semantic classification label of the user's eye shape.

[0063] A further implementation scheme, the adapted makeup analysis module searches the classified makeup image library for all lip makeup type makeup images containing the semantic classification label of the lip shape based on the semantic classification label of the user's lip shape.

[0064] A further implementation scheme, the adapted makeup analysis module searches the classified makeup image library for all eyebrow makeup type makeup images, contour makeup type makeup images, and blush makeup type makeup images containing the semantic classification label of the face shape based on the semantic classification label of the user's face shape respectively.

[0065] An implementation scheme, the virtual makeup module customizes various makeup images by automatically screening or / and manually checking according to various makeup images recommended by the adaptive makeup analysis module, combined with secondary or multi-level keywords, and uses local makeup transfer technology to generate makeup images in real time.

[0066] An implementation scheme, the virtual makeup module first uses a semantic segmentation model (such as U-Net, DeepLab) to generate segmentation masks for the customized classified makeup images and the user's frontal face image without makeup and without a hat respectively, and then uses the SSAT model for local makeup transfer and generates the makeup image after completing the local makeup transfer and outputs it to the display module.

[0067] An implementation scheme, the video teaching module sequentially plays the corresponding classified beauty makeup teaching videos step by step in the order of base makeup → eyebrow makeup → eye makeup → contouring → blush → lip makeup.

[0068] As a preferred scheme, the intelligent system is also provided with an intelligent beauty makeup assistant module for providing users with beauty makeup related information (such as: beauty makeup related tool and product information) and information services such as dressing and etiquette.

[0069] A method for realizing personalized beauty makeup teaching by applying the intelligent system of the present invention includes the following steps:

[0070] S1) Use the face image acquisition module to obtain the user's frontal face image without makeup and without a hat;

[0071] S2) Use the facial feature analysis module to identify and analyze the facial features and semantic classification of the user image;

[0072] S3) Use the adaptive makeup analysis module, and based on the semantic classification labels of the identified user's eye shape, search for eye makeup type makeup images in the classified makeup image library that contain the semantic classification label of this eye shape, based on the semantic classification labels of the identified user's lip shape, search for lip makeup type makeup images in the classified makeup image library that contain the semantic classification label of this lip shape, and based on the semantic classification labels of the identified user's face shape, respectively search for eyebrow makeup type makeup images, contouring type makeup images, and blush type makeup images in the classified makeup image library that contain the semantic classification label of this face shape;

[0073] S4) Use the virtual makeup module, according to various makeup images recommended by the adaptive makeup analysis module, and combine secondary or multi-level keywords to customize various makeup images by automatic screening or / and manual checking, and generate virtual makeup images in real time and output them to the display module;

[0074] S5) Use the video teaching module to sequentially play the corresponding classified beauty makeup teaching videos step by step in the order of base makeup → eyebrow makeup → eye makeup → contouring → blush → lip makeup.

[0075] A terminal device capable of realizing personalized beauty makeup teaching, which is installed with the intelligent system described in the present invention and is provided with a camera and a display unit.

[0076] An implementation scheme, the terminal device includes but is not limited to smart phones, tablets, computers, smart TVs, smart mirrors.

[0077] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0078] The present invention creates a video library consisting of several classified beauty makeup teaching video items with semantic classification labels and a classified makeup image library consisting of several video frame images from the classified beauty makeup teaching video items in the video library and having corresponding semantic classification labels. Then, it customizes the partial makeup from the classified makeup image library according to the semantic classification labels, thus ensuring that there are accurate corresponding teaching videos for the customized personalized makeup, solving the defect problem that the prior art cannot provide personalized beauty makeup teaching, enabling the customized personalized makeup to be effectively implemented, making everything visible learnable, and because the customized classified makeup images are all from the corresponding classified beauty makeup teaching video frames, the customized makeup effect can be highly consistent with the learned makeup effect, effectively avoiding the problem of differences between the actual makeup and the recommended makeup, enabling people to easily and quickly obtain personalized makeup guidance suitable for themselves; therefore, compared with the prior art, the present invention has significant progressiveness, application prospects and market value. Description of the Drawings

[0079] Figure 1 Shown is a structural block diagram of an intelligent system provided by the present invention that can realize personalized beauty makeup teaching.

[0080] Figure 2 Shown is a schematic diagram of the selection positions of key points on the eyes during eye shape recognition;

[0081] Figure 3 Shown is an interface diagram of the intelligent system provided by the embodiment when collecting a frontal face image of a person without wearing a hat and without makeup;

[0082] Figure 4 Shown is an interface diagram of the intelligent system provided by the embodiment for analyzing the facial features of the user's image;

[0083] Figure 5 Shown is an interface diagram of the intelligent system provided by the embodiment for recommending makeup images suitable for the user's eye makeup and lip makeup;

[0084] Figure 6 Shown is an interface diagram of the intelligent system provided by the embodiment for recommending makeup images suitable for the user's contouring and eyebrow makeup;

[0085] Figure 7The figure shows the interface diagram of the intelligent system provided by the embodiment for recommending the makeup images adapted to the user's blusher;

[0086] Figure 8 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the user's eye makeup customization and virtual makeup application;

[0087] Figure 9 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the user's eyebrow makeup customization and virtual makeup application;

[0088] Figure 10 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the user's lip makeup customization and virtual makeup application;

[0089] Figure 11 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the user's contour makeup customization and virtual makeup application;

[0090] Figure 12 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the user's blusher customization and virtual makeup application;

[0091] Figure 13 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the video teaching of base makeup and eyebrow makeup;

[0092] Figure 14 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the video teaching of eye makeup and contour makeup;

[0093] Figure 15 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the video teaching of blusher and lip makeup;

[0094] Figure 16 The figure shows the interface diagram of the intelligent system provided by the embodiment for realizing the skin type discrimination teaching. Detailed implementation manners

[0095] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings.

[0096] Please refer to Figure 1 As shown in the figure, an intelligent system capable of realizing personalized beauty makeup teaching provided by the present invention includes:

[0097] 1) A video library, which is provided with several classified beauty makeup teaching video items with semantic classification labels;

[0098] 2) A classified makeup image library, which is composed of several video frame images from the classified beauty makeup teaching video items in the video library and having corresponding semantic classification labels;

[0099] 3) A face image acquisition module for obtaining a frontal face image of the user without a hat and without makeup.

[0100] 4) A facial feature analysis module for identifying and analyzing the facial features and semantic classification of the user's image.

[0101] 5) A compatible makeup analysis module for analyzing the semantic classification makeup compatible with the results of the user's facial feature analysis.

[0102] 6) A virtual makeup module for customizing the classified makeup image based on the semantic classification label and intuitively displaying the customized classified makeup effect, so as to customize the personalized makeup that the user is satisfied with.

[0103] 7) A video teaching module for playing the corresponding classified beauty makeup teaching videos step by step according to the finally customized classified makeup image and makeup process.

[0104] Specifically, the classified beauty makeup teaching video items are video entities (i.e., the video files themselves) or / and video link addresses (i.e., the URLs or paths pointing to the videos); the classified beauty makeup teaching videos can be recorded and customized by inviting beauty bloggers, beauty professionals, beauty volunteers, etc., or / and, obtained by sorting and processing the video data obtained from the network platform; and the video library can be continuously increased and updated.

[0105] In addition, the video library contains at least the classified beauty makeup teaching video items of the first-level semantic classification labels of base makeup, eyebrow makeup, eye makeup, contouring, blush, and lip makeup. However, it should be noted that the video library may also include the classified beauty makeup teaching video items of other semantic classification labels, such as skin type discrimination teaching video items, makeup removal and skin care teaching video items, beauty tool use teaching video items, beauty product selection teaching video items, etc. They are not listed one by one here. In short, all kinds of classified beauty makeup teaching video items related to beauty makeup and with semantic classification labels can be in the video library.

[0106] As a preferred solution, each beauty makeup teaching video item of the base makeup category includes the teaching of each step from pre-makeup skin care → sunscreen and isolation → concealer → setting powder. Each beauty makeup teaching video item of the base makeup category is preferably provided with a second-level semantic classification label of skin type (such as: dry, oily, combination oily, etc.) to facilitate the accurate recommendation of corresponding base makeup products according to the user's skin type.

[0107] As a preferred solution, each beauty makeup teaching video item of the eyebrow makeup category is provided with a second-level semantic classification label of face shape (such as: square face, round face, oval face, triangular face, diamond face, rectangular face, etc.); as a further preferred solution, each beauty makeup teaching video item of the eyebrow makeup category is also provided with multi-level semantic classification labels of makeup style (such as: versatile, playful, gentle, Chinese style, ancient style, etc.) and eyebrow shape (such as: standard eyebrow, arched eyebrow, willow leaf eyebrow, crescent eyebrow, etc.).

[0108] As a preferred solution, each beauty makeup teaching video item in the eye makeup category is provided with a secondary semantic classification label for eye shapes (such as almond eyes, round eyes, phoenix eyes, peach blossom eyes, downturned eyes, puppy eyes, narrow eyes, etc.); as a further preferred solution, each beauty makeup teaching video item in the eye makeup category is also provided with multi-level semantic classification labels for makeup styles (such as cute puppy makeup, sweet and cool puppy makeup, pure and charming big eye makeup, cold and cool American makeup, etc.) and applicable scenarios (such as daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0109] As a preferred solution, each beauty makeup teaching video item in the contouring category is provided with a secondary semantic classification label for face shapes (such as square face, round face, oval face, triangular face, diamond face, rectangular face, etc.); as a further preferred solution, each beauty makeup teaching video item in the contouring category is also provided with a tertiary semantic classification label for makeup styles (such as low saturation contouring, Korean contouring, etc.).

[0110] As a preferred solution, each beauty makeup teaching video item in the blush category is provided with a secondary semantic classification label for face shapes (such as square face, round face, oval face, triangular face, diamond face, rectangular face, etc.); as a further preferred solution, each beauty makeup teaching video item in the blush category is also provided with multi-level semantic classification labels for makeup styles (such as sweet, fresh, pure and charming, cold and cool, etc.) and applicable scenarios (such as daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0111] As a preferred solution, each beauty makeup teaching video item in the lip makeup category is provided with a secondary semantic classification label for lip shapes (such as cherry small lips, standard lips, thick lips, thin lips, smiling lips, pouting lips, etc.); as a further preferred solution, each beauty makeup teaching video item in the lip makeup category is also provided with multi-level semantic classification labels for makeup styles (such as 3D stereoscopic lips, Korean smiling lips, cute pouting lips, fleshy comic lips, etc.) and applicable scenarios (such as daily makeup, banquet makeup, dating makeup, cos makeup, etc.).

[0112] It should be noted here that: the definition of the semantic classification label is not limited to the above description of this embodiment, and can be defined by oneself on the basis of the above inspiration; in addition, the semantic classification of face shapes, eye shapes, lip shapes, makeup styles, and applicable scenarios can also be defined by oneself on the basis of the above inspiration, and the classification can be increased or decreased and the arrangement order of the multi-level semantic labels can be changed, and the present application does not make special limitations on this.

[0113] In addition, in this embodiment, the video frame images in the classified makeup image library are sourced from the eye makeup, eyebrow makeup, lip makeup, contouring, and blush classified beauty makeup teaching video items in the video library; the construction of the classified makeup image library includes the following steps:

[0114] a) Extract the upper makeup and clear video frames of each category of makeup corresponding to the completed makeup teaching videos from the video library for eye makeup, eyebrow makeup, lip makeup, contouring, and blush respectively. Specifically, Python can be used to extract key frames based on the inter-frame difference;

[0115] b) Standardize the extracted video frames of each category of makeup, including size, color space, pixel values, and file format. Specifically, the OpenCV tool in the Python library can be used for standardization;

[0116] c) Make semantic classification marks on the video frame images of each category of makeup after standardization, which are the same as the semantic classification labels of the corresponding category of makeup teaching video items. For example, if the extracted video frame of a category of makeup is from a video file of an eyebrow makeup teaching video item with a semantic classification label of "eyebrow makeup / round face / willow leaf eyebrows", then the video frame image of this makeup after processing is also marked as "eyebrow makeup / round face / willow leaf eyebrows". The semantic classification marks for video frame images can be made using tools such as LabelMe or VoTT. Among them, LabelMe is an image annotation tool based on Python and QT, which supports semantic segmentation and instance segmentation tasks. It can export annotation data in VOC and COCO formats and is suitable for frame-by-frame annotation of video frames. VoTT (Visual Object Tagging Tool) is an open-source tool developed by Microsoft that supports the annotation of images and videos. It can export annotation data in multiple formats and supports integration with deep learning frameworks, making it suitable for semantic segmentation and object detection tasks.

[0117] In addition, in this embodiment, the facial feature analysis module includes semantic classification of face shape, eye shape, and lip shape. As a preferred solution, the facial feature analysis module further includes semantic classification of skin type.

[0118] In this embodiment, the method for the facial feature analysis module to implement face shape classification includes the following steps:

[0119] A) Input the image into a facial feature detection model (such as the dlib open-source model) for face key point localization and detection;

[0120] B) Calculate based on the detected face key points: the width of the forehead, denoted as line1; the length from the cheekbones to the mandibular line, denoted as line2; the width of the cheekbones, denoted as line3; the vertical length of the face, denoted as line4; the radian of the mandibular angle, denoted as θ;

[0121] C) Calculate the standard deviation (σ) of the lengths of the three line segments line1, line2, and line3. The specific calculation formula is as follows:

[0122] ;

[0123] D) Classify the face shape according to the following recognition conditions:

[0124]

[0125] An implementation scheme, the method for the facial feature analysis module to implement eye shape classification includes the following steps:

[0126] E) Input the image into a facial feature detection model (such as the dlib open-source model) to obtain the key points of the eyes;

[0127] F) Calculate the following parameters of the left or right eye according to the detected key points of the left or right eye:

[0128] ① The length l between the two eye corners;

[0129] ② The maximum width of the eye (i.e., the distance between the highest point and the lowest point of the eye) h;

[0130] ③ The eye shape ratio d = h / l;

[0131] ④ The angle α between the line connecting the two eye corners and the horizontal line;

[0132] Specifically, after obtaining the facial feature points, the eyes can be defined as shown in Figure 2 a1 to a 10 A total of 10 feature points, then the above parameters can be calculated according to the following formulas:

[0133] ;

[0134] G) Make the following definitions through the data provided by the face++ cloud service platform:

[0135]

[0136] H) Classify the eye shape according to the following recognition conditions:

[0137]

[0138] An implementation scheme, the method for the facial feature analysis module to implement lip shape classification includes the following steps:

[0139] I) Input the image into a facial feature detection model (such as the dlib model) to obtain the key points of the lips;

[0140] J) Calculate the mouth length, the upward angle of the mouth corners, the obviousness of the M shape, the thickness of the upper lip, and the thickness of the lower lip according to the detected key points of the lips;

[0141] K) Classify the lip shape according to the following recognition conditions:

[0142]

[0143] However, it should be noted here that the classification methods for face shape, eye shape, and lip shape are not limited to those described in this embodiment, and other existing technologies can also be adopted. For example, the DeepFace tool, which is an open-source face recognition project and supports multiple models (such as FaceNet, ArcFace, Dlib, etc.). In addition, the classification definitions of face shape, eye shape, and lip shape by this module are consistent with the corresponding classifications in the video library. The recognition conditions for face shape, eye shape, and lip shape can be obtained through empirical summary by analyzing the corresponding face shapes, eye shapes, and lip shapes in the video library. In the later stage, the features of the corresponding face shapes, eye shapes, and lip shapes in the machine learning video library can also be adopted and similarity can be introduced to improve the more refined matching classification of the user's facial features.

[0144] In addition, in this embodiment, the adapted makeup analysis module is implemented by using the semantic classification information of the user's facial features by the facial feature analysis module to search the classified makeup image library for keyword matching with semantic classification labels. Further, the adapted makeup analysis module searches the classified makeup image library for all eye makeup images containing the semantic classification label of the user's eye shape based on the semantic classification label of the user's eye shape. For example, if the semantic classification of the user's eye shape is "puppy eyes", then in the classified makeup image library, semantic search is performed with "eye makeup / puppy eyes", and all eye makeup images containing the semantic classification label "puppy eyes" of this eye shape are the adapted eye makeup analyzed by this module. Similarly, the adapted makeup analysis module searches the classified makeup image library for all lip makeup images containing the semantic classification label of the user's lip shape based on the semantic classification label of the user's lip shape. For example, if the semantic classification of the user's lip shape is "smiling lips", then in the classified makeup image library, semantic search is performed with "smiling lips", and all lip makeup images containing the semantic classification label "smiling lips" of this lip shape are the adapted lip makeup analyzed by this module. Similarly, the adapted makeup analysis module searches the classified makeup image library for all eyebrow makeup images, contour makeup images, and blush makeup images containing the semantic classification label of the user's face shape based on the semantic classification label of the user's face shape as the adapted eyebrow makeup, contour makeup, and blush makeup. That is to say, the eyebrow makeup images, contour makeup images, and blush makeup images are all searched for their respective adapted makeup images in the classified makeup image library according to the semantic classification label of the user's face shape.

[0145] In addition, in this embodiment, the virtual makeup module customizes various makeup images by automatically screening or / and manually checking according to various makeup images recommended by the adapted makeup analysis module, combined with secondary or multi-level keywords, and uses local makeup transfer technology to generate makeup images in real time. For example, if the user needs a "puppy dog eyes / daily" eye makeup look, the system will prioritize displaying eye makeup images that simultaneously contain the semantic classification labels of "puppy dog eyes / daily" based on the eye makeup images recommended by the adapted makeup analysis module for "puppy dog eyes" and the input secondary keyword "daily". Since there are different makeup styles classified for the eye makeup look suitable for "puppy dog eyes / daily", the user can further automatically screen by inputting keywords related to the makeup style, such as "cool and American", or directly manually check the interested makeup styles. After determining the eye makeup image for virtual makeup, the virtual makeup module first uses a semantic segmentation model (such as U-Net, DeepLab) to generate segmentation masks for the customized classified makeup image and the user's unmade-up frontal face image without a hat respectively, and then uses the SSAT (Symmetric Semantic-Aware Transformer Network) model for local makeup transfer and generates the makeup image after completing the local makeup transfer and outputs it to the display module. The user can replace the displayed makeup images of other makeup styles according to the makeup effect presented by the virtual makeup module until satisfied.

[0146] In the SSAT model, the mask of different regions of the human face can be extracted based on the semantic segmentation network through the following steps to achieve local makeup transfer. The main implementation steps are as follows:

[0147] I. Introduction of the semantic segmentation network

[0148] The SSAT model can extract the masks of different regions of the human face by introducing a pre-trained semantic segmentation network (such as DeepLab, U-Net or HRNet). These masks can divide the human face into multiple regions, such as eyes, lips, face, etc. The specific implementation steps are as follows:

[0149] ① Select the semantic segmentation model: Select a semantic segmentation model suitable for human face segmentation, such as DeepLab or HRNet;

[0150] ② Train or load the pre-trained model: Use a publicly available human face dataset (such as CelebA or FFHQ) to train the semantic segmentation model, or directly load the pre-trained model;

[0151] ③ Extract the mask: For the input human face image, use the semantic segmentation model to generate a segmentation mask and divide the human face into multiple regions.

[0152] II. Local Makeup Transfer

[0153] In the SSAT model, the specific steps for local makeup transfer using the extracted mask are as follows:

[0154] ① Feature extraction: Use the feature extraction module of the SSAT model to extract the features of the source image (i.e., the made-up image) and the target image (i.e., the non-made-up image);

[0155] ② Feature alignment: Use the semantic segmentation mask to align the makeup features of the source image (i.e., the made-up image) to the corresponding regions of the target image (i.e., the non-made-up image). For example, only perform feature alignment on the eye region;

[0156] ③ Feature fusion: Fuse the aligned makeup features with the features of the target image (i.e., the non-made-up image) to generate the result of local makeup transfer;

[0157] ④ Image generation: Use the generator of the SSAT model to generate the final local makeup transfer image.

[0158] III. Loss Function

[0159] To ensure the effect of local makeup transfer, a local loss term can be added to the loss function of the SSAT model. For example, pixel-level histogram loss can be used to constrain the makeup transfer effect in the local region. Specifically, the histogram loss can be calculated separately for local regions such as the eyes and lips to ensure the accurate transfer of makeup features.

[0160] In addition, in this embodiment, the video teaching module sequentially plays the corresponding classified beauty makeup teaching videos step by step in the order of base makeup → eyebrow makeup → eye makeup → contouring → blush → lip makeup.

[0161] As a preferred solution, the intelligent system is further provided with an intelligent beauty makeup assistant module for providing users with beauty makeup-related information (such as beauty makeup-related tools and product information) and information services such as dressing and etiquette.

[0162] The method for realizing personalized beauty makeup teaching by applying the intelligent system of the present invention includes the following steps:

[0163] S1) Use the face image acquisition module to obtain the frontal face image of the user without wearing a hat and without makeup; for example Figure 3 as shown, a high-quality frontal face image of the user without wearing a hat and without makeup can be obtained by embedding a frontal face shooting frame and providing semantic prompts such as "do not have bangs blocking";

[0164] S2) Use the facial feature analysis module to identify and analyze the facial features and semantic classification of the user image, such as Figure 4 the facial feature analysis result shown; Figure 4The analysis of the user's original eyebrow shape features is also provided herein. However, it should be noted that since the eyebrow shape in makeup is usually determined by the face shape to select a suitable eyebrow makeup, and it has little relation to the original eyebrow shape, the analysis content of the original eyebrow shape features can also be omitted.

[0165] S3) Use the adapted makeup analysis module, and based on the semantic classification labels of the recognized user's eye shape, search for eye makeup type makeup images in the classified makeup image library that contain the semantic classification label of this eye shape; based on the semantic classification labels of the recognized user's lip shape, search for lip makeup type makeup images in the classified makeup image library that contain the semantic classification label of this lip shape; based on the semantic classification labels of the recognized user's face shape, search for eyebrow makeup type makeup images, contour makeup type makeup images, and blush makeup type makeup images in the classified makeup image library that contain the semantic classification label of this face shape respectively. For example, based on Figure 4 the facial features shown, the makeup images of the adapted eye makeup and lip makeup are as shown in Figure 5 ; the makeup images of the adapted contour and eyebrow makeup are as shown in Figure 6 ; the makeup images of the adapted blush are as shown in Figure 7 . In addition, more adapted makeup images can also be viewed by clicking the "MORE" button below each type of makeup display column.

[0166] S4) Use the virtual makeup application module to customize various makeup images according to the various makeup images recommended by the adapted makeup analysis module, and combine secondary or multi-level keywords to perform automatic screening or / and manual selection, and generate virtual makeup application images in real time and output them to the display module. For example, as shown in Figure 8 , since the eye makeup images according to the semantic classification label of "puppy eyes / daily" are also classified into different makeup styles such as cute puppy makeup and sweet and cool puppy makeup, users can further perform automatic screening by inputting keywords about the makeup style, such as "cute", or directly manually select the interested makeup style of "cute puppy makeup". When the eye makeup image is determined, the corresponding virtual makeup application eye makeup image can be generated through local makeup transfer. Similarly, the personalized customization and real-time virtual makeup application display of the corresponding makeup effects of the eyebrow makeup shown in Figure 9 , the lip makeup shown in Figure 10 , the contour shown in Figure 11 , and the blush shown in Figure 12 can be carried out for each classified makeup image.

[0167] S5) Use the video teaching module to play the corresponding classified beauty makeup teaching videos step by step in the order of base makeup → eyebrow makeup → eye makeup → contour → blush → lip makeup, for example, as shown in Figures 13 to 15 .

[0168] In addition, it should be noted here that for skin type analysis, one can also conduct self-testing and discrimination of skin types through the corresponding skin type discrimination teaching video items provided in the video library. If there are difficulties in the discrimination, one can also rely on the AI assistant analysis function set by the system for auxiliary judgment. For example Figure 16 as shown

[0169] Furthermore, the intelligent system described in the present invention may also include a personalized data storage module and an intelligent beauty assistant. The personalized data may include information such as facial feature analysis data, favorite makeup images, favorite teaching videos, favorite cosmetics, etc.

[0170] A terminal device capable of realizing personalized beauty makeup teaching, which is installed with the intelligent system described in the present invention and is provided with a camera and a display unit. The terminal device includes but is not limited to smart phones, tablets, computers, smart TVs, smart mirrors.

[0171] Finally, it is necessary to note here that: The program modules described in this application may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions, and the computer-executable instructions, when executed, cause at least a part of the functions described in this application to be executed. The software components can be coded in any of various programming languages. It should be understood that, without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The protection scope of the present invention is not limited to the specific embodiments described above. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An intelligent system capable of implementing personalized makeup teaching, characterized in that: include: A video library, which has several classified beauty tutorial video items with semantic classification labels; A classified makeup image library, which is composed of a number of video frame images from classified makeup tutorial video items in the video library and having corresponding semantic classification labels; A face image acquisition module is used to obtain a frontal face image of the user without a hat or makeup; Facial feature analysis module, used to identify and analyze facial features and semantic classification of user images; Adaptive makeup analysis module, used to analyze the semantic classification makeup that is adapted to the user's facial feature analysis results; A virtual makeup module is used to customize classified makeup images based on semantic classification labels and intuitively display the customized classified makeup effects, so as to customize a personalized makeup look that satisfies the user; The video teaching module is used to play the corresponding classified makeup teaching videos step by step according to the final customized classified makeup images and makeup processes.

2. The intelligent system according to claim 1, characterized in that: The classified makeup tutorial video items in the video library are video entities and / or video link addresses.

3. The intelligent system according to claim 1, characterized in that: The video library at least includes classified makeup teaching video items with first-level semantic classification labels for base makeup, eyebrow makeup, eye makeup, contouring, blush, and lip makeup.

4. The intelligent system according to claim 1, characterized in that: The video frame images in the classified makeup image library are derived from the eye makeup, eyebrow makeup, lip makeup, contouring, and blush makeup teaching video items in the video library.

5. The intelligent system according to claim 1, characterized in that: The facial feature analysis module includes semantic classification of face shape, eye shape and lip shape.

6. The intelligent system according to claim 1, characterized in that: The adaptive makeup analysis module is based on the semantic classification information of the user's facial features by the facial feature analysis module, and uses semantic classification tags to search the classified makeup image library for keyword matching.

7. The intelligent system according to claim 1, characterized in that: The virtual makeup module customizes various makeup images based on the various makeup images recommended by the adaptive makeup analysis module, combined with secondary or multi-level keywords, using automatic screening and / or manual selection, and uses local makeup migration technology to generate makeup images in real time.

8. The intelligent system according to claim 1, characterized in that: The video teaching module plays the corresponding classified beauty makeup teaching videos step by step in the order of foundation makeup → eyebrow makeup → eye makeup → contouring → blush → lip makeup.

9. A method for implementing personalized makeup teaching using the intelligent system of claim 1, characterized in that: The method comprises the following steps: S1) using a face image acquisition module to obtain a frontal face image of the user without a hat or makeup; S2) using a facial feature analysis module to identify and analyze facial features and semantic classification of the user image; S3) using the adapted makeup analysis module, and based on the semantic classification label of the identified user's eye shape, searching the classified makeup image library for eye makeup images containing the semantic classification label of the eye shape, searching the classified makeup image library for lip makeup images containing the semantic classification label of the lip shape, and based on the semantic classification label of the identified user's face shape, searching the classified makeup image library for eyebrow makeup images, contouring makeup images, and blush makeup images containing the semantic classification label of the face shape; S4) using the virtual makeup module to customize various makeup images based on various makeup images recommended by the makeup analysis module and in combination with secondary or multi-level keywords by automatic screening or / and manual selection, and generating virtual makeup images in real time and outputting them to the display module; S5) Use the video teaching module to play the corresponding classified makeup teaching videos in order of foundation makeup → eyebrow makeup → eye makeup → contouring → blush → lip makeup.

10. A terminal device capable of implementing personalized makeup teaching, characterized in that: The intelligent system according to any one of claims 1 to 8 is installed and provided with a camera and a display unit.

Images