Facial flaw-based makeup suggestion intelligent recommendation method, system and equipment

Abstract

The invention discloses an intelligent makeup suggestion recommendation method, system and device based on facial flaws, and relates to the technical field of image processing, and the method comprises the steps: inputting a preprocessed front face image into a facial flaw segmentation model, so as to obtain a front face flaw segmentation result and a flaw type; based on the front face flaw segmentation result and flaw types, generating a face flaw evaluation result, including the positions of various types of flaws on the front face and the area of regions occupied by the various types of flaws; and inputting the preprocessed front face image and the corresponding face defect evaluation result into a preset question and answer BEiT V3 model to obtain a makeup recommendation result, including makeup, makeup skills and cosmetic categories. According to the invention, personalized make-up suggestions can be rapidly provided.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, and device for intelligent makeup suggestion recommendation based on facial blemishes. Background Technology

[0002] Makeup can enhance one's mental state in various daily situations, and more and more people are choosing to wear makeup in their daily lives, especially those with noticeable facial blemishes. Makeup can effectively cover up facial imperfections. However, the choice of makeup style, makeup techniques, and the selection of cosmetics are all perplexing issues that are crucial to the final makeup effect. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, and device for intelligently recommending makeup suggestions based on facial blemishes, so as to quickly provide personalized makeup advice.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] In a first aspect, the present invention provides a method for intelligently recommending makeup suggestions based on facial blemishes, comprising:

[0006] Obtain the preprocessed frontal face image;

[0007] The preprocessed frontal face image is input into the facial blemish segmentation model to obtain the frontal face blemish segmentation result and blemish type;

[0008] Based on the facial blemish segmentation results and blemish types, a facial blemish assessment result is generated; the facial blemish assessment result includes the location of each type of blemish on the front of the face and the area occupied by each type of blemish.

[0009] The preprocessed frontal image and the corresponding facial blemish assessment results are input into the preset question-answering BEiTV3 model to obtain makeup recommendation results; the makeup recommendation results include makeup style, makeup techniques and cosmetic categories.

[0010] Secondly, the present invention provides an intelligent recommendation system for makeup suggestions based on facial blemishes, comprising:

[0011] The frontal face image acquisition module is used to acquire preprocessed frontal face images;

[0012] The defect determination module is used to input the preprocessed frontal face image into the facial defect segmentation model to obtain the frontal face defect segmentation result and defect type;

[0013] The blemish assessment module is used to generate facial blemish assessment results based on the frontal blemish segmentation results and blemish types; the facial blemish assessment results include the location of each type of blemish on the frontal face and the area occupied by each type of blemish.

[0014] The makeup recommendation module is used to input the preprocessed frontal image and the corresponding facial blemish assessment results into a preset question-answering BEiT V3 model to obtain makeup recommendation results; the makeup recommendation results include makeup styles, makeup techniques and cosmetic categories.

[0015] Thirdly, the present invention provides an electronic device, characterized in that it includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to enable the electronic device to perform a smart recommendation method for makeup suggestions based on facial blemishes.

[0016] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0017] This invention discloses an intelligent makeup recommendation method, system, and device based on facial blemishes. A preprocessed frontal image is input into a facial blemish segmentation model to obtain the segmentation results and blemish types, thereby generating a facial blemish assessment result. This includes the location of each type of blemish on the frontal face and the area occupied by each type of blemish. In other words, by identifying facial skin blemishes, a structured facial assessment conclusion is generated. Then, the preprocessed frontal image and the corresponding facial blemish assessment result are input into a preset question-answering BEiTV3 model to obtain makeup recommendation results, including makeup styles, makeup techniques, and cosmetic categories. This invention can identify various facial blemishes of users. Through the preset question-answering BEiTV3 model, it can acquire a vast amount of beauty-related knowledge and quickly find personalized makeup recommendations. Specifically, by combining the facial image and facial blemish segmentation results, the preset question-answering BEiTV3 model outputs personalized beauty suggestions to help users solve makeup-related problems. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the intelligent makeup suggestion method based on facial blemishes according to the present invention.

[0020] Figure 2 This is a schematic diagram of the intelligent recommendation system for makeup suggestions based on facial blemishes according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To make the objectives, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1

[0024] like Figure 1 As shown, this invention provides a method for intelligent makeup recommendation based on facial blemishes, including:

[0025] Step 100: Obtain the preprocessed frontal face image, i.e., the face region image of the frontal face; specifically, obtain the frontal face image to be processed; using the face detection interface of Google's open-source Mediapipe, crop out the face region from the frontal face image to obtain the preprocessed frontal face image, i.e., the face region image of the frontal face. This invention crops out the face region, which can eliminate the interference of background elements on the performance of the facial blemish segmentation algorithm and allows the evaluation algorithm to focus on the state of the face, thus improving the accuracy of the evaluation algorithm.

[0026] Step 200: Input the preprocessed frontal face image into the facial blemish segmentation model to obtain the frontal face blemish segmentation result and blemish type; the blemish type includes acne pits, acne marks, sensitive redness, pigmentation, and enlarged pores.

[0027] The facial blemish segmentation model is obtained by training the SwinTransformer neural network using a first training sample set. The samples in the first training sample set include facial sample images with facial blemishes and corresponding facial blemish labels. The facial blemish labels include the blemish area and the blemish type, and are manually labeled, that is, the image data of facial blemishes are manually labeled, the range of the blemish area is defined and the type of blemish is labeled.

[0028] The SwinTransformer model adopts a hierarchical design, consisting of four stages. Except for the first stage, each stage first reduces the resolution of the input feature map through the PatchMerging layer and performs downsampling operations, expanding the receptive field layer by layer like a CNN in order to obtain global information.

[0029] Step 300: Based on the frontal blemish segmentation results and blemish types, generate facial blemish assessment results; the facial blemish assessment results include the location of each type of blemish on the frontal face and the area occupied by each type of blemish; the text structure of the facial blemish assessment results is: location of blemish on the frontal face + blemish type + area occupied by blemish; the area occupied by blemish is a percentage value.

[0030] Specifically, based on the facial blemish segmentation results (blemish areas and blemish types on the front of the face), the location and size of the facial blemishes (the size of the area is the number of pixels in the area) can be determined, generating a templated facial blemish assessment conclusion, such as "5% acne marks on the forehead, 10% sensitive redness on the left cheek".

[0031] Step 400: Input the preprocessed frontal image and the corresponding facial blemish assessment results into the preset question-answering BEiTV3 model to obtain makeup recommendation results; the makeup recommendation results include makeup style, makeup techniques and cosmetic categories.

[0032] The preset question-answering BEiTV3 model is obtained by training the multimodal BEiTV3 model using a second training sample set. The multimodal BEiTV3 model uses MultiwayTransformer as its core structure. The input of BEiTV3 is multimodal data, including images, text, and image-text pairs. The samples in the second training sample set include frontal facial blemish sample images and corresponding facial blemish assessment results, as well as makeup recommendation tags. The makeup recommendation tags include makeup styles, makeup techniques, and cosmetic categories.

[0033] The Multiway Transformer's structure consists of a shared multi-head self-attention mechanism (MSA) connecting the input layers, followed by three parameter-independent feedforward networks based on the different modalities of the input data. VLMo has 24 layers, each using the exact same three-expert structure. Unlike VLMo, BEiT v3 has 40 layers, but it only uses this three-expert (V-FFN, L-FFN, VL-FFN) structure in the last three layers, while the other layers use a structure consisting of a visual expert (V-FFN) and a language expert (L-FFN).

[0034] The process of constructing the cosmetic recommendation tags specifically includes:

[0035] 1) Crawling corpus information from internet beauty communities; the corpus information includes makeup techniques, makeup design, and cosmetic selection; 2) Annotators clean and verify the crawled corpus information, and annotate each type of blemish with the corresponding makeup, makeup techniques, and cosmetic categories to obtain a beauty recommendation database, which includes the correspondence between various skin problems (various types of blemishes) and suitable makeup, makeup techniques, and cosmetic categories; 3) Based on the frontal blemish sample images and the corresponding facial blemish assessment results, selecting corresponding makeup recommendation tags from the beauty recommendation database.

[0036] In a specific example, a preprocessed frontal image and the text of facial blemish assessment results are used as image-text pairs as input to the pre-defined question-answering BEiTV3 model. The model will output corresponding suitable makeup looks, makeup techniques, and cosmetic categories. The question-answering BEiTV3 model uses a hybrid model to jointly encode the image-text pairs using the input image and text. Based on the feature vectors obtained from both encodings, it searches the beauty recommendation database for the few records that best match the question and the user's blemish description. These records are then adjusted and optimized to ensure that the output question-answer statements are fluent and well-organized.

[0037] This invention uses both images and text (facial blemish assessment results) as input to the question-answering model, allowing the model to acquire information from two modalities simultaneously. This makes the model's output more targeted and provides personalized answers. Meanwhile, the manually cleaned beauty recommendation database ensures the reliability of the model's answers.

[0038] Example 2

[0039] like Figure 2 As shown, in order to implement the technical solution in Embodiment 1 and achieve the corresponding functions and technical effects, this embodiment also provides an intelligent recommendation system for makeup suggestions based on facial blemishes, including:

[0040] The frontal face image acquisition module is used to acquire preprocessed frontal face images.

[0041] The blemish determination module is used to input the preprocessed frontal face image into the facial blemish segmentation model to obtain the frontal face blemish segmentation result and blemish type.

[0042] The blemish assessment module is used to generate facial blemish assessment results based on the frontal blemish segmentation results and blemish types; the facial blemish assessment results include the location of each type of blemish on the frontal face and the area occupied by each type of blemish.

[0043] The makeup recommendation module is used to input the preprocessed frontal image and the corresponding facial blemish assessment results into a preset question-answering BEiT V3 model to obtain makeup recommendation results; the makeup recommendation results include makeup styles, makeup techniques and cosmetic categories.

[0044] Example 3

[0045] This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to cause the electronic device to execute the intelligent makeup suggestion method based on facial blemishes described in Embodiment 1. Optionally, the electronic device may be a server.

[0046] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the intelligent recommendation method for makeup suggestions based on facial blemishes as described in Embodiment 1.

[0047] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0048] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligently recommending makeup suggestions based on facial blemishes, characterized in that, The methods include: Obtain the preprocessed frontal face image; The preprocessed frontal face image is input into the facial blemish segmentation model to obtain the frontal face blemish segmentation result and blemish type; Based on the facial blemish segmentation results and blemish types, a facial blemish assessment result is generated; the facial blemish assessment result includes the location of each type of blemish on the front of the face and the area occupied by each type of blemish. The preprocessed frontal image and the corresponding facial blemish assessment results are input into the preset question-answering BEiTV3 model to obtain makeup recommendation results; the makeup recommendation results include makeup style, makeup techniques and cosmetic categories.

2. The intelligent makeup suggestion method based on facial blemishes according to claim 1, characterized in that, Obtain the preprocessed frontal face image, specifically including: Obtain the frontal image to be processed; The face detection interface of Google's open-source Mediapipe is used to crop out the face region in the frontal face image to be processed, so as to obtain the preprocessed frontal face image.

3. The intelligent makeup suggestion method based on facial blemishes according to claim 1, characterized in that, The facial blemish segmentation model was obtained by training the SwinTransformer neural network using the first training sample set; The samples in the first training sample set include facial sample images with facial blemishes and corresponding facial blemish labels; the facial blemish labels include the blemish area and the blemish type.

4. The intelligent makeup suggestion method based on facial blemishes according to claim 1, characterized in that, The types of blemishes include acne scars, acne marks, sensitivity and redness, pigmentation, and enlarged pores.

5. The intelligent makeup suggestion recommendation method based on facial blemishes according to claim 1, characterized in that, The text structure of the facial blemish assessment result is as follows: location of the blemish on the front of the face + blemish type + area occupied by the blemish; the area occupied by the blemish is a percentage value.

6. The intelligent makeup suggestion method based on facial blemishes according to claim 1, characterized in that, The preset question-answering BEiTV3 model is obtained by training the multimodal BEiTV3 model using the second training sample set; The multimodal BEiTV3 model uses MultiwayTransformer as its core structure. The samples in the second training sample set include frontal facial blemish sample images and corresponding facial blemish assessment results, as well as makeup recommendation tags; the makeup recommendation tags include makeup styles, makeup techniques, and cosmetic categories.

7. The intelligent makeup suggestion method based on facial blemishes according to claim 6, characterized in that, The process of constructing the cosmetic recommendation tags specifically includes: The data was crawled from online beauty communities, including information on makeup techniques, makeup designs, and cosmetic product selection. The crawled corpus information is cleaned and verified, and each type of flaw is labeled with its corresponding makeup, makeup techniques and cosmetic categories to obtain a beauty recommendation database. Based on the sample images of facial blemishes and the corresponding facial blemish assessment results, the corresponding makeup recommendation tags are selected from the beauty recommendation database.

8. A smart makeup suggestion system based on facial imperfections, characterized in that, The system includes: The frontal face image acquisition module is used to acquire preprocessed frontal face images; The defect determination module is used to input the preprocessed frontal face image into the facial defect segmentation model to obtain the frontal face defect segmentation result and defect type; The blemish assessment module is used to generate facial blemish assessment results based on the frontal blemish segmentation results and blemish types; the facial blemish assessment results include the location of each type of blemish on the frontal face and the area occupied by each type of blemish. The makeup recommendation module is used to input the preprocessed frontal image and the corresponding facial blemish assessment results into a preset question-answering BEiT V3 model to obtain makeup recommendation results; the makeup recommendation results include makeup styles, makeup techniques and cosmetic categories.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the intelligent recommendation method for makeup suggestions based on facial blemishes as described in any one of claims 1-7.

Images