Skin care suggestion intelligent recommendation method, system and equipment based on skin state detection

Abstract

The invention provides a skin care suggestion intelligent recommendation method, system and device based on skin state detection, and belongs to the field of intelligent computation.The method comprises the steps that a target skin image is obtained; detecting a skin state in the target skin image by adopting a pre-trained skin state detection model to obtain skin state data; determining a skin state conclusion according to the skin state data; acquiring a nursing problem input by a user; based on a pre-constructed skin care knowledge base, according to the care question, the skin state conclusion and the target skin image, determining an answer to the care question by adopting a question and answer model, and displaying the skin state conclusion and the answer to the care question; the question and answer model is obtained by training BEiTV3 by adopting a training sample set in advance; the accuracy of skin state detection is improved, and skin care suggestions can be recommended in a personalized manner.

Description

Technical Field

[0001] This invention relates to the field of intelligent computing, and in particular to a method, system, and device for intelligent recommendation of skin care suggestions based on skin condition detection. Background Technology

[0002] Various facial skin problems can easily affect and trouble people's daily lives. However, going to a professional medical institution for skin testing is time-consuming, laborious, and expensive. Related answers on the internet are varied and difficult to discern, and the sheer number of skincare products available makes it hard for users to choose the most suitable skincare routine. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, and device for intelligent skincare recommendation based on skin condition detection, which can improve the accuracy of skin condition detection and enable personalized skincare recommendations.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A smart skincare recommendation method based on skin condition detection includes:

[0006] Obtain the target skin image;

[0007] A pre-trained skin condition detection model is used to detect the skin condition in the target skin image to obtain skin condition data;

[0008] A conclusion on the skin condition is determined based on the aforementioned skin condition data;

[0009] Obtain the nursing questions input by the user;

[0010] Based on a pre-built skin care knowledge base, the answer to the nursing question is determined using a question-and-answer model according to the nursing question, the skin condition conclusion, and the target skin image, and the answer to the nursing question is displayed. The question-and-answer model is obtained by pre-training BEiT V3 with a training sample set. The training sample set includes multiple sample images, the skin condition conclusion corresponding to each sample image, and the sample nursing question.

[0011] Optionally, the skin condition detection model includes: an acne detection model, an acne scar segmentation model, an oily skin classification model, a skin color classification model, a sensitive skin classification model, a pore classification model, a blackhead classification model, and a dark circle segmentation model;

[0012] The acne detection model is obtained by training the end-to-end target detector DINO with an acne sample set in advance; the acne sample set includes multiple acne sample images, acne detection boxes in each acne sample image and corresponding acne types; the acne types include whiteheads, papules, pustules, nodules and cysts.

[0013] The acne scar segmentation model is obtained by training the UNet network with an acne scar sample set in advance; the acne scar sample set includes multiple acne scar sample images and the acne scar contours in each acne scar sample image.

[0014] The oil exudation classification model is obtained by training a residual neural network ResNet with an oil exudation sample set in advance; the oil exudation sample set includes multiple oil exudation sample images and the degree of oil exudation in each oil exudation sample image;

[0015] The skin color classification model is obtained by training a residual neural network ResNet with a skin color sample set in advance; the skin color sample set includes multiple skin color sample images and the skin color type of each skin color sample image;

[0016] The sensitive classification model is obtained by training a residual neural network ResNet with a sensitive sample set in advance; the sensitive sample set includes multiple sensitive sample images and the sensitivity type of each sensitive sample image;

[0017] The pore classification model is obtained by training a residual neural network ResNet with a pore sample set in advance; the pore sample set includes multiple pore sample images and the pore type of each pore sample image;

[0018] The blackhead classification model is obtained by training a residual neural network ResNet with a blackhead sample set in advance; the blackhead sample set includes multiple blackhead sample images and the blackhead type of each blackhead sample image;

[0019] The dark circle segmentation model is obtained by training the Swin Transformer with a dark circle sample set in advance; the dark circle sample set includes multiple dark circle sample images, dark circle detection boxes in each dark circle sample image and corresponding dark circle types.

[0020] The skin condition data includes: acne detection box, acne type, acne scar segmentation result, oiliness, skin tone type, sensitivity type, pore type, blackhead type, dark circle segmentation result, and dark circle type.

[0021] To achieve the above objectives, the present invention also provides the following solution:

[0022] A skincare recommendation system based on skin condition detection includes:

[0023] The image acquisition module is used to acquire the target skin image;

[0024] A state detection module, connected to the image acquisition module, is used to detect the skin state in the target skin image using a pre-trained skin state detection model to obtain skin state data;

[0025] The conclusion determination module is connected to the state detection module and is used to determine the skin state conclusion based on the skin state data.

[0026] The question retrieval module is used to retrieve nursing questions input by the user;

[0027] The answer determination module, connected to the image acquisition module, the conclusion determination module, and the question acquisition module, is used to determine the answer to the nursing question based on a pre-built skin care knowledge base, according to the nursing question, the skin condition conclusion, and the target skin image, using a question-and-answer model, and display the skin condition conclusion and the answer to the nursing question. The question-and-answer model is obtained by pre-training BEiTV3 using a training sample set. The training sample set includes multiple sample images, the skin condition conclusion corresponding to each sample image, and the sample nursing question.

[0028] To achieve the above objectives, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-described intelligent skincare suggestion recommendation method based on skin condition detection.

[0029] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The present invention uses a pre-trained skin condition detection model to detect the skin condition in a target skin image, obtains skin condition data, and then determines a skin condition conclusion based on the skin condition data. Only an image of the skin is needed to automatically provide a skin condition conclusion, thus improving the efficiency of skin condition detection. Subsequently, based on a pre-built skin care knowledge base, according to the user-input care question, the skin condition conclusion, and the target skin image, a question-answering model is used to determine the answer to the care question and display the skin condition conclusion and the answer to the care question. The present invention can provide personalized skin care suggestions and improve the user experience. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart of the intelligent skincare suggestion recommendation method based on skin condition detection provided by the present invention;

[0032] Figure 2 This is a schematic diagram of the intelligent skincare recommendation system based on skin condition detection provided by the present invention.

[0033] Symbol explanation: 1-Image acquisition module, 2-State detection module, 3-Conclusion determination module, 4-Question acquisition module, 5-Answer determination module. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] The purpose of this invention is to provide a method, system, and device for intelligent skincare advice recommendation based on skin condition detection. By assessing the condition of facial skin in various dimensions, it generates structured skin condition conclusions and, in conjunction with facial skin images, outputs answers to various questions related to facial skin care.

[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1

[0038] like Figure 1 As shown, this embodiment provides a method for intelligent skincare recommendation based on skin condition detection, including:

[0039] Step 100: Obtain the target skin image.

[0040] Specifically, a preliminary skin image is first acquired. Then, the face position in the preliminary skin image is detected. Next, the face region in the preliminary skin image is cropped based on the face position to obtain the target skin image.

[0041] The initial skin image contains a frontal view. The face detection interface of Google's open-source MediaPipe is used to detect the face location in the initial skin image. Cropping out the face region eliminates interference from background elements in the facial skin condition assessment. Furthermore, it allows the assessment algorithm to focus on the facial condition, improving the accuracy of facial skin condition detection.

[0042] Step 200: Use a pre-trained skin condition detection model to detect the skin condition in the target skin image to obtain skin condition data. The skin condition data includes: acne detection bounding box, acne type, acne scar segmentation result, oiliness level, skin color type, sensitivity type, pore type, blackhead type, dark circle segmentation result, and dark circle type.

[0043] Specifically, the skin condition detection model includes: an acne detection model, an acne scar segmentation model, an oily skin classification model, a skin color classification model, a sensitive skin classification model, a pore classification model, a blackhead classification model, and a dark circle segmentation model.

[0044] The acne detection model was pre-trained on an end-to-end target detector (DINO) using an acne sample set. The acne sample set includes multiple acne sample images, acne detection boxes in each image, and corresponding acne types. The acne types include whiteheads, papules, pustules, nodules, and cysts.

[0045] DINO is a Transformer-based object detection algorithm. Its structure mainly consists of an encoder and a decoder. This invention uses DINO to detect and identify whiteheads, papules, pustules, nodules, and cysts. During training, a large number of labeled acne sample images are used. The labeling rule is to use rectangular boxes to mark whiteheads, papules, pustules, nodules, and cysts in the acne sample images. When performing skin lesion detection, the input of the acne detection model is the target skin image, and the output is the coordinates of several acne detection boxes, the label corresponding to each acne detection box (whitehead, papule, pustules, nodules, cysts), and the confidence score. Based on the output of the acne detection model, the number of whiteheads, papules, pustules, nodules, and cysts can be counted.

[0046] The acne scar segmentation model is obtained by training the UNet network using an acne scar sample set. The acne scar sample set includes multiple acne scar sample images and the acne scar contours in each acne scar sample image.

[0047] When training the UNet network, a large number of labeled acne scar sample images are used. The labeling rule is to use polygons to mark the acne scars in the sample images. For acne scar segmentation, the input is the target skin image, and the output is the region of acne scars in the target skin image. Based on the detection results, the locations of obvious acne scars on the face can be marked, and the area ratio of the acne scars can be calculated.

[0048] The oil leakage classification model is obtained by training a residual neural network (ResNet) using an oil leakage sample set. The oil leakage sample set includes multiple oil leakage sample images and the degree of oil leakage for each image. The degree of oil leakage is categorized as no obvious oil leakage, slight oil leakage, moderate oil leakage, and severe oil leakage.

[0049] The skin color classification model is obtained by pre-training a ResNet residual neural network using a skin color sample set. The skin color sample set includes multiple skin color sample images and the skin color type of each image. Skin color types include translucent skin, fair skin, natural skin, wheat skin, dull skin, and dark skin.

[0050] The sensitivity classification model is obtained by pre-training a residual neural network ResNet using a set of sensitive samples. The set of sensitive samples includes multiple sensitive sample images and the sensitivity type of each sensitive sample image. The sensitivity types include no obvious sensitivity, mild sensitivity, severe sensitivity, and high sensitivity.

[0051] The pore classification model is obtained by pre-training a ResNet residual neural network using a pore sample set. The pore sample set includes multiple pore sample images and the pore type of each pore sample image. The pore types include no obviously enlarged pores, mildly enlarged pores, moderately enlarged pores, and severely enlarged pores.

[0052] The blackhead classification model is obtained by training a residual neural network (ResNet) using a blackhead sample set. The blackhead sample set includes multiple blackhead sample images and the blackhead type for each image. Blackhead types include no obvious blackheads, mild blackheads, moderate blackheads, and severe blackheads.

[0053] The dark circle segmentation model is obtained by training a Swin Transformer using a dark circle sample set. The dark circle sample set includes multiple dark circle sample images, dark circle detection boxes in each sample image, and the corresponding dark circle type. Dark circle types include vascular dark circles, pigmented dark circles, and shadow dark circles.

[0054] When training the Swin Transformer, a large number of labeled dark circle sample images are used. The labeling rule is to use polygons to mark the dark circles in the dark circle sample images, and at the same time label the corresponding dark circle type.

[0055] This invention, through extensive experimental testing, determined that UNet performs superiorly in acne scar segmentation and SwinTransformer performs superiorly in dark circle recognition. Therefore, UNet was chosen for acne scar detection, and SwinTransformer for dark circle recognition. By rationally selecting models for each detection dimension, the accuracy of the detection results was improved, and the skin detection results from each dimension led to a more comprehensive conclusion on skin condition.

[0056] Step 300: Determine the skin condition conclusion based on the skin condition data.

[0057] Specifically, the skin condition conclusions include: number of whiteheads, number of papules, number of pustules, number of nodules, number of cysts, percentage of acne scars, degree of oiliness, skin tone type, sensitivity type, pore type, and blackhead type.

[0058] Specifically, based on the acne detection bounding boxes and the acne types in each acne detection box in the target skin image, the number of whiteheads, papules, pustules, nodules, and cysts in the target skin image are determined. The proportion of acne marks in the target skin image is determined based on the area of ​​the acne mark segmentation results and the area of ​​the target skin image. Specifically, the proportion of acne marks is calculated by calculating the area of ​​the acne marks based on the acne mark detection boxes, and then dividing the area of ​​the acne marks by the area of ​​the target skin image.

[0059] This invention combines skin condition data to generate templated skin condition conclusions. For example: "Your facial condition is: 5 whiteheads, 2 papules, 0 pustules, 1 nodule, 0 cysts, acne scars accounting for 12.31%, no obvious oiliness, natural skin tone, mild sensitivity, no obvious enlarged pores, no obvious blackheads, vascular dark circles."

[0060] Step 400: Obtain the nursing question input by the user.

[0061] To enhance the user experience, this invention pre-generates multiple skincare questions for users to choose from, such as "What is the appropriate skincare routine / products?", "What skin problem is currently the most serious?", and "What are some ways to improve the current skin condition?".

[0062] Step 500: Based on a pre-built skin care knowledge base, and according to the nursing question, the skin condition conclusion, and the target skin image, a question-and-answer model is used to determine the answer to the nursing question, and the skin condition conclusion and the answer to the nursing question are displayed.

[0063] The question-answering model was trained on BEiTV3 using a training sample set. The training sample set includes multiple sample images, skin condition conclusions corresponding to each sample image, and sample care questions.

[0064] Specifically, step 500 includes:

[0065] (1) The nursing problem, the skin condition conclusion and the target skin image are encoded by the question-and-answer model to obtain the problem feature vector, the state feature vector and the graphic feature vector.

[0066] (2) Search the pre-built skin care knowledge base using a question-and-answer model for knowledge that has a similarity greater than a set threshold to the question feature vector, the state feature vector and the graphic feature vector, and use it as the answer to the care question.

[0067] The question-answering model uses question feature vectors, state feature vectors, and graphic feature vectors to search the skin care knowledge base for the most relevant and matching information about the user's facial condition. The skin care knowledge base contains a wealth of skincare-related web pages, popular science articles, and professional books, covering information on various skin problems and suggestions for improvement. The information has been manually cleaned to ensure its reliability and accuracy.

[0068] This invention uses both images (target skin images) and text (skin condition conclusions) as input to the question-answering model, enabling the model to acquire information from both modalities simultaneously and output targeted and personalized answers.

[0069] As a specific implementation method, preliminary skin images can be collected via mobile phone, and the algorithms of steps 100 to 500 can be packaged into a mobile application to process the preliminary skin images, generate skin condition conclusions and answers to care questions, and display them on the mobile phone.

[0070] Alternatively, a preliminary skin image can be captured using a camera, and the algorithms from steps 100 to 500 can be embedded into the computer's processor. The preliminary skin image is then processed by the processor to generate a conclusion on the skin condition and answers to care questions, which are then displayed on the screen.

[0071] This invention can identify various skin problems on a user's face. The question-and-answer model can acquire a vast amount of skin care-related knowledge. By combining the target skin image and skin condition conclusions, the question-and-answer model can output professional popular science content and provide personalized skin care advice for each user, helping users to improve their skin problems better and faster.

[0072] Example 2

[0073] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a skin care suggestion intelligent recommendation system based on skin condition detection is provided below.

[0074] like Figure 2 As shown, the intelligent skincare recommendation system based on skin condition detection provided in this embodiment includes: an image acquisition module 1, a condition detection module 2, a conclusion determination module 3, a question acquisition module 4, and an answer determination module 5.

[0075] The image acquisition module 1 is used to acquire the target skin image.

[0076] The state detection module 2 is connected to the image acquisition module 1. The state detection module 2 is used to detect the skin state in the target skin image using a pre-trained skin state detection model to obtain skin state data.

[0077] The conclusion determination module 3 is connected to the state detection module 2, and the conclusion determination module 3 is used to determine the skin state conclusion based on the skin state data.

[0078] The problem acquisition module 4 is used to acquire nursing problems input by the user.

[0079] The answer determination module 5 is connected to the image acquisition module 1, the conclusion determination module 3, and the question acquisition module 4 respectively. The answer determination module 5 is used to determine the answer to the nursing question based on the pre-built skin care knowledge base, the nursing question, the skin condition conclusion, and the target skin image, using a question-and-answer model, and to display the skin condition conclusion and the answer to the nursing question.

[0080] The question-answering model was trained on BEiTV3 using a training sample set. The training sample set includes multiple sample images, skin condition conclusions corresponding to each sample image, and sample care questions.

[0081] Compared to existing technologies, the skin condition detection-based intelligent skincare recommendation system provided in this embodiment has the same beneficial effects as the skin condition detection-based intelligent skincare recommendation method provided in Embodiment 1, and will not be repeated here.

[0082] Example 3

[0083] This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to execute the intelligent skincare recommendation method based on skin condition detection as described in Embodiment 1.

[0084] Alternatively, the aforementioned electronic device may be a server.

[0085] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the intelligent skincare recommendation method based on skin condition detection as described in Embodiment 1.

[0086] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0087] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligent skincare recommendation based on skin condition detection, characterized in that, The intelligent skincare recommendation method based on skin condition detection includes: Obtain the target skin image; A pre-trained skin condition detection model is used to detect the skin condition in the target skin image to obtain skin condition data; A conclusion on the skin condition is determined based on the aforementioned skin condition data; Obtain the nursing questions input by the user; Based on a pre-built skin care knowledge base, the answer to the nursing question is determined using a question-and-answer model according to the nursing question, the skin condition conclusion, and the target skin image, and the answer to the nursing question is displayed. The question-and-answer model is obtained by pre-training BEiT V3 with a training sample set. The training sample set includes multiple sample images, the skin condition conclusion corresponding to each sample image, and the sample nursing question.

2. The intelligent skincare recommendation method based on skin condition detection according to claim 1, characterized in that, Obtaining the target skin image specifically includes: Acquire preliminary skin images; Detect the face location in the preliminary skin image; The face region in the preliminary skin image is cropped based on the face location to obtain the target skin image.

3. The intelligent skincare recommendation method based on skin condition detection according to claim 1, characterized in that, The skin condition detection model includes: acne detection model, acne scar segmentation model, oily skin classification model, skin color classification model, sensitive skin classification model, pore classification model, blackhead classification model, and dark circle segmentation model; The acne detection model is obtained by training the end-to-end target detector DINO with an acne sample set in advance; the acne sample set includes multiple acne sample images, acne detection boxes in each acne sample image and corresponding acne types; the acne types include whiteheads, papules, pustules, nodules and cysts. The acne scar segmentation model is obtained by training the UNet network with an acne scar sample set in advance; the acne scar sample set includes multiple acne scar sample images and the acne scar contours in each acne scar sample image. The oil exudation classification model is obtained by training a residual neural network ResNet with an oil exudation sample set in advance; the oil exudation sample set includes multiple oil exudation sample images and the degree of oil exudation in each oil exudation sample image; The skin color classification model is obtained by training a residual neural network ResNet with a skin color sample set in advance; the skin color sample set includes multiple skin color sample images and the skin color type of each skin color sample image; The sensitive classification model is obtained by training a residual neural network ResNet with a sensitive sample set in advance; the sensitive sample set includes multiple sensitive sample images and the sensitivity type of each sensitive sample image; The pore classification model is obtained by training a residual neural network ResNet with a pore sample set in advance; the pore sample set includes multiple pore sample images and the pore type of each pore sample image; The blackhead classification model is obtained by training a residual neural network ResNet with a blackhead sample set in advance; the blackhead sample set includes multiple blackhead sample images and the blackhead type of each blackhead sample image; The dark circle segmentation model is obtained by training the Swin Transformer with a dark circle sample set in advance; the dark circle sample set includes multiple dark circle sample images, dark circle detection boxes in each dark circle sample image and corresponding dark circle types. The skin condition data includes: acne detection box, acne type, acne scar segmentation result, oiliness, skin tone type, sensitivity type, pore type, blackhead type, dark circle segmentation result, and dark circle type.

4. The intelligent skincare recommendation method based on skin condition detection according to claim 3, characterized in that, The skin condition conclusions include: number of whiteheads, number of papules, number of pustules, number of nodules, number of cysts, percentage of acne scars, oiliness, skin tone type, sensitivity type, pore type, and blackhead type; The conclusions regarding skin condition are determined based on the aforementioned skin condition data, specifically including: Based on the acne detection boxes in the target skin image and the acne type of each acne detection box, determine the number of whiteheads, papules, pustules, nodules, and cysts in the target skin image; The proportion of acne marks in the target skin image is determined based on the area of ​​the acne mark detection box in the target skin image and the area of ​​the target skin image.

5. The intelligent skincare recommendation method based on skin condition detection according to claim 1, characterized in that, Based on a pre-built skin care knowledge base, and according to the nursing question, the conclusion about the skin condition, and the target skin image, a question-answering model is used to determine the answer to the nursing question, specifically including: The nursing question, the skin condition conclusion, and the target skin image are encoded using a question-and-answer model to obtain question feature vector, state feature vector, and graphic feature vector. The question-answering model searches a pre-built skin care knowledge base for knowledge whose similarity to the question feature vector, the state feature vector, and the graphic feature vector is greater than a set threshold, and uses this knowledge as the answer to the care question.

6. A skincare recommendation system based on skin condition detection, characterized in that, The intelligent skincare recommendation system based on skin condition detection includes: The image acquisition module is used to acquire the target skin image; A state detection module, connected to the image acquisition module, is used to detect the skin state in the target skin image using a pre-trained skin state detection model to obtain skin state data; The conclusion determination module is connected to the state detection module and is used to determine the skin state conclusion based on the skin state data. The question retrieval module is used to retrieve nursing questions input by the user; The answer determination module, connected to the image acquisition module, the conclusion determination module, and the question acquisition module, is used to determine the answer to the nursing question based on a pre-built skin care knowledge base, according to the nursing question, the skin condition conclusion, and the target skin image, using a question-and-answer model, and display the skin condition conclusion and the answer to the nursing question. The question-and-answer model is obtained by pre-training BEiT V3 using a training sample set. The training sample set includes multiple sample images, the skin condition conclusion corresponding to each sample image, and the sample nursing question.

7. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the intelligent recommendation method for skin care suggestions based on skin condition detection as described in any one of claims 1 to 5.

Images