Detecting and visualizing skin signs using heat maps

Abstract

The present invention relates to a cosmetic and non-therapeutic method for detecting and quantifying skin signs in a region of interest of a user, in particular eye contours, cheeks, mouth contours, forehead, said method comprising: (i) acquiring an input image of the region of interest (112), (ii) analyzing the region of interest in order to provide a score of the skin signs, (iii) determining the score of the skin signs in a heat map (12, 14). (iv) visualizing the region of interest (111) in the form of the heat map (12, 22), (iv) merging the heat map (12, 22) with an input image of the region of interest (111).

Description

[0001] The subject of the present invention is a cosmetic and non-therapeutic method for detecting and quantifying skin signs in an area of ​​concern of a user. The invention also relates to a system for implementing such a method.

[0002] More generally, a cosmetic product is a product as defined in Regulation (EC) No 1223 / 2009 of the European Parliament and of the Council of 30 November 2009 on cosmetic products. Prior art

[0003] Computer-assisted analysis of the skin has become increasingly common over the past decade with the availability of controlled lighting systems and digital image capture and processing capabilities.

[0004] Because of the importance of facial appearance, computer-assisted analysis research has focused primarily on the skin of the face. There are commercially available systems for imaging the skin of the face that can capture digital images in a controlled manner. These systems are typically coupled to computer analysis systems that are used to display and quantify visible features of the skin in standard white light images (such as hyperpigmented spots, wrinkles, and texture), as well as invisible features in fluorescence or hyperspectral absorption images (such as UV spots).

[0005] Document US8218862 discloses a method for generating a skin mask defining a region of interest (ROI) in a skin image, the method comprising:

[0006] Detect skin in a skin image to generate a skin map. Skin detection includes:

[0007] performing a raw skin segmentation operation on the skin image;

[0008] Converting the original skin segmented skin image into an image in a color space having at least three channels; and

[0009] filtering two of the at least three channels;

[0010] generating a melanin index image based on the filtered channels; and

[0011] performing a thresholding operation on the melanin index image so as to separate a skin area and a non-skin area in the skin image;

[0012] providing an initial contour based on the skin map; and

[0013] The initial contour is optimized to generate a contour of a skin mask, the skin map and the skin mask each comprising at least one natural limit of the skin.

[0014] MED'COM is also well known for the six "Atlas des signes du viellissement cutané [Atlas of signs of skin aging]" published since 2007. These works systematically study and identify the characteristics and classifications of the skin in relation to age and geography.

[0015] By identifying several features of the face, they allow professionals to qualify and quantify the signs of aging based on the development of these features. They define these criteria (openness of pores, depth of forehead wrinkles and crow's feet) and assign them a severity score, which starts at zero and can go up to nine.

[0016] These atlases represent part of the knowledge on which tools for assessing skin signs are based. Their performance for assessing beauty products or for modeling the dynamics of aging progression based on hundreds of thousands of photographs is known.

[0017] To automate the diagnosis, algorithms are fed with data from the atlas by associating photos with scores. These algorithms are then trained to score these different areas of observation.

[0018] By studying thousands of selfies taken around the world, an AI-based algorithm has learned to identify various signs of facial aging in any type of photo.

[0019] By combining data derived from selfies with information about the user's rhythm of life, sleep or surroundings, these diagnostics now make it possible to provide the user with an estimated "aging curve" that can be compared with the average of people of the same age.

[0020] Patent application WO 2020169214 filed by the present applicant discloses predicting skin aging by taking into account a collection of images and the user's surroundings or habits (particularly sleep time, pollution and / or living space). By combining data from selfies with information such as the user's pace of life, sleep or surroundings, this prediction provides an "aging curve" based on these data.

[0021] The performance of the above methods needs to be further improved to better:

[0022] Combines different features to provide a complete and accurate facial diagnosis.

[0023] Personalize beauty treatments / products based on the diagnosis, either by creating a new product or by choosing from a given range.

[0024] Personalize the user experience by displaying the results of the diagnosis and / or by simulating its development to enable knowledge sharing. Summary of the invention

[0025] The present invention relates to a cosmetic and non-therapeutic method for detecting and quantifying skin signs in a user's area of ​​interest, particularly an eye contour, a cheek, a mouth contour, and a forehead, the method comprising: (i) obtaining an input image of the user's area of ​​interest, the area of ​​interest comprising an area of ​​interest containing skin signs; (ii) analyzing the area of ​​interest by comparing with reference scores pre-recorded in a reference database to provide scores for these skin signs; (iii) visualizing the area of ​​interest in the form of a heat map to detect these skin signs, the heat map being obtained by an algorithm focused on these skin signs; and (iv) merging the heat map with the input image of the area of ​​interest to provide an output image of the area of ​​interest and optimizing the visualization of these skin signs.

[0026] According to the method of the present invention, images are processed to obtain an accurate assessment of skin signs that exceeds the prior art in terms of accuracy and inclusiveness. The system is based on the knowledge of decades of research and data stored in reference databases covering all the various skin types.

[0027] A “reference database” may include:

[0028] data stored together with one or more image sequences comprising at least two images,

[0029] one or more atlases displayed on a screen or printed on a medium or stored in some form on a computer storage medium, such as the atlases described in patent application WO 2011 / 141769, each atlas representing various levels of at least one feature of body typology,

[0030] Synthetic images,

[0031] Internet publications.

[0032] The data can be processed by artificial intelligence (AI) algorithms, which can include fuzzy logic, neural networks, genetic programming, and decision tree programming.

[0033] All engines can be trained based on inputs such as information about products, expert advice, user profiles, or data based on sensory perception. Using the inputs, the AI ​​engine can implement an iterative learning process.

[0034] Training can be based on a variety of learning rules or training algorithms and scientific publications such as the following:

[0035] Frederic Flament et al., Effect of the sun on visible clinical signs of aging in Caucasians skin, Clin Cosmet Investig Dermatol, 27 Sep 2013, 6:221-32.

[0036] Frederic Flament et al., Assessing changes in some facial signs offatigue in Chinese women, induced by a single working day, International Journal of Cosmetic Science, 29 November 2018.

[0037] The present invention also relates to a cosmetic and non-therapeutic method for temporal monitoring of an area of ​​concern of a user, the method comprising: detecting and quantifying skin signs in the area of ​​concern as described above; temporal monitoring of output images obtained after comparing output images generated at different moments, and temporal monitoring of scores obtained after comparing scores generated at different moments.

[0038] The present invention also relates to a cosmetic and non-therapeutic method for detecting and quantifying skin signs in a user's area of ​​concern, the method comprising: a communication device suitable for communicating with a user computing device and configured to import at least one input image including a representation of the user's area of ​​concern from an image library of the user computing device; a processing device configured to: (ii) analyze the area of ​​concern by comparing with reference scores pre-recorded in a reference database to provide a score for the skin signs; (iii) visualize the area of ​​concern in the form of a heat map to detect the skin signs, the heat map being obtained by an algorithm focused on the skin signs; and (iv) merge the heat map with the input image of the area of ​​concern to provide an output image of the area of ​​concern and optimize the visualization of the skin signs.

[0039] The invention also relates to a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method as described above.

[0040] The invention finally relates to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement a method as described above. Preferred Embodiments

[0041] Preferably, the cosmetic and non-therapeutic method for detecting and quantifying skin signs in an area of ​​concern of a user according to the present invention exhibits one or more of the following features considered alone or in combination:

[0042] The method includes: (v) extracting skin signs of the region of interest by converting the heat map into a color space image; and (vi) generating an output image of the region of interest based on the color space image and the input image.

[0043] The learning libraries are inclusive, covering all ages, ethnic origins, and photo types, and all libraries are scored using common criteria to obtain comparable measurements.

[0044] Skin sign detection, evaluation of the scores of these skin signs, provision of the heat map and creation of the final image are achieved via image processing or computer vision algorithms.

[0045] The machine learning model is a pre-trained convolutional neural network.

[0046] The machine learning model is fed with data from an atlas that correlates pre-recorded images with pre-recorded severity scores.

[0047] These skin signs are selected from the group consisting of pore openness, skin texture, wrinkles, fine lines, crow's feet, sagging, color of under-eye shadows, depth of under-eye shadows, skin texture, facial expression, morphology, hydration, radiance or spot color.

[0048] The detection of the skin signs, the evaluation of the scores of the skin signs, the provision of the heat map and the creation of the output image are achieved via a machine learning model.

[0049] The machine learning model is a pre-trained convolutional neural network.

[0050] The machine learning model is fed with data from an atlas that correlates pre-recorded images with pre-recorded severity scores.

[0051] The machine learning model has been trained to score different skin signs.

[0052] The method includes filtering of at least one channel.

[0053] The method includes applying a contour to the heat map based on directional masking lines and reference points.

[0054] The method further includes recommending at least one beauty product based on the severity score.

[0055] Preferably, the cosmetic and non-therapeutic method for temporal monitoring of an area of ​​concern of a user according to the present invention exhibits one or more of the following features considered alone or in combination:

[0056] This heatmap is obtained from a Generative Adversarial Network (GAN).

[0057] The output image includes obtaining the data provided by the user when answering the questionnaire and processing the data through the machine learning model.

[0058] The method further includes recommending at least one beauty product or beauty procedure based on the output image and the score.

[0059] The invention will be better understood by reading the following detailed description of non-limiting exemplary embodiments and by studying the accompanying schematic and partial drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0060] [ Figure 1 ]

[0061] [ Figure 1 ] is a block diagram of a method for detecting and quantifying under-eye shadows according to the present invention.

[0062] [ Figure 2 ]

[0063] [ Figure 2 ] is a block diagram of a method for detecting and quantifying crow's feet according to the present invention.

[0064] [ Figure 3 ]

[0065] [ Figure 3 ] is a block diagram of aging prediction associated with beauty product recommendations.

[0066] [ Figure 4 ]

[0067] [ Figure 4 ] is a schematic diagram showing the various steps of the consumer experience based on implementing the method according to the present invention in an application on a mobile phone or tablet.

[0068] [ Figure 5 ]

[0069] [ Figure 5 ] shows[ Figure 4 ] is a screen of the application showing the questionnaire.

[0070] [ Figure 6 ]

[0071] [ Figure 6 ] shows that during the processing of input data, [ Figure 4 ] screen of the application in .

[0072] [ Figure 7 ]

[0073] [ Figure 7 ] shows[ Figure 4 ], which shows the user the results of the assessment of skin signs.

[0074] [ Figure 8 ]

[0075] [ Figure 8 ] shows[ Figure 4 ], which shows the user the screen of the application in [ Figure 7 ] for details of the results.

[0076] [ Fig. 9 ]

[0077] [ Fig. 9 ] shows[ Figure 4 ] showing the analysis report to the user.

[0078] [ Fig.10 ]

[0079] [ Fig.10 ] shows[ Figure 4 ] showing product recommendations to the user.

[0080] [ Fig.11 ]

[0081] [ Fig.11 ] shows[ Figure 4 ] showing a QR code assigned to a user.

[0082] like[ Figure 1 ] as shown, the method according to the present invention comprises the following steps: a processing unit extracts an image of a focus area 111 from a first input image of a focus area 112. In the example shown, the focus area 112 is formed by the face; the focus area 111 is formed by the lower contour of the eye. The skin signs present in the focus area 111 are wrinkles, fine lines and shadows.

[0083] The image of the region of interest 112 may be captured using an automated and controlled skin image capture system, such as the VISIA Skin Color Analysis System available from Canfield Scientific for analyzing facial skin.

[0084] exist[ Figure 1 ] In the exemplary embodiment of the present invention, the image is captured using standard light, which can be represented in the form of an RGB (red, green, blue) color image. However, the image can also be captured using a different illumination modality or using a multispectral imaging system, as long as the area of ​​interest 111 can be distinguished based on the skin index measurement results (e.g., the concentration of melanin and / or hemoglobin) derived from the captured image.

[0085] Several skin detection algorithms have been developed for various purposes, including for face detection, as described, for example, by RL Hsu et al. in "Face detection in color images," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 5, pp. 696-707, May 2002. If the detection algorithms provide a suitable level of granularity for such skin, then these algorithms can be used to detect regions of interest in accordance with the present invention.

[0086] The skin detection procedure uses measurements such as the individual type angle (ITA), which is used as a skin indicator. The ITA is calculated using the L* and b* channels of the transformed skin image CIE L*a*b* (hereafter referred to as L*a*b). (For a detailed description of this metric, see GN Stamatas et al., "Non-Invasive Measurements of Skin Pigmentation In Situ," Pigment Cell Research, vol. 17, pp. 618–626, 2004.) The ITA for each image pixel (i,j) is defined as arctan(L*[i,j]-50) / b*[i,j]) and is related to the concentration of melanin in the skin. The assumption is that the ITA values ​​of skin pixels will cluster around one value, while the ITA values ​​of pixels without skin will differ greatly from the ITA values ​​of skin pixels.

[0087] In the example shown, the method comprises step 1: a processing unit generates a heat map 12 based on an image of the region of interest 111. In step 2, the heat map 12 is converted into a color space image 13 having a green channel 131, a blue channel 132 and a yellow channel 133 in order to visualize shadow lines 130. The blue and green channels are best suited for showing spots and pigmented pores, as these features have a greater absorption in the blue and green spectrum.

[0088] from[ Figure 1]As can be seen, a heat map is a graphical representation of statistical data, which is assigned on a two-dimensional matrix according to the intensity of the variable magnitude, tonal range, or color chart.

[0089] In step 3 , the color space image 13 is combined with the input image of the region of interest 111 to produce an output image 14 showing the shading pattern 130 of the color space image on the image of the region of interest 111 .

[0090] [ Figure 2 ] The block diagram shown includes the steps of acquiring a first image of the face 21 via an image acquisition device.

[0091] Thereafter, the method comprises the following steps: the processing unit extracts an image of a region of interest 111 from the first acquired image of the region of interest (i.e. the face). In the example shown, the region of interest 111 is formed by the corners of the eyes. The skin signs present in the region of interest 111 are mainly wrinkles and fine crow's feet.

[0092] Then, the method comprises the following steps: the processing unit generates a heat map 22 based on the image of the region of interest 111 .

[0093] In step 22 , the heat map 22 is converted into a color space image 23 having a green channel 131 and a blue channel 132 in order to visualize crow's feet wrinkles 230 .

[0094] In step 23 , the color space image 23 is combined with the input image to produce an output image 24 which shows crow's feet 230 of the color space image 23 on the image of the region of interest 111 .

[0095] Thereafter, the method comprises the following steps: The processing unit may extract facial data of skin signs from the output image and / or assign a score. A time indication of the image acquisition time may be optionally associated with these facial data. The facial data and their time indication may be stored in a memory.

[0096] The method optionally includes temporal monitoring of the processing of shadow streaks based on input images acquired at different times (eg, every day).

[0097] Thus, the time monitoring may include the step of the image acquisition device acquiring a new image.

[0098] Thereafter, the time monitoring comprises the following steps: the processing unit compares the facial data extracted from the last acquired image with the facial data extracted from the second to last acquired image.

[0099] This comparison allows the processing unit to calculate a progression gradient of the skin sign. The progression gradient may be calculated based on the difference between the scores assigned to the two images.

[0100] The score can be calculated based on the “reference data of the database”, which is listed, for example, in the international application WO 2020169214A1 filed by the present applicant.

[0101] [ Figure 3 ] The block diagram shown generally illustrates the steps of the method according to the present invention.

[0102] For example, the image 50 is acquired in step 69 using a multifunction phone (typically a smartphone), a touch screen tablet or a computer.

[0103] First, in step 70, an overall analysis may be performed on various skin signs present on the image 50, such as wrinkles, texture, spots, pores, redness, sagging, pigmentation, hydration, rosy glow, facial expressions, and facial morphology.

[0104] The image of the region of interest is processed according to the method of the invention to optimize the visualization of selected skin signs in step 71. In the example shown, the selected skin signs are crow's feet and shadows visualized on the output image of the crow's feet region 51 and the shadow region 52.

[0105] Using artificial intelligence algorithms (classifiers, CNNs), a score was provided for each skin sign according to a grading scale provided by experts based on environmental exposure data53.

[0106] To visualize the information, the areas where the algorithm focused to provide the score were visualized in the form of a heat map. The heat map was then merged with the original image using image processing techniques to designate the area of ​​interest for the user.

[0107] The development of these signs (before and after effects) was simulated with the help of generative adversarial networks.

[0108] In step 72, the results of the various features are combined to predict aging and provide decisions regarding the selection of appropriate products or cosmetic procedures.

[0109] The method according to the invention allows a detailed assessment of the entire face, covering all skin manifestations, such as wrinkles, texture, sagging, pigmentation, pores, and irregularities in color and visual properties. Global attributes, such as rosy glow or apparent age, are easily obtained because other people can perceive these.

[0110] Combined with already published knowledge, this new diagnostic approach could provide consumers with a simulation not only of immediate or long-term efficacy, but also of the effects of environmental exposures53 and lifestyle on the appearance of the eye contour.

[0111] The method according to the invention facilitates decision making in the cosmetic industry, for example, in preparing sites for cosmetic injections and providing available treatment simulations.

[0112] Combined with the perception of the hair, such as volume or frizziness, the method according to the invention is key to defining the overall image and achieving a holistic look.

[0113] [ Figure 4 ] is a schematic diagram showing a non-limiting example of implementing recommendations and program partitioning according to various aspects of the present disclosure.

[0114] In step 61, a fixed or mobile computing device has captured at least one image 402 of a user. The image 402 may be presented on a display of the computing device. The image 402 is processed according to the method according to the invention.

[0115] The image 402 can be normalized, and the image normalization engine 212 uses a facial detection algorithm to detect the portion of the image 402 that represents the face. The image normalization engine can use the facial detection algorithm to find a bounding box that contains the face. In a second step of normalization, the image normalization engine can modify the image to center the bounding box. In a third step of normalization, the image normalization engine can scale the image 402 to make the bounding box as large as possible. By performing the normalization steps, the image normalization engine can reduce the differences in layout and size between multiple images, thereby improving the learning ability and accuracy of the automatic learning model. In some embodiments, different normalization steps can be taken.

[0116] In step 62, a questionnaire is shown on the screen of the user's device to collect data to be processed by an artificial intelligence algorithm to assign scores to different skin signs.

[0117] In step 63, the scores assigned to the skin signs are presented to the user. In the example, the scores are shown in the form of horizontal bars spaced apart from each other. Other graphical representation tools, such as curves or histograms, can be used, but bar graphs are particularly suitable for graphical representation of such discrete quantitative variable statistical series.

[0118] In step 64, a visualization of the clinical signs is provided to the user by applying filters to the input image based on the visualized skin signs.

[0119] In step 65 , recommendations for beauty products or beauty procedures are presented on the device screen based on the priorities determined by the skin analysis in step 63 .

[0120] Figures 5 to 11 It shows [ Figure 4 ] for detailed information of the application presented in .

[0121] [ Figure 5 ] shows the questionnaire page of the application; this page will be filled in by the user. This questionnaire allows the user to self-assess their skin condition. These answers are processed by the method according to the present invention to determine the score.

[0122] [ Figure 6 ] shows the loading page presented on the screen during questionnaire analysis and image evaluation. This loading page is usually displayed for 30 to 45 seconds.

[0123] [ Figure 7 ] shows a page showing the results. In the example shown, skin issues are shown in an intuitive color code, from Needs Attention on the left to Excellent Skin on the right. These issues are generally marked on a scale of 0% (left) to 100% (right).

[0124] [ Figure 8 ] shows a page showing skin problems visualized by inserting filters. By using the magnifying glass tool, the user can zoom in on specific areas of the face and even zoom in and out.

[0125] Each skin sign is associated with a filter or light. For example, for wrinkles, texture, and firmness, a standard white light polarizing filter is inserted. For spots or redness, a pair of polarizing filters is inserted. For UV damage and pores, UV light is added.

[0126] [ Fig. 9 ] shows a page displaying the final report highlighting the user's priorities and skin concerns.

[0127] [ Fig.10 ] shows a page showing recommended products, for example, by brand. Product 1 is provided by a first brand. Products 2, 3, and 4 are provided by a second brand. Products 5 and 6 are provided by a third brand.

[0128] [ Fig.11 ] shows a page displaying a QR, for example in case the user wishes to restore the evaluation results later.

[0129] Of course, the invention is not limited to the exemplary embodiments that have just been described.

Claims

1. A cosmetic and non-therapeutic method for detecting and quantifying skin signs in areas of concern of a user, in particular the eye contour, cheeks, mouth contour, forehead, said method include: (i) obtaining an input image of a user's area of ​​interest (112), the area of ​​interest comprising a region of interest (111) containing a skin sign (130, 230); (ii) analyzing the area of ​​interest by comparison with a reference score pre-recorded in a reference database to provide a score for the skin sign; (iii) visualizing the area of ​​interest (111) in the form of a heat map (12, 22) in order to detect the skin sign, the heat map being obtained by an algorithm focused on the skin sign; (iv) merging the heat map (12, 22) with an input image of the area of ​​interest (111) in order to provide an output image (14, 24) of the area of ​​interest (111) and optimizing the visualization of the skin sign (130, 230).

2. The method according to claim 1 comprises: (v) extracting skin signs (130, 230) of the area of ​​interest (111) by converting the heat map (12, 22) into a color space image (13, 23); (vi) generating an output image (14, 24) of the area of ​​interest (111) based on the color space image (13, 23) and the input image.

3. The method according to any one of the preceding claims, in, The reference database is inclusive, covering all ages, ethnic origins, and photo types, and all libraries are scored using common standards to obtain comparable measurement results.

4. The method according to any one of the preceding claims, in, Skin sign detection, evaluation of the score of said skin signs, provision of said heat map and creation of the final image are achieved via image processing or computer vision algorithms.

5. The method according to any one of the preceding claims, in, The detection of the skin signs, the assessment of the scores of the skin signs, the provision of the heat map and the creation of the output image are achieved via a machine learning model.

6. The method according to the preceding claim, in, The machine learning model is a pre-formed convolutional neural network.

7. The method according to any one of claims 5 and 6, in, The machine learning model is provided with data from an atlas that associates pre-recorded images with pre-recorded severity scores.

8. The method according to any one of the preceding claims, in, The skin signs (130, 230) are selected from the group consisting of pore openness, skin texture, wrinkles, fine lines, crow's feet, sagging, color of under-eye shadows, depth of under-eye shadows, skin texture, facial expression, morphology, hydration, radiance or spot color.

9. The method according to any one of claims 5 to 7, in, The machine learning model has been trained to score different skin signs (130, 230).

10. A method according to any preceding claim, comprising applying a contour to the heat map (12, 22) based on directional masking lines and reference points.

11. The method of any preceding claim, further comprising recommending at least one cosmetic product based on the severity score.

12. A cosmetic and non-therapeutic method for temporal monitoring of an area of ​​concern (112) of a user, the method include: Detecting and quantifying skin signs (130, 230) in an area of ​​interest (112) according to any of the preceding claims at different moments in time; temporally monitoring output images obtained after comparing output images (14, 24) generated at said different moments in time, and temporally monitoring scores obtained after comparing scores generated at said different moments in time.

13. The method according to any one of the preceding claims, in, The heatmap is obtained from a Generative Adversarial Network (GAN).

14. The method according to any of the preceding claims comprises obtaining data provided by the user when answering a questionnaire and processing the data through the machine learning model.

15. The method according to any of the preceding claims, further comprising recommending at least one beauty product or beauty procedure based on the output image (24) and the score.

16. A system for detecting and quantifying skin signs (130, 230) in an area of ​​concern (112) of a user, the system include: a communication device adapted to communicate with a user computing device and configured to import at least one input image comprising a representation of an area of ​​interest (112) of the user from an image library in the user computing device; a processing device configured to: (ii) analyze the area of ​​interest by comparison with reference scores pre-recorded in a reference database so as to provide a score for the skin sign; (iii) visualizing the area of ​​interest (111) in the form of a heat map (12, 22) in order to detect the skin sign, the heat map being obtained by an algorithm focused on the skin sign; (iv) merging the heat map (12, 22) with an input image of the area of ​​interest (111) in order to provide an output image (14, 24) of the area of ​​interest (111) and optimizing the visualization of the skin sign (130, 230).

17. A computer program comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 11.

18. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 11.

Images