Detection and visualization of skin signs using a thermal map
Patent Information
- Application Number
- FR2022004423
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-05-10
- Publication Date
- 2025-02-14
- Estimated Expiration
- 2042-05-10
Smart Images
Figure 00000015_0000 
Figure 00000015_0001 
Figure 00000016_0000
Abstract
Description
Title of the invention: Detection and visualization of cutaneous signs using a thermal map
[0001] The invention relates to a cosmetic and non-therapeutic method for detecting and quantifying cutaneous signs on an area of interest of a user. It also relates to a system for implementing this method.
[0002] More generally, a cosmetic product is a product as defined in Regulation EC No. 1223 / 2009 of the European Parliament and of the Council dated November 30, 2009, relating to cosmetic products. Prior art
[0003] Computer-assisted skin analysis has become widespread over the past decade with the availability of controlled lighting systems and digital image capture and processing capabilities.
[0004] Due to the importance placed on facial appearance, computer-aided analysis work has focused on facial skin. There are commercially available systems for facial skin imaging that can capture digital images in a controlled manner. These systems are often coupled with computer analysis systems for visualization and quantification of visible skin features in standard white light images such as hyperpigmented spots, wrinkles, and texture, as well as features not visible in hyperspectral absorption or fluorescence images such as UV spots.
[0005] Document US8218862 discloses a method for generating a skin mask delimiting a region of interest (ROI) in a skin image comprising:
[0006] detecting skin in the skin image to generate a skin map, the skin detection comprising:
[0007] performing a raw skin segmentation operation on the skin image;
[0008] converting the raw skin segmented skin image into a color space image having at least three channels; and
[0009] filtering two of the at least three channels;
[0010] generating a melanin index image from the filtered channels; and
[0011] performing a thresholding operation on the melanin index image to separate skin and non-skin areas in the skin image;
[0012] providing an initial contour based on the skin map; and
[0013] optimizing the initial contour to generate a contour of the skin mask, the skin map and the skin mask each comprising at least one natural boundary of the skin.
[0014] We also know the six “Atlases of the signs of skin aging#” published by MED'COM since 2007. These works systematically study and establish a characterization and classification of the skin linked to age and geography.
[0015] By identifying several facial characteristics, they allow professionals to qualify and quantify the signs of aging according to their evolution. They define criteria - pore opening, depth of forehead wrinkles, crow's feet - and assign them a severity score, which starts from zero and can go up to 9.
[0016] These atlases represent part of the knowledge on which the tools for evaluating skin signs are based. We know their performance for evaluating cosmetic products or for modeling kinetics of aging evolution based on hundreds of thousands of photos.
[0017] In order to automate the diagnoses, the atlas data was used to feed the algorithms, by associating a photo with a score. Then, they were trained to score these different observation areas.
[0018] Thanks to thousands of selfies studied across the globe, algorithms based on artificial intelligence have learned to recognize the different signs of facial aging in any type of photograph.
[0019] By cross-referencing data from selfies with information on users' lifestyle, sleep patterns or environment, these diagnoses now make it possible to offer a user an estimated "aging curve", which can be compared to an average of people of the same age.
[0020] Patent application WO2020169214 filed by the Applicant discloses the prediction of skin aging, taking into account atlases and also the environment or habits of a user, in particular sleep time, pollution and / or place of living. By cross-referencing data from selfies with information such as the users' lifestyle, sleep or environment, this prediction proposes an "aging curve" based on this data.
[0021] There is a need to further improve the performance of the above methods to better:
[0022] Provide comprehensive and accurate facial diagnostics combining various features.
[0023] Personalize a cosmetic treatment / product based on the diagnosis either by creating a new product or by selecting it from a given range.
[0024] Personalize the user experience by visualizing the results of the diagnosis and / or simulating their evolution for knowledge sharing.
[0025] Definition of the invention
[0026] The subject of the invention is a cosmetic and non-therapeutic method for detecting and quantifying cutaneous signs on an area of interest of a user, in particular the contour of the eyes, the cheeks, the contour of the mouth, the forehead, comprising (i) an acquisition of an input image of a user's area of interest comprising a region of interest encompassing cutaneous signs, (ii) an analysis of the area of interest to provide a score of the cutaneous signs, by comparison with reference scores pre-recorded in a reference database, (iii) a visualization of the region of interest in the form of a heat map to detect the cutaneous signs, the heat map being produced by an algorithm focused on the cutaneous signs, (iv) a fusion of the heat map with the input image of the region of interest to provide an output image of the region of interest and optimize the visualization of the cutaneous signs.
[0027] Accurate assessments of skin signs surpassing those of the state of the art in terms of accuracy and inclusiveness are obtained in accordance with the method according to the invention, thanks to image processing. The system is based on the knowledge of decades of research and data covering all different skin types stored in the reference database.
[0028] The “reference database” may include:
[0029] Data stored with one or more image sequences comprising at minus two images,
[0030] One or more atlases displayed on screen or printed on a medium or in a form stored in a computer storage medium, such as the atlas described in patent application WO2011 / 141769, each atlas representing different gradations of at least one characteristic of body typology,
[0031] Synthetic images,
[0032] Internet publications,
[0033] The data can be processed by artificial intelligence (AI) algorithms which may include fuzzy logic, neural networks, genetic programming and decision tree programming.
[0034] All engines can be trained based on inputs such as product information, expert advice, a user profile or data based on sensory perceptions. Using an input, an AI engine can implement an iterative learning process.
[0035] Training can be based on a wide variety of learning rules or training algorithms scientific publications, such as publications:
[0036] Frédéric Flament et al., “Effect of the sun on visible clinical signs of aging in Caucasian skin”, Clin Cosmet Investig Dermatol., 2013 Sep 27;6:221-32.
[0037] Frédéric Flament et al., Assessing changes in some facial signs of fatigue in Chinese women, induced by a single working day, International Journal of Cosmetic Science, November 29, 2018.
[0038] The invention also relates to a cosmetic and non-therapeutic method for temporal monitoring of an area of interest of a user comprising detection and quantification of cutaneous signs on an area of interest as described previously; temporal monitoring of the output images resulting from a comparison of the output images generated at different time instants and temporal monitoring of the scores resulting from a comparison of the scores generated at different time instants.
[0039] The invention also relates to a system for detecting and quantifying cutaneous signs on an area of interest of a user, comprising communication means adapted to communicate with a user computing device and configured to import, from an image library of the user computing device, at least one input image comprising a representation of the area of interest of the user;processing means configured to perform, (ii) an analysis of the area of interest to provide a score of the cutaneous signs, by comparison with reference scores pre-recorded in a reference database, (iii) a visualization of the region of interest in the form of a heat map to detect the cutaneous signs, the heat map being derived from an algorithm focused on the cutaneous signs, (iv) a fusion of the heat map with the input image of the region of interest to provide an output image of the region of interest and optimize the visualization of the cutaneous signs.;
[0040] The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method described above.
[0041] The invention finally relates to a computer-readable medium comprising instructions which, when executed by a computer, cause the latter to implement the method described above.
[0042] Preferred Embodiments
[0043] Preferably, the cosmetic and non-therapeutic method for detecting and quantifying cutaneous signs on an area of interest of a user according to the invention has one or more of the following characteristics, taken alone or in combination:
[0044] It comprises (v) an extraction of the skin signs of the region of interest by converting the thermal map into a color space image, (vi) a production of an output image of the region of interest from the color space image and the input image.
[0045] The learning base is inclusive by covering all ages, ethnic origins and phototypes and all bases are scored by a universal scale to obtain comparable measurements.
[0046] The detection of skin signs, the estimation of the score of the skin signs, the output of the heat map, or the generation of the final image are done via an image processing or computer vision algorithm.
[0047] The machine learning model is a pre-trained convolutional neural network.
[0048] The machine learning model is fed with data from an atlas combining pre-recorded images to pre-recorded severity scores.
[0049] The cutaneous signs are chosen from a pore opening, a skin texture, a wrinkle, a fine line, a crow's feet, a sagging, a dark circle color, a dark circle depth, a skin texture, a facial expression, a morphology, a degree of hydration, a degree of shine or a spot color.
[0050] A detection of skin signs, an estimation of the score of skin signs, an output of the heat map, or a generation of the output image are carried out via a machine learning model.
[0051] The machine learning model is a pre-trained convolutional neural network.
[0052] The machine learning model is fed with data from an atlas combining pre-recorded images to pre-recorded severity scores.
[0053] The machine learning model was trained to score different skin signs.
[0054] It comprises filtration of at least one channel.
[0055] It involves applying a contour to the heat map from a line masking guideline and a reference point.
[0056] It further includes a recommendation of at least one cosmetic product based on the severity score.
[0057] Preferably, the cosmetic and non-therapeutic method of temporal tracking of an area of interest of a user according to the invention has one or more of the following characteristics, taken alone or in combination:
[0058] The heat map is derived from a generative adversarial network (GAN).
[0059] The output image comprises an acquisition of data provided by the user in response to a questionnaire and a processing of this data by the machine learning model.
[0060] The method further comprises recommending at least one cosmetic product or cosmetic routine based on the output image and the score.
[0061] The invention may be better understood by reading the detailed description which follows, a non-limiting example of its implementation, and by examining the appended schematic and partial drawing, in which: Brief description of the drawings
[0062] [Fig.l]
[0063] [Fig.l] represents a functional diagram of a method for detecting and quantifying dark circles under the eyes in accordance with the present invention,
[0064] [Fig.2]
[0065] [Fig.2] represents a functional diagram of a method for detecting and quantifying crow's feet wrinkles in accordance with the present invention.
[0066] [Fig.3]
[0067] [Fig.3] represents a functional diagram of a prediction of aging associated with a recommendation of a cosmetic product.
[0068] [Fig.4]
[0069] [Fig.4] is a schematic drawing which illustrates the different stages of a consumer experience based on the implementation of the method according to the invention in an application for a mobile phone or tablet.
[0070] [Fig.5]
[0071] [Fig.5] illustrates a screen of the application of [Fig.4] showing a questionnaire.
[0072] [Fig.6]
[0073] [Fig.6] illustrates a screen of the application of [Fig.4] during processing of input data.
[0074] [Fig.7]
[0075] [Fig.7] illustrates a screen of the application of [Fig.4] showing a presentation of the results of the evaluation of the skin signs to the user.
[0076] [Fig.8]
[0077] [Fig.8] illustrates a screen of the application of [Fig.4] detailing to the user a result of [Fig.7].
[0078] [Fig.9]
[0079] [Fig.9] illustrates a screen of the application of [Fig.4] showing an analysis report for the user.
[0080] [Fig. 10]
[0081] [Fig.10] illustrates a screen of the application of [Fig.4] showing product suggestions for the user.
[0082] [Fig.l 1]
[0083] [Fig. 11] illustrates a screen of the application of [Fig.4] showing a QR code assigned to the user.
[0084] As shown in [Fig.l], the method according to the invention comprises a step in which a processing unit extracts an image of the region of interest 111 from a first input image of an area of interest 112. In the example shown, the area of interest 112 is formed by the face; the region of interest 111 is formed by the lower contour of the eye. The cutaneous signs present in the region of interest 111 are wrinkles, fine lines, dark circles.
[0085] The image of the area of interest 112 may be captured with an automated and controlled skin image capture system, such as the VISIA Skin Tone Analysis System for facial skin analysis, available from Canfield Scientific, Inc.
[0086] In the exemplary embodiment of [Fig.l], the image is captured with standard light that can be expressed as an RGB (red, green, blue) color image. However, the image can also be captured in a different illumination modality, or with a multispectral imaging system, provided that the regions of interest 111 can be discriminated based on a skin index measurement (e.g., melanin and / or hemoglobin concentrations) derived from the captured image.
[0087] Several skin detection algorithms have been developed for various purposes, including face detection. For example, such algorithms are described in the article RL Hsu, et al., "Face detection in color images," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 5, pp. 696-707, May 2002.). If such skin detection algorithms provide an adequate level of granularity, they can be used for detection of regions of interest in accordance with the present invention.
[0088] The skin detection process uses, for example, the measurement of the individual typology angle (ITA), which is used as a skin indicator. The ITA is calculated using the L* and b* channels of the CIE L*a*b* transformed skin image (hereinafter referred to as L*a*b). (For a detailed description of this metric, see GN Stamatas, et al., "Non-Invasive Measurements of Skin Pigmentation In Situ," Pigment Cell Research, Vol. 17, pp: 618-626, 2004.) The ITA is defined for each image pixel (i,j) as arctan((L*[i,j]-50) / b*[i,j]) and is related to the melanin concentration of the skin. The assumption is that ITA values for skin pixels will be clustered around one value, while ITA values for non-skin pixels are significantly far from the ITA value of skin pixels.
[0089] In the example shown, the method comprises a step 1 in which the processing unit generates a thermal map 12 from the image of the area of interest 111. In a step 2, the thermal map 12 is converted into a color space image 13 having green 131, blue 132 and yellow 133 channels to visualize a dark circle line 130. The blue and green channels ideally display hyperpigmented spots and pores, since these features have greater absorption in the blue and green spectra.
[0090] As can be seen in [Fig.l], the heat map or frequency map is a graphical representation of statistical data which corresponds to the intensity of a variable quantity, a range of tones or a color chart on a two-dimensional matrix.
[0091] In a step 3, the color space image 13 and the input image of the region of interest 111 are combined to produce an output image 14 highlighting the ring line 130 of the color space image on the image of the area of interest 111.
[0092] The functional diagram shown in [Fig.2] comprises a step in which a first image of the face 21 is acquired by an image acquisition device.
[0093] Next, the method comprises a step in which a processing unit extracts an image of the area of interest 111 from the first acquired image of an area of interest, namely a face. In the example shown, the area of interest 111 is formed by the corner of the eye. The cutaneous signs present in the area of interest 111 are mainly wrinkles and crow's feet.
[0094] Then, the method comprises a step in which the processing unit generates a thermal map 22 from the image of the area of interest 111.
[0095] In a step 22, the thermal map 22 is converted into a color space image 23 having green 131 and blue 132 channels to visualize the crow's feet wrinkles 230.
[0096] In a step 23, the color space image 23 and the input image are combined to produce an output image 24 highlighting the crow's feet wrinkles 230 of the color space image 23 on the region of interest image 111.
[0097] Next, the method comprises a step in which the processing unit can extract physiognomic data of the cutaneous signs from the output image and / or assign a score. A temporal indication of the time of acquisition of the image is optionally associated with these physiognomic data. The physiognomic data and their temporal indication can be stored in a memory.
[0098] Optionally, the method comprises temporal tracking of the treatment of the dark circle line from input images acquired at different times, for example every day.
[0099] Thus, the temporal tracking may comprise a step, in which the image acquisition device acquires a new image.
[0100] Then, the temporal tracking comprises a step in which the processing unit compares the physiognomic data extracted from the last acquired image with the physiognomic data extracted from the penultimate image.
[0101] This comparison allows the processing unit to calculate a gradient of evolution of the cutaneous sign. This gradient of evolution can be calculated from a difference between the scores attributed to the two images.
[0102] The score can be calculated from “reference data from a database” listed for example in international application WO2020169214A1 filed by the Applicant.
[0103] The functional diagram shown in [Fig.3] illustrates, in general, the steps of the method according to the invention.
[0104] An image 50 is acquired for example by a multifunction telephone (usually “smartphone”), a touch tablet, or a computer in a step 69.
[0105] Firstly, in a step 70, a holistic analysis of the different cutaneous signs present in the image 50 can be carried out, such as wrinkles, texture, spots, pores, redness, sagging, pigmentation, hydration, radiance, facial expressions, facial morphology.
[0106] The images of the regions of interest are processed in accordance with the method according to the invention, to optimize the visualization of the cutaneous signs chosen in a step 71. In the example shown, the cutaneous signs chosen are the crow's feet wrinkles and the dark circles visualized on the output images of the crow's feet region 51 and the dark circles region 52.
[0107] Artificial intelligence algorithms (classifiers, CNN) are used to provide a score for each skin sign based on a rating scale given by an expert from data from an exposome 53.
[0108] To visualize the information, the areas on which the algorithms focus to provide the score are visualized as a heat map. This is then merged with the initial image using image processing techniques to designate the areas of interest to the user.
[0109] Simulations of the evolution of these signs (“before-after effect”) are carried out using generative adversarial networks.
[0110] The results of the different characteristics are combined to predict aging and provide a decision to select an appropriate product at step 72 or a cosmetic routine.
[0111] The method according to the invention can lead to a comprehensive assessment of the complete face by covering all skin manifestations, such as wrinkles, texture, sagging, pigmentation, pores, as well as color inhomogeneities and optical properties. Holistic attributes such as radiance or apparent age as they might be perceived by other people are within reach.
[0112] Coupled with published knowledge, this new diagnosis is a way to offer consumers a simulation not only of immediate or long-term effectiveness but also of the impact of exposome 53 and lifestyles on the appearance of the eye contour.
[0113] The method according to the invention is decisive in the field of beauty, for example to prepare the location of cosmetic injections and to propose an accessible treatment simulation.
[0114] Merged with the appreciation of hair, such as volume, or frizz, the method according to the invention is key to defining an overall look and appreciating an overall appearance.
[0115] [Fig.4] is a schematic drawing illustrating a non-limiting exemplary embodiment of routine recommendation and sharing according to various aspects of the present disclosure.
[0116] In step 61, a fixed or mobile computing device has captured at least one image 402 of the user. The image 402 may be presented on a display device of the computing device. The image 402 is processed in accordance with the method according to the invention.
[0117] The image 402 may undergo normalization, with the image normalization engine 212 using a face detection algorithm to detect the portion of the image 402 that represents the face. The image normalization engine may use the face detection algorithm to find a bounding box that includes the face. In a second normalization action, the image normalization engine may modify the image to center a bounding box. In a third normalization action, the image normalization engine may zoom the image 402 to cause the bounding box to be as large as possible. By performing the normalization actions, the image normalization engine may reduce differences in layout and size between multiple images, and may therefore improve the training and accuracy of the machine learning model.In some embodiments, different normalization actions may occur.
[0118] In step 62, a questionnaire is proposed on the screen of the user's device, to collect data which will be processed by the artificial intelligence algorithm, with the aim of assigning scores to the different cutaneous signs.
[0119] In step 63, the scores assigned to the different skin signs are presented to the user. In the example, the scores are represented by horizontal bars spaced apart from each other. Other graphical representation tools could be used, such as curves or histograms, but bar charts are particularly suitable for this graphical representation of a statistical series of discrete quantitative variables.
[0120] In step 64, a visualization of the clinical signs is proposed to the user, by applying filters to an input image, according to the cutaneous sign visualized.
[0121] In step 65, recommendations for cosmetic products or routines are presented on the screen of the device, based on the priorities highlighted by the skin analysis in step 63.
[0122] Figures 5 to 11 detail the application presented in [Fig.4].
[0123] [Fig.5] represents a questionnaire page of the application, this page being to be completed by the user. This questionnaire is a self-assessment by the user of the condition of his skin. The responses are processed according to the method according to the invention for determining the scores.
[0124] [Fig.6] shows a loading page presented on the screen during questionnaire analysis and image scoring. This loading page is usually displayed for 30 to 45 seconds.
[0125] [Fig.7] shows a results display page. In the example shown, skin problems are displayed with an intuitive color code ranging from "needs attention" on the left to "excellent skin" on the right. They are generally rated on a scale of 0% (left) to 100% (right).
[0126] [Fig.8] shows a display page of a skin problem visualized by interposing a filter. By using the magnifying glass tool, the user can enlarge a specific area of the face, or even zoom in and out.
[0127] Each skin sign is associated with a filter or light. For example, for wrinkles, texture and firmness, a standard filter polarized to white light is interposed. For spots or redness, a pair of polarizing filters is interposed. For UV damage and pores, UV light is added.
[0128] [Fig.9] shows a display page of a final report, which emphasizes the user's priorities and skin problems.
[0129] [Fig. 10] represents a display page of recommended products, for example classified according to brands. Product 1 is offered by a first brand. Products 2, 3, 4 are offered by a second brand. Products 5 and 6 are offered by a third brand.
[0130] [Fig. 11] represents a display page of a QR, for example in the case where the user wishes to find the results of the evaluation later.
[0131] Of course, the invention is not limited to the exemplary embodiments which have just been described.
Claims
Claims
1. Cosmetic and non-therapeutic method for detecting and quantifying cutaneous signs on an area of interest of a user, in particular the contour of the eyes, the cheeks, the contour of the mouth, the forehead, comprising (i) an acquisition of an input image of an area of interest (112) of a user comprising a region of interest (111) encompassing cutaneous signs (130, 230), (ii) an analysis of the area of interest to provide a score of the cutaneous signs, by comparison with reference scores pre-recorded in a reference database, (iii) a visualization of the region of interest (111) in the form of a heat map (12, 22) to detect the cutaneous signs, the heat map being derived from an algorithm focused on the cutaneous signs, (iv) a fusion of the heat map (12, 22) with the input image of the region of interest (111) to provide a output image (14,24) of the region of interest (111) and optimizing the visualization of the skin signs (130, 230), the method comprising (v) extracting the skin signs (130, 230) from the region of interest (111) by converting the heat map (12, 22) into a color space image (13, 23), (vi) producing an output image (14, 24) of the region of interest (111) from the color space image (13, 23) and the input image.,
2. A method according to any preceding claim, wherein the reference database is inclusive by covering all ages, ethnic origins and phototypes and all bases are scored by a universal scale to obtain comparable measurements.
3. Method according to any one of the preceding claims, in which the detection of skin signs, the estimation of the score of the skin signs, the outcome of the heat map, or the generation of the final image are done via an image processing or computer vision algorithm.
4. A method according to any preceding claim, wherein a detection of the skin signs, an estimation of the score of the skin signs, an output of the heat map, or a generation of the output image are carried out via a machine learning model.
5. Method according to the preceding claim, wherein the machine learning model is a pre-trained convolutional neural network.
6. A method according to any one of claims 4 or 5, wherein the machine learning model is fed with data from an atlas associating pre-recorded images with pre-recorded severity scores.
7. A method according to any preceding claim, wherein the skin signs (130, 230) are selected from pore opening, skin texture, wrinkle, fine line, crow's feet, sagging, dark circle color, dark circle depth, skin texture, facial expression, morphology, degree of hydration, degree of shine or spot color.
8. A method according to any one of the preceding claims 4 to 6, wherein the machine learning model has been trained to score different skin signs (130, 230).
9. A method according to any preceding claim, comprising applying a contour to the thermal map (12, 22) from a masking guideline and a reference point.
10. A method according to any preceding claim, further comprising a recommendation of at least one cosmetic product based on the severity score.
11. Cosmetic and non-therapeutic method for temporal tracking of an area of interest (112) of a user comprising detection and quantification of cutaneous signs (130, 230) on an area of interest (112) according to any one of the preceding claims at different time instants; temporal tracking of the output images resulting from a comparison of the output images (14, 24) generated at the different time instants and temporal tracking of the scores resulting from a comparison of the scores generated at the different time instants.
12. A method according to any preceding claim, wherein the heat map is derived from a generative adversarial network (GAN).
13. A method according to any preceding claim, comprising acquiring data provided by the user in response to a questionnaire and processing this data by the machine learning model.
14. Method according to one of the preceding claims, further comprising a recommendation of at least one cosmetic product or cosmetic routine based on the output image (24) and the score.
15. System for detecting and quantifying cutaneous signs (130, 230) on an area of interest (112) of a user, comprising communication means adapted to communicate with a user computing device and configured to import, from an image library of the user computing device, at least one input image comprising a representation of the area of interest (112) of the user;processing means configured to perform, (ii) an analysis of the area of interest to provide a score of the cutaneous signs, by comparison with reference scores pre-recorded in a reference database, (iii) a visualization of the region of interest (111) in the form of a heat map (12, 22) to detect the cutaneous signs, the heat map being derived from an algorithm focused on the cutaneous signs, (iv) a fusion of the heat map (12, 22) with the input image of the region of interest (111) to provide an output image (14, 24) of the region of interest (111) and optimize the visualization of the cutaneous signs (130, 230), (v) an extraction of the cutaneous signs (130, 230) from the region of interest (111) by converting the heat map (12, 22) into a color space image (13, 23), (vi) producing an output image (14, 24) of the region of interest (111) from the color space image (13, 23) and the input image;
16. A computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to one of claims 1 to 10.
17. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement the method according to one of claims 1 to 10.