Method for predicting development of skin signs over time

Abstract

The method relates to a cosmetic and non-therapeutic method for predicting the appearance of a region of interest of a user, the method comprising (i) acquiring an input image of the region of interest; the input image is processed (31) on the one hand with (ii) image segmentation and analysis algorithms for determining optical characteristics of the region of interest and extracting features of the skin signs, and on the other hand with (iii) algorithms for predicting and / or displaying the development of the skin signs.

Description

[0001] The subject of the invention is a cosmetic and non-therapeutic method for predicting the development of skin signs in an area of ​​concern of a user over time and over movement. The invention also relates to a system for implementing such a method.

[0002] More generally, a cosmetic product is a product as defined in Regulation (EC) No 1223 / 2009 of the European Parliament and of the Council of 30 November 2009 on cosmetic products. Background Art

[0003] Computer-assisted analysis of the skin has become increasingly common over the past decade with the availability of controlled lighting systems and digital image capture and processing capabilities.

[0004] Because of the importance of facial appearance, computer-assisted analysis research has focused primarily on the skin of the face. There are commercially available systems for imaging the skin of the face that can capture digital images in a controlled manner. These systems are typically coupled to computer analysis systems that are used to display and quantify visible features of the skin in standard white light images (such as hyperpigmented spots, wrinkles, and texture), as well as invisible features in fluorescence or hyperspectral absorption images (such as UV spots).

[0005] Document US8218862 discloses a method for generating a skin mask defining a region of interest (ROI) in a skin image, the method comprising:

[0006] Detect skin in a skin image to generate a skin map. Skin detection includes:

[0007] performing a raw skin segmentation operation on the skin image;

[0008] Converting the original skin segmented skin image into an image in a color space having at least three channels; and

[0009] filtering two of the at least three channels;

[0010] generating a melanin index image based on the filtered channels; and

[0011] performing a thresholding operation on the melanin index image so as to separate a skin area and a non-skin area in the skin image;

[0012] providing an initial contour based on the skin map; and

[0013] The initial contour is optimized to generate a contour of a skin mask, the skin map and the skin mask each comprising at least one natural limit of the skin.

[0014] MED'COM is also well known for the six "Atlas des signes du viellissement cutané [Atlas of signs of skin aging]" published since 2007. These works systematically study and identify the characteristics and classifications of the skin in relation to age and geography.

[0015] By identifying several features of the face, they allow professionals to qualify and quantify the signs of aging based on the development of these features. They define these criteria (openness of pores, depth of forehead wrinkles and crow's feet) and assign them a severity score, which starts at zero and can go up to nine.

[0016] These atlases represent part of the knowledge on which tools for assessing skin signs are based. Their performance for assessing beauty products or for modeling the dynamics of aging progression based on hundreds of thousands of photographs is known.

[0017] To automate the diagnosis, algorithms are fed with data from the atlas by associating photos with scores. These algorithms are then trained to score these different areas of observation.

[0018] By studying thousands of selfies taken around the world, an AI-based algorithm has learned to identify various signs of facial aging in any type of photo.

[0019] By combining data derived from selfies with information about the user's rhythm of life, sleep or surroundings, these diagnostics now make it possible to provide the user with an estimated "aging curve" that can be compared with the average of people of the same age.

[0020] Patent application WO 2020169214 filed by the present applicant discloses predicting skin aging by taking into account a collection of images and the user's surroundings or habits (particularly sleep time, pollution and / or place of residence). By combining data from selfies with information such as the user's pace of life, sleep or surroundings, this prediction provides an "aging curve" based on these data.

[0021] The performance of the above methods needs to be further improved to better:

[0022] Combines different features to provide a complete and accurate facial diagnosis.

[0023] Personalize beauty treatments / products based on the diagnosis, either by creating a new product or by choosing from a given range.

[0024] Personalize the user experience by displaying the results of the diagnosis and / or by simulating its development to enable knowledge sharing.

[0025] Predict the future development of the user's area of ​​interest. Summary of the invention

[0026] The subject of the invention is a cosmetic and non-therapeutic method for predicting the appearance of an area of ​​interest of a user, the area of ​​interest being divided into zones of interest comprising skin signs, the method comprising (i) acquiring an input image of the area of ​​interest; processing the input image on the one hand using (ii) an image segmentation and analysis algorithm for determining the optical properties of the area of ​​interest and extracting the features of the skin signs and on the other hand using (iii) an algorithm for predicting and / or displaying the future development of skin signs from the input image by comparing the features of the skin signs extracted from the input image with reference data previously stored in a reference database; (iv) simulating, ideally or at an accelerated pace, the future appearance of the area of ​​interest over time. By means of the method according to the invention, the skin signs are analyzed one by one to significantly improve the prediction in terms of reliability and in terms of overall performance, in particular with respect to WO 2020169214. The features of each skin sign are detected, analyzed and compared one by one with the same features extracted from the reference database. In fact, the analysis of the skin signs is more sophisticated and more complete than the analysis of the prior art, which is a global analysis of the area of ​​interest.

[0027] The present invention combines the advantages of clinical evaluation and instrumental evaluation.

[0028] The present invention brings together the advantages of two objective means for measuring facial features in the field of beauty, namely:

[0029] Automated clinical visual assessment is just around the corner, leveraging AI-based deep learning algorithms trained by dermatologists and experts calibrated against published photographic clinical scales.

[0030] Instrumentation using specific image segmentation and analysis software for measuring optical properties such as light / skin interaction.

[0031] “Appearance” can be understood as the overall appearance of the region of interest, which can be moving or fixed.

[0032] In addition, it is conceivable that each data newly collected by the method according to the invention can be fed into reference data previously stored in the reference database:

[0033] To better simulate aging and appearance,

[0034] In order to further improve the software's detection capabilities and accuracy adjustments.

[0035] According to the method of the present invention, images are processed to obtain an accurate assessment of skin signs that exceeds the prior art in terms of accuracy and inclusiveness. The system is based on the knowledge of decades of research and data stored in reference databases covering all the various skin types.

[0036] Examples of "areas of concern" may include the hair, nails, face, scalp, neck, eye area, ears, torso, arms, legs, and / or other body parts.

[0037] A "region of interest" is a portion of a region of interest. For example, if the region of interest is a face, then the eye contour, forehead, cheekbones, and lip contours are associated regions of interest.

[0038] Preferably, the external body part is the face.

[0039] A “reference database” may include:

[0040] data stored together with one or more image sequences comprising at least two images,

[0041] one or more atlases displayed on a screen or printed on a medium or stored in some form on a computer storage medium, such as the atlases described in patent application WO 2011 / 141769, each atlas representing various levels of at least one feature of body typology,

[0042] Synthetic images,

[0043] Internet publications.

[0044] The data can be processed by artificial intelligence (AI) algorithms, which can include fuzzy logic, neural networks, genetic programming, and decision tree programming.

[0045] All engines can be trained based on inputs such as information about products, expert advice, user profiles, or data based on sensory perception. Using the inputs, the AI ​​engine can implement an iterative learning process.

[0046] Training can be based on a variety of learning rules or training algorithms from scientific publications, such as the following:

[0047] Frederic Flament et al., "Effect of the sun on visible clinical signs of aging in Caucasian skin," Clinical, Cosmetic and Investigational Dermatology, 2013,6 ,221-232.

[0048] Frederic Flament et al., "Effects of seasonal and daily photoprotection on certain signs of facial appearance in Chinese women", International Journal of Cosmetic Science, 2017, 39 ,256-268.

[0049] Frédéric Flament et al., "Comment on fatigue and fatigue in certain ways. How fatigue from a day at work changes certain facial signs in Caucasian women of different ages”, International Journal of Cosmetic Science, 2017, 39, 467-475.

[0050] Frédéric Flament et al., “Extreme chronological pollution alters certain signs of aging on the face of Chinese women [Severe chronic external urban pollution alters certain signs of aging on the face of Chinese women],” International Journal of Cosmetic Science, 2018, 40, 467-481.

[0051] The present invention also relates to a cosmetic and non-therapeutic method for predicting the appearance of a user's area of ​​interest, the area of ​​interest being divided into areas of interest including skin signs, the method comprising (i) acquiring an input image of the area of ​​interest; processing the input image using, on the one hand, (ii) an image segmentation and analysis algorithm for determining the optical properties of the area of ​​interest and extracting features of these skin signs and, on the other hand, (iii) an algorithm for predicting and / or displaying the rejuvenation of skin signs from the input image by comparing the features of these skin signs extracted from the input image with reference data pre-stored in a reference database; and (iv) ideally or accelerated simulation of the future appearance of the area of ​​interest over time.

[0052] The present invention also relates to a cosmetic and non-therapeutic method for predicting the appearance of an area of ​​concern of a user, the area of ​​concern being divided into a zone of concern including skin signs, the method comprising (i) acquiring an input image of the area of ​​concern; processing the input image using, on the one hand, (ii) an image segmentation and analysis algorithm for determining the optical properties of the area of ​​concern and extracting features of these skin signs and, on the other hand, (iii) an algorithm for predicting and / or displaying the effect on skin signs from the input image by comparing the features of these skin signs extracted from the input image with reference data pre-stored in a reference database given known product performance; and (iv) ideally or accelerated simulation of the future appearance of the area of ​​concern over time.

[0053] The present invention also relates to a cosmetic and non-therapeutic system for predicting the appearance of a user's area of ​​concern, the system comprising: a communication device suitable for communicating with a user computing device and configured to import at least one input image including a representation of the user's area of ​​concern from an image library of the user computing device; a processing device configured to perform on the input image (ii) an image segmentation and analysis algorithm for determining the optical properties of the area of ​​concern and extracting features of skin signs, and (iii) an algorithm for predicting and / or displaying the future development of skin signs from the input image by comparing the features of these skin signs extracted from the reference image with reference data pre-stored in a reference database; and (iv) a device for ideally or accelerated simulation of the future appearance of the area of ​​concern over time.

[0054] The invention also relates to a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method as described above.

[0055] The invention finally relates to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement a method as described above. Preferred Embodiments

[0056] Preferably, the cosmetic and non-therapeutic method for detecting and quantifying skin signs in an area of ​​concern of a user according to the present invention exhibits one or more of the following features considered alone or in combination:

[0057] Processing the input image includes an overall analysis of the region of interest as well as an analysis of the region of interest.

[0058] The algorithms used for prediction and / or display are deep learning algorithms based on artificial intelligence.

[0059] The algorithms for predicting and / or displaying the future development, rejuvenation or resulting effects of skin signs from input images are based on photorealistic or realistic physically based 3D / 2D rendering algorithms.

[0060] The algorithm for predicting and / or displaying the future development, rejuvenation or resulting effect of skin signs from an input image is based on a neural rendering algorithm.

[0061] Processing the input image includes evaluating at least two skin signs.

[0062] The method includes (v) processing one or more pieces of information describing the user's lifestyle.

[0063] The method includes (vi) estimating an aging acceleration rate for each skin sign based on information describing the user's lifestyle, the input image and pre-stored reference data.

[0064] The method includes (vii) evaluating the appearance of each skin sign over time based on the aging acceleration rate and based on the input image.

[0065] The method includes (viii) simulating the future appearance of the area of ​​interest under exposure conditions of the user, and possibly (viii) simulating the future appearance of the area of ​​interest under ideal exposure conditions.

[0066] These areas of concern are selected from the forehead, eye contour, nose, cheekbones, lip contour, chin, lips and cheeks.

[0067] The machine learning model is a pre-trained convolutional neural network.

[0068] These skin signs are selected from pores, skin texture, wrinkles, fine lines, crow's feet, sagging, under-eye shadows, skin texture, facial expressions, morphology, redness, and shine.

[0069] The characteristics of these skin signs are selected from openness, surface area, volume, depth, width, length, color, moisturization, gloss, distribution, intensity, reflectivity, frequency, uniformity, suppleness, firmness, translucency, brightness, pigmentation, matteness, elasticity or density.

[0070] The simulation was generated by an unsupervised learning algorithm, specifically a generative adversarial network (GAN).

[0071] A cosmetic product is selected and the future appearance of the area of ​​concern to which the product is applied is simulated.

[0072] The invention may be better understood by reading the following detailed description of non-limiting exemplary embodiments and by studying the accompanying schematic and partial drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0073] [ Figure 1 ]

[0074] [ Figure 1 ] is a block diagram of the prediction method according to the present invention.

[0075] like[ Figure 1 ] as shown in FIG. 3 , the method according to the present invention comprises a step 31 of evaluating various characteristics of skin signs based on an input image.

[0076] According to the present invention, overall analysis is performed through multi-region overall mapping, thereby achieving overall evaluation and target evaluation.

[0077] For example, the weight of the eye region relative to the entire face is calculated based on the apparent age.

[0078] More than 20 features of the face can be captured automatically: more signs analyzed means a more specific and accurate assessment of each skin sign.

[0079] Each part of the face will be analyzed:

[0080] The top is the wrinkles on the forehead,

[0081] Droopy eyes, crow's feet, eye bags, periorbital wrinkles, under-eye shadows,

[0082] Nose / cheeks with texture, pores, pigment spots, nasolabial folds,

[0083] A mouth with lip corners,

[0084] Drooping lower face.

[0085] In step 11, these features evaluated based on the input image are compared with reference features pre-stored in a database.

[0086] In step 41, the user's answers to the questionnaire about his or her lifestyle habits may be stored in a database and processed.

[0087] These answers may, for example, belong to categories such as the user's physical characteristics, their place of residence, demographics of their residence, nutritional information, information on cosmetic use, environmental information, information on beauty product use, lifestyle, eating habits, geographical location of residence, location of workplace, work habits, sleeping habits, exercise habits, relaxation habits, beauty care habits, smoking and drinking habits, sun exposure habits, sunscreen use, tanning tendency, number of tans and severe sunburns, dietary requirements, dietary supplements or vitamins, travel habits, physical condition information, adverse reactions to products, previously used beauty care products and their efficacy, leisure, citizenship, country and region of birth, whether the subject is an urban, suburban or rural resident, the size of the urban area where the subject lives, and whether the subject is retired.

[0088] In step 42, the user's life context may be stored in the same database and processed additionally.

[0089] In step 43, the database is fed with scientific data, such as publications, reviews, articles, encyclopedias, atlases, patents, data on exposure and aging, testimonies.

[0090] In step 32, the information collected at the end of steps 11, 41, 42 and 43 is processed to estimate the aging acceleration rate according to each skin sign.

[0091] “Accelerators” are parameters that speed up aging, such as sunlight, pollution or certain medications.

[0092] The accelerator modifies the predicted information regarding aging of the user's external body parts relative to predicted information generated in the absence of any accelerator.

[0093] Typically, a user without an associated accelerator would be one who leads a very healthy or ideal life.

[0094] The "acceleration rate" is a numerical value that quantifies the effect of an accelerator on aging.

[0095] When the influence of the accelerator (such as sunlight) on the predicted information is high, the acceleration rate is high, while if the accelerator has only a weak influence, the acceleration rate is low.

[0096] In step 33, the future development of the skin signs is assessed based on the information derived from steps 31 and 32, ie the characteristics and acceleration rates of the various skin signs.

[0097] In step 34 , the appearance of the user's face in his or her foreseeable actual exposure situation is simulated, in particular based on the information communicated by the user in steps 41 and 42 .

[0098] In step 35, the appearance of the user's face under hypothetical ideal exposure conditions may be simulated.

[0099] Of course, the invention is not limited to the exemplary embodiments that have just been described.

Claims

1. A cosmetic and non-therapeutic method for predicting the appearance of an area of ​​concern of a user, the area of ​​concern being divided into a zone of concern including skin signs, the method comprising (i) acquiring an input image of the area of ​​concern; processing (31) the input image using (ii) an image segmentation and analysis algorithm on the one hand and (iii) an artificial intelligence-based deep learning algorithm on the other hand, the image segmentation and analysis algorithm being used to determine the optical properties of the area of ​​concern and extract features of the skin signs, the artificial intelligence-based deep learning algorithm being used to predict and / or display the future development (33) of the skin signs from the input image by comparing (11) the features of the skin signs extracted from the input image with reference data pre-stored in a reference database (43); and (iv) ideally or accelerated simulation (34, 35) of the future appearance of the area of ​​concern over time.

2. A cosmetic and non-therapeutic method for predicting the appearance of an area of ​​concern of a user, wherein the area of ​​concern is divided into a zone of concern including skin signs, the method comprising: (i) acquiring an input image of the area of ​​concern; processing (31) the input image using (ii) an image segmentation and analysis algorithm on the one hand and (iii) an artificial intelligence-based deep learning algorithm on the other hand, wherein the image segmentation and analysis algorithm is used to determine the optical properties of the area of ​​concern and extract features of the skin signs, and the artificial intelligence-based deep learning algorithm is used to predict and / or display the rejuvenation of skin signs from the input image by comparing (11) the features of the skin signs extracted from the input image with reference data pre-stored in a reference database (43); and (iv) ideally or accelerated simulation (34, 35) of the future appearance of the area of ​​concern over time.

3. A cosmetic and non-therapeutic method for predicting the appearance of an area of ​​concern of a user, the area of ​​concern being divided into a zone of concern including skin signs, the method comprising (i) acquiring an input image of the area of ​​concern; processing (31) the input image using (ii) an image segmentation and analysis algorithm on the one hand and (iii) an artificial intelligence-based deep learning algorithm on the other hand, the image segmentation and analysis algorithm being used to determine the optical properties of the area of ​​concern and extract features of the skin signs, the artificial intelligence-based deep learning algorithm being used to predict and / or display the effect on the skin signs from the input image by comparing (11) the features of the skin signs extracted from the input image with reference data pre-stored in a reference database (43) when product performance is known; and (iv) ideally or accelerated simulation (34, 35) of the future appearance of the area of ​​concern over time.

4. The method according to any one of the preceding claims, characterized in that The algorithm for prediction and / or display is a deep learning algorithm based on artificial intelligence.

5. The method according to any one of the preceding claims, characterized in that Processing the input image includes an overall analysis of the region of interest and an analysis of the region of interest.

6. The method according to any one of the preceding claims, characterized in that Processing the input image includes evaluating at least two skin signs.

7. The method according to any one of the preceding claims, characterized in that The method includes (v) processing one or more pieces of information (41, 42) describing the user's lifestyle.

8. The method according to any one of the preceding claims, characterized in that The method comprises (vi) estimating (32) an aging acceleration rate for each skin sign based on information describing the user's lifestyle, the input image and pre-stored reference data.

9. The method according to any one of the preceding claims, characterized in that The method includes (vii) evaluating (33) the appearance of each skin sign over time based on the aging acceleration rate and based on the input image.

10. The method according to any one of the preceding claims, characterized in that The method includes (viii) simulating a future appearance of the area of ​​interest under exposure conditions of the user (34), and possibly (viii) simulating a future appearance of the area of ​​interest under ideal exposure conditions (35).

11. The method according to any one of the preceding claims, characterized in that The area of ​​concern is selected from the group consisting of forehead, eye contour, nose, cheekbones, lip contour, chin, lips and cheeks.

12. A method according to any one of the preceding claims, wherein: The algorithm for predicting and / or displaying the future development, rejuvenation or resulting effect of skin signs from the input image is based on an artificial intelligence based deep learning algorithm.

13. A method according to any one of the preceding claims, wherein: The algorithm for predicting and / or displaying the future development, rejuvenation or resulting effect of skin signs from the input image is based on a photorealistic or realistic physically based 3D / 2D rendering algorithm.

14. A method according to any one of the preceding claims, wherein: The algorithm for predicting and / or displaying the future development, rejuvenation or resulting effect of skin signs from the input image is based on a neural rendering algorithm.

15. A method according to any one of the preceding claims, wherein: The machine learning model is a pre-trained convolutional neural network.

16. A method according to any one of the preceding claims, wherein: The skin signs are selected from the group consisting of pores, skin texture, wrinkles, fine lines, crow's feet, sagging, under-eye shadows, skin texture, facial expressions, morphology, redness, shine.

17. A method according to any one of the preceding claims, wherein: The characteristic of the skin sign is selected from openness, surface area, volume, depth, width, length, color, moisturization, gloss, distribution, intensity, reflectivity, frequency, uniformity, smoothness, firmness, translucency, brightness, pigmentation, matteness, elasticity or density.

18. A method according to any one of the preceding claims, wherein: The simulations are generated by unsupervised learning algorithms, in particular generative adversarial networks (GANs).

19. A method according to any one of the preceding claims, wherein: A cosmetic product is selected and the future appearance of the area of ​​concern with the product applied is simulated.

20. A cosmetic and non-therapeutic system for predicting the appearance of an area of ​​concern of a user, the system comprising: A communication device, the communication device is suitable for communicating with a user computing device and is configured to import at least one input image including a representation of the user's area of ​​interest from an image library of the user computing device; a processing device, the processing device is configured to perform on the input image (ii) an image segmentation and analysis algorithm, the image segmentation and analysis algorithm is used to determine the optical properties of the area of ​​interest and extract the characteristics of the skin signs, and (iii) an artificial intelligence-based deep learning algorithm, the artificial intelligence-based deep learning algorithm is used to predict and / or display the future development of the skin signs from the input image by comparing the characteristics of the skin signs extracted from the reference image with reference data pre-stored in a reference database; (iv) a device for ideally or accelerated simulation of the future appearance of the area of ​​interest over time.

21. A computer program comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 19.

22. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 19.

Images