Method for diagnosing skin by 3D facial analysis

Abstract

The subject matter of the present invention is a method for modeling an element of interest in a skin region of a user, the method comprising: (i) acquiring at least one RGB or grayscale input image of the region of interest, (ii) acquiring at least one high-resolution or medium-resolution 2D input image of the region of interest, (iii) reconstructing the region of interest of the user in 3D form.

Description

Technical Field

[0001] The present invention relates to a cosmetic method without therapeutic intent for predicting aging of an element of interest of a user's skin. The present invention further relates to a data processing system comprising a processor configured to execute the method. Background Art

[0002] Individuals are always curious to know what will happen in the future.

[0003] One area of ​​particular concern relates to individuals’ curiosity about their future appearance.

[0004] Given that "crystal balls" or "time machines" exist only in fairy tales and science fiction, it is impossible to obtain an accurate prediction of an individual's future appearance.

[0005] However, with the advances in modern research techniques and data processing equipment, a somewhat accurate assessment of an individual's prognosis is becoming increasingly possible.

[0006] In the field of beauty, the use of information related to personal prospects has attracted special attention.

[0007] During the aging process, people may develop problems such as wrinkles, fine lines, hair loss, and skin pigmentation, such as those caused by excessive exposure to the sun or more generally by their lifestyle.

[0008] Document US2017270593 discloses a method for providing personalized product recommendations to consumers, the method comprising: a) capturing an image of a user using an image capture device; b) analyzing the image using a computing device coupled to the image capture device, wherein the computing device analyzes the image using a convolutional neural network to predict the age of the user, and wherein predicting the age of the user includes identifying at least one pixel indicating the age of the user; c) using at least one pixel to generate a two-dimensional heat map, the two-dimensional heat map identifying skin areas in the image that contribute to the predicted age of the user; d) recommending to the user a product to be applied to the skin area in order to achieve a target skin age; and e) providing the user with the possibility of purchasing the product.

[0009] The article "Evaluation of ring projection and laser scanning for 3D reconstruction of dental pieces" by Natalia Múnera published in Dyna, Medellin, Colombia, in February 2012, discloses 3D scanning of human teeth based on ring projection or laser scanning. In order to show the outer shape of the teeth, human teeth can be reconstructed.

[0010] A widely used method for 3D reconstruction of objects is the moire effect. This phenomenon occurs when two regular patterns are superimposed on a surface. It has been used in optical metrology since the 1920s. The moire effect has been used in several fields such as life sciences and engineering. In life sciences, the moire effect provides valuable information in facial plastic surgery, spinal deformity research, and facial morphometry.

[0011] Kawano, Y.'s article "Three dimensional analysis of the face in respect of zygomatic fractures and evaluation of the surgery with the aid of Moirétopography", Journal of Craniomaxillofacial Surgery, 15, 68-74, 1987, discloses the use of Moirétopography, a three-dimensional morphometric method, to analyze facial distortion in patients with zygomatic fractures. Moirétopography of the patient's face was taken at an angle of 45 degrees before and after surgery, and the facial moiré photos were quantitatively analyzed by computer processing the data to determine the degree of cheek distortion. Each data was compared in terms of the interval between injury and surgery and in relation to the surgical procedure. For comparison purposes, radiological findings and normal individuals were also studied.

[0012] Improved methods for predicting skin aging are needed.

[0013] Improvements in methods used to diagnose skin are needed, particularly in terms of reliability, validity, and accuracy.

[0014] There is a need to evaluate the performance of beauty products and other factors that have an impact on aging. Summary of the invention

[0015] The subject of the present invention is a method for modeling an element of interest of a user's skin, the method comprising: (i) obtaining at least one RGB or grayscale input image of the area of ​​interest, (ii) obtaining at least one high-resolution or medium-resolution 2D input image of the area of ​​interest, and (iii) reconstructing the area of ​​interest of the user in 3D form.

[0016] Due to the specific way in which the images are acquired, an improvement in skin diagnosis for all elements of interest is observed. Thus, there is an opportunity to accurately monitor changes in elements of interest of a person's skin over time.

[0017] According to the invention, elements of concern are, for example, wrinkles, fine lines, spots, scars, pores, texture, dark circles, eye bags.

[0018] Areas of concern are e.g. cheeks, lips, forehead, crow's feet.

[0019] Red, green and blue (abbreviated as RGB) is the computer color encoding system closest to hardware. Computer screens reproduce colors by additive color synthesis based on the three primary colors (red, green and blue), forming a small and imperceptible mosaic on the screen. RGB encoding indicates the value of each of these primary colors.

[0020] Low-resolution images have approximately 72 pixels, or color squares, per inch.

[0021] A high-resolution image has at least 300 pixels per inch (ppi or dpi).

[0022] Medium resolution images have between 72 pixels per inch and 300 pixels per inch.

[0023] The 3D information is used, among other things, to compensate for the effects of viewing angle and orientation of light, which change the appearance of the person in the 2D input image, such as shadows or specular reflections which have an impact on the assessment of clinical signs.

[0024] The algorithms that make possible deep high-resolution reconstruction of a face in 3D are not explicitly described here, as there are many different technologies that make such scanning possible.

[0025] Generally speaking, many data sources may be combined to make it possible to monitor the details of the protrusions on the skin.

[0026] The method according to the invention is cosmetic and non-therapeutic.

[0027] Advantageously, the method for diagnosing an element of interest in a user's skin area comprises: (i) acquiring at least one RGB or grayscale input image of the area of ​​interest, (ii) acquiring at least one high-resolution or medium-resolution 2D input image of the area of ​​interest, (iii) reconstructing the area of ​​interest in 3D form, and (iv) extracting data based on the 3D reconstruction.

[0028] The modeling approach can use 3D facial measurements to accurately extract the 3D topology of the face via facial reconstruction.

[0029] The present invention also relates to a cosmetic and non-therapeutic method for monitoring a user's region of interest over time, the method comprising: (i) acquiring at least one RGB or grayscale input image of the region of interest, (ii) acquiring at least one high-resolution or medium-resolution 2D input image of the region of interest, (iii) reconstructing the region of interest in 3D, and (v) monitoring the generated 3D reconstruction over time by comparing the 3D reconstructions generated at different moments in time.

[0030] The modeling method can use 3D facial measurements to accurately extract the 3D topology of the face via facial reconstruction. Recording 3D facial reconstructions of the same person over time makes it possible to superimpose parts that have not changed significantly (such as bone structure) and to evaluate differences in parts that may move or change over time (such as elements of interest such as wrinkles and fine lines or the firmness of the skin). In this way, changes in a person's skin over time can be monitored, and in addition, the performance of various beauty products applied can be evaluated, as well as all other factors that have an impact on the aging of a person (such as exposure or nutrition). Recommendations for beauty products can be generated from this.

[0031] The invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method as described above.

[0032] The present invention also relates to a data processing system, comprising:

[0033] a memory, wherein the computer program product as described above is stored in the memory,

[0034] a processing unit configured to implement the computer program product,

[0035] a photographic device configured to acquire an RGB or grayscale input image of the region of interest and a high-resolution or medium-resolution 2D input image of the region of interest,

[0036] A screen is configured to display a 3D reconstruction of the area of ​​interest of the user generated by the processing unit.

[0037] The invention also relates to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the above method. Preferred Embodiments

[0038] Modeling approach

[0039] Preferably, the modeling method according to the present invention has one or more of the following features considered individually or in combination:

[0040] The reconstruction in 3D is a high-resolution reconstruction in 3D.

[0041] Reconstruction of 3D forms is performed by combining global or local high-resolution 3D acquisition with high-resolution local / global depth analysis.

[0042] The method includes 3D scanning based on annular projection or laser scanning.

[0043] The method consists in triangulation based on images acquired from different viewpoints.

[0044] The method comprises a high-resolution estimation based on a lower-resolution image using a machine learning model. The method comprises obtaining at least one additional data describing the user's personal information, in particular obtaining at least one additional data based on a questionnaire filled in by the user, and processing the additional data.

[0045] Methods for monitoring over time

[0046] Preferably, the method for monitoring a user's area of ​​interest over time has one or more of the following features considered individually or in combination:

[0047] The method comprises (vi) processing reconstructions of the 3D form generated at different moments in time by superimposing predetermined time-invariant parts, in particular bone structures, and detecting the development of the element of interest between the different moments in time based on the superposition.

[0048] Monitoring over time involves calculating the gradient of development of the area of ​​interest based on comparing parameters recorded at two moments in time.

[0049] The method includes recommending beauty products based on a 3D reconstruction of a user's area of ​​interest.

[0050] Computer program product

[0051] The computer program product includes instructions, which, when executed by a computer, cause the computer to: obtain at least one RGB or grayscale input image of a user's area of ​​interest, obtain at least one high-resolution or medium-resolution 2D input image of the area of ​​interest, and reconstruct a 3D form of the user's area of ​​interest.

[0052] The computer program product comprises instructions which, when the program is executed by a computer, cause the computer to: compare reconstructions of the 3D form generated at different times.

[0053] The computer program product comprises instructions which, when executed by a computer, cause the computer to: recommend beauty products based on the reconstruction of the 3D form of the user's focus area, and present the products recommended to the user on a screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Further features and advantages of the present invention will become apparent by reading the following detailed description of non-limiting exemplary embodiments of the present invention and by examining the accompanying drawings, in which:

[0055] [ Figure 1 ]

[0056] [ Figure 1 ] schematically illustrates an embodiment and implementation of the present invention.

[0057] In the example shown, the method comprises, on the one hand, the step of importing three RGB input photographic images 1 , 2 , 3 of the region of interest, which images were taken at various camera angles, and, on the other hand, the step of importing a high-resolution or low-resolution local scan 4 .

[0058] For example, an imported photographic image may be produced by a capture performed by a user in a self-portrait mode or perhaps by photographing their reflection in a mirror.

[0059] Next, the method comprises a segmentation step in scanning 4 in order to identify at least one region of interest 5 of the user's face.

[0060] The segmentation step may be automatically performed by an image segmentation algorithm, for example, suitable for identifying objects, defining contours, identifying patterns, removing background, or performing other suitable and known image processing operations.

[0061] The segmentation step enables, for example, the detection of elements of interest such as wrinkles.

[0062] Of course, the elements of interest identified in the scan depend on the exposure.

[0063] As an alternative, the segmentation step may be performed "manually" by a user, for example by sending a request to the user identifying the location of the at least one region of interest in the image.

[0064] A machine learning model may advantageously be provided to correct for color distortions, for example, by taking into account the geographical location of the exposure (inside or outside), the timestamp of the exposure (day or night), the weather at the time of the exposure (sunny, cloudy, rainy, snowy), making it possible to estimate the parameters of the correction to be performed that may be more complex than the basic elements presented above.

[0065] The machine learning model may be, for example, of the convolutional neural network type. Using a machine learning model makes it possible to benefit from extremely precise accuracy in determining the color of the area of ​​interest.

[0066] After importing the image, the method comprises processing the image in order to perform a reconstruction 6 of the 3D form of the user's face based on a processing unit (eg a processor or a microprocessor).

[0067] The processing unit may be configured to monitor over time the reconstructions 6 and 7 of the 3D representation acquired at two different moments in time based on images of the user taken at different moments in time.

[0068] In particular, images may be acquired periodically (eg, annually). For monitoring purposes, each new 3D modeling 8 may be recorded.

[0069] The data extracted from each 3D modeling 6, 7, 8 may be associated with a time indication, which is related to the time when the image was acquired. For example, the time indication may be the date when the image was acquired.

Claims

1. A method for modeling a user's focus area, the method comprising: (i) obtaining at least one RGB or grayscale input image of the area of ​​interest, (ii) obtaining at least one high-resolution or medium-resolution 2D input image of the area of ​​interest, and (iii) reconstructing the area of ​​interest of the user in 3D form.

2. The method according to the preceding claim, the 3D reconstruction being a high-resolution 3D reconstruction.

3. The method according to one of the preceding claims, wherein the reconstruction of the 3D form is performed by combining global or local high-resolution 3D acquisition with a high-resolution local / global depth analysis.

4. The method according to any of the preceding claims, comprising 3D scanning based on annular projection or laser scanning.

5. A method according to any preceding claim, comprising triangulation based on images acquired from different viewpoints.

6. A method according to any of the preceding claims, comprising high resolution estimation based on lower resolution images using a machine learning model.

7. The method according to any of the preceding claims, comprising obtaining at least one additional data describing the personal information of the user, in particular obtaining the at least one additional data based on a questionnaire filled in by the user, and processing the additional data.

8. A cosmetic and non-therapeutic method for monitoring an area of ​​concern of a user over time, the method comprising: (i) acquiring at least one RGB or grayscale input image of the region of interest, (ii) acquiring at least one high-resolution or medium-resolution 2D input image of the region of interest, (iii) reconstructing the region of interest in 3D form, and (v) monitoring the generated 3D reconstruction over time by comparing the 3D reconstructions generated at different times.

9. The method according to the preceding claim, comprising: (vi) processing the reconstructions of the 3D form generated at different moments in time by superimposing predetermined time-invariant parts, in particular bone structures, and detecting the development of the element of interest between different moments in time based on the superposition.

10. The method according to any one of claims 8 and 9, wherein: Monitoring over time comprises calculating a gradient of development of the area of ​​interest based on a comparison of parameters recorded at two moments in time.

11. A method according to any preceding claim, comprising recommending a beauty product based on the reconstruction of the 3D form of the user's area of ​​interest.

12. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

13. A computer program product comprising instructions which, when executed by a computer, cause the computer to: Acquire at least one RGB or grayscale input image of the user's area of ​​interest, acquire at least one high-resolution or medium-resolution 2D input image of the area of ​​interest, A 3D form of the user's focus area is reconstructed.

14. A computer program product according to claim 13, comprising instructions which, when the program is executed by a computer, cause the computer to: Compare the reconstructions of the 3D form generated at different times.

15. A computer program product according to claim 13 or 14, comprising instructions which, when the program is executed by a computer, cause the computer to: recommending beauty products based on the 3D reconstruction of the user's area of ​​interest, The product recommended to the user is presented on a screen.

16. A data processing system comprising: - a memory having stored therein a computer program product according to any one of claims 13 to 15, - a processing unit configured to implement the computer program product, - a photographic device configured to acquire an RGB or grayscale input image of the region of interest and a high-resolution or medium-resolution 2D input image of the region of interest, - a screen configured to display a reconstruction in 3D of the area of ​​interest of the user generated by the processing unit and which can be configured to present products recommended to the user.

Images