Skin evaluation

Abstract

According to one aspect, there is provided an apparatus for assessing a subject's skin, the apparatus comprising: a processor configured to receive one or more images of the subject's skin from an imaging unit configured to acquire images of the subject's skin, the processor configured to process the one or more images to determine a skin type and / or a melanin index of the subject's skin based on one or more characteristics of a structure of the skin's pigment network, and to output a first signal indicative of the determined skin type and / or melanin index.

Description

[Technical field]

[0001] The present disclosure relates to assessing skin, and more particularly to determining the skin type and / or melanin index of skin. [Background technology]

[0002] Intense Pulsed Light (IPL) technology is a popular solution for many applications, including but not limited to photoepilation, lesion treatment, photorejuvenation, at-home personal care, professional personal care, and medical settings. For photoepilation in a home environment, an IPL photoepilation device applies broad spectrum light to the surface of the skin, targeting melanin in the hair. The hair and hair follicle (in the anagen phase of the growth cycle) absorb this energy and transition to the quiescent phase. This prevents hair regrowth. For effective use of the above IPL technology for hair removal (e.g., to minimize skin damage or irritation), the energy setting of the IPL photoepilation device must be adjusted based on the skin's complexion. Some IPL photoepilation devices, such as the Philips Lumea Prestige, detect skin color to select the appropriate energy setting. Skin color can be detected and classified into one of, for example, six different types. Skin types 1 through 6 can be loosely labeled as "white," "beige," "light brown," "medium brown," "dark brown," and "brownish black and darker." Because the skin, not the hair or hair follicles, absorbs the energy in the light pulses, IPL photoepilation devices should not typically be used on darker skin tones. In this case, the device should not trigger a flash when, for example, brownish black and darker skin tones are detected.

[0003] In addition to hair removal, light-based technology can also be used for other types of dermatological treatments, including treating acne and skin lesions. Summary of the Invention [Problem to be solved by the invention]

[0004] Current methods of skin type detection in IPL devices use reflectance spectroscopy, as described for example in "A portable reflectometer for a rapid quantification of cutaneous haemoglobin and melanin" by Feather JW, Ellis DJ and Leslie G., Phys. Med. Biol. 1988, 33, 711-722. In this technique, the ratio between two reflected wavelengths (red and near infrared - at those two wavelengths melanin has a high absorption coefficient compared to water and hemoglobin) is used to calculate the melanin index, which is then used to calculate the skin type. These reflected energy signals are affected by temperature, ambient light, and the waveguide of the IPL device. Furthermore, the ratio can be calibrated against a reference synthetic skin sample. Another alternative to this method is to use smartphone camera images of the skin area to calculate skin color. This is also very difficult due to the variation in specular reflectance of skin, the variation in lighting conditions, and the characteristics of the image sensor. Furthermore, image measurements require the use of color correction cards.

[0005] Therefore, improvements in determining the skin type and / or melanin index of skin, for use in setting the energy level of an IPL device, are desirable. [Means for solving the problem]

[0006] According to a first aspect, there is provided an apparatus for assessing the skin of a subject, the apparatus comprising a processor configured to receive one or more images of the skin of the subject from an imaging unit configured to acquire images of the skin of the subject, the processor configured to process the one or more images to determine a skin type and / or a melanin index of the skin of the subject based on one or more characteristics of a structure of a pigment network of the skin, and to output a first signal indicative of the determined skin type and / or melanin index, whereby the apparatus is able to determine the skin type and / or melanin index of the skin from the images of the skin.

[0007] In some embodiments, one or more characteristics of the structure of the skin's pigment network relate to any of the following: the presence of melanin in the structure; the presence of melanin in keratinocytes and / or melanocytes in the structure; the presence of melanin in keratinocytes and / or melanocytes along the dermal-epidermal junction of the skin; and the presence of dermal papillae and / or projections of the skin.

[0008] In some embodiments, the processor is configured to determine a presence and / or density of melanin in keratinocytes and / or melanocytes of the skin from the one or more images, in these embodiments, the processor is configured to determine the skin type and / or melanin index by comparing the determined presence and / or density of melanin to respective threshold values ​​corresponding to the respective skin type and / or melanin index.

[0009] In some embodiments, the processor is configured to determine a pattern of dermal papilla flanges and / or protrusions from the one or more images, in these embodiments, the processor is configured to determine the skin type and / or melanin index by comparing the determined pattern of flanges and / or protrusions with respective patterns corresponding to the respective skin type and / or melanin index.

[0010] In some embodiments, the processing unit is configured to use a trained machine learning model to process the one or more images to determine a skin type and / or a melanin index of the subject's skin based on one or more characteristics of the structure of the pigment network. The trained machine learning model may be an artificial neural network, such as a deep neural network.

[0011] In some embodiments, the determined skin type comprises any of normal skin, dry skin, oily skin, tanned skin, non-tanned skin, combination skin, and skin types on the Fitzpatrick scale.

[0012] In some embodiments, the determined melanin index comprises a melanin degree.

[0013] In some embodiments, the device further comprises a user interface configured to receive the first signal, the first signal configured to cause the user interface to output feedback to the user indicative of the determined skin type and / or melanin index.

[0014] In some embodiments, the first signal is output to a controller of the treatment device, and the controller uses the first signal to determine one or more operational settings of the treatment device, whereby the operational settings can be automatically applied by the treatment device in response to the first signal, which are useful when settings for treatment operation by the treatment device are changed based on skin type and / or melanin index.

[0015] In some embodiments, the processor is configured to normalize the received image or images with respect to color and process the normalized image or images to determine a skin type and / or a melanin index of the skin. In this way, it can be ensured that the color of the skin does not influence the determination of the skin type and / or melanin index, which is instead based on the properties of the structure of the pigment network.

[0016] According to a second aspect, there is provided a system comprising an imaging unit arranged to acquire an image of a subject's skin and an apparatus according to the first aspect or an embodiment thereof.

[0017] In some embodiments, the system further comprises a treatment device for performing a treatment operation on the subject's skin.

[0018] In some embodiments, the treatment device comprises a controller configured to receive a first signal, the controller configured to determine one or more operational settings for the treatment device based on the skin type and / or melanin index indicated in the first signal.

[0019] In these embodiments, the treatment device may comprise the imaging unit and / or the apparatus. In alternative embodiments, the treatment device may be separate from the apparatus.

[0020] In some embodiments, the system further comprises one or more light sources for illuminating the skin. These light sources may be used to illuminate the skin when the image is acquired.

[0021] In some embodiments, the light source is configured to emit light having wavelengths in the visible spectrum, and the emitted blue light has a higher brightness than the other colors of light.

[0022] In some embodiments, the system further comprises a first polarizer positioned to polarize light emitted by the one or more light sources. In some embodiments, the system further comprises a second polarizer positioned to polarize light incident on the imaging unit from the skin. The use of polarized light and / or the polarization of the light incident on the imaging unit may improve the visible skin depth in the acquired image.

[0023] In embodiments comprising a first polarizer and a second polarizer, the first polarizer and the second polarizer may be positioned orthogonal, or nearly orthogonal, with respect to one another.

[0024] According to a third aspect, there is provided a computer-implemented method for assessing the skin of a subject, the method comprising the steps of receiving one or more images of the subject's skin from an imaging unit acquiring images of the subject's skin, processing the one or more images to determine a skin type and / or a melanin index of the subject's skin based on one or more characteristics of a structure of the skin's pigment network, and outputting a first signal indicative of the determined skin type and / or melanin index, whereby the method is able to determine the skin type and / or melanin index of the skin from the images of the skin.

[0025] In some embodiments, one or more characteristics of the structure of the skin's pigment network relate to any of the following: the presence of melanin in the structure; the presence of melanin in keratinocytes and / or melanocytes in the structure; the presence of melanin in keratinocytes and / or melanocytes along the dermal-epidermal junction of the skin; and the presence of dermal papillae and / or projections of the skin.

[0026] In some embodiments, the processing step comprises determining a presence and / or density of melanin in keratinocytes and / or melanocytes of the skin from the one or more images, in these embodiments, the processing step comprises determining a skin type and / or melanin index by comparing the determined presence and / or density of melanin to respective threshold values ​​corresponding to the respective skin type and / or melanin index.

[0027] In some embodiments, the processing step comprises determining a pattern of dermal papilla flanges and / or protrusions from the one or more images, in these embodiments, the processing step comprises determining a skin type and / or melanin index by comparing the determined pattern of flanges and / or protrusions with respective patterns corresponding to respective skin types and / or melanin indexes.

[0028] In some embodiments, the processing step comprises using a trained machine learning model to process the one or more images to determine a skin type and / or a melanin index of the subject's skin based on one or more characteristics of the structure of the pigment network. The trained machine learning model may be an artificial neural network, such as a deep neural network.

[0029] In some embodiments, the determined skin type comprises any of normal skin, dry skin, oily skin, tanned skin, non-tanned skin, combination skin, and skin types on the Fitzpatrick scale.

[0030] In some embodiments, the determined melanin index comprises a melanin degree.

[0031] In some embodiments, the outputting step comprises outputting the first signal to a user interface, and the method further comprises outputting, by the user interface, feedback to the user indicative of the determined skin type and / or melanin index.

[0032] In some embodiments, the outputting step includes outputting the first signal to a controller of the treatment device, and the method further includes using the first signal by the controller to determine one or more operational settings of the treatment device, whereby the operational settings can be automatically applied by the treatment device in response to the first signal, such that the embodiments are useful when settings for treatment operation by the treatment device are changed based on skin type and / or melanin index.

[0033] In some embodiments, the method further comprises the step of normalizing the received image or images with respect to color and processing the normalized image or images to determine a skin type and / or melanin index of the skin. In this way, it can be ensured that the color of the skin does not influence the determination of the skin type and / or melanin index, which is instead based on the properties of the structure of the pigment network.

[0034] In some embodiments, the method further comprises acquiring one or more images using an imaging unit, in these embodiments, the method further comprises illuminating the skin using one or more light sources when acquiring the one or more images.

[0035] In some embodiments, the illuminating step includes the light source emitting light having wavelengths in the visible spectrum, and emitting blue light with a greater intensity than light of other colors.

[0036] In some embodiments, the method further comprises polarizing the light emitted by the one or more light sources. In some embodiments, the method further comprises polarizing the light incident from the skin to the imaging unit. The use of polarized light and / or the polarization of the light incident to the imaging unit may improve the visible skin depth in the acquired image.

[0037] According to a fourth aspect, there is provided an apparatus for training a machine learning model, MLM, for use in an apparatus for assessing a subject's skin according to the invention, comprising a processing unit configured to obtain, for one or more test subjects, a plurality of images of skin, each image annotated with an indication of a skin type and / or melanin index, and to train the MLM using the plurality of images to distinguish between images of different skin types and / or melanin indexes based on one or more properties of the structure of the skin's pigment network.

[0038] In some embodiments, the processing unit is further configured to normalize the plurality of images with respect to color, and train the MLM using the plurality of normalized images.

[0039] According to a fifth aspect, there is provided an apparatus for using a machine learning model, MLM, comprising a processing unit, the apparatus receiving one or more images of an area of ​​skin from an imaging unit acquiring images of the skin of a subject, the processing unit configured to use the MLM trained according to the fourth aspect or any embodiment thereof to analyze the one or more images to determine a skin type and / or a melanin index of the skin of the subject based on one or more properties of a structure of the skin's pigment network, and to output a first signal indicative of the determined skin type and / or melanin index.

[0040] According to a sixth aspect, there is provided a method of training a machine learning model, MLM, for use in an apparatus for assessing a subject's skin according to the invention, the method comprising obtaining a plurality of images of the skin of one or more test subjects, each image being annotated with an indication of skin type and / or melanin index, the method comprising training the MLM using the plurality of images to distinguish between images of different skin types and / or melanin index based on one or more properties of the structure of the skin's pigment network.

[0041] In some embodiments, the method further comprises normalizing the plurality of images with respect to color and training the MLM using the plurality of normalized images.

[0042] According to a seventh aspect, there is provided a method of using a machine learning model, MLM, the method comprising the steps of receiving one or more images of an area of ​​skin from an imaging unit acquiring images of the subject's skin, said method comprising the steps of using the MLM trained according to the sixth aspect or any embodiment thereof to analyze the one or more images to determine a skin type and / or a melanin index of the subject's skin based on one or more properties of a structure of the skin's pigment network, and outputting a first signal indicative of the determined skin type and / or melanin index.

[0043] According to an eighth aspect there is provided a computer program comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured, when executed by a suitable device, computer or processing unit, to cause the device, computer or processing unit to perform a method according to the third aspect, the sixth aspect or the seventh aspect, or any embodiment thereof.

[0044] These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0045] Example embodiments are described, by way of example only, with reference to the following drawings: [Brief description of the drawings]

[0046] [Figure 1] FIG. 1 is a diagram of an exemplary treatment device with which the present invention can be used. [Diagram 2] FIG. 1 is a block diagram of an example system including an imaging unit and a device, in accordance with various embodiments. [Diagram 3] 1 is a plot illustrating an example light spectrum for a light source illuminating skin. [Figure 4] FIG. 1 shows six images of different skin types. [Diagram 5] FIG. 1 illustrates the pigment network in skin. [Figure 6] 1 is a flow chart illustrating an exemplary method for assessing skin. [Figure 7] FIG. 1 shows four images of different skin types. [Figure 8] FIG. 1 shows four images of different skin types after color normalization. [Figure 9] 13 is a plot showing inference results for different skin types. [Figure 10] 1 is a flowchart illustrating a method for training a machine learning model for use in assessing skin. [Figure 11] 13 is a plot showing inference results on training folds. [Figure 12] 13 is a plot showing inference results for other training folds. [Figure 13] 13 is a plot showing inference results for other training folds. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0047] As mentioned above, the techniques described herein may be used to evaluate skin shown in an image to determine the skin type and / or melanin index of the skin. The techniques may be implemented by a treatment device that utilizes the determined skin type and / or melanin index (e.g., the techniques may be implemented by a processor in the treatment device) or may be implemented by a processor in a separate device. As described below, an exemplary type of treatment device in which the skin type and / or melanin index may be used is a treatment device that uses light pulses for hair removal and / or hair growth reduction. It will be understood that the determined skin type and / or melanin index need not be used by the treatment device or for purposes associated with the treatment device. For example, the skin type and / or melanin index may be used for recommending skin products (e.g., makeup products or concealers) and / or for evaluating pigmentation disorders.

[0048] An imaging unit (e.g., a camera) is used to capture one or more images of an area of ​​the subject's skin. The imaging unit may be part of the treatment device (if applicable), or part of a separate apparatus, or may be separate from both the treatment device and the apparatus implementing the techniques described herein. For example, the imaging unit may be or comprise a camera on a device such as a smartphone.

[0049] It has been found that skin type and / or melanin index are related to the structure of the skin, which is seen in the image of the skin. In particular, one or more characteristics of the structure of the pigment network in the skin (such as the density of the pigment network) are related to the skin type and / or melanin index. For dark skin types, the pigment network is usually well connected (i.e., densely connected) and has a dense honeycomb structure, while for light skin types, the pigment network is disconnected and more sparse (i.e., less dense), and in some cases the pigment is not perceptible. Therefore, according to the technique described herein, one or more images of skin are analyzed to determine the skin type and / or melanin index of the subject's skin based on one or more characteristics of the structure of the pigment network of the skin.

[0050] Figure 1 is a diagram of an exemplary treatment device 2 that can be used to apply light pulses to an area of ​​skin. It will be understood that the treatment device 2 of Figure 1 is provided only as an example of a handheld treatment device 2 with which the present invention can be used, and that the treatment device 2 is not limited to the form or handheld treatment device shown in Figure 1. As noted above, the present invention is similarly not limited to being implemented in or with the treatment device 2.

[0051] The treatment device 2 of Fig. 1 is for use on the body of a subject (e.g., a human or animal) and is held in one or both hands of a user during use. With the treatment device 2 in contact with the subject's body part, the treatment device 2 performs some treatment action on the hair on the subject's body using one or more light pulses. The treatment action can be the removal of unwanted hair by laser and / or light therapy (known as photoepilation treatment or intense pulsed light (IPL) treatment).

[0052] As described herein, the treatment device 2 is operated or used by a "user" and the treatment device 2 is used on the body of a "subject." In some cases, the user and the subject are the same person, i.e., the treatment device 2 is held by the user in the hand and used on the user himself (e.g., on the skin of his leg). In other cases, the user and the subject are different people, e.g., the treatment device 2 is held by the user in the hand and used on another person. In either case, changes in the skin or hair are less or not at all perceptible to the user for a short period of time after application of the light pulse, making it difficult for the user to achieve complete coverage of the body part and / or to avoid over-treating certain areas of the body part.

[0053] The exemplary treatment device 2 comprises a housing 4 including at least a handle portion 5 and a head portion 6. The handle portion 5 has a shape that allows a user to hold the treatment device 2 in one hand. The head portion 6 resides at a head end 8 of the housing 4, and is positioned to contact a subject such that a personal care action is performed on the subject's body or skin at a position where the head portion 6 is in contact with the body or skin.

[0054] The treatment device 2 is for performing a treatment action using energy or energy pulses (e.g. light or light pulses). Thus, in FIG. 1, the head portion 6 comprises an aperture 10 disposed in or on the housing 4, which may be disposed adjacent to or on (i.e. in contact with) the skin of the subject. The treatment device 2 comprises one or more light sources 12 for generating light pulses that are applied to the skin of the subject via the aperture 10 to achieve the treatment action. The one or more light sources 12 are disposed in the housing 4, whereby light pulses are provided from the one or more light sources 12 via the aperture 10. The aperture 10 has the shape of an opening in the head end 8 of the housing 4, or has the shape of a window (including a waveguide) that is transparent or semi-transparent for the light pulses (i.e. the light pulses can pass through the window).

[0055] In the exemplary embodiment shown in Figure 1, the aperture 10 has a generally rectangular shape, resulting in a generally rectangular skin treatment area on the skin. It will be understood that the aperture 10 may have any other desired shape. For example, the aperture 10 can be square, oval, circular, or any other polygonal shape.

[0056] The one or more light sources 12 can generate light pulses at any suitable or desired wavelength (or range of wavelengths) and / or intensity. For example, the light sources 12 can generate visible light, infrared (IR) light, and / or ultraviolet (UV) light. Each light source 12 can comprise any suitable type of light source, such as one or more light emitting diodes (LEDs), (xenon) flash lamps, lasers, etc. The light sources 12 can provide light pulses having spectral content in the range of 560 to 1200 nanometers (nm) for a duration of approximately 2.5 milliseconds (ms), as these wavelengths heat the melamine in the hair and hair roots by absorption, transitioning the hair follicle into a resting phase and preventing hair regrowth.

[0057] The light source(s) 12 are configured to provide pulses of light, i.e., the light source 12 is configured to generate high intensity light for a short period of time (e.g., less than one second). The intensity of the light pulse should be high enough to achieve a treatment action on the skin or body area adjacent the aperture 10.

[0058] The illustrated treatment device 2 further comprises two skin contact sensors 14, 16 disposed on or in the head 6 that are used to determine whether the head 6 is in contact with the skin. The skin contact sensors 14, 16 measure parameters indicative of whether the head 6 is in contact with the skin and generate respective measurement signals that include a time series of the measurement results of the parameters. The measurement signals can be processed to determine whether the head 6 is in contact with the skin. Typically, skin contact sensors are used in treatment devices 2, particularly photoepilators, to ensure that the treatment device 2 is in proper contact with the skin before the light pulse is generated to avoid the light pulse being directed toward the user's or subject's eyes.

[0059] In some embodiments, the parameter may be capacitance, such that the skin contact sensors 14, 16 can measure capacitance via respective pairs of electrical contacts or electrodes on the surface of the head portion 6, with the measured capacitance indicating whether skin contact is present. In alternative embodiments, the parameter may be light intensity or amount, such that the skin contact sensors 14, 16 can be light sensors that measure the intensity or amount of light incident on the light sensor, with the measured intensity or amount indicating whether skin contact is present (e.g., little light or no light may indicate skin contact when the skin covers the light sensor 14, 16, or vice versa). In other alternative embodiments, the parameter may be a measurement of skin pressure, such that the skin contact sensors 14, 16 can measure skin pressure via respective pressure sensors or mechanical switches, with the measured skin pressure indicating whether skin contact is present.

[0060] The illustrated treatment device 2 further comprises a user control 20 operable by a user to activate the treatment device 2, thereby causing the head portion 6 to perform a desired treatment action on the subject's body (e.g., generation of light pulses by the one or more light sources 12). The user control 20 may be in the form of a switch, a button, a touch pad, or the like.

[0061] 2 is a block diagram of an exemplary system 40 comprising an apparatus 42 for assessing the subject's skin and an imaging unit 44. In some embodiments, the treatment device 2 may be considered as part of the system 40, although the treatment device 2 is not shown in FIG. 2. As mentioned above, the apparatus 42 may be a separate device from the treatment device 2, and thus the apparatus 42 may have the form of an electronic device such as a smartphone, a smart watch, a tablet, a personal digital assistant (PDA), a laptop, a desktop computer, a remote server, a smart mirror, etc. In other embodiments, the apparatus 42, and in particular the functionality provided by the apparatus 42 according to the invention, is part of the treatment device 2.

[0062] The imaging unit 44 is provided to generate one or more images (or video sequences) of one or more areas of the subject's skin. The imaging unit 44 includes any suitable components for capturing an image, such as, for example, a charge-coupled device (CCD) and one or more lenses and / or mirrors. In some embodiments, the imaging unit 44 is a camera, such as a digital camera. In some embodiments, the imaging unit 44 is a camera configured to provide a microscopic image of the skin (i.e., the imaging unit 44 provides a magnification or zoom on the image of the skin). In some embodiments, one or more additional optical components are associated with the imaging unit 44 or are associated with a part of the imaging unit 44. For example, a polarizer placed in front of the imaging unit 44 may be provided to polarize the light entering the imaging unit 44. The use of a polarizer may improve the depth of the skin area visible in the image (i.e., the polarizer allows the light to be observed at different penetration depths in the skin). In some embodiments, one or more light sources may be provided to illuminate the skin area to be imaged as the image is being acquired. These light sources are separate from the light source 12 of the treatment device 2 used to perform the treatment operation. In these embodiments, a polarizer may be provided to polarize the light emitted by the light source before illuminating the skin. The light source for illuminating the skin for the purposes of acquiring the image may be any suitable light source, such as, for example, one or more LEDs.

[0063] In certain embodiments, a first polarizer can be provided to polarize light emitted by one or more light sources before the light shines on the skin, and a second polarizer can be provided to polarize light incident on the imaging unit 44 from the skin. The first and second polarizers can be arranged to be "crossed", i.e., the first and second polarizers are arranged such that their respective polarization directions are orthogonal (i.e., 90° to each other) or nearly orthogonal (i.e., approximately 90° to each other). The use of crossed polarizers can improve the depth of the skin area in the image, thereby making the structure of the skin visible.

[0064] 2 as being separate from the apparatus 42, it will be understood that in other embodiments the imaging unit 44 may be integral with or part of the apparatus 42. In embodiments in which the imaging unit 44 is separate from the apparatus 42, the imaging unit 44 may be part of the treatment device 2 or may be separate from the treatment device 2. In embodiments in which the imaging unit 44 is part of the treatment device 2, the imaging unit 44 may be positioned on the treatment device 2 near the aperture 10 such that images can be obtained when the treatment device 2 is on or close to the skin.

[0065] In an exemplary embodiment in which one or more light sources are provided to illuminate the skin area to be imaged, the one or more light sources can emit light at a specific wavelength or a specific wavelength range. The selected light spectrum allows skin structures such as pigment networks to be identified and / or analyzed in the resulting image. FIG. 3 is a plot showing an exemplary light illumination spectrum of a light source illuminating the skin during image acquisition. It can be seen that the light source can emit light having a wavelength in the range of 400 nanometers (nm) to 760 nm (i.e. visible light), with the highest brightness of the emitted light being around 450 nm (i.e. blue light). Light in the range of 500 nm to 700 nm (i.e. green-red light) is emitted, but with a (significantly) lower brightness than blue light. In the example of FIG. 3, the brightness of the emitted light decreases significantly towards the reddish color range. A light source having the light illumination spectrum of FIG. 3 (or a similar spectrum), together with the cross polarizers described above, allows images to be acquired from the depth of the skin where the melanin / pigment network is located. These images do not contain the specular component or the red component from deeper skin regions caused by the presence of blood vessels.

[0066] The device 42 comprises a processing unit 46 that generally controls the operation of the device 42 and enables the device 42 to perform the methods and techniques described herein. Briefly, the processing unit 46 receives one or more images from the imaging unit 44, processes the images to determine a skin type and / or melanin index of the subject's skin based on one or more characteristics of the structure of the skin's pigment network as shown in the images, and outputs a first signal indicative of the determined skin type and / or melanin index.

[0067] Processing unit 46 may thereby be configured to receive an image from imaging unit 44 either directly, in an embodiment in which imaging unit 44 is part of device 42, or via other components in an embodiment in which imaging unit 44 is separate from device 42. In either case, processing unit 46 may include or comprise one or more input ports or wiring for receiving an image (or a signal carrying information representative of that image) from imaging unit 44 or other components as appropriate. Processing unit 46 may further include or comprise one or more output ports or wiring for outputting a signal indicative of whether hairs in the area of ​​skin have been treated with a light pulse.

[0068] Processing unit 46 can be implemented in many ways using software and / or hardware to perform the various functions described herein. Processing unit 46 comprises one or more microprocessors or digital signal processors (DSPs) that are programmed using software or computer program code to perform the desired functions and / or to control the components of processing unit 46 to achieve the desired functions. Processing unit 46 is implemented as a combination of dedicated hardware (e.g., amplifiers, preamplifiers, analog-to-digital converters (ADCs) and / or digital-to-analog converters (DACs)) to perform some functions and processors (e.g., one or more programmed microprocessors, controllers, DSPs, and associated circuitry) to perform other functions. Examples of components used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), hardware for implementing neural networks, and / or so-called artificial intelligence (AI) hardware accelerators (i.e., processors or other hardware designed specifically for AI applications that can be used in conjunction with a main processor).

[0069] The processing unit 46 may include or be associated with a memory unit 48. The memory unit 48 may store data, information, and / or signals (including images) used by the processing unit 46 in controlling the operation of the device 42 and / or in implementing or performing the methods described herein. In some implementations, the memory unit 48 stores computer readable code executable by the processing unit 46 to cause the processing unit 46 to perform one or more functions including the methods described herein. In certain embodiments, the program code may have the form of an application for a smart phone, tablet, laptop, computer, or server. The memory portion 48 may comprise any type of non-transitory machine-readable medium, such as cache or system memory, including volatile and non-volatile computer memory, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), and electrically erasable PROM (EEPROM), and the memory portion may be embodied in the form of memory chips, optical disks (such as compact disks (CDs), digital versatile disks (DVDs) or Blu-ray disks), hard disks, tape storage solutions, or solid-state devices, including memory sticks, solid-state drives (SSDs), memory cards, and the like.

[0070] In the embodiment shown in FIG. 2, the device 42 is shown as being separate from the imaging unit 44, and therefore the device 42 further includes an interface circuit 50 that allows the device 42 to receive images from the imaging unit 44. The interface circuit 50 of the device 42 allows data connection and / or data exchange with other devices, including any one or more of the imaging unit 44, the treatment device 2, a server, a database, a user device, and a sensor. The connection to the imaging unit 44 (or an electronic device such as the treatment device 2) may be direct or indirect (e.g., via the Internet), whereby the interface circuit 50 can provide a connection between the device 42 and a network or a direct connection between the device 42 and other devices (such as the imaging unit 44 and / or the treatment device 2) by a desired wired or wireless communication protocol. For example, the interface circuit 50 can operate using WiFi, Bluetooth, Zigbee, or any cellular communication protocol, including but not limited to Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), LTE-Advanced, etc. In the case of a wireless connection, the interface circuit 50 (and thus the device 42) includes one or more suitable antennas for transmitting and receiving over a transmission medium (e.g. air). Alternatively, in the case of a wireless connection, the interface circuit 50 includes means (e.g. connectors or plugs) allowing the interface circuit 50 to be connected to one or more suitable antennas external to the device 42 for transmitting and receiving over a transmission medium (e.g. air). The interface circuit 50 is connected to the processing unit 46.

[0071] 2, device 42 may include one or more user interface components including one or more components that enable a user of device 42 to input information, data, and / or commands to device 42 and / or enable device 42 to output information or data to a user of device 42. The user interface may include any suitable input component including, but not limited to, a keyboard, a keypad, one or more buttons, switches or dials, a mouse, a trackpad, a touch screen, a stylus, a camera, a microphone, etc., and the user interface may include any suitable output component including, but not limited to, a display unit or display screen, one or more lights or light elements, one or more speakers, a vibrator, etc.

[0072] It will be appreciated that practical implementations of device 42 may include additional components to those shown in Figure 2. For example, device 42 may further include a power source, such as a battery, or components that enable device 42 to be connected to a mains power source.

[0073] Figure 4 shows six example images of different skin types, where the skin was illuminated with light according to the spectrum shown in Figure 3 and captured using imaging unit 44 with crossed polarizers. Each of the images shows one of the six different skin types outlined above (i.e. skin types 1-6, i.e. "white", "beige", "light brown", "medium brown", "dark brown", and "brownish black and darker", respectively).

[0074] A general description of the structure of the skin is given in "Standardization of terminology in dermoscopy / dermatoscopy: Results of the third consensus conference of the International Society of Dermoscopy", Kittler et al., and a diagram of the normal pigment network is shown in Figure 5. Figure 5 can be found at https: / / dermoscopedia.org / pigment_network.

[0075] The pigment network 60 consists of a lattice of intersecting pigmented "lines" 62 that form a honeycomb pattern 64. The anatomical basis of the pigment network 60 is melanin in keratinocytes and / or melanocytes along the dermal-epidermal junction (epidermis 66 and dermis 68), which represents how the interpapillary ridge pattern of the epidermis 66 appears when viewed in a horizontal plane. The less pigmented "pores" 70 of the pigment network 60 correspond to the tips of the dermal papillae of the epidermis 66 and the overlying epidermis above the papillae. A wider diameter of the dermal papillae corresponds dermomicroscopically to a wider "pore" 70 of the network, while narrower dermal papillae result in a denser "sieve" of the lattice. The pigment network 60 may not be visible if the interpapillary ridge pattern contains less melanin pigment. The pattern shown in the six microscopic images of FIG. 4 corresponds to the melanin distribution along the interpapillary ridges when viewed down.

[0076] Thus, one or more characteristics of the structure of the pigment network in the skin (such as the density of the pigment network) can be observed or determined from the skin image, the characteristics being related to the skin type and / or melanin index. For dark skin types (e.g. skin type 5.0 or 6.0), the pigment network is usually well-connected (e.g. densely connected) and dense honeycomb structure, whereas for light skin types (e.g. skin types 1.0 and 2.0), the structure of the pigment network is choppy and more sparse, and in some cases the pigment is not perceptible. Skin structure is also generally described in "Advances in Dermoscopy of Pigmented Lesions", P. Kumarasinghe, Pigmentary Skin Disorders, pp. 79-92.

[0077] 6 illustrates an example method described herein for assessing a subject's skin. One or more of the steps of the method may be performed by a processor 46 of the device 42, in conjunction with either a memory portion 48 and an interface circuit 50 of the device 42, and / or an imaging unit 44, as appropriate. The processor 46 performs one or more steps in response to the execution of computer program code, which may be stored on a computer-readable medium, such as the memory portion 48.

[0078] In step 101, one or more images of an area of ​​skin are received. The images can be received directly from the imaging unit 44 as the images are generated by the imaging unit 44, for example in real time or near real time. Alternatively, the images are generated in advance by the imaging unit 44 and stored for subsequent analysis, for example in the memory unit 48, a memory unit associated with the treatment device 2 or the imaging unit 44, or a remote database, in which case step 101 may include the processing unit 46 obtaining or retrieving the images from a storage location (e.g., the memory unit 48, etc.). The images received in step 101 relate to a particular subject. If multiple images are received, they include multiple images of the same area of ​​skin. The multiple images may also or alternatively include images of skin on different parts of the subject's body.

[0079] Figure 7 illustrates images of four different skin types that may be received or acquired in step 101. It will be appreciated that such different images are not acquired from a single subject in a single execution of step 101, rather, Figure 7 merely illustrates different skin types that may be shown in images that may be acquired in step 101. It will be appreciated that the four images in Figure 7 are labelled according to a floating point continuous classification (similar to Fitzpatrick skin types), however this classification is not known at this stage of the method.

[0080] In step 103, the image or images are processed to determine the skin type and / or melanin index of the skin in the image based on one or more properties of the structure of the skin's pigment network (eg density).

[0081] Step 103 may include determining the skin type as either normal skin, dry skin, oily skin, tanned skin, non-tanned skin, combination skin (i.e. a mixture of the aforementioned skin types), and the skin type on the Fitzpatrick scale. Additionally or alternatively, step 103 may include determining a melanin index as a score of melanin degree (e.g. a score ranging from 0 to 1000, or the like).

[0082] In some embodiments, step 103 may include determining a skin type and / or melanin index based on one or more characteristics of the structure of the skin pigment network related to the presence of melanin in the keratinocytes and / or melanocytes in the structure of the skin pigment network. In some embodiments, the skin type and / or melanin index may be based on the presence of melanin in the keratinocytes and / or melanocytes along the dermal-epidermal junction of the skin. In particular, step 103 may comprise determining the presence and / or density of melanin in the keratinocytes and / or melanocytes of the skin. In some embodiments, the skin type and / or melanin index may be determined by comparing the determined presence and / or density of melanin to a respective threshold value corresponding to the respective skin type and / or melanin index.

[0083] Additionally or alternatively, step 103 may comprise determining the skin type and / or melanin index based on one or more characteristics of the structure of the skin's pigment network related to the presence of dermal papillae edges and / or protrusions of the skin. In particular, step 103 may comprise determining the skin type and / or melanin index by determining a pattern of dermal papillae edges and / or protrusions from the one or more images and comparing the determined pattern of edges and / or protrusions with respective patterns corresponding to the respective skin types and / or melanin indices.

[0084] In some preferred embodiments of step 103, a trained machine learning model (MLM) is used to process one or more images to determine the skin type and / or melanin index of the skin. The MLM is trained to determine the skin type and / or melanin index of the skin based on one or more properties (e.g., density) of the structure of the pigment network. The MLM can be any suitable type of MLM, such as a traditional machine learning model, such as feature extraction using support vector machines, decision trees, random forests, etc., or an artificial neural network, such as a neural network, having multiple layers between the input layer and the output layer and identifying a linear or non-linear relationship between the input layer and the output layer. The MLM performs an evaluation for each image or each set of images of a skin area to determine the skin type and / or melanin index. In some embodiments, the MLM receives the images directly and performs all the necessary analysis and processing of the images (e.g., determining the properties of the structure of the pigment network) to determine the skin type and / or melanin index. This is especially the case for MLMs that are artificial neural networks, such as deep neural networks. In other embodiments, for example when using a conventional MLM, the image may be processed before being provided to the MLM, for example to determine values ​​of properties of the pigment network structure, and those values ​​(optionally in addition to the image) may be provided to the MLM for analysis to determine skin type and / or melanin index. In some embodiments, multiple separate trained MLMs (homogeneous or heterogeneous) may be provided to respectively determine skin type and melanin index.

[0085] In some embodiments, prior to or as an initial operation in step 103, the received image or images may be color normalized (i.e., color information is removed from the image, leaving a grayscale image), and the color normalized images may be processed to determine the skin type and / or melanin index of the skin. Normalizing the image for color removes color from the decision process, meaning that the decision regarding the skin type and / or melanin index may be based on properties of the structure of the pigment network. FIG. 8 illustrates images of four different skin types normalized for color. Note that these images are of different skin regions of a different subject than the image in FIG. 7. It will be appreciated that such different images are not obtained from a single subject in one execution of step 101, but rather FIG. 8 merely illustrates the skin types that may be shown in the color normalized images. With reference to FIG. 7, it will be appreciated that the four images in FIG. 8 are labeled according to a floating-point continuous classification (similar to Fitzpatrick skin types), but the classification is not known until step 103 is completed.

[0086] After the skin type and / or melanin index are determined in step 103, a signal indicative of the determined skin type and / or melanin index is output in step 105. This signal is provided to a user interface component of the apparatus 42 or treatment device 2, where the signal is configured to cause the user interface component to indicate the determined skin type and / or the determined melanin index. A user of the treatment device 2 uses the indications to determine whether to trigger a light pulse at the current location of the treatment device 2 or to adjust the power settings of one or more light sources 12 in the treatment device 2 according to the skin type. As another example, if the apparatus 42 is in the form of a smartphone or similar type of device, feedback regarding the determined skin type and / or the determined melanin index may be provided to the user or subject by an app (software application) running on the apparatus 42. Those skilled in the art will recognize other ways in which feedback regarding the determined skin type and / or the determined melanin index may be provided to the user, including, for example, a display screen, a speaker, haptic feedback, etc.

[0087] Alternatively (or in addition), if the treatment device 2 is capable of automatically triggering a light pulse or other treatment action if conditions are appropriate (e.g., the treatment device 2 is in contact with skin, the light source 12 is charged so that it can generate a light pulse, etc.), a signal can be provided to a controller of the treatment device 2, and the controller can use the first signal to determine one or more operational settings of the treatment device 2. For example, the controller can use an indication of the determined skin type and / or melanin index in the first signal to make a decision whether to treat the area of ​​skin currently adjacent the aperture 10 with a light pulse and / or use energy / power to do so.

[0088] In alternative embodiments, the signal indicative of the determined skin type and / or melanin index may be used to provide recommendations or suggestions for particular skin care and / or makeup products that the subject may use. For example, the determined skin type may be used to identify skin care products suitable for that skin type (or vice versa). As another example, the determined melanin index may be used to recommend a particular concealer.

[0089] The plot in Figure 9 illustrates the operation of a deep neural network in identifying skin type from many different color-normalized images for different skin types based on the properties of the structure of the skin pigment network. Figure 9 plots the actual skin type ("expected skin type") against the skin type identified by the deep neural network ("inferred skin type"). It can thus be seen that the neural network is able to accurately identify skin type from images, with accuracy being particularly high for skin types at either end of the range (i.e., skin types 2.0 and 6.0).

[0090] The MLM used in the embodiment of step 103 is trained prior to use. The flow chart of Figure 10 illustrates a method of training an MLM for use in an apparatus for assessing skin in accordance with the present invention to determine skin type and / or melanin index. It will be appreciated that the training method of Figure 10 need not be performed by the same apparatus or device that performs Figure 6, but the training method of Figure 10 may be performed by any suitable apparatus or device, including apparatus 42. For example, the training method of Figure 10 may be performed by a server or computer at a central location, and the trained MLM (or computer code representing the trained MLM) may be distributed to various apparatus 42 for use in assessing skin in accordance with the method of Figure 6.

[0091] 6, one or more of the method steps may be performed by processing unit 46 in conjunction with either memory unit 48 and interface circuitry 50 of device 42 and / or imaging unit 44, as appropriate. Processing unit 46 performs one or more steps in response to the execution of computer program code, which may be stored on a computer-readable medium, such as memory unit 48.

[0092] To train the MLM, a training dataset is required. This training dataset includes multiple images of skin of one or more test subjects (which may or may not include the subject for whom the method of FIG. 6 is used). The multiple images are of skin at different sites on the body of the one or more test subjects. Each image in the training dataset is annotated with an indication of skin type and / or melanin index (depending on the parameters the MLM is trained to identify). This annotation is made manually by a user or other person. The training dataset should include at least one image (but preferably multiple images) relating to each of the different skin types and / or melanin indexes to be identified by the learned MLM. It will be appreciated that the larger the training dataset, the more accurate the resulting MLM may be.

[0093] A training data set is obtained in step 111 of Figure 13. This step may involve collecting images and associated annotations (e.g., using an imaging unit and a user interface) or retrieving the images of the training data set from a database or other electronic storage device.

[0094] Next, in step 113, the MLM is trained using a number of images in the training dataset, such that the MLM can distinguish between skin types and / or melanin indices (depending on the parameters the MLM is trained to identify). The MLM is trained to distinguish between images based on properties of the structure of the pigment network visible in the images. As mentioned above, for dark skin types (e.g., skin types 5.0 or 6.0), the pigment network is usually a well-connected, dense honeycomb structure, while for light skin types (e.g., skin types 1.0 and 2.0), the pigment network is choppy, sparser, and in some cases pigment is not perceptible. Thus, the MLM can be trained to distinguish between skin types and / or melanin indices based on the density of the pigment network.

[0095] In particular, the MLM may be trained to determine the skin type and / or melanin index based on one or more characteristics related to the presence of melanin in the keratinocytes and / or melanocytes in the structure of the pigment network and / or the presence of melanin in the keratinocytes and / or melanocytes along the dermal-epidermal junction of the skin, for example, by determining the presence and / or density of melanin in the keratinocytes and / or melanocytes of the skin. Additionally or alternatively, the MLM may be trained to determine the skin type and / or melanin index based on one or more characteristics related to the presence of dermal papillae ridges and / or protuberances of the skin, such as, for example, the pattern of dermal papillae ridges and / or protuberances. Techniques for training the MLM using a training dataset are known to those skilled in the art and will not be provided in detail herein. However, as an example, the MLM may be trained using cross-validation, where the MLM is trained using a subset of images in the training dataset and the learned MLM is tested for accuracy using one or more of the other images in the training dataset. This training and testing may be repeated on different subsets of images in the training dataset to arrive at a final trained MLM, which may then be used in step 103 of FIG.

[0096] In some embodiments, the images in the training dataset may be normalized with respect to color before being used to train the MLM, so that subsequently during training of the MLM, the MLM is biased towards skin structure in the images rather than color differences.

[0097] In an exemplary implementation, skin data (images) of 27 subjects were collected across all different skin types, with at least 5 subjects for each skin type. These images included images of skin both before and after seasonal tanning, as well as images of skin from several different body locations, such as cheeks, inner arms, outer arms, and legs. A baseline melanin index was also measured for each skin patch shown in each image. An MLM (e.g., a deep neural network) was trained with the data collected above using a common training technique. To prove the generalizability of skin structure for melanin index (and skin type) quantification, the dataset (images of 27 subjects) was split into "folds" for cross-fold validation experiments. Each fold consists of a training set (24 subjects out of the 27 subjects) and a validation set (3 subjects out of the 27 subjects). The training and validation sets are mutually exclusive. Each fold consists of a different permutation of the subjects split into the training and validation sets. For each fold, the training set was used to train the MLM and the validation set was used to test and evaluate the trained MLM. Thus, the MLM is tested and evaluated with images of skin structures from three subjects that were not included in the training set and that the MLM has not seen before. The plots in Figures 11, 12, and 13 show the inference results obtained by the trained MLM according to three different folds of the original dataset. Thus, each plot shows the behavior of the deep neural network in identifying the melanin index for a validation set of images of a particular fold. Each figure plots the actual melanin index ("expected melanin index") against the melanin index identified by the deep neural network ("inferred melanin index"). Thus, it can be seen that the neural network was able to accurately identify the melanin index from each image with minimal error spread.

[0098] Thus, improvements in determining the skin type and / or melanin index of skin are realized, which can be used, for example, in setting the energy level of an IPL device or in adjusting the energy level of an IPL treatment for each location where light is emitted, improving the comfort-to-benefit ratio for an IPL user.

[0099] By considering the drawings, the disclosure, and the appended claims, variations of the disclosed embodiments can be understood and realized by those skilled in the art in practicing the principles and techniques described herein. In the claims, the word "comprises" does not exclude other elements or steps, and the singular does not exclude a plurality. A single processor or other unit performs the functions of several elements recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program may be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems. Any reference numbers in the claims should not be interpreted as limiting the scope.

Claims

1. An apparatus for evaluating the skin of a subject, the apparatus comprising a processing unit, the processing unit comprising: receiving one or more images of the subject's skin from an imaging unit that acquires images of the subject's skin; processing the one or more images to determine a skin type and / or a melanin index of the skin of the subject based on one or more characteristics of a structure of a pigment network of the skin; outputting a first signal indicative of the determined skin type and / or melanin index. Device.

2. The one or more characteristics of the structure of the pigment network of the skin are: the presence of melanin in said structure; the presence of melanin in keratinocytes and / or melanocytes in said structure; the presence of melanin in keratinocytes and / or melanocytes along the dermal-epidermal junction of the skin; the presence of dermal papillae and / or projections of said skin; relating to any of the 2. The apparatus of claim 1.

3. The apparatus according to claim 1 or 2, wherein the processing unit is adapted to determine the presence and / or density of melanin in keratinocytes and / or melanocytes of the skin from the one or more images.

4. 4. The apparatus of claim 1, wherein the processor is adapted to determine a pattern of dermal papilla ridges and / or protrusions from the one or more images.

5. 5. The apparatus of claim 1 , wherein the processing unit uses a trained machine learning model to process the one or more images to determine the skin type and / or melanin index of the skin of the subject based on one or more characteristics of the structure of the pigment network.

6. 6. The apparatus of claim 1, wherein the determined skin type comprises any of the following: normal skin, dry skin, oily skin, tanned skin, non-tanned skin, combination skin, and skin types on the Fitzpatrick scale.

7. 7. The apparatus of claim 1, further comprising a user interface that receives the first signal, the first signal causing the user interface to output feedback to a user indicative of the determined skin type and / or melanin index.

8. 8. The apparatus of claim 1 , wherein the first signal is output to a control unit of a treatment device, the control unit using the first signal to determine one or more operational settings of the treatment device.

9. 9. The device according to claim 1 , wherein the processing unit is adapted to normalize the received image or images with respect to color and to process the normalized image or images to determine the skin type and / or melanin index of the skin.

10. an imaging unit for acquiring an image of the subject's skin; A device according to any one of claims 1 to 9. A system comprising:

11. The system of claim 10 , further comprising one or more light sources for illuminating the skin.

12. 12. The system of claim 11, wherein the one or more light sources emit light having wavelengths in the visible spectrum, and wherein the emitted blue light has a higher brightness than light of other colors.

13. 1. A computer-implemented method for assessing the skin of a subject, the computer-implemented method comprising: receiving one or more images of the subject's skin from an imaging unit that acquires images of the subject's skin; processing the one or more images to determine a skin type and / or a melanin index of the skin of the subject based on one or more characteristics of a structure of a pigment network of the skin; outputting a first signal indicative of the determined skin type and / or melanin index; 13. A computer-implemented method comprising:

14. A computer readable medium having computer readable code embodied therein, the computer readable code, when executed by a suitable computer or processor, causing the computer or processor to perform the method of claim 13.

15. 10. An apparatus for training a machine learning model for use in an apparatus for assessing the skin of a subject according to claim 5, the apparatus for training the machine learning model comprising: obtaining a plurality of images of skin for one or more test subjects, each image being annotated with an indication of skin type and / or melanin index; training the machine learning model using a plurality of images to distinguish between images of different skin types and / or melanin indices based on one or more characteristics of the structure of the skin pigment network; An apparatus comprising a processing unit.

Images