How to evaluate your skin

Abstract

To provide a novel skin evaluation method capable of evaluating an estimated value on a structure inside the skin of a living body, by utilizing information obtained easily from the living body.SOLUTION: Multiple pairs of a feature amount obtained from data showing an inner structure of a subject's skin, and a skin image obtained by imaging a living surface of the same subject are utilized, and the skin image is inputted into a model formed by leaning a relation between the feature amount and the skin image, to thereby output an estimated value of the feature amount.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present invention relates to a method for evaluating skin. [Background technology]

[0002] In recent years, advances in measurement techniques have made it possible to measure the internal structure of the skin non-invasively. For example, in Non-Patent Document 1, the structure of blood vessels is revealed by a tissue transparency technique.

[0003] Also known are methods for diagnosing skin conditions based on non-invasively measured internal structures of the skin. For example, Patent Document 1 discloses a method for diagnosing skin burns, gangrene, and bedsores by non-invasively measuring the internal structure of the skin using photoacoustic waves. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Shibata T, Kajiya K, Sato K, Yoon J, Kang HY. 3D microvascular analysis reveals irregularly branching blood vessels in the hyperpigmented skin of solar lentigo. Pigment cell & melanoma research.2018;31(6):725 7. [Patent documents]

[0005] [Patent Document 1] Patent No. 5860534 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, it is known that the internal morphological and physiological characteristics of the skin, such as the thickness and quantity of collagen fibers and the amount of water retained within the skin, i.e., the internal structure of the skin, affect the appearance of the skin (age spots, wrinkles, dullness, etc.). However, analysis of the internal structure of the skin using tissue clearing methods requires the removal of skin, which means that blood and oxygen are lost, making it impossible to measure the state of the tissue while it is still alive. In addition, it is difficult to ensure a sufficient number of samples for analysis. In other words, the above-mentioned techniques are limited in their ability to measure the internal structure of skin in a living body, and skin evaluations based on indicators obtained by such techniques do not necessarily reflect the condition of the living body.

[0007] In addition, although a method for diagnosing skin conditions (presence or absence of burns, gangrene, bedsores, etc.) based on non-invasively measured internal structure of a subject's skin, such as that described in Patent Document 1, is known, this method has the problem that expensive equipment is required to measure the internal structure of the skin.

[0008] In view of the above circumstances, an objective of the present invention is to provide a novel skin evaluation method that can output an estimated value regarding the internal structure of a living body's skin by utilizing information that can be easily obtained from the living body without using expensive equipment. [Means for solving the problem]

[0009] The present invention, which solves the above-mentioned problems, provides a skin evaluation method in which multiple pairs of feature quantities obtained from data showing the internal structure of a subject's skin and skin images obtained by photographing the biological surface of the same subject are used as learning data, and the skin images are input into a model that has learned the relationship between the feature quantities and the skin images, thereby outputting estimated values ​​of the feature quantities.

[0010] In a preferred embodiment of the present invention, the data indicative of the internal structure is non-invasively obtained data.

[0011] In a preferred embodiment of the present invention, the model is a convolutional neural network model created by performing deep learning using the skin image as input data and the feature amount as a response variable.

[0012] In addition, in a preferred form of the present invention, the method includes estimating the contribution of the model to the estimation result for each region of the input skin image by analyzing the model from the output side to the input side based on the input skin image.

[0013] In addition, a preferred embodiment of the present invention includes displaying the contribution of each region of the skin image to the estimation result by superimposing it on the skin image.

[0014] In a preferred embodiment of the present invention, the feature amount is at least one of a feature amount relating to a blood vessel structure of the skin, a feature amount relating to an oxygen saturation level of the skin, and a feature amount relating to a melanin structure of the skin.

[0015] In a preferred embodiment of the present invention, the feature amount relating to the blood vessel structure is a feature amount relating to the amount of microvascular structure in the upper part of the skin.

[0016] In a preferred embodiment of the present invention, the feature amount relating to the vascular structure is selected from an average thickness of blood vessels, a number of branches of blood vessels, and a volume of blood vessels in the upper part of the skin.

[0017] In a preferred embodiment of the present invention, the feature amount relating to oxygen saturation is blood vessel oxygen saturation.

[0018] In a preferred embodiment of the present invention, the feature quantity relating to the melanin structure is a volume of a tubular pattern structure of melanin.

[0019] In a preferred embodiment of the present invention, the data indicating the internal structure is measurement data obtained by a photoacoustic method or data generated based on the measurement data.

[0020] In a preferred embodiment of the present invention, the oxygen saturation of the skin of the subject is estimated based on the feature amount relating to the blood vessel structure of the skin.

[0021] In a preferred embodiment of the present invention, the oxygen saturation of the skin is estimated based on a feature amount relating to the melanin structure of the skin.

[0022] In a preferred embodiment of the present invention, the method further includes estimating a feature amount related to the blood vessel structure of the skin and / or a feature amount related to the melanin structure of the skin based on the oxygen saturation level of the front skin.

[0023] In a preferred embodiment of the present invention, the method further includes estimating an index of an aging state of the subject's skin based on the estimated value output by the model.

[0024] In a preferred embodiment of the present invention, the aging state is evaluated by inputting the estimated value into a correlation model created by providing a plurality of pairs of the feature amount and age, and calculating an estimated age.

[0025] The present invention also provides a skin evaluation program that uses multiple pairs of feature amounts obtained from non-invasively obtained data indicating the internal structure of a subject's skin and skin images obtained by photographing the biological surface of the same subject as learning data, and inputs the skin images into a model that has learned the relationship between the feature amounts and the skin images, thereby causing a computer to function to output estimated values ​​of the feature amounts. Effect of the Invention

[0026] According to the present invention, there is provided a method for evaluating skin that easily outputs an estimated value of a feature quantity related to an internal structure of skin based on a skin image. In particular, when measurement data obtained by a non-invasive method such as a photoacoustic method is used as learning data, estimation that reflects the results of learning based on the internal structure of living skin is possible. [Brief description of the drawings]

[0027] [Figure 1]Figure 1 shows the results of deep learning training to discriminate photoacoustic measurements using facial skin image data as input. (A) Training and validation accuracy. (B) Loss and data validation. [Diagram 2] (A) Examples of analysis of vascular structures of subjects with thinner (top) and thicker (bottom) average vascular structures, showing input (left), Grad-CAM (center), and Guided Grad-CAM (right) images, respectively. (B) Examples of analysis of oxygen saturation of subjects with high oxygen saturation (top) and low oxygen saturation (bottom) images, showing input (left), Grad-CAM (center), and Guided Grad-CAM (right) images, respectively. [Diagram 3] These images show an input image (left), a Grad-CAM image (center), and a Guided Grad-CAM image (right) as examples of the analysis of melanin tubular pattern structures in subjects with small (top row) and large (bottom row) melanin tubular pattern structures. [Figure 4] (A) A merged image of an image obtained from a 575 nm photoacoustic signal and an image obtained from a 650 nm ultrasound signal. (B) A diagram showing the vascular structure in the upper part of the skin of a subject in his 20s, generated from measurement data obtained by the photoacoustic method. (C) A diagram showing the vascular structure in the upper part of the skin of a subject in his 60s, generated from measurement data obtained by the photoacoustic method. [Diagram 5] (A) Measured image data, annotated image data, and predicted image data of vascular structure by deep learning, which were used as training data in deep learning. (B) Measured image data, annotated image data, and predicted image data of vascular structure by deep learning, which were not used as training data in deep learning. (C) A representative example of three-dimensional data generated by combining predicted image data (xy plane) in the z-axis direction. [Figure 6] FIG. 1 is a diagram showing the relationship between subject age and the average thickness of the vascular structure in each layer of the skin. [Figure 7](A) A diagram showing the spatial distribution of oxygen saturation in the measurement area of ​​a subject in his 20s (B) A diagram showing the spatial distribution of oxygen saturation in the measurement area of ​​a subject in his 60s (A) and (B) . [Figure 8] FIG. 1 is a diagram showing the relationship between the age of a subject and the oxygen saturation in each layer of the skin. [Figure 9] FIG. 1 shows the relationship between the average thickness of the vascular structure in each layer of the skin and the oxygen saturation in the corresponding layer. [Figure 10] (A) A diagram showing the distribution of melanin structures in the measurement area of ​​a subject in his 20s (B) A diagram showing the distribution of melanin structures in the measurement area of ​​a subject in his 60s. [Figure 11] (A) Measured image data, annotated image data, and predicted image data of melanin structure by deep learning, which were used as training data in deep learning. (B) Measured image data, annotated image data, and predicted image data of melanin structure by deep learning, which were not used as training data in deep learning. (C) A representative example of three-dimensional data generated by combining predicted image data (xy plane) in the z-axis direction. [Figure 12] FIG. 1 shows the relationship between subject age and melanin tubular patterning present in the upper part of the skin. [Figure 13] FIG. 1 shows the relationship between oxygen saturation in each layer of the skin and the volume of melanin tubular markings in the corresponding layer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] In the present invention, the term "skin" refers to biological tissue including the epidermis layer, the dermis layer, and the subcutaneous fat layer, and is used when focusing particularly on the structure of the tissue. In the present invention, the term "skin" is synonymous with "dermis" as a biological tissue, but is used without particular attention being paid to the structure of the tissue. Additionally, "evaluation" in this invention does not include medical diagnosis.

[0029] <1> How to evaluate your skin The skin evaluation method of the present invention uses multiple pairs of feature quantities obtained from data showing the internal structure of the subject's skin and skin images obtained by photographing the biological surface of the same subject as learning data, and inputs the skin images into a model that has learned the relationship between the feature quantities and the skin images, thereby outputting estimated values ​​of the feature quantities. Specifically, skin images obtained by photographing the subject's facial skin in vivo (not a collected sample) are used as input data, and a convolutional neural network model is created by performing deep learning using multiple feature quantities extracted from data showing the internal structure of the same subject's skin as objective variables. A skin image of a subject, whose features are to be estimated, is input to the model, and an estimated value of the features relating to the subject is output.

[0030] That is, in this embodiment, first, in creating a model, features including features relating to the blood vessel structure of the skin, features relating to the oxygen saturation of the skin, and features relating to the melanin structure of the skin are extracted based on data indicating the internal structure of the skin obtained by a non-invasive method such as a photoacoustic method. Then, skin images are taken of the same subject's body, and a set of the skin image and the feature amount is provided as learning data to create a model that outputs an estimate of the feature amount based on the skin image. The data and feature amount showing the internal structure of the skin will be described later.

[0031] Here, the inventors have demonstrated, by using the photoacoustic method, that there is a correlation between the average blood vessel thickness in a measurement area of ​​the skin and the oxygen saturation in the measurement area, and between the oxygen saturation in the measurement area of ​​the skin and the volume of the melanin streak structure in the measurement area (see Figures 9 and 13). Therefore, by utilizing this correlation, a part of the learning data may be created by mutually estimating the features of other skin structures. For example, a feature related to the blood vessel structure of the skin may be extracted from data showing the internal structure of the skin, and the oxygen saturation of the skin may be estimated based on the feature related to the blood vessel structure of the skin, and each of the features may be used as a feature in the learning data. Similarly, correlations may be used to estimate features of other skin structures from the estimated values ​​of the features output by the model.

[0032] In addition, in a preferred embodiment of the present invention, the model is analyzed from the output side to the input side based on the input skin image, and the contribution of the model to the estimation result is estimated for each region of the input skin image and displayed superimposed on the skin image. Preferred examples of analysis methods include the Grad-CAM method, the Guided BackPropagation method, and the Guided Grad-CAM method, which is a combination of these methods. These methods are known as methods for visualizing the area that a model focuses on in deep learning. Based on the input image, the method uses class-specific gradient information flowing into the final convolution layer and performs analysis by backpropagation. The Guided Grad-CAM method is a technology that extracts features of interest in deep learning for each pixel of the input image by tracing backward from the output to the input. This makes it possible to visualize how features related to the internal structure of the skin relate to expressions in a skin image.

[0033] <2> Acquisition of data and features showing the internal structure of the skin In this embodiment, the data showing the internal structure of the skin used to create the learning data is obtained by a non-invasive measurement method, but the present invention is not limited to this, and model learning may be performed based on invasively obtained measurement data. The data can be obtained using a known device and a known method as appropriate. As a known non-invasive measurement method, for example, a high-frequency current method, an ATR-FTIR method, a near-infrared spectroscopy, a magnetic resonance imaging method (MRI), an in vivo confocal Raman spectroscopy, an ultrasonic tomography, an optical coherence tomography, an in vivo confocal laser scanning microscope, a multiphoton microscope, an SHG microscope, a CARS microscope, an optical ultrasonic microscope, etc. can be selected. Furthermore, so long as it can be measured using the above-mentioned non-invasive measurement method, the type of data indicating the internal structure of the skin is not particularly limited, and examples include data regarding the vascular structure in a certain measurement area of ​​the subject's skin, data regarding blood flow, data regarding oxygen saturation, data regarding the microspatial distribution of melanin, data regarding moisture, data regarding the viscoelasticity of the skin, data regarding specific substances, etc. The obtained data showing the internal structure of the skin may be used as raw data, or may be subjected to general statistical processing (such as obtaining the average or standard deviation) or processing such as imaging. In a preferred embodiment of the present invention, a photoacoustic method is employed as the non-invasive measurement method, and the data indicating the internal structure of the skin is image data acquired by the photoacoustic method.

[0034] Here, any part can be selected as the measurement region for obtaining data showing the internal structure of the skin, but it is preferable to select a part of the skin that is easily exposed to light in daily life (for example, the face or the back of the hand). It is also preferable to select a part of the skin that is not easily exposed to light in daily life (for example, the inside of the upper arm). If the former is selected, the overall skin condition taking into account the effects of light can be evaluated, and if the latter is selected, the (physiological) skin condition can be evaluated while minimizing the effects of light. In addition, a normal area free of visible blemishes, pigmentation, wounds, or other abnormalities is set as the measurement area. Moreover, the size of the measurement region can be preferably set to 5 mm×5 mm to 15 mm×15 mm.

[0035] In the present invention, when image data is used as data showing the internal structure of the skin, the image data for extracting the feature amount is preferably three-dimensional data. Such three-dimensional data may be the measured image data itself obtained by a photoacoustic method or the like, or may be processed image data obtained by processing the measured image data.

[0036] When image data is used as data indicating the internal structure of the skin, the image data of the skin may be image data including all layers, namely the epidermis layer, dermis layer, and subcutaneous fat layer, or image data obtained by arbitrarily dividing this in the depth direction of the skin may be used. For example, image data arbitrarily divided in the depth direction of the skin can be image data in which the epidermis layer to the subcutaneous fat layer is divided into an upper portion (the region from the epidermis to the papillary dermis), a middle portion (the boundary region from the reticular dermis to the subcutaneous fat layer), and a lower portion (the upper layer of the subcutaneous fat layer).

[0037] The skin features obtained from data indicating the internal structure of the skin and used in creating the model of the present invention include, for example, features related to blood vessel structure, features related to blood flow, features related to oxygen saturation, features related to melanin structure, and features related to moisture. Among these, preferred forms include a feature amount relating to a blood vessel structure, a feature amount relating to oxygen saturation, and a feature amount relating to a melanin structure. Of these, when using the feature amount related to the blood vessel structure, it is preferable to use data including measurement data of the skin, and the data is preferably image data. When using features related to the melanin structure, it is preferable to use data including measurement data of the epidermis layer, and the data is preferably image data. When the feature amount related to the oxygen saturation is used, it is preferable to use data including measurement data of the epidermis layer or measurement data of the subcutaneous fat layer. The data is preferably image data.

[0038] In the evaluation method of the present invention, one or more of the feature amount relating to the blood vessel structure, the feature amount relating to the oxygen saturation, and the feature amount relating to the melanin structure may be used.

[0039] As the feature quantity related to the blood vessel structure, it is preferable to use the feature quantity related to the fine blood vessel structure, and in particular, it is preferable to use the feature quantity related to the fine blood vessel structure in the upper part of the skin. As such feature quantity, the number of blood vessel branches, the curvature of blood vessels, the total amount (volume) of blood vessels, the number of fine blood vessels, the density of blood vessels, the average thickness of blood vessels, etc. are preferably mentioned, and among them, it is preferable to use the number of blood vessel branches, the total amount (volume) of blood vessels, the number of fine blood vessels, and the average thickness of blood vessels.

[0040] The feature quantity relating to the blood vessel structure can be extracted from data relating to the blood vessel structure in the measurement region measured by the non-invasive measurement method. Image data for extracting the feature amount related to the blood vessel structure can be generated by, for example, irradiating a measurement site with pulsed light including two wavelengths of 575 nm and 650 nm, and using a difference value obtained by subtracting the photoacoustic signal of 650 nm from the photoacoustic signal of 575 nm, to generate three-dimensional image data showing the blood vessel structure inside the skin. For example, such three-dimensional image data (measurement image data) can be generated using HadatomoZ (Hadatomo is a registered trademark) by Advantest Corporation. Then, by analyzing the generated image, the feature amount related to the blood vessel structure can be extracted. Such measurement image data may be used to quantify features related to the blood vessel structure. In a preferred embodiment, the measurement image data is processed to improve the accuracy of extraction of features relating to the blood vessel structure.

[0041] For example, a method can be used to use a deep learning model to decompose measured image data into horizontal image data (planar image data), generate extracted planar image data by extracting elements that indicate vascular structure from this decomposed measured planar image data, and combine this extracted planar image data to generate extracted three-dimensional image data from which the vascular structure has been extracted. A deep learning model can be used to generate the extracted planar image data from the measured planar image data. In this case, a data curator visually evaluates the measurement planar image data to create annotated image data that extracts the vascular structure. The required number of measured planar image data and the corresponding annotated image data are used as training image data to perform deep learning. As a deep learning model, Unet, a network with an encoder-decoder model structure, can be used.

[0042] As the feature amount related to the oxygen saturation, the oxygen saturation of the measurement area (for example, any layer of the skin), the amount of the area where microvessels with high oxygen saturation are present, and the like can be used.

[0043] Image data for extracting the feature amount related to oxygen saturation can be obtained by, for example, irradiating a measurement site with pulsed light including two wavelengths of 575 nm and 586 nm, and using the difference in absorption coefficient between oxyhemoglobin and deoxyhemoglobin to generate a measurement image in which the ratio of oxyhemoglobin to the whole is visualized by color coding or the like. For example, such three-dimensional image data (measurement image data) can be generated using HadatomoZ (Hadatomo is a registered trademark) by Advantest Corporation. Then, by analyzing the generated image, the feature amount related to oxygen saturation can be extracted. Such measurement image data may be used to quantify the feature amount related to the oxygen saturation level. Furthermore, the oxygen saturation level at a certain position (region) may be calculated based on the amplitude ratio of the photoacoustic signals at each wavelength at that position (region) (see the examples).

[0044] The feature quantity related to the melanin structure is a feature quantity related to a structure formed by the aggregation of melanin granules. Here, in the present invention, the feature quantity related to the melanin structure does not mean a structure such as a melanin aggregation related to visible spots or freckles, but means a structure in a space of less than 1 mm square that is usually invisible to the naked eye. Photoacoustic methods generally cannot measure individual melanin granules. This structure has a size ranging from 10 to several hundreds of μm, and is preferably a tubular patina structure (see FIG. 10(B)). The feature amount relating to the melanin structure includes a feature amount relating to the amount of the structure present, such as volume, density, and number.

[0045] Image data for extracting features related to the melanin structure can be obtained, for example, by irradiating a measurement site with 650 nm pulsed light and generating a measurement image based on the obtained photoacoustic data. For example, such three-dimensional image data (measurement image data) can be generated using HadatomoZ (Hadatomo is a registered trademark) by Advantest Corporation. Then, by analyzing the generated image, features related to the melanin structure can be extracted. Such measured image data may be used to quantify features related to the melanin structure.

[0046] <3> Evaluation method (internal structure estimation) Furthermore, the inventors have demonstrated, by using the photoacoustic method, that there is a correlation between the average blood vessel thickness in a measurement area of ​​the skin and the oxygen saturation in the measurement area, and between the oxygen saturation in the measurement area of ​​the skin and the volume of the melanin streak structure in the measurement area (see Figures 9 and 13). Therefore, the present invention provides a skin evaluation method that includes estimating features related to each other's skin structures by analyzing at least one of features related to the blood vessel structure of the skin, features related to the oxygen saturation of the skin, and features related to the melanin structure in the skin.

[0047] Specifically, the higher the average thickness of blood vessels in a measurement area of ​​the skin, the lower the oxygen saturation of the blood vessels in that measurement area (see FIG. 9). This relationship is particularly noticeable in the upper and lower parts of the skin. Therefore, by measuring or estimating the average thickness of blood vessels in a measurement area of ​​the skin, the oxygen saturation of the blood vessels in that measurement area can be estimated. As described above, the oxygen saturation in the measurement area of ​​the skin decreases with age, so the oxygen saturation of the blood vessels in the measurement area of ​​the skin can be estimated based on the measurement data of the average thickness of the blood vessels in the measurement area of ​​the skin, thereby making it possible to evaluate the skin (particularly the aging state of the skin). The evaluation of the aging state will be described later.

[0048] Furthermore, based on the relationship between the average thickness of blood vessels in the measurement area and the oxygen saturation in the measurement area, the average thickness of blood vessels in the measurement area can be estimated by measuring or estimating the oxygen saturation in the measurement area of ​​the skin. As described above, the average thickness of blood vessels in a measurement area of ​​the skin increases with age, so the skin (in particular, the aging state of the skin) can be evaluated by estimating the average thickness of blood vessels in the measurement area of ​​the skin based on the measurement data of oxygen saturation in that measurement area.

[0049] In addition, there is a correlation that the higher the oxygen saturation in the measurement area of ​​the skin, the smaller the volume of the melanin streak structure present in the measurement area (see FIG. 13). Therefore, by measuring or estimating the oxygen saturation in a measurement area of ​​the skin, it is possible to estimate the volume of the melanin markings observed in said measurement area. As described above, the volume of the melanin striated structure observed in a measurement area of ​​the skin increases with age, so the skin (particularly the aging state of the skin) can be evaluated by estimating the volume of the melanin striated structure observed in the measurement area of ​​the skin based on the measurement data of oxygen saturation in that measurement area.

[0050] Furthermore, based on the relationship between the oxygen saturation in the measurement area and the volume of the melanin streak structure present in the measurement area, the oxygen saturation in the measurement area can be estimated by measuring the volume of the melanin streak structure observed in the measurement area of ​​the skin. As described above, the oxygen saturation in the measurement area of ​​the skin decreases with age, so the skin (particularly the aging state of the skin) can be evaluated by estimating the oxygen saturation in the measurement area based on the measurement data of the volume of the melanin pattern structure observed in the measurement area of ​​the skin.

[0051] In this way, by mutually estimating feature quantities related to the internal structure of the skin, in the creation of learning data for a model that estimates feature quantities from a skin image in the present invention, learning data related to one feature quantity can be created from another feature quantity. Also, after a feature quantity is estimated from a skin image using a model, another feature quantity can be further estimated. A preferred embodiment of each feature quantity and its extraction method is as follows: <2> As stated above.

[0052] <4> Assessment of aging status The skin evaluation method of the present invention may further include a step of evaluating the aging state of the subject's skin based on estimated values ​​of features related to the internal structure of the subject's skin output by the model. In a preferred embodiment of the present invention, the aging state is evaluated by inputting the estimated value into a correlation model created by providing a plurality of pairs of the feature amount and age, and calculating an estimated age. According to the present invention, since the aging state of the skin can be evaluated from an estimated value based on a skin image, the aging state of the skin of a subject can be evaluated simply and non-invasively.

[0053] In the present invention, examples of "skin aging state" include the degree (e.g., size, number, depth, etc.) of skin aging traits (e.g., spots, wrinkles, dullness, etc.) visible to a third party, prediction of the occurrence of aging traits that may appear in the future, the subject's impression of aging (apparent age), etc.

[0054] As shown in the examples, the inventors have clarified that each of the above-mentioned features has a relationship with age. Therefore, by utilizing this relationship, it is possible to evaluate the aging state of the skin based on the features.

[0055] In this embodiment, the aging state is evaluated from features estimated based on skin images using a trained convolutional neural network model. In creating learning data used for model training, the method of extracting features from data showing the internal structure of the skin is not particularly limited, and may be a form in which features are extracted by applying general statistical processing (e.g., calculation of average, standard deviation, etc.) to data showing the internal structure of the skin obtained by a non-invasive measurement method, or a form in which features are extracted by analyzing the image after processing such as imaging.

[0056] In the present invention, in the evaluation of the aging state of the skin, it is preferable to create a correlation model showing the correlation between the feature amount and age in advance from the viewpoint of simplicity. In this way, the feature amount estimated from the skin image can be inputted into the correlation model to calculate the estimated age. It should be noted that, in addition to age, any index related to any aging state having a correlation with the feature amount can be used. The analysis of correlations and the creation of correlation models are described below.

[0057] The feature amount (evaluation index) used for evaluating the aging state in the present invention can be determined as follows. First, a non-invasive measurement method is used to measure the measurement area of ​​the skin of multiple subjects of different ages to obtain data on the internal structure of the skin. The data on the internal structure of the skin is preferably image data or the obtained data is visualized. When the data on the internal structure of the skin is image data, the image may be a two-dimensional image such as a cross-sectional image of the skin, or a three-dimensional image.

[0058] As a non-invasive measurement method, the various devices and methods described above can be used, but it is preferable to employ a photoacoustic method. By adopting the photoacoustic method, it is possible to more accurately measure structures deep inside the body that could not be measured using conventional methods. For example, when obtaining image data of the internal structure of the skin, pulsed light is irradiated onto the measurement area of ​​the skin of multiple subjects of different ages, the generated ultrasound is measured, and image data of the internal structure of the skin is generated for each subject of each age based on the measured ultrasound data inside the skin.

[0059] Next, parameters contained in the obtained measurement data are extracted, and parameters correlated with actual age or apparent age are searched for. The method of extracting parameters from the measurement data is not particularly limited, and known analysis methods can be used. For example, when image data is used as the measurement data, a method of visually comparing images by age group and extracting parameters of skin that shows changes can be adopted. In addition, parameters can be extracted by extracting features including color distribution, luminance distribution, spatial frequency intensity, etc. through analysis using general-purpose image processing software.

[0060] Here, when determining the features (evaluation index) to be used in evaluating the aging state, features estimated from a skin image using a model may be used instead of parameters (features) extracted from data indicating the internal structure of the skin measured by a non-invasive method.

[0061] Next, the correlation between the extracted parameters and an index indicating an aging state such as apparent age or actual age is analyzed. In this analysis, a feature amount estimated from a skin image by a model may be used, or a feature amount obtained from data indicating the internal structure of the skin measured by a non-invasive method may be used. Note that, as the apparent age, a value obtained by appropriately converting age into a generation, score, or the like may be used. As the analysis method, a method generally used for analyzing correlation, such as regression analysis, can be adopted.

[0062] In addition, when image data is used as the measurement data, correlations can also be analyzed by machine learning using training data that is image data showing the internal structure of the skin obtained by a non-invasive measurement method, labeled with data showing the degree of aging, such as apparent age or actual age.

[0063] As a result of the analysis, the feature quantities that are determined to be correlated with age are determined as skin evaluation indices, and a correlation model that explains the correlation between the feature quantities and age is created. Examples of correlation models include multiple regression equations, discriminant equations, and neural networks.

[0064] The inventors' analysis specifically found a relationship in which the older the actual age of the subject, the thicker the average thickness of the blood vessels at the measurement site (the thicker the blood vessels) (see FIG. 6). They also found that the older the person is, the smaller the volume of blood vessels is, and the older the person is, the fewer the number of blood vessel branches in the upper part of the skin is. Additionally, it was found that the higher the actual age, the lower the oxygen saturation in the measurement area, and this relationship was particularly strong in the upper and lower areas of the skin (see Figure 8).

[0065] Furthermore, a correlation was found between higher actual age and more melanin streak structures (larger volume) observed per unit skin area (see FIG. 12).

[0066] Therefore, these feature amounts can be particularly preferably used as feature amounts (evaluation indexes) for use in the method for evaluating the aging state in the present invention.

[0067] In the present invention, in addition to the above feature amounts, items known as indices for evaluating the aging state of the skin may also be measured and taken into consideration when evaluating the aging state of the skin. For example, the moisture content, viscoelasticity, and degree of aging characteristics of the skin can be non-invasively measured using known methods, and these measurement results can be combined with at least any of the measurement results of the above-mentioned features using a photoacoustic method to evaluate the aging state of the subject's skin.

[0068] A method of evaluating skin from features is preferably used, for example, by storing in advance in a computer an equation or model showing the correlation between features and estimated values ​​of the aging state (apparent age or score) based on the above-mentioned correlation analysis, inputting the features into the equation or model, and outputting an estimated value of the aging state. In the present invention, by using features estimated based on a skin image as the features input here, the aging state can be estimated from the skin image through the estimation of the features. EXAMPLES

[0069] Photoacoustic measurement image data was obtained from normal cheek areas (9mm x 9mm) of subjects (80 Japanese women in their 20s to 60s), excluding areas where blemishes or red spots were visible to the naked eye, using the following method, and various features were extracted. Furthermore, an image (1200px x 1200px) of the area including the photoacoustic measurement area was extracted from the facial image of the same subject, and an enlarged skin image (640px x 480px) of the cheek area was obtained using a 30x video microscope, and deep learning was performed using the skin image and features as learning data. In addition, the correlation between the extracted features and age was analyzed to analyze features related to the aging state.

[0070] <1> Acquiring photoacoustic measurement images For the photoacoustic measurements, we used a HadatomoZ photoacoustic measurement device manufactured by Advantest Corporation. This device is capable of dual-wavelength photoacoustic imaging and ultrasonic imaging. The measurements were performed with a scanning step of 30 μm.

[0071] Two dual wavelength lasers were used for the measurements: (i) 575 nm and 586 nm, and (ii) 575 nm and 650 nm. For vascular imaging, the 575 nm photoacoustic signal minus the 650 nm photoacoustic signal was used. For imaging of oxygen saturation distribution, the difference in absorption coefficient between oxyhemoglobin and deoxyhemoglobin obtained from photoacoustic signals at 575 nm and 586 nm was used. For imaging of melanin distribution, a photoacoustic signal of 650 nm was used.

[0072] <2> Image generation for analysis of vascular and melanin structures To extract features related to vascular and melanin structures, analysis image data was generated based on image data obtained by photoacoustic measurements using MATLAB (registered trademark) 2020a (MathWorks, USA), Python 3.7, and Tensorflow. Three-dimensional image data obtained by photoacoustic measurements was reconstructed into horizontal (x-y plane) images from images stacked on the z-axis. To extract vascular and melanin structures from the reconstructed images, Unet, a network with an encoder-decoder model structure, was used as a deep learning segmentation method. Images of the measurement area where vascular structures and melanin structures were observed were cut out from the xy plane image at multiple arbitrary positions on the z axis, and extracted images were created as annotated image data (training data) for areas visually regarded as vascular structures or melanin aggregate structures by the data curator. These extracted images were used as output data in deep learning and their class labels. These extracted images were 90 and 30 examples for vascular structures and melanin structures, respectively. Next, a network with 5 iterations and 2 × 2 pooling was constructed to extract vascular and melanin structures using the original xy plane images as input data. The objective function was assumed to be a Dice function calculated by the following equation (1).

number

[0073] Using the constructed network, xy planar image data in which vascular structures and melanin structures were extracted from the photoacoustic data acquired from the subject was extracted, and this extracted planar image data was reintegrated in the depth direction to create three-dimensional data.

[0074] Based on the three-dimensional data created, parameters such as length, thickness, number of branches, and volume were measured. Various parameters were measured for each of the three equal parts in the depth direction of the skin: upper, middle, and lower.

[0075] <3> Oxygen Saturation Calculation Spatial distribution of oxygen saturation SO 2 (r) is oxygenated hemoglobin (HbO) as shown in formula (2). 2 It can be calculated based on the ratio of hemoglobin to deoxygenated hemoglobin Hb.

number

[0076] <4> Creation of a feature estimation model based on skin images Facial images were acquired using VISIA (Canfield, USA). From the facial images, an image (1200px x 1200px) of the area including the photoacoustic measurement area was extracted, and from there, an enlarged skin image (640px x 480px) of the cheek area, which was the measurement area, was acquired using a 30x video microscope.

[0077] Deep learning was performed using MATLAB (registered trademark) 2020a and a deep learning toolbox. First, a top group of 20 people and a bottom group of 20 people were extracted from 80 Japanese female subjects in their 20s to 60s based on the measured values ​​of the features. Then, images of the area including the measurement area of ​​the features and enlarged skin images extracted from the face images of a total of 40 subjects in the top and bottom groups were used as input data, and a convolutional neural network model was trained to determine whether the measured values ​​were in the top or bottom group. Here, the volume of the vascular structure, oxygen saturation, and the volume of the tubular pattern structure of melanin obtained by the photoacoustic method were adopted as the feature values, and a neural network model was created to distinguish the top group and the bottom group from the image of the area including the measurement area by the photoacoustic method and the enlarged skin image for each of them. Furthermore, we used GoogLeNet, a pre-trained model for transfer learning, to mitigate the impact of small sample sizes. Furthermore, through repeated training, the prediction accuracy improved (to over 95%). It was also verified that the accuracy was about 75% when using validation data that was not used for learning. Figure 1 shows a representative example of the results of deep learning on the relationship between melanin structure and skin images.

[0078] <5> Analysis of the relationship between skin images and features related to the internal structure of the skin In order to visualize what image features affect classification in deep learning, the inventors performed visualization using the Grad-CAM method.

[0079] The relationship between vascular structures and facial and skin features is shown in Figure 2(A). In subjects with thinner average vascular structures, the heat map shown in the Grad-CAM image tended to expand uniformly (upper image in Figure 2(A)), whereas in subjects with thicker average vascular structures, the heat map became localized (lower image in Figure 2(A)). A heterogeneous distribution of cloud-like features was observed in subjects with thicker average vascular structures, as shown by Guided Grad-CAM images, whereas such features were not observed in subjects with fine vascular structures. Furthermore, in cases where the average thickness of the vascular structures was thicker, the presence of color shading and unevenness was found to be perceived in the facial image (input image), which was consistent with the areas shown in the Guided Grad-CAM image. It is suggested that the thickness of the vascular structures observed in this study may lead to uneven coloring on the face, which is perceived as sagging skin. The arrows in the Guided Grad-CAM image of the subject with the thicker average thickness of the vascular structures indicate an uneven distribution of cloud-like features, which is consistent with the presence of color shading and unevenness in the input skin image.

[0080] Next, we investigated the relationship between oxygen saturation and appearance (Figure 2(B)). The heat maps shown in the Grad-CAM images of subjects with high oxygen saturation tended to be broad and uniform (top image in Figure 2(B)). In contrast, the heat maps in the Grad-CAM images of subjects with low oxygen saturation tended to be separated (bottom image in Figure 2(B)). Furthermore, as shown in the Guided Grad-CAM images, the distribution of features is unevenly distributed in the shape of a cloud in subjects with low oxygen saturation, whereas it is uniformly distributed in subjects with high oxygen saturation. This suggests that the oxygen saturation obtained in this study leads to relatively macroscopic color unevenness in the face. The arrows in the Guided Grad-CAM images indicate the uneven distribution of features in the shape of a cloud in subjects with low oxygen saturation, which is consistent with the presence of color shading and nonuniformity in the input images.

[0081] We also investigated the relationship between the melanin tubular pattern structure and facial skin images. We found that the features were widely distributed in a cloud-like manner, as shown in the Guided Grad-CAM image of a subject with a large amount of melanin structures (lower image in Figure 3). In addition, color unevenness was observed in the area where the input image had features, suggesting that the melanin tubular pattern structure found here is expressed as the viewpoint of color unevenness in the skin image taken by the skin camera (arrow in the figure). In this way, we demonstrated the possibility that the presence of the melanin tubular pattern structure is involved in the generation of uneven color spots on the skin.

[0082] <6> statistical analysis The various parameters obtained above and the age of the subjects were analyzed using JMP software version 14.0 (SAS Institute, USA). Correlations between each data were evaluated by Pearson's correlation analysis.

[0083] <7> Analysis of vascular structure Figure 4(A) shows a merged image of the 575 nm photoacoustic signal and the 650 nm ultrasound signal, which shows that the photoacoustic measurement can identify vascular structures from the upper to lower depths of the subcutaneous tissue.

[0084] Figure 4 (B) and (C) are representative measurement images showing vascular structures extracted based on the difference value obtained by subtracting the photoacoustic signal at 650 nm from the photoacoustic signal at 575 nm. (B) is an image of a subject in his 20s, and (C) is an image of a subject in his 60s. In particular, as is clear from the top view, Figure 4 shows that the area containing the microvascular structures seen in the subject in his 20s (blood vessels indicated by white triangles in Figure 4(B)) was not seen in the subject in his 60s (area indicated by white arrows in Figure 4(C)).

[0085] Figure 5 shows a representative example of measured image data in the xy plane including vascular structures obtained by photoacoustic measurement, and predicted image data of the vascular structures obtained by inputting the measured image data into the network constructed using the above-mentioned deep learning. Annotated image data of the vascular structures visually extracted by a data curator from the measured image data is also shown. (A) is a predicted image obtained from measured image data used as training data in deep learning, and (B) is a predicted image obtained from measured image data that was not used as training data in deep learning. This confirmed that it is possible to generate predicted image data with a high similarity to the correct data created by the analyst even from measured image data that has not been used as learning data.

[0086] (C) is a representative example of three-dimensional data generated by combining predicted image data (xy plane) in the z-axis direction. As a result of correlation analysis between various parameters extracted from this three-dimensional data and the subjects' ages, correlations between some parameters and ages were confirmed.

[0087] As shown in FIG. 6, the average thickness of the vascular structure in each layer of the skin was observed to increase with age. It was also observed that with increasing age, the volume of the vasculature in the upper part decreased and the volume of the vasculature in the lower part increased (data not shown). It was also observed that the number of branches decreased with increasing age (data not shown). In other words, with age, the number of microvessels in the upper part of the skin decreases and the volume of the entire vascular structure also decreases, whereas in the lower part of the skin, the number of microvessels decreases but the volume of the entire vascular structure does not decrease, suggesting that the changes in the vascular structure that occur with age differ depending on the layer of the skin (upper, middle, and lower).

[0088] From the above, it became clear that the feature value indicating the amount of microvessels can be used as an index for evaluating the state of aging. It was also revealed that the average thickness of the vascular structure and the number of branches can be used as the feature quantities. It was also revealed that the volume of the vascular structure in the upper part of the skin can be used as an index for evaluating the state of aging. It was also revealed that any combination of the average thickness, number of branches, and volume of the vascular structure in the upper part of the skin can be used as an index for evaluating the aging state.

[0089] <8> Oxygen saturation analysis The photoacoustic measurement image data of subjects in their 20s and 60s are shown in Figure 7. It became clear that photoacoustic imaging can visualize the spatial distribution of oxygen saturation.

[0090] In FIG. 7, (A) is measurement image data showing the spatial distribution of oxygen saturation in blood vessels of a subject in his 20s, and (B) is measurement image data showing the spatial distribution of oxygen saturation in blood vessels of a subject in his 60s. In Fig. 7, comparing (A) and (B), the oxygen saturation in the measurement area of ​​the subject in his 60s was generally lower than the oxygen saturation of blood vessels in the measurement area of ​​the subject in his 20s. In addition, in the measurement image data of the subject in his 20s, an area where microvessels exist with high oxygen saturation (indicated by a triangle in (A)) was observed, but in the measurement image data of the subject in his 60s, this area did not exist (indicated by an arrow in (B)).

[0091] For statistical analysis, oxygen saturation was calculated using the method described above, and the correlation with the previously measured vascular structure features and age was analyzed.

[0092] As shown in FIG. 8, it was revealed that oxygen saturation in each layer of the skin tends to decrease with age. In other words, it became clear that oxygen saturation can be used as an indicator for evaluating the aging state of the skin.

[0093] In addition, as shown in Figure 9, a negative correlation was found between the average blood vessel thickness and oxygen saturation in the upper and lower parts of the skin.

[0094] Therefore, it is possible to estimate the oxygen saturation level from the average blood vessel thickness, and conversely, it is possible to estimate the average blood vessel thickness by measuring the oxygen saturation level. The analysis also suggests that age-related loss of microvasculature induces a decrease in oxygen saturation, at least in the upper and lower layers of the skin.

[0095] From the above, it became clear that the feature quantity indicating the distribution of oxygen saturation in the measurement area can be used as an index for evaluating the aging state. It was also revealed that the oxygen saturation and the volume of microvascular regions with high oxygen saturation can be used as the feature quantities. In particular, because oxygen saturation in the upper and lower layers is also related to the average thickness of blood vessels, which is related to the aging state, it was suggested that a more accurate evaluation could be made by combining the oxygen saturation in these layers with features related to the vascular structure.

[0096] <9> Analysis of melanin structure FIG. 10 shows the measurement image data acquired by the photoacoustic method using pulsed light with a wavelength of 650 nm.

[0097] In FIG. 10, (A) is measurement image data showing the distribution of melanin structures of a subject in his 20s, and (B) is measurement image data showing the distribution of melanin structures of a subject in his 60s. As shown in (A), melanin structures were less abundant in the skin of subjects in their 20s. In contrast, as shown in (B), a tubular striated structure (indicated by an arrow) with a toroidal cavity was observed in the skin of a subject in his 60s.

[0098] Figure 11 shows a representative example of measured image data in the xy plane including melanin structures obtained by photoacoustic measurement using a network constructed using the above-mentioned deep learning, and predicted image data of the melanin structures obtained by inputting the measured image data into the network. Annotated image data of the melanin structures visually extracted by a data curator from the measured image data is also shown. (A) is predicted image data obtained from measured image data used as training data in deep learning, and (B) is predicted image data obtained from measured image data not used as training data in deep learning. This confirmed that it is possible to generate predicted image data with a high similarity to the correct data created by the analyst even from measured image data that has not been used as learning data.

[0099] (C) is a representative example of three-dimensional data generated by combining predicted image data (xy plane) in the z-axis direction. Correlation analysis of various parameters extracted from this three-dimensional data and the subjects' ages confirmed a correlation between the volume of the melanin tubular pattern structure and age.

[0100] As shown in FIG. 12, it was revealed that the volume of the melanin structure in the tubular pattern of melanin present in the surface layer of the skin increases with aging. The three-dimensional distribution of melanin in normal skin has not been studied previously, and there have been no reports on the melanin structure on the scale of 10 to several hundred μm demonstrated in this study. This newly discovered feature value related to the characteristic melanin structure has been shown to be useful for evaluating the state of aging.

[0101] From the above, it became clear that the volume of melanin in the surface layer, in particular the volume of the tubular melanin pattern, can be used as an evaluation index to evaluate the state of aging.

[0102] We also analyzed the correlation between the above-mentioned oxygen saturation and the volume of the tubular pattern structure of melanin. As shown in FIG. 13, it was found that the volume of melanin tubular striations increased with decreasing oxygen saturation. Therefore, the spatial distribution of oxygen saturation can be estimated based on the volume of the melanin tubular structures. Conversely, the volume of melanin tubular structures can be estimated based on oxygen saturation. [Industrial Applicability]

[0103] The present invention can be used in a method for evaluating skin in relation to cosmetics.

Claims

1. a step of inputting a plurality of pairs of feature amounts obtained from data showing an internal structure of the skin of a subject and skin images obtained by photographing a biological surface of the same subject as learning data into a model that has learned a relationship between the feature amounts and the skin images, thereby outputting an estimate of the feature amounts; A skin evaluation method, wherein the feature is a feature relating to a tubular pattern structure of melanin in the skin.

2. a step of inputting a plurality of pairs of feature amounts obtained from data showing an internal structure of the skin of a subject and skin images obtained by photographing a biological surface of the same subject as learning data into a model that has learned a relationship between the feature amounts and the skin images, thereby outputting an estimate of the feature amounts; A skin evaluation method, wherein the feature is a feature related to the vascular structure of the skin selected from an average thickness of blood vessels, a number of blood vessel branches, a volume of blood vessels in the upper part of the skin, and a curvature of blood vessels.

3. a step of inputting a plurality of pairs of feature amounts obtained from data showing an internal structure of the skin of a subject and skin images obtained by photographing a biological surface of the same subject as learning data into a model that has learned a relationship between the feature amounts and the skin images, thereby outputting an estimate of the feature amounts; A method for evaluating skin, wherein the feature amount is the amount of area in the skin where microvessels with high oxygen saturation are present.

4. The skin evaluation method according to claim 1 , wherein the feature amount is a volume of a tubular melanin pattern structure.

5. The method for evaluating skin according to claim 1 or 4, further comprising a step of estimating an oxygen saturation level of the skin based on an estimated value of the feature amount indicating a volume of a tubular pattern structure of melanin in the skin.

6. The skin evaluation method according to claim 2 , further comprising a step of estimating an oxygen saturation level of the subject's skin based on the estimated value of the feature amount indicating an average thickness of blood vessels in the skin.

7. The method for evaluating skin according to any one of claims 1 to 6, wherein the data indicating the internal structure is non-invasively obtained data.

8. The skin evaluation method according to any one of claims 1 to 7, wherein the model is a convolutional neural network model created by performing deep learning using the skin image as input data and the feature amount as a response variable.

9. The skin evaluation method according to claim 8, further comprising estimating the contribution of each region of the input skin image to the estimation result by the model by analyzing the input skin image from the output side to the input side of the model based on the input skin image.

10. The skin evaluation method according to claim 9 , further comprising: displaying, on the skin image, a contribution of each region of the skin image to the estimation result, superimposed thereon.

11. The skin evaluation method according to any one of claims 1 to 9, wherein the data indicating the internal structure is measurement data obtained by a photoacoustic method or data generated based on the measurement data.

12. The skin evaluation method according to any one of claims 1 to 11, further comprising estimating an index of the subject's skin aging state based on the estimated value output by the model.

13. The skin evaluation method according to claim 12 , wherein the aging state is evaluated by inputting the estimated value into a correlation model created by providing a plurality of pairs of the feature amount and age, and calculating an estimated age.

14. 1. A skin assessment program, comprising: a computer is caused to function so as to output an estimate of the feature amount by inputting the skin image into a model that has learned the relationship between the feature amount and the skin image, the model using a plurality of pairs of feature amounts obtained from data showing the internal structure of the skin of the subject and skin images obtained by photographing the biological surface of the same subject as learning data; A skin evaluation program, wherein the feature is a feature relating to a tubular pattern structure of melanin in the skin.

15. 1. A skin assessment program, comprising: a computer is caused to function so as to output an estimate of the feature amount by inputting the skin image into a model that has learned the relationship between the feature amount and the skin image, the model using a plurality of pairs of feature amounts obtained from data showing the internal structure of the skin of the subject and skin images obtained by photographing the biological surface of the same subject as learning data; A skin evaluation program, wherein the feature is a feature related to the vascular structure of the skin selected from an average thickness of blood vessels, a number of blood vessel branches, a volume of blood vessels in the upper part of the skin, and a curvature of blood vessels.

16. 1. A skin assessment program, comprising: a computer is caused to function so as to output an estimate of the feature amount by inputting the skin image into a model that has learned the relationship between the feature amount and the skin image, the model using a plurality of pairs of feature amounts obtained from data showing the internal structure of the skin of the subject and skin images obtained by photographing the biological surface of the same subject as learning data; A skin evaluation program, wherein the feature amount is the amount of area in the skin where microvessels with high oxygen saturation are present.

Images