Skin image analysis method

Abstract

To provide a skin image analysis technique to acquire an index value of a state or an attribute of skin for every small section of the skin.SOLUTION: In a skin image analysis method, one or more processors capable of using a learned discrimination model that is machine-learned on the basis of a plurality of pieces of teaching data including a plurality of combinations of correct answer information indicating a common attribute or a state of the skin to be photographed in a skin image for teaching and a plurality of skin patch image groups for teaching in a predetermined image size extracted from the skin image for teaching, acquire a skin image for evaluation, acquire the skin patch image groups in a predetermined image size from the acquired skin image for evaluation, perform normalization of a pixel value for each of the acquired skin patch images, and input the normalized skin patch images to the discrimination model to acquire index values of a skin attribute or a skin state related to the skin patch images.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a skin image analysis technique that analyzes a human skin image using a discriminant model obtained by machine learning. [Background technology]

[0002] The following Patent Document 1 discloses a method for detecting a face region from a person's image, extracting image features (gradient histograms) from areas within the detected face region where wrinkles are likely to occur, and using the features and a learning device (SVM (Support Vector Machine)) to identify classifications of person attributes (age and gender). The following Patent Document 2 discloses a method for using a convolutional neural network (CNN) to analyze an image of a user to predict the user's age and recommend products to be applied to areas of the skin to achieve a target skin age. The following Patent Document 3 discloses a method for determining and displaying the apparent skin age by performing a process of identifying the position of a face and masking macroscopic features (eyes, nose, mouth, etc.) on a digital image containing a human face, and then analyzing the image using a convolutional neural network (CNN). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-58060 [Patent Document 2] Special Publication No. 2019-512797 [Patent Document 3] Special Publication No. 2020-522810 Summary of the Invention [Problem to be solved by the invention]

[0004] As in the above-described method, when the entire face image of a user or an image of a region to be evaluated is input into a neural network or a learning device, the skin attributes and skin age for the entire input image are predicted. However, the condition of the skin may vary locally. For example, when applying cosmetics, it is difficult to apply them evenly to the entire face, and even if they are applied evenly, the appearance may vary in parts. From this perspective, there is still room for improvement in each of the above-mentioned methods, in that they mainly only evaluate the entire skin region that is the subject of evaluation. The present invention provides a skin image analysis technique that obtains skin condition or attribute indices for each skin patch. [Means for solving the problem]

[0005] According to the present invention, a skin image analysis method can be provided in which one or more processors that can use a trained discrimination model that has been machine-learned based on multiple training data including multiple combinations of correct answer information indicating common attributes or conditions of the subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image perform the following steps: acquiring an evaluation skin image showing the skin of a subject; acquiring a group of skin patch images of a predetermined image size from the acquired evaluation skin image; normalizing the pixel values ​​of each of the acquired skin patch images; and inputting each normalized skin patch image into the discrimination model to acquire an index value for the skin attribute or skin condition for each skin patch image. Furthermore, according to the present invention, there can be provided a method for training a discriminant model that discriminates the skin attributes or skin condition of an input skin patch image of a predetermined image size, the method including the steps of: acquiring a plurality of combinations of teacher skin images and correct answer information indicating common attributes or conditions of the subject skin of the teacher skin images; acquiring a group of teacher skin patch images of a predetermined image size from the acquired teacher skin images, each of which contains a skin area at a predetermined rate or more; normalizing the pixel values ​​of each of the acquired teacher skin patch images; generating a group of teacher data that associates the correct answer information corresponding to the original teacher skin image from which each teacher skin patch image was acquired with each of the normalized teacher skin patch images; and training the discriminant model using the group of teacher data. In addition, a skin image analysis device that is equipped with at least one or more of the above-mentioned processors and memories and is capable of executing the above-mentioned skin image analysis method, and a discriminant model learning device that is capable of executing the above-mentioned discriminant model learning method may also be provided. [Effects of the Invention]

[0006] According to the above aspect, it is possible to provide a skin image analysis technique that acquires index values ​​of skin conditions or attributes for each small section of skin. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device capable of executing a skin image analysis method and a discriminant model learning method according to an embodiment of the present invention. [Figure 2] 4 is a conceptual diagram illustrating a processing flow of a skin image analysis method according to the present embodiment. [Figure 3] 1 is a flowchart of a skin image analysis method according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a color map image as index value distribution information of a group of skin patch images. [Figure 5] FIG. 2 is a conceptual diagram illustrating a processing flow of a discriminant model learning method according to the present embodiment. [Figure 6]10 is a flowchart of a method for learning a discriminant model according to the present embodiment. [Figure 7] 10 is a graph showing the relationship between the size of a skin region and the discrimination accuracy of a discriminant model. [Figure 8] 10 is a graph showing the relationship between the makeup feel score obtained by human visual evaluation and the makeup feel score obtained by the skin image analysis method according to the present embodiment. [Figure 9] 10 is a graph showing representative index values ​​of makeup feel obtained by the skin image analysis method according to the present embodiment for each of eight types of skin images for evaluation that differ in the manner in which cosmetics are applied. [Figure 10] FIG. 10 is a diagram showing a color map image showing the index value distribution of a group of skin patch images for each of eight types of skin images for evaluation that have different cosmetic application patterns. [Figure 11] 10 is a graph showing the relationship between estimated age ranges and actual age ranges obtained by the skin image analysis method according to the present embodiment. [Figure 12] FIG. 10 is a diagram showing the relationship between the evaluation skin images of five subjects and a color map image showing the index value distribution of a group of skin patch images. [Figure 13] FIG. 10 is a diagram showing color map images of skin images for evaluation taken immediately after application of two types of foundation cosmetics and 4 hours, 8 hours, and 10 hours after application. [Figure 14] 10 is a graph showing the relationship between the number of epochs and the discrimination accuracy of the discriminant model in the discriminant model learning method according to the present embodiment. [Figure 15] This is a graph comparing the estimation accuracy of a trained model that estimates a visual skin assessment score by inputting a face image and a trained model that estimates a visual skin assessment score by inputting a skin patch image. [Figure 16] 10 is a radar chart showing the correlation between the estimated visual evaluation score obtained from the skin patch type discrimination model in Example 8 and age, and the change in the estimated visual evaluation score following the application of a cosmetic material. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, examples of preferred embodiments of the present invention (hereinafter referred to as the present embodiments) will be described. The embodiments given below are merely examples, and the present invention is not limited to the configurations of the following embodiments.

[0009] The skin image analysis method (hereinafter referred to as the present analysis method) and the discriminant model learning method (hereinafter referred to as the present learning method) according to this embodiment are executed by one or more processors included in one or more information processing devices. FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device 10 that can execute the present analysis method and the present learning method. The information processing device 10 is a so-called computer, and includes a CPU 11, a memory 12, an input / output interface (I / F) 13, a communication unit 14, etc. The information processing device 10 may be a desktop PC (Personal Computer), a portable terminal such as a portable PC, a smartphone, or a tablet, or a dedicated computer.

[0010] The CPU 11 is a so-called processor, and may include not only a general CPU (Central Processing Unit) but also an application specific integrated circuit (ASIC), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), etc. The memory 12 is a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device (such as a hard disk), etc. The input / output I / F 13 can be connected to user interface devices such as a display device 15 and an input device 16. The display device 15 is a device, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display, that displays a screen corresponding to drawing data processed by the CPU 11 or the like. The input device 16 is a device, such as a keyboard or a mouse, that accepts input of user operations. The display device 15 and the input device 16 may be integrated and realized as a touch panel. The communication unit 14 communicates with other computers via a communication network and exchanges signals with other devices such as a printer, etc. The communication unit 14 can also be connected to a portable recording medium, etc.

[0011] The hardware configuration of the information processing device 10 is not limited to the example in FIG. 1. The information processing device 10 may include other hardware elements not shown. Furthermore, the number of each hardware element is not limited to the example in FIG. 1. For example, the information processing device 10 may have multiple CPUs 11. Furthermore, the information processing device 10 may be realized by multiple computers each consisting of multiple housings.

[0012] The information processing device 10 can execute the present analysis method and the present learning method by having the CPU 11 execute a computer program stored in the memory 12. This computer program is installed from a portable recording medium such as a CD (Compact Disc) or a memory card, or from another computer on a network, via the input / output I / F 13 or the communication unit 14, and stored in the memory 12. For convenience of explanation, this specification provides an example in which the analysis method and the learning method are executed on a common information processing device 10, but the analysis method and the learning method may also be executed on separate information processing devices having the same or different hardware configurations. Furthermore, the analysis method and the learning method may each be implemented as separate computer programs.

[0013] The information processing device 10 (CPU 11) can use a trained discrimination model that has been machine-learned based on training data, and can also train the discrimination model by executing the present learning method. The "discrimination model" here is a model obtained by machine learning using training data, i.e., supervised learning, and can be expressed as an AI (Artificial Intelligence) model, a machine learning (ML) model, etc. The discriminant model used in this embodiment is only required to be able to output information capable of discriminating the skin attributes or skin conditions of a skin patch image in response to the input of the skin patch image, and there are no limitations on its data structure, learning algorithm, etc. Note that the skin attributes and skin conditions that can be discriminated based on the skin patch images input to the discriminant model and the output from the discriminant model will be described later.

[0014] The discriminant model may be, for example, a regression equation obtained by regression analysis, or may be composed of a convolutional neural network including a deep neural network, and may be realized by a combination of a computer program and parameters, a combination of multiple functions and parameters, etc. Furthermore, when the discriminant model is constructed using a neural network and the input layer, intermediate layer, and output layer are considered as units of a single neural network, the discriminant model may refer to a single neural network or a combination of multiple neural networks. Furthermore, the discriminant model may be composed of a combination of multiple multiple regression equations, or a single multiple regression equation. Furthermore, the discriminant model may be composed of a combination of a neural network and another model such as a regression equation or a discriminant equation, and may output a value obtained by inputting a value obtained as an output of the neural network into the other model. In this embodiment, an example is given in which the discriminant model is composed of a deep neural network.

[0015] This discrimination model may be stored in the memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication. In this way, the information processing device 10 can be described as a skin image analysis device that can use the discriminant model and execute the present analysis method, and can also be described as a discriminant model learning device that can learn the discriminant model. In the following description, the CPU 11 will be described as the entity that executes the present analysis method and the present learning method.

[0016] [Skin image analysis method (this analysis method)] Here, this analysis method will be explained with reference to FIGS. Fig. 2 is a conceptual diagram of the processing flow of the skin image analysis method according to this embodiment, and Fig. 3 is a flowchart of the skin image analysis method according to this embodiment. This analysis method is executed by an information processing device 10 (CPU 11) as exemplified in Fig. 1. As described above, the CPU 11 can use a trained discrimination model that discriminates the skin attributes or skin conditions of a skin patch image of a predetermined image size in response to the input of the skin patch image. This discrimination model is machine-trained based on multiple training data sets that include multiple combinations of correct answer information indicating common attributes or conditions of the subject skin of the training skin images and a group of training skin patch images of a predetermined image size extracted from the training skin image. Note that this discrimination model can be trained using the present training method, which will be described later.

[0017] The teacher skin image is an image used as teacher data and shows human skin. The skin shown in the teacher skin image may include skin that is the target of skin attribute or skin condition discrimination, such as the skin of a human face, the skin of a human neck, the skin of a human arm, the skin of the back or palm of a human hand, the skin of a human leg, etc. The teacher skin image may also show other body parts other than the skin that is the target of discrimination, such as the eyes, nostrils, mouth, etc. The attributes of the skin include, for example, attributes of the person who owns the skin, such as age and sex, and the body part to which the skin belongs (cheek, forehead, face, neck, leg, etc.). The skin condition may be, for example, a condition indicating whether the skin is bare or made up, a condition indicating whether the makeup has come off, a condition indicating whether the skin is moisturized, a condition indicating whether the skin is dry, etc. The skin condition targeted in this embodiment is preferably one that indicates the overall impression of appearance rather than one that indicates the presence of local features such as blemishes or wrinkles. The teacher skin patch image group is an image collection including a plurality of skin patch images extracted from one teacher skin image. A skin patch image is a small section image that contains a skin region (image region showing skin) at a predetermined rate or more. The predetermined rate is set to be at least 90%. The skin patch image has a predetermined shape, and is preferably rectangular for ease of processing, but the shape is not limited to a circle, ellipse, etc. The image size of the skin patch image is predetermined, and is set to, for example, 224 pixels x 224 pixels. However, the size of the skin patch image is not limited to this example and may be determined appropriately depending on the configuration of the discriminant model, etc.

[0018] In this analysis method, first, the CPU 11 acquires a skin image for evaluation (S31). The evaluation skin image may be an image showing the skin of the subject, or may show other parts of the subject. The image size of the evaluation skin image is not limited as long as multiple skin patch images can be extracted from the evaluation skin image. The skin image for evaluation acquired in step (S31) is acquired as image data in, for example, JPEG (Joint Photographic Experts Group) format, BMP (Bitmap image) format, TIFF (Tagged Image File Format), GIF (Graphic Interchange Format), etc. However, the data format of the skin image for evaluation is not limited. Furthermore, the acquired skin image for evaluation is preferably a color image captured with a camera that captures visible light in order to determine the appearance or attributes of the skin. The CPU 11 may acquire the skin image for evaluation from a camera that captured the image, or may acquire the image from another computer or a portable recording medium.

[0019] Next, the CPU 11 acquires a group of skin patch images from the evaluation skin images acquired in step S31 (S32). The number of skin patch images included in the group of skin patch images is preferably large because it corresponds to the number of acquired index values, which will be described later, but is not particularly limited. In addition, it is preferable that the predetermined image size of each skin patch image obtained in step (S32) is the same as the image size of each skin patch image forming the above-mentioned teacher skin patch image group, but it does not necessarily have to be the same as that image size.

[0020] The skin area of ​​the subject appearing in each skin patch image included in the skin patch image group acquired in step (S32) and each teacher skin patch image included in the teacher skin patch image group of the teacher data is within 1 cm 2 More than 20cm 2 Preferably, the skin area is less than 3cm 2 over 19cm 2 As will be described later as an example, the inventors have verified the discrimination accuracy of the discriminant model by varying the size of the skin area of ​​the subject depicted in the teacher skin patch image, and have newly discovered that the size of the skin area affects the discrimination accuracy of the discriminant model, and that using a skin area size in the above-mentioned range improves the discrimination accuracy of the discriminant model. Therefore, it is preferable that the evaluation skin images and teacher skin images from which the individual skin patch images and individual teacher skin patch images are obtained are captured under shooting conditions such as resolution that the skin patch images and teacher skin patch images having a predetermined image size show the skin area in the range described above. Furthermore, by capturing an image with a color chart of a predetermined size (for example, 1 cm square) attached to the subject's skin, it is possible to grasp the relationship between the pixel size on the evaluation skin images and teacher skin images and the actual size of the subject's skin.

[0021] In step (S32), the set of skin patch images can be obtained by various methods. For example, one method is to divide the evaluation skin image into rectangular small section images of a predetermined image size, and then cut out all rectangular small section images that contain a skin region (image region showing skin) at a predetermined ratio or more from the group of rectangular small section images as skin patch images. In this method, small section images may be cut out at an image size larger than the final image size, and then skin patch images of the final image size may be further cut out from the small section images. Another method involves acquiring a group of skin patch images by moving a cropping window of a predetermined image size on the evaluation skin image and cropping small section images within the window when a predetermined percentage or more of the skin region is included within the window. In this method, the same skin region may be partially included in overlapping skin patch images. In this case, determining the skin region when the cropping window is moved by a predetermined amount or more from the previously cropped position can prevent the same skin region from occupying a large portion of the skin patch images. Furthermore, the skin patch image group may be acquired by other methods, and the method for acquiring the skin patch image group is not limited.

[0022] Next, CPU 11 normalizes the pixel values ​​of each skin patch image acquired in step (S32) (S35). For example, CPU 11 calculates the average value and variance of the pixel values ​​(RGB values) of one skin patch image, and performs linear calculations on each pixel value of the skin patch image using the average value and variance, thereby normalizing the pixel values ​​of the skin patch image. In this case, the average value and variance of the pixel values ​​calculated for all skin patch images acquired in step (S32) may be used. Furthermore, as for the normalization method, not only a general normalization method that sets the average value to zero and the variance to one, but also a normalization method based on normalization parameters determined for each discriminant model may be used. Various known methods may be used for the normalization method in step (S32).

[0023] Next, the CPU 11 inputs each normalized skin patch image into a discrimination model (S37), and obtains index values ​​of skin attributes or skin conditions for each skin patch image (S38). The skin attribute or skin condition indicated by the acquired index value is determined by the discriminant model used. For example, when a discriminant model trained with correct answer information indicating either bare skin or made-up skin as the common skin condition of the subject skin of the teacher skin image is used, in (S38), an index value of bare skin feel (bare skin-likeness) or made-up feel (made-up skin-likeness) is acquired as the skin condition index value for each skin patch image. When a discriminant model trained with correct answer information indicating either male or female as the common attribute of the subject skin of the teacher skin image is used, an index value of male skin-likeness or female skin-likeness is acquired as the skin attribute index value for each skin patch image. When a discriminant model trained with correct answer information indicating the subject's actual age or actual age range (age group) as the common attribute of the subject skin of the teacher skin image is used, an index value indicating the apparent age or apparent age range of the skin, or an index value indicating the degree of likelihood for each apparent age range of the skin, is acquired as the skin attribute index value for each skin patch image. Furthermore, when a discrimination model trained with correct answer information indicating either immediately after application of a cosmetic product or a predetermined time after application of a cosmetic product as a common skin condition for all subject skin images is used, the degree of makeup smearing is acquired as a skin condition index value for each skin patch image. Furthermore, when a discrimination model trained with correct answer information indicating either before application of a skin care product or after application of a skin care product is used as a common skin condition for all subject skin images, the index value of skin moisturization is acquired as a skin condition index value for each skin patch image.

[0024] In this way, the index values ​​obtained for each skin patch image in step (S38) include one of a first index value indicating the degree of male or female skin-likeness, a second index value indicating the degree of bare skin-likeness or made-up skin-likeness, a third index value indicating the degree of skin-likeness for each age group, a fourth index value indicating the degree of makeup breakdown, or a fifth index value indicating the degree of skin moisturization. However, the index value output from the discriminant model is not limited to such examples, and may be any value that can be an index of a skin attribute or skin condition. For example, in (S38), the index value may be a binary value indicating whether or not a certain skin condition or certain skin attribute exists. However, it is preferable that the index value obtained in this embodiment indicates the overall impression of appearance rather than indicating the presence of local features such as blemishes or wrinkles.

[0025] 3 shows a plurality of steps (processes) in sequence, but the order in which the steps of the present analysis method are performed is not limited to the example of FIG. 3. For example, rather than performing steps (S37) and (S38) after normalization has been completed for all skin patch images in the group of skin patch images, normalization (S35), input to the discriminant model (S37), and acquisition of index values ​​(S38) may each be performed in a single flow for each individual skin patch image. Furthermore, rather than performing steps (S35) and subsequent steps after acquiring all of the group of skin patch images in step (S32), steps (S35) and subsequent steps may be performed each time a skin patch image is acquired in step (S32).

[0026] In this analysis method, as described above, a group of skin patch images of a predetermined image size obtained from the evaluation skin images is input to a trained discrimination model that has been machine-learned using a group of training data consisting of ground truth information indicating common attributes or conditions of the subject skin in the training skin images and a group of training skin patch images extracted from the training skin images, thereby obtaining index values ​​for the skin attributes or skin conditions for each skin patch image. By obtaining index values ​​for the skin attributes or skin conditions for each skin patch image using a discrimination model trained using ground truth information indicating common attributes or conditions of the entire skin, it is possible to analyze the apparent skin conditions or skin attributes obtained from the entire subject skin captured in the evaluation skin images, while suppressing the effects of variations in the shooting environment of the evaluation skin images, such as lighting, and differences in the characteristics of local parts of the face.

[0027] The CPU 11 can process the index values ​​of each skin patch image acquired in step (S38) as follows. For example, the CPU 11 can generate index value distribution information that indicates the distribution of index values ​​of each skin patch image according to the position of the skin region of each skin patch image. This index value distribution information may be, for example, a graph or map in which index values ​​are arranged according to the position of the skin region of each skin patch image. This index value distribution information allows the distribution of index values ​​of skin conditions or skin attributes to be understood.

[0028] FIG. 4 is a diagram showing an example of a color map image as index value distribution information of a group of skin patch images. As shown in Figure 4, CPU 11 can generate a color map image as the index value distribution information in which the index values ​​of each skin patch image are mapped to the evaluation skin image acquired in step (S31) according to the position of the subject skin area of ​​each skin patch image. Although Fig. 4 is difficult to understand due to the constraints of the drawing, it is necessary to use a binary image, which indicates that the closer to black the image, the higher the index value of the skin condition or skin attribute, and the closer to white the image, the lower the index value. While Fig. 4 is shown in black and white, in the color map image, the position of each skin patch image on the evaluation skin image is represented by a color corresponding to its index value. This color map image allows users to visually grasp the distribution of index values, and because the distribution is displayed on the evaluation skin image, users can intuitively grasp the skin condition or skin attribute index for each skin region.

[0029] CPU 11 can also calculate a representative evaluation value for the skin attributes or skin condition of the person being evaluated based on the index values ​​of each acquired skin patch image. A single representative evaluation value may be calculated for the evaluation skin image acquired in step (S31), or the subject skin area of ​​the evaluation skin image may be divided into several parts (e.g., cheeks, forehead, around the mouth, nose, etc.) that are larger than each skin patch image, and a representative evaluation value for each part may be calculated using the index values ​​of the skin patch images belonging to each part. The representative evaluation value may be, for example, a statistical value such as the average, standard deviation, or variance of the index values ​​of each skin patch image. By obtaining a representative evaluation value from the index values ​​of each skin patch image in this way, it is possible to easily grasp the overall evaluation of the skin condition or skin attributes of the entire skin or each part of the skin of the person being evaluated that appears in the evaluation skin image.

[0030] Furthermore, it is also possible to specify a skin region for which the representative evaluation value is to be calculated. In this case, the CPU 11 further executes a step of acquiring region designation information for designating a desired image region in the skin image for evaluation acquired in step (S31), and calculates a representative evaluation value for the skin attribute or skin condition of the image region based on index values ​​acquired for a group of skin patch images corresponding to the image region indicated by the region designation information acquired in that step. The CPU 11 displays the skin image for evaluation on the display device 15 and identifies a designated image region on the displayed skin image for evaluation using the input device 16, thereby obtaining the region designation information. The CPU 11 can also acquire such region designation information from another computer via communication. In this way, a representative evaluation value of the skin condition or skin attribute can be obtained for a skin area that is of interest to the person being evaluated or other users.

[0031] [Discrimination model training method (this training method)] Next, a method for learning the discriminant model used in the present analysis method will be described with reference to FIGS. Fig. 5 is a conceptual diagram of the process flow of the discriminant model learning method according to this embodiment, and Fig. 6 is a flowchart of the discriminant model learning method according to this embodiment. This learning method is executed by an information processing device 10 (CPU 11) as exemplified in Fig. 1. As described above, this embodiment utilizes a discriminant model configured with a deep neural network that discriminates the skin attributes or skin condition of a skin patch image of a predetermined image size in response to the input of the skin patch image. When this learning method is executed, this discriminant model may be stored in memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication.

[0032] First, the CPU 11 acquires a plurality of combinations of teacher skin images and correct answer information indicating attributes or states common to the subject skin of the teacher skin images (S61). This teacher skin image is a teacher skin patch image having a predetermined image size, which is 1 cm 2 of the subject as described above. 2 More than 20cm 2 It is preferable to capture images under shooting conditions such as resolution that show the following skin area: In addition, by capturing an image with a color chart of a predetermined size (for example, 1 cm square) attached to the skin of the subject of the teacher skin image, it is possible to grasp the relationship between the pixel size on the teacher skin image and the actual size of the subject's skin. The "teacher skin image" and the "attributes" and "conditions" common to all subject skins are as described above. For example, the correct answer information indicating the common condition of all subject skins in the teacher skin image indicates either bare skin or made-up skin. Other examples of correct answer information for skin conditions include either the condition immediately after makeup application or a predetermined time after makeup application, or either the condition before or after application of a skin care product. Furthermore, examples of correct answer information for skin attributes include either male or female, or the actual age or actual age range of the subject. Note that "common to subject skin" means that one correct answer information indicating the attributes or state of the subject skin of one teacher skin image is assigned to one teacher skin image, and that one correct answer information is commonly associated with a group of teacher skin patch images obtained from that teacher skin image as described below.

[0033] Next, the CPU 11 executes steps (S63) to (S67) for each combination of the teacher skin image and the correct answer information acquired in step (S61).

[0034] The CPU 11 acquires a group of teacher skin patch images of a predetermined image size that contain a predetermined ratio or more of skin regions from the teacher skin image related to the target combination (S63). The number of teacher skin patch images acquired from one teacher skin image is preferably as large as possible from the viewpoint of increasing the number of teacher data, but is not particularly limited. Furthermore, the predetermined image size of each skin patch image forming the skin patch image group is determined according to the configuration of the discriminant model to be trained. Here, the skin area of ​​the subject in each of the teacher skin patch images included in the teacher skin patch image group acquired in step (S63) is within 1 cm 2 More than 20cm 2 Preferably, the skin area is less than 3cm 2 over 19cm 2 It is more preferable that the skin area is as follows: The range of the skin area of ​​the subject appearing in such a teacher skin patch image is as described above.

[0035] The method for acquiring the teacher skin patch image group from the teacher skin image in step (S63) may be the same as or different from step (S32) in this analysis method. Various methods can be used to acquire the teacher skin patch image group, similar to the method for acquiring the skin patch image group described in step (S32). Examples of each acquisition method are the same as those described in step (S32), so a description thereof will be omitted here. The method for acquiring the teacher skin patch image group is not limited.

[0036] Next, CPU 11 normalizes the pixel values ​​of each teacher skin patch image acquired in step (S63) (S65). For example, CPU 11 calculates the average value and variance of the pixel values ​​(RGB values) of one teacher skin patch image, and then performs linear calculations on each pixel value of the teacher skin patch image using the average value and variance, thereby normalizing the pixel values ​​of the teacher skin patch image. The normalization method used in step (S65) is preferably the same as the normalization method used for the skin patch images in step (S35) of this analysis method. Various known methods can be used for the normalization method used in step (S65).

[0037] When normalization (S65) is completed for all teacher skin images included in the teacher skin patch image group obtained in step (S63), CPU 11 generates teacher data (S67) in which correct answer information corresponding to the teacher skin images related to the target combination is commonly associated with the teacher skin patch images normalized in step (S65).

[0038] In this way, by executing steps (S63) to (S67) for all combinations of the teacher skin image and correct answer information obtained in step (S61), a group of teacher data is generated in which the correct answer information corresponding to the original teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images. The CPU 11 trains the discriminant model using the generated training data group (S69). In this embodiment, the discriminant model is trained by deep learning. However, the specific learning algorithm for the discriminant model is not limited in any way.

[0039] While Figure 6 shows multiple steps (processes) in sequence, the order in which the steps of the present learning method are performed is not limited to the example of Figure 6. For example, rather than performing step (S67) after normalization has been completed for all teacher skin patch images in the teacher skin patch image group, normalization (S65) and association with correct answer information (S67) may be performed for each teacher skin patch image in a single flow. Furthermore, rather than performing step (S65) and subsequent steps after acquiring all of the teacher skin patch image group in step (S63), step (S65) and subsequent steps may be performed each time a teacher skin patch image is acquired in step (S63).

[0040] In this analysis method, as described above, a discriminant model is trained using training data in which correct answer information indicating the attributes or condition of the subject skin of a group of training skin patch images obtained from a single training skin image is commonly associated with the group of training skin patch images obtained from the training skin image. This allows for the efficient construction of a highly accurate discrimination model, since a large number of training skin patch images can be obtained from a small number of training skin images, whereas a large amount of training data is required to improve the accuracy of the discrimination model. Furthermore, by making the correct answer information associated with each skin patch image correct answer information that indicates the common attributes or state of the entire skin appearing in the original teacher skin image, it is possible to suppress the effects of variations in the shooting environment of the teacher skin image, such as lighting, and differences in the characteristics of local areas of the face, thereby constructing a highly accurate discrimination model.

[0041] Some or all of the above-described embodiments and modifications can be specified as follows: However, the above-described embodiments and modifications are not limited to the following descriptions.

[0042] <1> one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating common attributes or states of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image; A step of acquiring an evaluation skin image showing the skin of the person to be evaluated; acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation; normalizing pixel values ​​of each of the acquired skin patch images; inputting each normalized skin patch image into the discriminant model to obtain a skin attribute or skin condition index value for each skin patch image; Perform skin image analysis method.

[0043] <2> the one or more processors: generating index value distribution information indicating a distribution of the acquired index values ​​according to the position of the subject skin region of each skin patch image; Run the following again: <1> The skin image analysis method according to claim 1. <3> generating a color map image as the index value distribution information by mapping each of the acquired index values ​​onto the acquired skin image for evaluation according to the position of the subject skin region of each skin patch image; <2> The skin image analysis method according to claim 1. <4> the one or more processors: A step of calculating a representative evaluation value of the skin attributes or skin condition of the subject based on the index value for each of the acquired skin patch images; Run the following again: <1> from <3> 10. The skin image analysis method according to claim 9, wherein <5> the one or more processors: acquiring area designation information that designates a desired image area in the acquired skin image for evaluation; calculating a representative evaluation value of a skin attribute or a skin condition of an image region based on the index values ​​acquired for a group of skin patch images corresponding to the image region indicated by the acquired region designation information; Run the following again: <1> from <4> 10. The skin image analysis method according to claim 9, wherein <6> the correct answer information indicating a common skin state of the subject skin of the teacher skin image included in the teacher data indicates either bare skin or made-up skin, a bare skin feel or a makeup feel index value is acquired for each of the skin patch images as a skin condition index value; <1> from <5> 10. The skin image analysis method according to claim 9, wherein <7> The correct answer information indicating a common attribute of subject skin of the teacher skin image included in the teacher data indicates either male or female, An index value of male skin-likeness or female skin-likeness is acquired for each of the skin patch images as an index value of a skin attribute. <1> from <5> 10. The skin image analysis method according to claim 9, wherein <8> The correct answer information indicating a common attribute of the subject's skin of the teacher skin image included in the teacher data indicates the actual age or actual age range of the subject of the teacher skin image, For each of the skin patch images, an index value for each skin appearance age range or an appearance age is acquired as a skin attribute index value. <1> from <5> 10. The skin image analysis method according to claim 9, wherein <9> the correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state immediately after makeup application or a state after a predetermined time has elapsed since makeup application, A makeup smearing degree is acquired for each of the skin patch images as an index value of skin condition. <1> from <5> 10. The skin image analysis method according to claim 9, wherein <10> the correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state before or after application of a skin care agent; A skin moisturizing index value is acquired for each of the skin patch images as a skin condition index value. <1> from <5> 10. The skin image analysis method according to claim 9, wherein <11> The skin area in the skin patch image is 1 cm 2 More than 20cm 2 The following skin areas: <1> from <10> 10. The skin image analysis method according to claim 9, wherein <12> The discrimination model is composed of a convolutional neural network including a deep neural network. <1> from <11> 10. The skin image analysis method according to claim 9, wherein

[0044] <13> A method for learning a discriminant model that discriminates skin attributes or skin conditions of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising: obtaining a plurality of combinations of teacher skin images and correct answer information indicating attributes or states common to the subject skin of the teacher skin images; a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images; normalizing pixel values ​​of each of the acquired teacher skin patch images; generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images; a step of training the discriminant model using the training data group; A method for training discriminant models including: <14> A skin image analysis device comprising at least the one or more processors and a memory, <1> from <12> A skin image analysis device capable of executing any one of the skin image analysis methods described above. <15> <13> A discriminant model learning device capable of executing the discriminant model learning method described in claim 1.

[0045] The above content will be explained in more detail below with reference to several examples, but the description of each example below does not impose any limitations on the above content. In the following examples, a discriminant model constructed by transfer learning (fine tuning including fine adjustment of weights) based on a 16-layer convolutional neural network model called VGG16 was used. VGG16 is a model that inputs rectangular images of 224 x 224 pixels and classifies them into approximately 1,000 categories, and this model was adjusted by transfer learning to classify the images into two categories indicating skin attributes or skin conditions. However, as mentioned above, the configuration of the discriminant model is not limited to this example. [Example]

[0046] In Example 1, the size of the skin area of ​​the subject appearing in the teacher skin patch image (hereinafter referred to as skin area size) was varied to verify the discrimination accuracy of the discrimination model. In Example 1, each teacher skin patch image and each evaluation skin patch image was a rectangular image of 224 x 224 pixels, and nine types of skin area sizes were used: 4.5 mm square (mm square), 8.9 mm square, 11.1 mm square, 13.4 mm square, 16.0 mm square, 17.8 mm square, 26.7 mm square, 35.6 mm square, and 44.5 mm square. Teacher skin images of 269 people were captured under shooting conditions that allowed each teacher skin patch image to have these nine skin region sizes, and a teacher skin patch image group was obtained from each teacher skin image, and a discriminant model was trained using the above-mentioned learning method. In other words, nine discriminant models were constructed that were trained using nine teacher data groups corresponding to the nine skin region sizes. Meanwhile, evaluation skin images of the same 269 people were captured under four different lighting conditions, and a group of evaluation skin patch images was obtained from each evaluation skin image. Each evaluation skin patch image was input into a trained discrimination model, and the output was evaluated. Specifically, each skin patch image corresponding to nine different skin region sizes was input into a discrimination model trained with training data for the corresponding skin region size.

[0047] 7 is a graph showing the relationship between the size of the skin region and the discrimination accuracy of the discrimination model. As shown in FIG. 7, Example 1 demonstrated that the size of the skin region affects the discrimination accuracy of the discrimination model. In particular, it was confirmed that the highest discrimination accuracy was achieved when the skin region size was 17.8 mm square. In addition, the skin area size is 8.9mm square (approximately 0.79cm 2 ), a discrimination accuracy of more than 60% can be achieved. 2 ) or larger and 44.5mm square (approximately 19.8cm 2 It has been demonstrated that a discrimination accuracy of 90% or more can be achieved if the range is within the range below. As a result, the skin area of ​​the subject in the evaluation skin patch image and the teacher skin patch image is within 1 cm of the subject. 2 More than 20cm 2 Preferably, the skin area is less than 3cm 2 over 19cm 2 It is more preferable that the skin area is as follows: [Example]

[0048] In Example 2, the accuracy of the makeup feel index values ​​for each skin patch image obtained by this analysis method was verified by comparing with the discrimination accuracy based on human visual evaluation. In the visual evaluation, five trained judges looked at the facial images and judged whether or not base makeup was applied. For the facial images, frontal images of the 269 subjects were taken both with and without base makeup (bare skin), resulting in a total of 538 frontal facial images. On the other hand, the 538 frontal facial images were used as training skin images, and correct answer information on whether or not base makeup cosmetics had been applied to each training skin image was attached to each training skin image.A trained discrimination model was constructed by learning using a training data group in which a group of skin patch images and their correct answer information were associated with each training skin image. The discriminant model was a model that outputs two index values: bare skin resemblance and made-up skin resemblance (resemblance that base makeup has been applied). In contrast, for the 269 subjects, left and right side-facing facial images not used as training data were taken with and without base makeup applied, resulting in a total of 1,076 side-facing facial images prepared as evaluation skin images. Then, for each evaluation skin image, the present analysis method using the trained discriminant model was executed to obtain index values ​​for bare skin-likeness and made-up skin-likeness for each skin patch image, and the skin condition (bare skin or made-up skin) with the highest average index value for each subject was determined as the estimated skin condition (bare skin or made-up skin) in the present analysis method.

[0049] As a result, the highest discrimination accuracy among the five judges was 93.1%, while the discrimination accuracy using the index values ​​obtained by this analysis method was 97.2%. This demonstrates that the accuracy of the index values ​​for bare skin or made-up skin (makeup look) for each skin patch image obtained by this analysis method is higher than the discrimination accuracy of human visual evaluation.

[0050] In Example 2, the correlation between the judgment based on human visual evaluation and the judgment based on the index value obtained by this analysis method was further verified. In the visual evaluation, each judge rated the makeup feel using a 6-point score (bare skin being 0, and the makeup feel (degree of makeup-likeness) being indicated on a 5-point scale from 1 to 5). The index value (a value ranging from -5 to 20) for bare skin-likeness or makeup-likeness (makeup feel) obtained by this analysis method was used directly as the makeup feel score. The correlation between the average makeup feel scores of the five judges and the makeup feel score obtained by this analysis method was then analyzed. Figure 8 is a graph showing the relationship between the makeup feel score obtained by human visual evaluation and the makeup feel score obtained by the present analysis method. In Figure 8, the vertical axis shows the makeup feel score obtained by the present analysis method, and the horizontal axis shows the makeup feel score obtained by human visual evaluation. As a result, as shown in Figure 8, a correlation was found between the average makeup feel scores of the five judges and the makeup feel score obtained by this analysis method, with a correlation coefficient of 0.72. [Example]

[0051] In Example 3, it was verified whether the index value of the makeup feel for each skin patch image obtained by this analysis method is applicable to various cosmetic application modes. As skin images for evaluation, in addition to facial images of the subject's bare skin, makeup-applied facial images of seven different cosmetic application modes were prepared. The makeup-applied facial images of the seven cosmetic application modes were: a makeup-applied facial image after application of a skin care cosmetic (denoted as "skin care"), a makeup-applied facial image after application of a skin care cosmetic and a sunscreen cosmetic (denoted as "+sunscreen"), a makeup-applied facial image after application of a skin care cosmetic, a sunscreen cosmetic, and a first base makeup cosmetic (denoted as "base"), a makeup-applied facial image after application of a skin care cosmetic, a sunscreen cosmetic, a first base makeup cosmetic, and a powder foundation cosmetic (denoted as "+PFD"), a makeup-applied facial image after application of a skin care cosmetic and a second base makeup cosmetic (denoted as "base 2"), a makeup-applied facial image after application of a skin care cosmetic, a second base makeup cosmetic, and a cream foundation cosmetic (denoted as "+cream FD"), and a makeup-applied facial image after application of a skin care cosmetic, a second base makeup cosmetic, a cream foundation cosmetic, and a face powder cosmetic (denoted as "loose PD"). Then, a group of skin patch images was obtained for each of the eight evaluation skin images, including the bare skin face image, and each skin patch image was input into the discriminant model to obtain an index value for the makeup look of each skin patch image. Finally, a representative index value (average value) for the makeup look of each skin patch image was calculated for each original evaluation skin image.

[0052] FIG. 9 is a graph showing representative index values ​​of the makeup feel obtained by this analysis method for each of eight types of skin images for evaluation that have different cosmetic application patterns. 9, the representative index value of the makeup feel calculated for the bare face image is the lowest, followed by the representative index value of the makeup feel calculated for the face image after the skin care cosmetic has been applied. Regarding the representative index values ​​of the makeup feel for the makeup face images with other cosmetic application modes, the representative index values ​​increase in the following order: the makeup face image labeled +Sunscreen, the makeup face image labeled +Primer 2, the makeup face image labeled +Primer, the makeup face image labeled +PFD, the makeup face image labeled +Cream FD, and the makeup face image labeled +Loose PD. These results were generally consistent with the visual impression of the made-up face for each cosmetic application mode, and it can be said that this analysis method can detect the makeup feel associated with the application of skin care cosmetics, sunscreen cosmetics, and base makeup cosmetics, and can also appropriately detect the increase in makeup feel caused by the application of multiple cosmetics. In other words, it was demonstrated that the makeup feel index values ​​obtained for each skin patch image using this analysis method can be applied to various cosmetic application modes.

[0053] Fig. 10 is a diagram showing a color map image showing the index value distribution of a group of skin patch images for each of eight types of evaluation skin images with different cosmetic application patterns. Fig. 10 is difficult to understand because it is a binary image due to drawing restrictions, but it shows that the closer to black the image, the higher the makeup index value (the more likely it is to resemble made-up skin), and the closer to white the image, the lower the makeup index value (the more likely it is to resemble natural skin). 10 also shows that the makeup look index values ​​for each skin patch image obtained by this analysis method are applicable to various cosmetic application patterns. Furthermore, the color map image showing the distribution of index values ​​for the group of skin patch images makes it easy to grasp which areas of the face have a high and low makeup look for each small, divided region. [Example]

[0054] In Example 4, the accuracy of the index values ​​for each skin appearance age range of each skin patch image obtained by this analysis method was verified. In Example 4, 512 women in various age groups from their 20s to 70s were photographed, with bare skin images and facial images after application of foundation cosmetics, and a group of skin patch images was obtained from each facial image, resulting in a total of 43,000 skin patch images. 70% of the group of skin patch images were used as a training group of skin patch images for training a discriminant model, and the remaining 30% were used as an evaluation group of skin patch images for this analysis method using the discriminant model. The discriminant model was a model that output index values ​​for each of six age ranges from 20s to 70s. In this analysis method, the discriminant model was used to obtain index values ​​for each skin appearance age range for each skin patch image included in the group of evaluation skin patch images. In this example, an estimated age for each skin patch was obtained by linear regression using each index value obtained from the discriminant model as an independent variable, and the average estimated age for all skin patches obtained for each subject was determined as the estimated age in this analysis method.

[0055] Figure 11 is a graph showing the relationship between estimated ages and actual ages obtained by this analysis method. In Figure 11, the vertical lines indicate the range of estimated ages obtained by this analysis method for each actual age range, the horizontal lines indicate the range of actual ages of the subjects for each actual age range, and the circles indicate the average actual ages of the subjects for each actual age range and the average estimated ages obtained by this analysis method. As a result, it was confirmed that the estimated age obtained using this analysis method correlated with actual age with a correlation coefficient of 0.77, and the predicted standard error was 10.77. In other words, it was demonstrated that the index values ​​and estimated ages for each skin appearance age range of each skin patch image obtained by this analysis method can be obtained with high accuracy. [Example]

[0056] In Example 5, the accuracy of the index values ​​of male skin-likeness or female skin-likeness of each skin patch image obtained by this analysis method was verified. In Example 5, bare skin facial images were captured for a total of 52 subjects, 26 men and 26 women, and a group of skin patch images was obtained from each facial image, resulting in a total of 1,200 skin patch images. 70% of the group of skin patch images were used as a training group of skin patch images for training a discriminant model, and the remaining 30% were used as an evaluation group of skin patch images for this analysis method using the discriminant model. The discrimination model was a model that outputs two index values, one for male skin-likeness and one for female skin-likeness. In this analysis method, index values ​​for male skin-likeness and one for female skin-likeness were obtained for each skin patch image included in the evaluation skin patch image group, and the gender with the larger index value was determined as the estimated gender for each skin patch image.

[0057] As a result, the estimated gender obtained for each skin patch image using this analysis method showed a discrimination accuracy of 97.1%, demonstrating that the index value for the male or female skin-likeness of each skin patch image obtained using this analysis method shows high accuracy. Fig. 12 is a diagram showing a color map image showing the index value distribution of the skin patch image group for each of the evaluation skin images of the five subjects. Fig. 12 is difficult to understand because it is a binary image due to drawing restrictions, but it shows that the closer to black the color is, the higher the index value of male skin-likeness (the lower the index value of female skin-likeness), and the closer to white the color is, the lower the index value of male skin-likeness (the higher the index value of female skin-likeness). In this way, the color map image shown in FIG. 12 makes it possible to easily grasp the areas of the entire face that are highly or less likely to be male or female skin, for each small divided area. [Example]

[0058] In Example 6, the accuracy of the degree of makeup smearing of each skin patch image obtained by this analysis method was verified. In Example 6, 60 women were photographed as subjects, with facial images taken immediately after application of a foundation cosmetic and 6 to 10 hours after application of the foundation cosmetic. A group of skin patch images was obtained from each facial image, resulting in a total of 5,000 skin patch images. 70% of the group of skin patch images were used as a training group of skin patch images for training a discriminant model, and the remaining 30% were used as an evaluation group of skin patch images for this analysis method using the discriminant model. The discriminant model was a model that output two index values: one immediately after cosmetic application and one after a predetermined time had elapsed since cosmetic application. In this analysis method, two index values ​​were obtained for each skin patch image included in the group of skin patch images for evaluation: one immediately after cosmetic application and one after a predetermined time had elapsed since cosmetic application, and the state with the larger index value (either immediately after cosmetic application or the predetermined time had elapsed since cosmetic application) was determined to be the estimated cosmetic application state for each skin patch image.

[0059] As a result, the estimated cosmetic application state obtained for each skin patch image using this analysis method demonstrated a discrimination accuracy of 78.4%. While makeup smearing can occur uniformly across the entire face, it is often the case that it occurs only in parts. Considering this, this discrimination accuracy value (78.4%) can be said to be sufficiently high. In other words, it was demonstrated that the two index values ​​obtained for each skin patch image obtained using this analysis method, immediately after cosmetic application and after a specified time has passed since cosmetic application, demonstrate high accuracy. Furthermore, these two index values ​​can be said to indicate the degree of makeup smearing. 13A and 13B are diagrams showing color map images showing the index value distribution of the skin patch image group for each evaluation skin image immediately after application of two types of foundation cosmetics and 4, 8, and 10 hours after application. The subject is the same woman, but different foundation cosmetics were used in FIGS. 13A and 13B. Figure 13 is difficult to understand because it has to be a binary image due to drawing restrictions, but it shows that the closer to black the image, the higher the index value indicating a predetermined time after application (the greater the degree of makeup smearing), and the closer to white the image, the lower the index value indicating a predetermined time after application (the less the degree of makeup smearing). 13, it can be seen that the change in the degree of makeup smearing over time is smaller for the foundation cosmetic used in FIG. 13(a) than for the foundation cosmetic used in FIG. 13(b). In this way, the cosmetic smearing characteristics can be objectively evaluated using the color map image shown in Fig. 13. Furthermore, this color map image makes it easy to grasp the areas of the face that are more likely to smear and the areas that are less likely to smear, divided into small regions. [Example]

[0060] In Example 7, the accuracy of the index values ​​of the skin moistness of each skin patch image obtained by this analysis method was verified. In Example 7, facial images of eight women were taken before and after the application of a skin care agent, and a skin patch image group was obtained from each facial image, resulting in a total of 231 skin patch images. 70% of the skin patch image group were used as a training skin patch image group for training a discriminant model, and the remaining 30% were used as an evaluation skin patch image group for this analysis method using the discriminant model. The discriminant model was a model that outputs two index values, one before application of a skin care agent and one after application of a skin care agent. In this analysis method, two index values, one before application of a skin care agent and one after application of a skin care agent, were obtained for each skin patch image included in the group of skin patch images for evaluation, and the state with the larger index value (before application of a skin care agent or after application of a skin care agent) was determined to be the estimated skin care application state of each skin patch image.

[0061] 14 is a graph showing the relationship between the number of epochs and the discrimination accuracy of the discriminant model in this learning method. The discrimination accuracy of the discriminant model indicates the accuracy of the estimated skin care application state obtained for each skin patch image, as described above. In Figure 14, the discrimination accuracy of the discriminant model in response to the input of a teacher skin patch image is shown by a broken line labeled "teacher data," and the discrimination accuracy of the discriminant model in response to the input of an evaluation skin patch image is shown by a broken line labeled "verification data." The results confirmed that when the discriminant model was trained using the optimal number of epochs (64), the accuracy of discrimination between before and after skin care product application was 88.6%. This demonstrates that the two index values ​​obtained from this analysis method for each skin patch image, before and after skin care product application, exhibit high accuracy. These two index values ​​can be called index values ​​for skin moisture sensation. [Example]

[0062] In Example 8, a trained model for estimating the visual skin assessment score was used as a discriminant model to verify the accuracy of the visual skin assessment score for each skin patch image obtained by this analysis method. First, 341 facial images were prepared, showing the entire bare face of 87 Japanese women aged between 20 and 39. These 341 facial images were obtained by photographing each of the 87 subjects (with some exceptions) up to four times at different times. Furthermore, 10 judges visually evaluated the skin of 341 facial images, and as evaluation results, a 7-level evaluation score ranging from -3 points to +3 points (average value of the 10 judges) was assigned to each of the 10 evaluation items.Finally, the evaluation scores assigned to each facial image for each of the 10 evaluation items were considered to be the correct answer information. The evaluation items were 10 items: overall evaluation, whiteness of the skin, yellowness of the skin, moisture, dryness, firmness, luster, transparency, smoothness of the skin surface, and evenness of the skin texture. This resulted in 341 combinations of face images and 10 visual evaluation scores as correct answer information. The 10 visual evaluation scores obtained as correct answer information were normalized so that the mean and variance for each evaluation item were 0 and 1, respectively.

[0063] Next, face detection and skin color detection were performed on the face images, and a group of skin patch images of a predetermined image size, each containing 90% or more of the skin area, was obtained from each face image. In this example, the skin area of ​​the subject in each skin patch image was determined to be a 17.8 mm square skin area of ​​the subject, and 9,306 skin patch images were obtained from all face images. Furthermore, the pixel values ​​of each skin patch image were normalized, and for each normalized skin patch image, the 10 visual evaluation scores of the face image from which the skin patch image was derived were assigned as labels (correct answer information). In this example, 70% of the 341 combinations of the skin patch images thus obtained and the 10 visual evaluation scores assigned as their labels were used as training data, and the remaining 30% were used as verification data.

[0064] The discriminant model used in Example 8 (hereinafter referred to as the skin patch model) is a model that inputs a skin patch image and outputs a 10-dimensional vector having estimated values ​​of visual evaluation scores for 10 items as elements. This discriminant model was trained using the training data described above so as to minimize the average value between evaluation items of the root mean square error (RMSE) between the estimated values ​​of all 10 items, which are elements of the output vector, and the actual visual evaluation scores. As a comparative example, a comparative discriminant model (hereinafter referred to as the full face model) was also generated, which inputs the above-mentioned face image and outputs a 10-dimensional vector having estimated values ​​of the visual evaluation scores of 10 items as elements.

[0065] Fig. 15 is a graph for comparing the estimation accuracy of a trained model (all-face type discriminant model) that inputs a face image to estimate a visual skin evaluation score, and a trained model (skin patch type discriminant model) that inputs a skin patch image to estimate a visual skin evaluation score. Fig. 15 shows the correlation coefficient between the estimated visual evaluation score output by the skin patch type discriminant model or the all-face type discriminant model and the correct answer information (visual evaluation score) of the verification data for each of the 10 visual evaluation items. According to Figure 15, the skin patch type discrimination model shows estimation accuracy with a correlation coefficient exceeding 0.6 for all 10 evaluation items, whereas the all-face type discrimination model as a comparative example shows a significant drop in estimation accuracy for the two evaluation items of sideways glance, namely skin color and yellowish skin. From the above, the effectiveness of using skin patch images was demonstrated even when using a discriminant model to estimate visual skin assessment scores.

[0066] In this example, the estimation accuracy of the above-mentioned skin patch type discrimination model was further verified using a new group of face images obtained by photographing a subject different from the above-mentioned face images in a different shooting environment. The new facial image collection features 269 Japanese women in their 20s to 70s, and includes two facial images of each subject: one with bare skin and one with foundation applied. On the other hand, since the skin patch type discrimination model is trained using the training data described above, it has not yet been trained on facial images of subjects aged 40 or over or facial images after applying foundation.

[0067] A group of skin patch images was obtained from these new face images in the same manner as described above, and each skin patch image was input into the skin patch type discrimination model to obtain estimated visual evaluation scores for the 10 items for each skin patch image. The skin patch images were then classified into two skin groups: bare skin and skin after foundation application. Each skin group was further classified into six age groups (20s to 70s) based on the age of the subject, and the average estimated visual evaluation score obtained by the skin patch type discrimination model was calculated for each group.

[0068] Figure 16 is a radar chart showing the correlation between age and the estimated visual evaluation score obtained from the skin patch type discriminant model in Example 8, and the change in the estimated visual evaluation score following the application of a cosmetic product. Figure 16(a) is a radar chart showing the average value of the estimated visual evaluation score obtained from the skin patch type discriminant model for each of the six age groups in the bare skin group, and Figure 16(b) is a radar chart showing the average value of the estimated visual evaluation score obtained from the skin patch type discriminant model for each of the six age groups in the skin group after the application of foundation. In Figure 16, the wider the area surrounded by connecting the estimated values ​​of the visual evaluation scores for each of the 10 evaluation items (hereinafter referred to as the chart area), the higher the estimated value of the visual evaluation score for each evaluation item. 16, it can be seen that for both the bare skin group and the foundation-applied skin group, the chart area for the 20s group is the widest, and the chart area becomes narrower as the age group increases. This means that the skin patch type discrimination model accurately captures the decline in visual skin evaluation scores with age. Furthermore, when comparing between the same age groups, it can be seen that the chart area shown in Figure 16(b) is wider than the chart area shown in Figure 16(a) for all age groups. This means that the skin patch type discrimination model accurately captures the improvement in the skin visual evaluation score that accompanies the application of foundation. From the above, it was demonstrated that the skin patch type discrimination model of Example 8 can obtain highly accurate estimates of visual evaluation scores even for new groups of face images, including face images from untrained ages and face images after untrained foundation application. [Explanation of symbols]

[0069] 10. Information processing equipment 11 CPU 12 Memory 13 Input / Output Interface 14 Communication Unit 15 Display device 16 Input Devices

Claims

1. one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating a common state of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image; A step of acquiring an evaluation skin image showing the skin of the person to be evaluated; acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation; normalizing pixel values ​​of each of the acquired skin patch images; an index value acquisition step of inputting each normalized skin patch image into the discriminant model to acquire a skin condition index value for each skin patch image; Run the correct answer information indicating a common skin state of the subject skin of the teacher skin image included in the teacher data indicates either bare skin or made-up skin, In the index value acquisition step, an index value of a bare skin look or a makeup look is acquired for each of the skin patch images as an index value of a skin condition. Skin image analysis method.

2. one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating a common state of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image; A step of acquiring an evaluation skin image showing the skin of the person to be evaluated; acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation; normalizing pixel values ​​of each of the acquired skin patch images; an index value acquisition step of inputting each normalized skin patch image into the discriminant model to acquire a skin condition index value for each skin patch image; Run the correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state immediately after makeup application or a state after a predetermined time has elapsed since makeup application, In the index value acquisition step, a makeup smear degree is acquired for each of the skin patch images as an index value of a skin condition. Skin image analysis method.

3. one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating a common state of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image; A step of acquiring an evaluation skin image showing the skin of the person to be evaluated; acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation; normalizing pixel values ​​of each of the acquired skin patch images; an index value acquisition step of inputting each normalized skin patch image into the discriminant model to acquire a skin condition index value for each skin patch image; Run the correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state before or after application of a skin care agent; In the index value acquisition step, an index value of skin moisturizing feeling is acquired as an index value of skin condition for each of the skin patch images. Skin image analysis method.

4. one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating common attributes or states of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image; A step of acquiring an evaluation skin image showing the skin of the person to be evaluated; acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation; normalizing pixel values ​​of each of the acquired skin patch images; inputting each normalized skin patch image into the discriminant model to obtain a skin attribute or skin condition index value for each skin patch image; Run The skin area shown in the skin patch image is a skin area of ​​the subject that is 1 cm 2 or more and 20 cm 2 or less. Skin image analysis method.

5. the one or more processors: generating index value distribution information indicating a distribution of the acquired index values ​​according to the position of the subject skin region of each skin patch image; The skin image analysis method according to claim 1 , further comprising:

6. generating a color map image as the index value distribution information by mapping each of the acquired index values ​​onto the acquired skin image for evaluation according to the position of the subject skin region of each skin patch image; The skin image analysis method according to claim 5 .

7. the one or more processors: calculating a representative evaluation value of the skin condition of the subject based on the index value for each of the acquired skin patch images; The skin image analysis method according to claim 1 , further comprising:

8. the one or more processors: A step of calculating a representative evaluation value of the skin attributes or skin condition of the subject based on the index value for each of the acquired skin patch images; The method of claim 4 further comprising:

9. the one or more processors: acquiring area designation information that designates a desired image area in the acquired skin image for evaluation; calculating a representative evaluation value of the skin condition of the image region based on the index values ​​acquired for a group of skin patch images corresponding to the image region indicated by the acquired region designation information; The skin image analysis method according to claim 1 , further comprising:

10. the one or more processors: acquiring area designation information that designates a desired image area in the acquired skin image for evaluation; calculating a representative evaluation value of a skin attribute or a skin condition of an image region based on the index values ​​acquired for a group of skin patch images corresponding to the image region indicated by the acquired region designation information; The method of claim 4 further comprising:

11. The correct answer information indicating a common attribute of subject skin of the teacher skin image included in the teacher data indicates either male or female, An index value of male skin-likeness or female skin-likeness is acquired for each of the skin patch images as an index value of a skin attribute. The method for analyzing skin images according to claim 4, 8 or 10.

12. The correct answer information indicating a common attribute of the subject's skin of the teacher skin image included in the teacher data indicates the actual age or actual age range of the subject of the teacher skin image, For each of the skin patch images, an index value for each skin appearance age range or an appearance age is acquired as a skin attribute index value. The method for analyzing skin images according to claim 4, 8 or 10.

13. The discrimination model is composed of a convolutional neural network including a deep neural network. The method for analyzing skin images according to any one of claims 1 to 12.

14. A method for learning a discriminant model for discriminating a skin condition of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising: obtaining a plurality of combinations of teacher skin images and correct answer information indicating a common state of subject skin in the teacher skin images; a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images; normalizing pixel values ​​of each of the acquired teacher skin patch images; generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images; a step of training the discriminant model using the training data group; Including, The correct answer information indicating a common state of subject skin in the teacher skin image indicates either bare skin or made-up skin. How to train a discriminant model.

15. A method for learning a discriminant model for discriminating a skin condition of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising: obtaining a plurality of combinations of teacher skin images and correct answer information indicating a common state of subject skin in the teacher skin images; a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images; normalizing pixel values ​​of each of the acquired teacher skin patch images; generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images; a step of training the discriminant model using the training data group; Including, The correct answer information indicating a common state of the subject skin of the teacher skin image indicates either a state immediately after applying makeup or a state after a predetermined time has elapsed since applying makeup. How to train a discriminant model.

16. A method for learning a discriminant model for discriminating a skin condition of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising: obtaining a plurality of combinations of teacher skin images and correct answer information indicating a common state of subject skin in the teacher skin images; a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images; normalizing pixel values ​​of each of the acquired teacher skin patch images; generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images; a step of training the discriminant model using the training data group; Including, the correct answer information indicating a common state of the subject skin of the teacher skin image indicates either a state before application of a skin care agent or a state after application of a skin care agent; How to train a discriminant model.

17. A method for learning a discriminant model that discriminates skin attributes or skin conditions of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising: obtaining a plurality of combinations of teacher skin images and correct answer information indicating attributes or states common to the subject skin of the teacher skin images; a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images; normalizing pixel values ​​of each of the acquired teacher skin patch images; generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images; a step of training the discriminant model using the training data group; Including, a skin area of ​​the subject appearing in each of the teacher skin patch images included in the teacher skin patch image group is equal to or larger than 1 cm 2 and equal to or smaller than 20 cm 2 ; How to train a discriminant model.

18. A skin image analysis device comprising at least the one or more processors and a memory, A skin image analysis device capable of executing the skin image analysis method according to any one of claims 1 to 13.

19. A discriminant model learning device capable of executing the discriminant model learning method according to any one of claims 14 to 17.

Images