Nasal and lip profile analysis method

Abstract

This invention provides a technique for appropriately analyzing nasolabial shadows, which are shadows that appear at the boundary between the nose, the area below the nose, and the cheek. [Solution] In the nasolabial shadow analysis method, one or more processors perform an image acquisition step to acquire an evaluation face image showing the face of the person to be evaluated, and an evaluation value acquisition step to input the acquired evaluation face image into a trained model that has been machine-learned based on training data from multiple people, thereby acquiring a first evaluation value indicating the state of elevation of the area near the nasolabial folds on the cheek of the person to be evaluated and a second evaluation value indicating the state of the nasolabial folds on the person to be evaluated. The first evaluation value indicates the state of elevation of the area near the nasolabial folds, which is the part of the cheek adjacent to the outside of the nasolabial folds and is a part that can produce nasolabial shadows in the same way as the nasolabial folds. The nasolabial shadow is analyzed using the first evaluation value and the second evaluation value.

Description

Technical Field

[0001] The present invention relates to a technique for analyzing a nasolabial shadow, which is a shadow that appears at the boundary between the nose and below the nose and the cheeks, using a face image.

Background Art

[0002] The following Patent Document 1 discloses a method for analyzing at least one of skin pores, porphyrin of comedones, wrinkles, stains, and skin color from an image of a subject's face. In the analysis of wrinkles, after performing distortion correction on the difference image between the R image and the G image using a Gaussian filter or the like, image processing for emphasizing wrinkles is performed, and the number and total area of wrinkles are quantified. The following Patent Document 2 discloses a method for evaluating wrinkles of a subject by extracting position information and size information for each wrinkle in a target region in an image of the entire face or a partial face of the subject. As wrinkle analysis parameters, a wrinkle area ratio, a total wrinkle average depth, a maximum wrinkle average depth, a maximum wrinkle maximum depth (μm), a wrinkle total volume (mm3), an ISO standard surface roughness parameter, etc. are analyzed. The following Patent Document 3 discloses a method for generating an aging simulation image of wrinkles and simulating the change over time of wrinkles. The following Patent Document 4 discloses a method for evaluating the degree of progression of sagging of the jaw and a method for evaluating the effectiveness of various beauty treatments for improving sagging of the jaw using the same. In this method, the shape of the subject's jaw to the neck is observed from the side, and the observed shape of the jaw to the neck is classified into the grade of the shape that most matches among the evaluation criteria of grades 0 to 5, and an increase in the grade number indicates the progression of sagging of the jaw. Patent Document 5 discloses a method for reducing false detection of wrinkles by generating multiple wrinkle component extraction data by extracting wrinkle components within a predetermined angle range where wrinkles are easily stretched for each predetermined part of the face from the analysis area of ​​the subject's facial data; generating multiple wrinkle component enhancement data by emphasizing the wrinkle components from each of the multiple wrinkle component extraction data; generating composite data by combining the multiple wrinkle component enhancement data; and detecting wrinkle components in the composite data that have an intensity of above a predetermined threshold as the subject's wrinkles. Patent Document 6 below discloses a method for appropriately analyzing the skin condition from the facial wrinkles and skin creasing during skincare. This method involves tracking the amount of change of multiple tracking points pre-arranged in the analysis region of the face based on changes in facial expressions contained in a video image of the subject's face, obtaining the skin compression ratio in the analysis region from the amount of change, and analyzing the subject's skin condition based on the obtained compression ratio. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2007-152084 [Patent Document 2] Japanese Patent Publication No. 2010-119431 [Patent Document 3] Japanese Patent Publication No. 2012-181843 [Patent Document 4] Japanese Patent Publication No. 2013-59529 [Patent Document 5] Japanese Patent Publication No. 2013-69122 [Patent Document 6] Japanese Patent Publication No. 2014-193197 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] As mentioned above, there are methods for analyzing the number, size, and depth of wrinkles from the facial images of those being evaluated, as well as methods for evaluating the degree of sagging in the jaw area. Along with such wrinkles and sagging cheeks, nasolabial folds are known as one of the factors that give the face an aged appearance. Nasolabial folds refer to wrinkles or grooves at the boundary between the nose and the area between the nose and upper lip and the cheeks, and are evaluated by the depth of these wrinkles (grooves). This is because the depth of nasolabial folds is related to the appearance of shadows around them, and as a result, affects the impression of aging on the face.

[0005] The inventors have newly discovered that, separate from nasolabial folds, the prominence of the cheek area adjacent to the nasolabial folds (referred to as the area of ​​the cheek near the nasolabial folds) also creates shadows at the boundary between the nose and the area below the nose and the cheek, similar to those of nasolabial folds, and thus influences the impression of aging of the face. In other words, it has been newly discovered that evaluating the nasolabial shadow, which is the shadow that occurs at the boundary between the nose and the area below the nose and the cheek, is not sufficient if only the condition of the nasolabial folds is evaluated.

[0006] This invention was made from this perspective and provides a technique for appropriately analyzing nasolabial shadows. "Nasolabial shadow" refers to the shadow that appears at the boundary between the nose and the area below the nose and the cheek. [Means for solving the problem]

[0007] According to the present invention, a method for analyzing nasolabial shadows, which are shadows that appear at the boundary between the nose and the cheek, is provided, comprising: an image acquisition step in which one or more processors acquire an evaluation face image showing the face of a person to be evaluated; and an evaluation value acquisition step in which the acquired evaluation face image is input to a trained model that has been machine-learned based on training data from multiple people, thereby acquiring a first evaluation value indicating the state of elevation of the area near the nasolabial folds on the cheek of the person to be evaluated and a second evaluation value indicating the state of the nasolabial folds on the person to be evaluated, wherein the first evaluation value indicates the state of elevation of the area near the nasolabial folds, which is an area of ​​the cheek adjacent to the outside of the nasolabial folds and which can produce nasolabial shadows in the same way as the nasolabial folds, and a nasolabial shadow analysis method is provided that analyzes the nasolabial shadow using the first evaluation value and the second evaluation value. Furthermore, according to the present invention, a method for learning a nasolabial shadow evaluation model is provided, which can output a first evaluation value indicating the state of elevation of the area near the nasolabial folds of the cheek in a face image in response to an input face image, the method comprising: one or more steps of one or more processors to acquire training data having multiple combinations of a face image and correct information including visual evaluation values ​​of the state of elevation of the area near the nasolabial folds of the cheek and visual evaluation values ​​of the state of the nasolabial folds for the faces of the people depicted in the face image; and training the nasolabial shadow evaluation model using the acquired training data, wherein the nasolabial shadow is a shadow that appears at the boundary between the nose and the area below the nose and the cheek, and the area near the nasolabial fold is the area of ​​the cheek adjacent to the outside of the nasolabial fold and is an area that can produce a nasolabial shadow in the same way as the nasolabial fold. Furthermore, we can also provide a method for evaluating the visual impression of nasolabial shadows, which includes at least the nasolabial shadow analysis method described above. Furthermore, a nasolabial shadow analysis device equipped with at least one of the above-mentioned processors and memory and capable of performing the above-mentioned nasolabial shadow analysis method, and a nasolabial shadow evaluation model learning device capable of performing the above-mentioned nasolabial shadow evaluation model learning method may also be provided. [Effects of the Invention]

[0008] According to the present invention, a technique for appropriately analyzing nasolabial shadows can be provided.

Brief Description of the Drawings

[0009] [Figure 1] It is a diagram conceptually showing a hardware configuration example of an information processing apparatus capable of executing a nasolabial shadow analysis method and a learning method of a nasolabial shadow evaluation model according to this embodiment. [Figure 2] It is a flowchart of the nasolabial shadow analysis method according to this embodiment. [Figure 3] It is a conceptual diagram showing a method for evaluating the raised state of a region near the nasolabial fold in this embodiment. [Figure 4] It is a graph showing the relationship between the range of the face region in a face image and the estimation accuracy of an AI model. [Figure 5] It is a graph showing the relationship between the vertical position of the face region in a face image and the estimation accuracy of an AI model. [Figure 6] It is a diagram showing an output example of the evaluation result of a nasolabial shadow. [Figure 7] It is a flowchart of the nasolabial shadow analysis method according to a modification example. [Figure 8] It is a diagram showing an output example of a first evaluation value, a second evaluation value, a first target value, and a second target value. [Figure 9] It is a flowchart of the learning method of the nasolabial shadow evaluation model according to this embodiment. [Figure 10] FIG. 10(a) is a graph showing the relationship between a visual evaluation value regarding a nasolabial fold grade and an estimated value (second evaluation value) of an AI model, and FIG. 10(b) is a graph showing the relationship between a visual evaluation value regarding a bulge grade and an estimated value (first evaluation value) of an AI model. [Figure 11] FIG. 11(a) is a graph showing the distribution of each evaluation value regarding a nasolabial fold grade and a bulge grade estimated by an AI model based on a test face image, and FIG. 11(b) is a graph showing the result of applying linear correction to the evaluation values in FIG. 11(a). [Figure 12]FIG. 12(a) is a diagram showing representative nasolabial profile images in each of nine nasolabial profile classes, and FIG. 12(b) is a diagram showing the distribution of nine nasolabial profile classes among 213 sample providers excluding sample providers with a nasolabial fold grade of "0". [Figure 13] It is a diagram showing the relationship between the distribution of the sample providers shown in FIG. 12(b) and the satisfaction of each sample provider with a specific formulation.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, examples of preferred embodiments of the present invention (hereinafter referred to as "the present embodiment") will be described. The embodiments described below are illustrative, and the present invention is not limited to the configurations of the following embodiments.

[0011] The nasolabial profile analysis method according to the present embodiment (hereinafter referred to as "the present analysis method") and the learning method of the nasolabial profile evaluation model (hereinafter referred to as "the present learning method") are executed by one or more processors included in one or more information processing devices. FIG. 1 conceptually shows an example of the hardware configuration of an information processing device 10 capable of executing the present analysis method and the present learning method. The information processing device 10 is a so-called computer and has a CPU 11, a memory 12, an input / output interface (I / F) 13, a communication unit 14, and the like. The information processing device 10 may be a desktop PC (Personal Computer), a portable terminal such as a portable PC, a smartphone, or a tablet, or a dedicated computer.

[0012] The CPU 11 is a so-called processor, and in addition to a general CPU (Central Processing Unit), it may also include an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), and the like. Memory 12 includes RAM (Random Access Memory), ROM (Read Only Memory), and auxiliary storage devices (such as hard disks).

[0013] The input / output interface 13 can be connected to user interface devices such as a display device 15 and an input device 16. The display device 15 is a device that displays a screen corresponding to drawing data processed by the CPU 11, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display. The input device 16 is a device that accepts user input such as a keyboard or mouse. The display device 15 and the input device 16 may be integrated and implemented as a touch panel. The communication unit 14 handles communication with other computers via a communication network and exchanges signals with other devices such as printers. Portable recording media and the like may also be connected to the communication unit 14.

[0014] The hardware configuration of the information processing device 10 is not limited to the example in Figure 1. The information processing device 10 may include other hardware elements not shown. Furthermore, the number of each hardware element is not limited to the example in Figure 1. For example, the information processing device 10 may have multiple CPUs 11. Also, the information processing device 10 may be implemented by multiple computers consisting of multiple enclosures.

[0015] The information processing device 10 can execute the analysis method and the learning method by having the CPU 11 execute a computer program stored in the memory 12. The computer program is installed, for example, from a portable recording medium such as a CD (Compact Disc) or memory card, or from another computer on a network, via the input / output interface 13 or communication unit 14, and stored in the memory 12. For the sake of explanation, this specification provides an example in which the analysis method and the learning method are performed on a common information processing device 10. However, the analysis method and the learning method may be performed on separate information processing devices having the same or different hardware configurations. Furthermore, the analysis method and the learning method may be implemented as separate computer programs.

[0016] Next, we will explain the overview of this analysis method, and then provide further details on this analysis method and this learning method. This analysis method analyzes nasolabial shadows, which are shadows that appear at the boundary between the nose and the cheek, and is executed by an information processing device 10 (CPU 11) as illustrated in Figure 1. In the following description, the CPU 11 will be used as the main entity executing this analysis method and this learning method.

[0017] This analysis method includes at least an image acquisition step and an evaluation value acquisition step. In the image acquisition process, the CPU 11 acquires an evaluation face image that shows the face of the person being evaluated. As will be explained in detail later, the facial image used for evaluation can be any image showing the face of the person being evaluated, and may also include other parts of the body (such as the neck or hair). The facial image used for evaluation is acquired as image data in formats such as JPEG (Joint Photographic Experts Group), BMP (Bitmap image), TIFF (Tagged Image File Format), or GIF (Graphic Interchange Format). However, the data format of the facial image used for evaluation is not restricted. Furthermore, it is preferable that the acquired facial image used for evaluation be a color image captured by a camera that captures visible light, but it may also be a grayscale image if it is possible to analyze the nasolabial shadow. The CPU 11 may acquire the evaluation facial image from the camera that captured the image, or it may acquire the image from another computer or portable recording medium.

[0018] In the evaluation value acquisition process, the CPU 11 inputs the evaluation face image acquired in the image acquisition process into a trained model that has been machine-learned based on training data from multiple people, thereby obtaining a first evaluation value that indicates the degree of elevation of the area near the nasolabial folds on the cheeks of the person being evaluated. The trained model outputs an evaluation value (first evaluation value) that indicates the degree of elevation in the area near the nasolabial folds of the cheek in the input facial image. For this reason, the trained model can also be referred to as a trained nasolabial shadow evaluation model. The first evaluation value indicates the degree of elevation of the cheek area adjacent to the nasolabial folds, which can create shadows (nasolabial shadows) at the boundary between the nose and the area below the nose and the cheek, similar to nasolabial folds.

[0019] Thus, in this embodiment, as a factor of nasolabial shadows that affect the impression of facial aging, attention is paid to the elevation of the area near the nasolabial folds on the cheeks, separate from the nasolabial folds themselves, and an evaluation value of that elevation is obtained. This allows for a proper analysis of the condition of nasolabial shadows, even in cases where the depth and length of the nasolabial folds are not particularly large, but the prominent elevation of the area near the nasolabial folds on the cheeks makes the nasolabial shadows noticeable, giving the face an aged appearance.

[0020] However, as will be described later, the trained model may be configured to output an evaluation value (second evaluation value) that indicates the state of the nasolabial folds in addition to the first evaluation value. According to this method, since evaluation values ​​can be obtained for both the condition of the nasolabial folds and the degree of elevation of the area near the nasolabial folds on the cheeks as factors that influence the impression of facial aging, the condition of the nasolabial folds can be analyzed with greater accuracy.

[0021] In this embodiment, the "trained model" is a model obtained through machine learning using training data, i.e., supervised learning, and can be described as an AI (Artificial Intelligence) model, a Machine Learning (ML) model, etc. A trained model may be, for example, a regression equation obtained from regression analysis, or it may consist of a convolutional neural network including a deep neural network, and it can be formed by a combination of a computer program and parameters, or a combination of multiple functions and parameters. Furthermore, when the trained model is constructed using a neural network, and the input layer, hidden layer, and output layer are considered as a single neural network unit, it may refer to a single neural network or a combination of multiple neural networks. Also, the trained model may consist of a combination of multiple multiple regression equations or a single multiple regression equation. Furthermore, the trained model may consist of a combination of a neural network and other models such as regression equations or discriminant equations, and the value obtained as the output of the neural network may be input to the other model and the resulting value may be used as the output.

[0022] The trained model may be stored in the memory 12 within the information processing device 10, or it may be stored in the memory of another computer that the information processing device 10 can access via communication. Thus, the information processing device 10 can be described as a nasolabial shadow analysis device that can utilize a trained model and perform the analysis method, or it can be described as a nasolabial shadow evaluation model learning device that can learn the nasolabial shadow evaluation model.

[0023] The following will provide a more detailed explanation of this analysis method and learning method. In the following explanation, we will omit any content that is the same as what was outlined above.

[0024] [Nasolabial shadow analysis method (this analysis method)] Figure 2 is a flowchart of the nasolabial shadow analysis method according to this embodiment. As described above, this analysis method is performed by an information processing device 10 (CPU 11) as illustrated in Figure 1. As mentioned above, CPU11 can utilize the pre-trained nasolabial shadow evaluation model. Hereafter, the pre-trained nasolabial shadow evaluation model will be abbreviated as the AI ​​model.

[0025] Before detailing this analysis method, let's first explain the AI ​​model. The AI ​​model takes a facial image as input and outputs an evaluation value indicating the nasolabial lateral shadow state of the facial image. As outlined, the output of the AI ​​model may consist only of a first evaluation value indicating the prominence of the area near the nasolabial folds of the cheek, or it may further include a second evaluation value indicating the state of the nasolabial folds. In this embodiment, the AI ​​model takes a facial image as input and outputs a first evaluation value indicating the prominence of the area near the nasolabial folds of the cheek and a second evaluation value indicating the state of the nasolabial folds.

[0026] The first and second evaluation values ​​output by the AI ​​model may be integer values ​​similar to the visual evaluation values ​​given as correct information for the training data described later, or they may be expressed as decimal numbers. For example, in the former case, the first and second evaluation values ​​are scores (integer values ​​between 0 and 7) that score the degree of cheek fullness (protrusion) on the outer side near the nasolabial folds on an 8-point scale, and scores (integer values ​​between 0 and 7) that score the degree of depth of the nasolabial fold grooves on an 8-point scale. In the latter case, the first and second evaluation values ​​are, for example, numerical values ​​including decimals between 0 and 7.

[0027] The AI ​​model is trained using machine learning based on training data that includes multiple datasets of face images and ground truth information about the state of the nasolabial shadows of the faces in those images. In this embodiment, the correct information for the state of the nasolabial shadow includes the visual evaluation value of the elevation state of the area near the nasolabial folds on the cheek and the visual evaluation value of the nasolabial folds on the face of the human face captured in the facial image. However, as outlined, if only the first evaluation value indicating the elevation state of the area near the nasolabial folds on the cheek is obtained from the AI ​​model, the correct information may include only the visual evaluation value of the elevation state of the area near the nasolabial folds on the cheek of the human face captured in the facial image.

[0028] The visual evaluation value for the state of nasolabial folds included in the correct answer information is a score obtained by the evaluator assessing the state of nasolabial folds by looking at the facial image. This score is set by evaluating the degree of depth of the nasolabial fold grooves according to a predetermined evaluation method. In this embodiment, as an example of this evaluation method, the evaluation range is defined as the area extending from the outer edge of the nostrils to both ends of the lips in the nasolabial folds, and the depth of the wrinkles referred to as nasolabial folds is the evaluation target. If multiple wrinkles exist, the deepest wrinkle is the evaluation target. In this evaluation method, an 8-level score from Grade 0 to Grade 7 is assigned as a visual evaluation value. That is, the shallower the nasolabial fold groove, the lower the grade (score) assigned, and the deeper the nasolabial fold groove, the higher the grade (score) assigned. For this reason, in the following explanation, the evaluation value indicating the condition of the nasolabial folds may be referred to as the nasolabial fold grade.

[0029] However, the method for evaluating the condition of nasolabial folds is not limited to these examples. For example, the evaluation method may involve the evaluator directly observing the data provider's face in the facial image. Furthermore, the evaluation method may include not only the depth of the nasolabial folds but also their length, or the number of nasolabial folds (such as double or triple) as an evaluation criterion. Also, the grading of the nasolabial fold condition is not limited to eight levels; it may be graded in fewer than eight levels or in nine or more levels.

[0030] The visual evaluation value of the elevation of the area near the nasolabial folds of the cheeks, included in the correct answer information, is a score obtained by the evaluator looking at the facial image and evaluating the elevation of the area near the nasolabial folds of the cheeks. This score is set by evaluating the degree of elevation of the area near the nasolabial folds of the cheeks according to a predetermined evaluation method.

[0031] Figure 3 is a conceptual diagram illustrating the method for evaluating the degree of prominence in the area near the nasolabial folds of the cheeks in this embodiment. In this embodiment, as shown in Figure 3, as an example of this evaluation method, the area outside the vicinity of the nasolabial folds (the oval frame in Figure 3), extending from the outside of the nasolabial wing to both sides of the lips, is used as the evaluation range, and the degree of cheek fullness (prominence) in the area outside the vicinity of the nasolabial folds is scored on an 8-point scale from grade 0 to grade 7. That is, the smaller the cheek fullness (prominence) in the evaluation range, the lower the grade (score) is assigned, and the larger the fullness (prominence), the higher the grade (score) is assigned. For this reason, in the following explanation, the evaluation value indicating the degree of prominence in the area near the nasolabial folds of the cheeks may be referred to as the fullness grade. Figure 3 shows images of the prominence corresponding to grades 0, 4, and 7.

[0032] However, the method for evaluating the degree of elevation in the area near the nasolabial folds of the cheeks is not limited to these examples. For example, the evaluation method may involve the evaluator directly observing the face of the data provider as it appears in the facial image. Furthermore, the grading of the degree of elevation in the area near the nasolabial folds of the cheeks is not limited to eight levels; it may be graded on fewer than eight levels or on nine or more levels.

[0033] While a facial image used as input for an AI model only needs to show a human face, particularly the area where the nasolabial shadow appears (the nose) and the boundary between the area below the nose and the cheek, the inventors have newly discovered that the extent of the facial region within the facial image used as input for the AI ​​model affects the estimation accuracy of that AI model.

[0034] Figure 4 is a graph showing the relationship between the extent of the facial region in the facial image and the estimation accuracy of the AI ​​model, and Figure 5 is a graph showing the relationship between the vertical position of the facial region in the facial image and the estimation accuracy of the AI ​​model. In Figures 4 and 5, the symbols S1 to S7 and L1 to L6 represent the morphology of the facial image used as input to the AI ​​model, respectively, and the image size is standardized for each morphology.

[0035] The face image labeled S7 includes the entire face, head, and neck, displaying approximately the entire head. Each of the face images labeled S6 to S1 is an enlarged image in which the overall image corresponds to a partial image area of ​​800 pixels, 600 pixels, 550 pixels, 500 pixels, 450 pixels, and 400 pixels within the face image labeled S7, which has a total size of 1200 pixels. In the graph in Figure 4, the horizontal axis shows the size of the human face area range in pixels, based on the face image labeled S7, and the vertical axis shows the estimation error of the nasolabial fold grade and puffiness grade of the AI ​​model when each type of face image is input.

[0036] The face image labeled S6 does not include the head, but displays the eyebrows, eyes, and the area below the eyes, including the overall contour of the face. In contrast, the face images labeled S5 to S1 do not include the eyes or the area above the eyes. Furthermore, the face image labeled S1 does not include the contour of the chin, while the face images labeled S2 to S5 include more facial contours, mainly around the chin, in the order of S2, S3, S4, and S5. On the other hand, the estimation error of the AI ​​model was smallest for the face image with code S5, and increased in the order of code S4, code S3, code S6, code S2, code S7, and code S1.

[0037] In Figure 5, the images at each position are shown, labeled L1 to L6, when the reference cropping range of the face image labeled S5 relative to the face image labeled S7 in Figure 4 is moved upward (towards the head) while maintaining the size of that reference cropping range. The horizontal axis of Figure 5 shows the distance moved upward (towards the head) and downward (towards the neck) from the reference position (0), with the position of the face region indicated by the face image labeled L3 as the reference position. Specifically, positive values ​​indicate the distance moved downward (towards the neck), and negative values ​​indicate the distance moved upward (towards the head). In contrast, the vertical axis of Figure 5 shows the estimation error of the nasolabial fold grade and puffiness grade of the AI ​​model when each type of face image is input.

[0038] As a result, the estimation error of the AI ​​model was smallest for the face image with code S5, and increased in the order of code L6, code L5, code L4, code L3, code L2, and code L1. In this way, by normalizing the facial images used as input to the AI ​​model so that the area below the eyes, excluding the eyes and the area above the eyes, contains more of the facial contour lines, mainly around the jawline, the estimation accuracy of the AI ​​model can be improved. These requirements for facial images apply not only to the facial images used as training data, but also to the facial images of the individuals being evaluated, as will be discussed later.

[0039] Furthermore, it is preferable that the training data utilizes multiple facial images obtained by photographing the same person under various shooting conditions. For example, various shooting conditions can be adopted, such as differences in hairstyle, whether the face is bare or made up, differences in makeup application, differences in shooting location (including differences in lighting), and whether or not the image is a selfie. In this case, the same group of facial images of the same person will be labeled with the same correct information (nasolabial fold grade and puffiness grade). By using such facial images as training data, it becomes possible to perform highly accurate analysis of nasolabial shadows on evaluation facial images taken under various shooting conditions, without limiting the shooting conditions of the evaluation facial images of the person being evaluated. This AI model only needs to be trained using the learning method described below.

[0040] In this analysis method, first, the CPU 11 acquires an evaluation face image showing the face of the person being evaluated (S21). As mentioned above, there are no restrictions on the image size, data format, or shooting conditions of the facial images used for evaluation acquired in step (S21). However, as also mentioned above, it is preferable that the facial images used for evaluation are normalized so that the parts of the face below the eyes, excluding the eyes and the parts above the eyes, are displayed with the facial contour line centered around the jaw.

[0041] The CPU 11 may acquire evaluation face images from a camera that has captured the face of the person being evaluated, or it may acquire evaluation face images from another computer or portable recording medium. Furthermore, the CPU 11 may apply face detection processing to images acquired from a camera or other computer, extract an appropriate face image region from that image, and then apply further image processing such as enlargement, reduction, and cropping to the extracted face image to obtain a normalized evaluation face image. Such normalization processing, including face detection and image processing, may be performed by the CPU 11 or by another computer.

[0042] Next, the CPU 11 inputs the facial image for evaluation acquired in process (S21) into the AI ​​model (S22) to obtain a first evaluation value (puffiness grade) indicating the degree of prominence of the area near the nasolabial folds on the cheeks of the person being evaluated, and a second evaluation value (nasolabial fold grade) indicating the condition of the nasolabial folds on the person being evaluated (S23).

[0043] Next, the CPU 11 outputs the evaluation result of the nasolabial shadow of the person being evaluated based on the first and second evaluation values ​​acquired in process (S23) (S24). The CPU 11 can display the evaluation result on the display device 15, output the evaluation result to a printer connected via the input / output I / F 13 or communication unit 14, or record or transmit the evaluation result to a portable recording medium or another computer via the input / output I / F 13 or communication unit 14. Thus, the destination and method of outputting the evaluation result are not limited in any way.

[0044] Furthermore, the evaluation results for the nasolabial shadow may be output in any format, as long as they are based on the first and second evaluation values ​​obtained in process (S23). For example, the first and second evaluation values ​​obtained in process (S23) may be output along with their item names (nasolabial fold grade and puffiness grade), or the first and second evaluation values ​​obtained in process (S23) may be converted into a visual evaluation value format (integer value) and output along with the item name. For example, in the latter case, if 2.35 is obtained as the first evaluation value, the number "2" obtained by rounding or the like may be output as the puffiness grade value.

[0045] In this embodiment, the evaluation results for the nasolabial shadow are output based on a first evaluation value (puffiness grade) indicating the degree of elevation of the area near the nasolabial folds of the cheeks of the person being evaluated, and a second evaluation value (nasolabial fold grade) indicating the condition of the nasolabial folds in the person being evaluated. Therefore, according to this embodiment, it is possible to distinguish between the state of nasolabial folds and the elevation of the area near the nasolabial folds on the cheek, which are factors of nasolabial shadows that affect the impression of facial aging, and to obtain evaluation results of nasolabial shadows based on the evaluation values ​​of each of the two factors, thereby obtaining results that evaluate the state of nasolabial shadows in more detail.

[0046] Furthermore, the evaluation results for the nasolabial shadow may be presented as an evaluation graph on which the evaluation data of the person being evaluated, shown by the first and second evaluation values ​​obtained in process (S23), are plotted on a graph that has two axes: one axis showing the degree of elevation of the area near the nasolabial folds of the cheek and another axis showing the state of the nasolabial folds. By outputting such an evaluation graph, the person being evaluated can easily understand the position of their nasolabial shadow condition based on the two factors of the nasolabial shadow.

[0047] Figure 6 shows an example of the output of the evaluation results for nasolabial shadows. Figure 6 also shows an example of the display screen on the display device 15. In the display screen of Figure 6, the first and second evaluation values ​​obtained in process (S23) are shown in the lower left corner along with their respective item names, "nasolabial fold grade" and "puffiness grade." Specifically, the first evaluation value is "1.58" and the second evaluation value is "2.59." Additionally, a two-axis evaluation graph is displayed on the right. This evaluation graph has a horizontal axis representing the nasolabial fold grade and a vertical axis representing the fullness grade, and the evaluation data of the person being evaluated is plotted based on the first and second evaluation values. Furthermore, an image of the person being evaluated is displayed in the upper left corner.

[0048] In the example shown in Figure 6, the CPU 11 sequentially acquires evaluation face images from the displayed image and inputs these evaluation face images into the AI ​​model, thereby sequentially acquiring and displaying the first and second evaluation values. In this way, the person being evaluated can view their own face on a displayed image and obtain real-time evaluation results for their nasolabial shadows based on that image. This allows them to change their facial expressions or apply makeup while obtaining evaluation results for their nasolabial shadows each time.

[0049] Furthermore, between steps (S23) and (S24), the CPU 11 may further execute a class determination step to determine the nasolabial shadow class to which the person being evaluated belongs from among a plurality of nasolabial shadow classes classified by a combination of the elevation state of the area near the nasolabial folds of the cheek and the state of the nasolabial folds, using the first and second evaluation values ​​obtained in step (S23), and output the determined nasolabial shadow class as the evaluation result. For example, identification information (ID, name, etc.) of the nasolabial shadow class to which the person being evaluated belongs can be output.

[0050] For example, the information processing device 10 may store a discrimination formula or discrimination criteria for the nasolabial shadow class in memory 12 in advance, and the CPU 11 may determine the nasolabial shadow class to which the person being evaluated belongs by applying the first evaluation value and the second evaluation value acquired in process (S23) to the discrimination criteria or discrimination criteria. As will be described later as an example, the multiple nasolabial shadow classes can be associated with each of the multiple regions formed by dividing the first quadrant of a graph formed by two axes: an axis indicating the nasolabial fold grade and an axis indicating the puffiness grade. For example, when classified into four nasolabial shadow classes, the above discrimination conditions can be conditioned as follows: a nasolabial shadow class where the puffiness grade is 0 or more and 4 or less and the nasolabial fold grade is 0 or more and 3 or less; a nasolabial shadow class where the puffiness grade is 5 or more and 7 or less and the nasolabial fold grade is 4 or more and 7 or less; and a nasolabial shadow class where the puffiness grade is 5 or more and 7 or less and the nasolabial fold grade is 4 or more and 7 or less. However, the discrimination conditions or discriminant formulas for nasolabial shadow classes are not limited to these examples.

[0051] Furthermore, the AI ​​model described above may be formed as a discrimination model that determines the nasolabial shadow class for the input evaluation face image, or the CPU 11 may also utilize a discrimination model that determines the nasolabial shadow class by inputting the first and second evaluation values ​​or the evaluation face image obtained in step (S23), separate from the AI ​​model described above. In the former case, the CPU 11 may execute a step to acquire information on the nasolabial shadow class to which the person being evaluated belongs, instead of step (S23), and in step (S24), it may output information on the nasolabial shadow class to which the person being evaluated belongs as the evaluation result.

[0052] According to this embodiment, each nasolabial shadow class can be characterized by the state of two factors of the nasolabial shadow (nasolabial fold grade and puffiness grade). Therefore, by outputting a nasolabial shadow class as an evaluation result, the person being evaluated can easily understand the position of their nasolabial shadow state based on the two factors of the nasolabial shadow.

[0053] Furthermore, the CPU 11 may also perform a step of outputting recommended information for the person being evaluated, in addition to or instead of outputting the evaluation results of the nasolabial shadow in step (S24). The information processing device 10 stores in memory 12 multiple recommendation pieces of information in advance, corresponding to each combination of the elevation state of the area near the nasolabial folds of the cheek and the state of the nasolabial folds, regarding formulations for reducing nasolabial shadows, methods of applying each formulation, or cosmetic procedures (excluding medical procedures), or combinations thereof. The CPU 11 can then extract and output the recommendation pieces from among these multiple recommendation pieces that correspond to the first evaluation value and the second evaluation value obtained in process (S23). In this case, each recommendation piece of information is stored in association with one or more nasolabial shadow classes as described above, and the CPU 11 may, after identifying the nasolabial shadow class of the person being evaluated, extract and output the recommendation pieces from memory 12 that correspond to that nasolabial shadow class. Alternatively, the CPU 11 may weight the first and second evaluation values ​​obtained in process (S23) based on the characteristics of the person being evaluated (age, gender, blood type, etc.), and extract recommendation information based on these weighted values.

[0054] In this embodiment, the specific content of the recommended information is not limited. For example, the recommended information may include information on formulations that reduce nasolabial shadows by being applied to fill in the nasolabial folds, or formulations that reduce nasolabial shadows by pulling the skin of the cheeks to suppress the protrusion of the area near the nasolabial folds. Furthermore, the application method for each formulation may include information specifying the area to which the corresponding formulation is applied, the range and size of the application area, etc. Cosmetic procedures may include procedures to suppress the protrusion of the area near the nasolabial folds on the cheeks. By outputting this kind of recommendation information, we can provide meaningful information to those being evaluated who are troubled by nasolabial shadows.

[0055] [Variation] As mentioned above, this analysis method is not limited to the flow shown in Figure 3, and the flow shown in Figure 3 can be modified as appropriate. For example, a method for evaluating the visual impression of the nasolabial shadow can be realized, which includes at least steps (S21), (S22), and (S23). In this evaluation method, the visual impression of the nasolabial shadow is evaluated using the first and second evaluation values ​​obtained in step (S23). The visual impression of the nasolabial shadow may be output as binary information (good or bad), as an evaluation value with three or more levels (young, age-appropriate, aged), or as information indicating apparent age. It may be said that the term "visual impression of nasolabial folds" is sometimes used to describe an evaluation that uses at least the degree of elevation of the area near the nasolabial folds on the cheeks and the condition of the nasolabial folds (wrinkles or grooves). However, even in this case, if the evaluation uses at least the degree of elevation of the area near the nasolabial folds on the cheeks and the condition of the nasolabial folds (wrinkles or grooves), then the evaluation subject would fall under the "visual impression of nasolabial shadows" mentioned above.

[0056] Figure 7 is a flowchart of the nasolabial shadow analysis method for modified shapes. Hereafter, the nasolabial shadow analysis method for modified shapes may also be referred to as the deformation analysis method. This deformation analysis method, like the analysis method described above, is executed by an information processing device 10 (CPU 11) as illustrated in Figure 1. The deformation analysis method further includes steps (S71), (S72), (S73), and (S74) in addition to the steps (S21), (S22), (S23), and (S24) described above.

[0057] In step (S71), the CPU 11 acquires a target face image of the person being evaluated. The target face image is a face image in which the state of the nasolabial shadow is in the state desired by the person being evaluated (target state). Preferably, the image size and data format of the target face image are the same as those of the evaluation face image described above, and the facial area captured in it is also the same as that of the evaluation face image.

[0058] The target face image may be an image taken while the subject's face is deformed, for example, by pulling the skin with their hands. In this case, for example, the subject may deform their face to achieve their desired facial state (target state) while looking at the displayed image of the subject's face on the screen as shown in Figure 6, and the CPU 11 may acquire the image of that deformed face as the target face image. Alternatively, the target face image may be obtained by the CPU 11 performing image processing that deforms the face image in response to input operations on the face image of the person being evaluated displayed on the display device 15.

[0059] The CPU 11 may acquire target face images from the camera, similar to the evaluation face images, or it may acquire target face images from another computer or portable recording medium. Furthermore, the CPU 11 can apply face detection processing to images acquired from the camera or other computers, extract an appropriate face image region from that image, and then perform further image processing such as enlargement, reduction, and cropping on the extracted face image to obtain a normalized target face image. Such normalization processing, including face detection and image processing, may be performed by the CPU 11 or by another computer.

[0060] In step (S72), the CPU 11 inputs the target face image acquired in step (S71) into the AI ​​model, and in step (S73), the CPU 11 obtains from the AI ​​model a first target value (target puffiness grade) indicating the degree of elevation of the area near the nasolabial folds of the cheek, which corresponds to the target state of the nasolabial folds, and a second target value (target nasolabial fold grade) indicating the state of the nasolabial folds. The first target value and the second target value are output values ​​of the same AI model as the first evaluation value and the second evaluation value. The only difference between them and the first evaluation value and the second evaluation value is whether the face image input to the AI ​​model is an evaluation face image or a target face image. Therefore, the first target value can be rephrased as the first evaluation value obtained from the AI ​​model by inputting a target face image, and the second target value can be rephrased as the second evaluation value obtained from the AI ​​model by inputting a target face image.

[0061] In step (S74), the CPU 11 outputs the first evaluation value and second evaluation value obtained in step (S23) and the first target value and second target value obtained in step (S73) in a comparable manner. The output destination, output method, and output format in step (S74) are not limited in any way, as long as the first evaluation value and second evaluation value (hereinafter sometimes collectively referred to as evaluation values) and the first target value and second target value (hereinafter sometimes collectively referred to as target values) are comparable. The evaluation value and target value may be output side by side, or they may be output as plotted on a graph with two axes: one indicating the nasolabial fold grade and the other indicating the fullness grade. Furthermore, the CPU 11 may calculate the distance between the first target value and the first evaluation value, and the distance between the second target value and the second evaluation value. Alternatively, it may calculate the target achievement rate from the distance between the evaluation value and the target value obtained from the facial image for evaluation at any given time, with each distance representing 100% of the target achievement rate. In this case, the CPU 11 may output each distance, or it may output the target achievement rate.

[0062] Figure 8 shows an example of the output of the first evaluation value and second evaluation value, and the first target value and second target value. Figure 8 also shows an example of the display screen on the display device 15. This display screen presents the effectiveness of recommended methods for reducing nasolabial shadows (hereinafter sometimes referred to as "recommended methods") to the person being evaluated. Recommended methods include using a preparation that can reduce nasolabial shadows, the method of applying the preparation, or a cosmetic procedure, and are displayed in display area G5. Additionally, the target face image is displayed in display area G3, the evaluation face image before using the recommended method is displayed in display area G1, and the evaluation face image after using the recommended method is displayed in display area G2.

[0063] Display area G4 shows a graph formed by two axes: one indicating the nasolabial fold grade and another indicating the puffiness grade. On this graph, points representing the target value obtained from the target face image in display area G3 (labeled "Target"), points representing the evaluation value obtained from the evaluation face image in display area G1 (labeled "Initial"), and points representing the evaluation value obtained from the evaluation face image in display area G2 (labeled "Current") are plotted. Furthermore, a 46% achievement rate is also displayed in display area G4. This achievement rate is calculated from the distance between the "Current" evaluation value and the target value, assuming that the distance between the "Initial" evaluation value and the target value represents a 100% achievement rate. Display area G10 shows the date and time information of when the evaluation face image in display area G2 was captured, as well as the achievement rate calculated for that evaluation face image.

[0064] This deformation analysis method allows the person being evaluated to easily understand the difference between the desired state of nasolabial shadow and the state of nasolabial shadow at a given point in time. Furthermore, it can clearly present the effectiveness of recommended methods for reducing nasolabial shadow.

[0065] Figure 7 shows multiple processes in sequence, but the execution order of each process in the deformation analysis method is not limited to the example in Figure 7. For example, processes (S71) to (S73) may be executed in parallel with processes (S21) to (S24), or before them, or they may be executed only once in advance, asynchronously with the timing of their execution. Alternatively, an image generation process may be performed that takes a target first evaluation value (also called the first target value) and a target second evaluation value (also called the second target value) as inputs and generates a face image having a nasolabial lateral shadow state corresponding to those evaluation values, and the output face image may be acquired as the target face image. In this case, after executing step (S73), the CPU 11 may execute the image generation process using the first target value and second target value acquired in step (S73) as inputs, and thereby execute step (S71).

[0066] [Training method for nasolabial shadow evaluation model (this training method)] Next, we will explain the training method of the AI ​​model used in the analysis method and deformation analysis method described above, using Figure 9. Figure 9 is a flowchart of the learning method for the nasolabial shadow evaluation model according to this embodiment. This learning method is executed by an information processing device 10 (CPU 11) as illustrated in Figure 1. Here, an example is given of an AI model composed of a deep neural network that outputs a first evaluation value indicating the degree of elevation of the area near the nasolabial folds on the cheeks and a second evaluation value indicating the state of the nasolabial folds in a face image, in response to an input face image. When this learning method is executed, this AI model may be stored in the memory 12 of the information processing device 10, or it may be stored in the memory of another computer that the information processing device 10 can access via communication.

[0067] First, the CPU 11 acquires training data for multiple people, which consists of combinations of a face image and correct information including visual evaluation values ​​of the degree of elevation of the area near the nasolabial folds on the cheeks and visual evaluation values ​​of the state of the nasolabial folds related to the face image (S91). The visual evaluation values ​​of the degree of elevation of the area near the nasolabial folds on the cheeks and the state of the nasolabial folds included in the correct information are as described above. The face images used in the training data are also as described above.

[0068] Next, CPU 11 trains the AI ​​model using the training data acquired in step (S91) (S92). Here, the AI ​​model is trained using deep learning. However, the specific training algorithm for the AI ​​model is not limited in any way.

[0069] Some or all of the above embodiments and modifications may also be specified as follows; however, the above embodiments and modifications are not limited to those described below.

[0070] <1> A method for analyzing nasolabial shadows, which are shadows that appear at the boundary between the nose and the area below the nose and the cheek, One or more processors The process involves acquiring an image of the person being evaluated, which includes obtaining an evaluation image of their face, An evaluation value acquisition step involves inputting the acquired facial image for evaluation into a trained model that has been machine-learned based on training data from multiple individuals, thereby acquiring a first evaluation value that indicates the degree of elevation of the area near the nasolabial folds of the cheeks of the person being evaluated. A method for performing nasolabial shadow analysis.

[0071] <2> In the evaluation value acquisition step, in addition to the first evaluation value, a second evaluation value indicating the condition of the nasolabial folds of the person being evaluated is further acquired. <1> The method for analyzing nasolabial shadows as described above. <3> The aforementioned one or more processors A class determination step in which, using the first and second evaluation values ​​obtained, the nasolabial shadow class to which the person being evaluated belongs is determined from among a plurality of nasolabial shadow classes classified by the combination of the elevation state of the area near the nasolabial folds of the cheek and the state of the nasolabial folds. Perform further <2> The method for analyzing nasolabial shadows as described above. <4> The aforementioned one or more processors A step of outputting an evaluation graph on which the evaluation data of the person being evaluated, indicated by the acquired first evaluation value and second evaluation value, is plotted on a graph that has two axes: one axis indicating the degree of elevation of the area near the nasolabial folds of the cheek and another axis indicating the state of the nasolabial folds. Perform further <2> or <3> The method for analyzing nasolabial shadows as described above. <5> The facial image for evaluation acquired in the aforementioned image acquisition process is normalized so that the part of the face of the person being evaluated, excluding the eyes and the part above the eyes, is displayed with the facial contour line centered around the jawline. <1> from <4> The nasolabial shadow analysis method described in any one of the following. <6> The aforementioned one or more processors A step of outputting recommended information corresponding to the obtained first and second evaluation values ​​from among multiple recommended information regarding a formulation for reducing nasolabial shadows, a method of applying the formulation, a cosmetic procedure (excluding medical procedures), or a combination thereof, corresponding to each combination of the degree of elevation of the area near the nasolabial folds of the cheek and the state of the nasolabial folds. Perform further <2> from <5> The nasolabial shadow analysis method described in any one of the following. <7> The aforementioned one or more processors A step of acquiring a target facial image of the person to be evaluated, in which the state of the nasolabial shadow indicates the target state of the person to be evaluated, The process involves inputting the acquired target face image into the trained model to obtain a first target value indicating the elevation state of the area near the nasolabial folds of the cheek corresponding to the target state of the nasolabial shadow, and a second target value indicating the state of the nasolabial folds. A step of outputting the acquired first evaluation value and second evaluation value and the acquired first target value and second target value in a way that allows for comparison, Perform further <2> from <6> The nasolabial shadow analysis method described in any one of the following.

[0072] <8> A method for learning a nasolabial shadow evaluation model that can output a first evaluation value indicating the elevation state of the area near the nasolabial folds of the cheek in a facial image, and a second evaluation value indicating the state of the nasolabial folds, in response to an input facial image, One or more processors A step of acquiring training data that has multiple combinations of a face image and correct information including visual evaluation values ​​of the degree of elevation of the area near the nasolabial folds of the cheeks and visual evaluation values ​​of the state of the nasolabial folds of the face of the person depicted in the face image, A step of training the nasolabial shadow evaluation model using the acquired training data, A method for training a nasolabial shadow evaluation model.

[0073] <9> A nasolabial shadow analyzer comprising at least one processor and memory, <1> from <8> A nasolabial shadow analysis device capable of performing any one of the nasolabial shadow analysis methods described therein.

[0074] <10> A nasolabial shadow evaluation model learning device comprising at least one processor and memory, <8> A nasolabial shadow evaluation model learning device capable of performing the learning method for the nasolabial shadow evaluation model described above.

[0075] <11> A method for evaluating the visual impression of nasolabial shadows, which are shadows that appear at the boundary between the nose and the area below the nose and the cheek, One or more processors The process involves acquiring an image of the person being evaluated, which includes obtaining an evaluation image of their face, The evaluation value acquisition step involves inputting the acquired facial images for evaluation into a trained model that has been machine-learned based on training data from multiple individuals, thereby acquiring a first evaluation value indicating the degree of elevation of the area near the nasolabial folds of the cheeks of the person being evaluated, and a second evaluation value indicating the state of the nasolabial folds of the person being evaluated. A method for evaluating the visual impression of nasolabial shadows.

[0076] The following examples illustrate the above points in more detail. However, the following examples do not limit the scope of the above-mentioned points. [Examples]

[0077] In this example, an AI model was used that was constructed by fine-tuning a pre-trained model for ImageNet. This AI model was designed to take a facial image as input and output a nasolabial fold grade and a puffiness grade scored on a scale from 0 to 7. In this embodiment, 273 Japanese women aged 20 to 70 were used as sample providers, and 1166 facial images of these sample providers, along with visual evaluation values ​​indicating the degree of elevation in the area near the nasolabial folds of the cheeks and visual evaluation values ​​indicating the condition of the nasolabial folds, were prepared.

[0078] The facial images mentioned above were taken of the same person under various shooting conditions. Of the 1166 images mentioned above, 540 were obtained by researchers taking pictures of the same person's bare face and face with makeup on using a tablet device camera in the same environment, 273 were obtained by each individual taking a picture of their bare face using a shared smartphone camera in the same environment, and 353 were obtained by each individual taking pictures of their bare face and face with makeup on using their own smartphone camera at their own home. On the other hand, the visual evaluation values ​​were obtained by visually evaluating images of the profile faces of 273 individuals taken with a skin measurement device, and assigning a score from 0 to 7 to each person's nasolabial fold grade and puffiness grade.

[0079] The 1166 prepared facial images were divided into training (teaching data), validation, and test sets in a ratio of approximately 5:2.5:2.5. Each facial image was labeled with a visual evaluation value indicating the degree of elevation of the area near the nasolabial folds on the cheeks of the person depicted, and a visual evaluation value indicating the state of the nasolabial folds themselves. Furthermore, each facial image was normalized so that, as indicated by the symbol S5 in Figures 4 and 5, the area below the eyes, excluding the eyes and the area above the eyes, contains more of the facial contour lines, mainly around the jawline.

[0080] As described above, the AI ​​model was trained using the facial images and correct information (visual evaluation values) separated as training data. The AI ​​model with the smallest mean squared error (MSE) was selected using the data separated for validation, and the accuracy of that AI model was evaluated using the data separated for testing.

[0081] Figure 10(a) is a graph showing the relationship between the visually assessed value for nasolabial fold grade and the estimated value (second assessment value) from the AI ​​model, and Figure 10(b) is a graph showing the relationship between the visually assessed value for puffiness grade and the estimated value (first assessment value) from the AI ​​model. According to Figure 10, the coefficient of determination (R) is used to estimate the grade of nasolabial folds. 2The coefficient of determination (R) was shown to be 0.6848, and the estimation of the bulge grade was determined by the coefficient of determination (R). 2 ) is shown to be 0.5869. Thus, this embodiment demonstrates the high estimation accuracy of the AI ​​model.

[0082] Figure 11(a) is a graph showing the distribution of evaluation values ​​for nasolabial fold grade and puffiness grade estimated by the AI ​​model based on test facial images, and Figure 11(b) is a graph showing the results after applying linear correction to the evaluation values ​​in Figure 11(a). In Figure 11(b), nine data regions are shown by thick dashed lines, each representing a classification of the puffiness grade and nasolabial fold grade using thresholds of 2.5 and 4.5, respectively. Each data region corresponds to a nasolabial shadow class.

[0083] The evaluation values ​​in Figure 11(a) are the output of an AI model optimized to minimize the mean squared error (MSE). However, when the objective is to determine the nasolabial shadow class to which the person being evaluated belongs, the error within the same class estimated by the AI ​​model has little impact on this objective, so there is still room for optimization of the AI ​​model. For example, when classifying based on a nasolabial fold grade of 2.5, even if the visual evaluation value of the nasolabial fold grade for an image is estimated to be 1.5 or 0, the class to which it is classified will be the same. Therefore, an optimal linear correction coefficient was derived for the evaluation values ​​output from the AI ​​model so as to maximize the agreement rate for the nasolabial shadow class. Figure 11(b) shows the result of multiplying the evaluation values ​​in Figure 11(a) by the optimal linear correction coefficient derived in this way, and corresponds to an enlarged version of the dashed area in Figure 11(a).

[0084] Based on the evaluation values ​​of the puffiness grade and nasolabial fold grade, to which the linear correction shown in Figure 11(b) was applied, the classification of the nasolabial shadow class showed a classification accuracy of 55.5%. Thus, if the AI ​​model is constructed solely for the purpose of identifying the nasolabial shadow class to which the person being evaluated belongs, without outputting nasolabial fold grades and puffiness grades, the AI ​​model may be trained using optimization methods such as Cross Entropy Loss to maximize the class classification accuracy.

[0085] Figure 12 shows an example of one classification of nasolabial shadow classes. Figure 12(a) shows a representative image of the nasolabial shadow state in each of the nine nasolabial shadow classes, and Figure 12(b) shows the distribution of the nine nasolabial shadow classes among 213 sample providers, excluding those with a nasolabial fold grade of "0". As shown in the distribution of nasolabial shadow state images in Figure 12(a), it can be seen that the nasolabial shadow state is characterized by each nasolabial shadow class. Furthermore, as shown in the distribution of sample providers in Figure 12(b), although there is some collinearity in the nasolabial fold grade and puffiness grade, it can be seen that the 213 sample providers are widely distributed across the nine nasolabial shadow classes. This shows that the nasolabial shadow condition can be appropriately classified into nine nasolabial shadow classes based on the nasolabial fold grade and puffiness grade.

[0086] Figure 13 shows the relationship between the distribution of sample providers shown in Figure 12(b) and each sample provider's satisfaction with the specific formulation. In Figure 13, satisfaction with the effectiveness of the specific formulation was surveyed among the 213 sample providers shown in Figure 12. As a specific treatment, a preparation was used that reduces nasolabial shadows by pulling the skin of the cheek to suppress the protrusion near the nasolabial folds. As a result, among sample providers whose puffiness grade and nasolabial fold grade were both 3 or higher, less than 15% responded with "neither agree nor disagree," "not very satisfied," or "not satisfied," meaning that the number of dissatisfied sample providers was less than 15%. Therefore, it can be seen that the target of a specific drug can be identified according to the classification of nasolabial shadow classes shown in Figure 12. [Explanation of symbols]

[0087] 10 Information Processing Devices 11 CPU 12 memory 13 Input / Output Interfaces 14 Communication Unit 15 Display device 16 Input device

Claims

1. A method for analyzing nasolabial shadows, which are shadows that appear at the boundary between the nose and the area below the nose and the cheek, One or more processors The process involves acquiring an image of the person being evaluated, which includes obtaining an evaluation image of their face, The evaluation value acquisition step involves inputting the acquired facial images for evaluation into a trained model that has been machine-learned based on training data from multiple individuals, thereby acquiring a first evaluation value indicating the degree of elevation of the area near the nasolabial folds on the cheeks of the person being evaluated, and a second evaluation value indicating the state of the nasolabial folds on the person being evaluated. Execute, The aforementioned first evaluation value indicates the state of elevation in the area near the nasolabial fold, which is the part of the cheek adjacent to the outside of the nasolabial fold and is capable of producing the aforementioned nasolabial shadow in the same way as the nasolabial fold. The nasolabial shadow is analyzed using the first evaluation value and the second evaluation value. Nasolabial shadow analysis method.

2. The aforementioned one or more processors A class determination step in which, using the first and second evaluation values ​​obtained, the nasolabial shadow class to which the person being evaluated belongs is determined from among a plurality of nasolabial shadow classes classified by the combination of the elevation state of the area near the nasolabial folds of the cheek and the state of the nasolabial folds. The method for analyzing nasolabial shadows according to claim 1, further comprising the steps described above.

3. The aforementioned one or more processors A step of outputting an evaluation graph on which the evaluation data of the person being evaluated, indicated by the acquired first evaluation value and second evaluation value, is plotted on a graph that has two axes: one axis indicating the degree of elevation of the area near the nasolabial folds of the cheek and another axis indicating the state of the nasolabial folds. The method for analyzing nasolabial shadows according to claim 1, further comprising the steps described above.

4. The facial image for evaluation acquired in the aforementioned image acquisition process is normalized so that the part of the face of the person being evaluated, excluding the eyes and the part above the eyes, is displayed with the facial contour line centered around the jawline. A method for analyzing nasolabial shadows according to any one of claims 1 to 3.

5. The aforementioned one or more processors A step of outputting recommended information corresponding to the obtained first and second evaluation values ​​from among multiple recommended information regarding a preparation for reducing nasolabial shadows, a method of applying the preparation, or a cosmetic procedure (excluding medical procedures), or a combination thereof, corresponding to each combination of the degree of elevation of the area near the nasolabial folds of the cheek and the state of the nasolabial folds. A method for analyzing nasolabial shadows according to any one of claims 1 to 3, further comprising the steps described above.

6. A method for learning a nasolabial shadow evaluation model that can output a first evaluation value indicating the elevation state of the area near the nasolabial folds of the cheek in a facial image, and a second evaluation value indicating the state of the nasolabial folds, in response to an input facial image, One or more processors A step of acquiring training data that has multiple combinations of a face image and correct information including visual evaluation values ​​of the degree of elevation of the area near the nasolabial folds of the cheeks and visual evaluation values ​​of the state of the nasolabial folds of the face of the person depicted in the face image, A step of training the nasolabial shadow evaluation model using the acquired training data, Execute, Nasolabial shadows are shadows that appear at the boundary between the nose and the area below the nose and the cheek. The aforementioned area near the nasolabial fold is the part of the cheek adjacent to the outside of the nasolabial fold and is an area that, like the nasolabial fold, can produce a nasolabial shadow. A training method for a nasolabial shadow evaluation model.

7. A nasolabial shadow analyzer comprising at least one processor and memory, A nasolabial shadow analysis apparatus capable of performing the nasolabial shadow analysis method according to any one of claims 1 to 3.

8. A nasolabial shadow evaluation model learning device comprising at least one processor and memory, A nasolabial shadow evaluation model learning device capable of performing the learning method for the nasolabial shadow evaluation model described in claim 6.

9. A method for evaluating the visual impression of nasolabial shadows, which are shadows that appear at the boundary between the nose and the area below the nose and the cheek, One or more processors The process involves acquiring an image of the person being evaluated, which includes obtaining an evaluation image of their face, The evaluation value acquisition step involves inputting the acquired facial images for evaluation into a trained model that has been machine-learned based on training data from multiple individuals, thereby acquiring a first evaluation value indicating the degree of elevation of the area near the nasolabial folds of the cheeks of the person being evaluated, and a second evaluation value indicating the state of the nasolabial folds of the person being evaluated. Execute, The aforementioned first evaluation value indicates the state of elevation in the area near the nasolabial fold, which is the part of the cheek adjacent to the outside of the nasolabial fold and is capable of producing the aforementioned nasolabial shadow in the same way as the nasolabial fold. The first evaluation value and the second evaluation value are used to evaluate the visual impression of the nasolabial shadow. A method for evaluating the visual impression of nasolabial shadows.

Images