Computer program

Abstract

To provide a technique to improve estimation accuracy of an estimation model for estimating, from an input face image, the skin state of a human face captured in the input face image.SOLUTION: A computer program is provided, which causes a computer comprising one or more processors to execute a skin state estimation method including the steps of: acquiring a face image of a subject; converting the face image so that a plurality of points on a contour of a face area of a human face captured in the acquired face image is located on an outer edge of the image, so as to conform to an input rule in an estimation model, thereby obtaining a post-conversion face image; and acquiring skin state information of the face of the subject that is output from the estimation model when the post-conversion face image is input to the estimation model as an input face image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for estimating the skin condition of a human face appearing in an input face image from the input face image by using a trained estimation model. [Background technology]

[0002] The following Patent Document 1 discloses a system that determines a person's apparent skin age by masking macroscopic facial features in an image of the person and then analyzing the image using a convolutional neural network, and displays the determined apparent skin age on a display device that is visible to the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 222808 Summary of the Invention [Problem to be solved by the invention]

[0004] In the existing method of estimating the skin condition of a human face captured in a face image using a trained estimation model like the above system, the face image input to the estimation model is normalized to a predetermined image size and rectangular shape, and the input face image is analyzed. Meanwhile, the face image input to the estimation model includes, in addition to the face region, regions other than the face, such as the background and hair.

[0005] For this reason, in the learning process, the estimation model needs to learn not only the skin condition of the facial region, but also not to use regions that do not resemble skin, and needs to retain such acquired knowledge resources. However, for the estimation model, calculations and information on regions other than the face are unnecessary when estimating the skin condition of a human face. In addition, if the image size is compressed while the region other than the face is included in the normalization of the face image, the resolution of the face region decreases, which may reduce the estimation accuracy of the learning model.

[0006] The present invention takes into consideration the above-mentioned viewpoints and provides a technique for improving the estimation accuracy of an estimation model that estimates the skin condition of a human face captured in an input face image from the input face image. [Means for solving the problem]

[0007] According to the present invention, there is provided a skin condition estimation method using an estimation model that has been trained to output the skin condition of a human face reflected in an input facial image from the input facial image, in which one or more processors execute an image acquisition step of acquiring a facial image of a subject, a conversion step of converting the facial image so that multiple points on the contour of the facial area of ​​the human face reflected in the acquired facial image are positioned at the outer edge of the image to conform to input rules in the estimation model, thereby obtaining a converted facial image, and an information acquisition step of inputting the converted facial image as the input facial image, thereby obtaining skin condition information of the subject's face that is output from the estimation model.

[0008] In addition, according to the present invention, there is provided a skin condition estimation system comprising an estimation model that has been trained to output the skin condition of a human face reflected in an input facial image from the input facial image, an image acquisition means for acquiring a facial image of a subject, a conversion means for converting the facial image so that multiple points on the contour of the facial area of ​​the human face reflected in the acquired facial image are located at the outer edge of the image to conform to the input rules of the estimation model, thereby obtaining a converted facial image, an information acquisition means for inputting the converted facial image into the estimation model as the input facial image, and acquiring skin condition information of the subject's face that is output from the estimation model, a skin condition output means for generating image information for presenting the acquired skin condition information, and an information display means for displaying the image information.

[0009] Furthermore, according to the present invention, it is possible to provide a user terminal for the above-mentioned skin condition estimating system. Furthermore, it is possible to provide a program for causing a computer to execute the above-mentioned skin condition estimation method, i.e., for causing a computer to function as all or a part of the above-mentioned skin condition estimation system, and it is also possible to provide a computer-readable recording medium having such a program recorded thereon. This recording medium includes a non-transitory tangible medium. Effect of the Invention

[0010] According to the present invention, it is possible to provide a technique for improving the estimation accuracy of an estimation model that estimates the skin condition of a human face captured in an input face image from the input face image. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device capable of executing a skin condition estimating method according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram for explaining the function of each unit of the skin condition estimating device according to the present embodiment. [Diagram 3] 4 is a flowchart showing a skin condition estimating method according to the present embodiment. [Figure 4] 5A to 5C are diagrams illustrating a flow of face image conversion in the skin condition estimating method according to the present embodiment. [Diagram 5] 5 is a flowchart showing a method for learning a skin condition estimation model according to the embodiment. [Figure 6] 13A and 13B are diagrams illustrating a flow of face image conversion in a skin condition estimating method according to a modified example. [Figure 7] 13 is a flowchart showing a skin condition estimating method according to a modified example. [Figure 8] FIG. 2 is a functional diagram illustrating an embodiment in which the skin condition estimating system is configured as a combination of a user terminal and a server. [Figure 9] FIG. 13 is a functional diagram showing another embodiment in which the skin condition estimating system is configured as a combination of a user terminal and a server. [Figure 10] FIG. 13 is a diagram showing an example of a display screen of skin condition information on a user terminal. [Figure 11] 6A to 6C are diagrams illustrating a flow of face image conversion in the skin condition estimating method according to the embodiment. [Figure 12] 1 is a graph showing the difference in prediction error between the skin condition estimating method according to the present embodiment (this method), a conventional method, and a prior patent method. [Figure 13] 13 is a diagram showing an example of face image transformation using a coordinate transformation rule for increasing the magnification rate around the corner of the eye compared to other parts in order to estimate wrinkles at the corner of the eye in more detail. FIG. [Figure 14] FIG. 13 is a diagram showing an example of face image transformation using a coordinate transformation rule that increases the magnification rate around the nasolabial folds more than other parts in order to estimate the state of the nasolabial folds in more detail. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] A preferred embodiment of the present invention (hereinafter, referred to as the present embodiment) will be described below. Note that the present embodiment described below is merely an example, and the present invention is not limited to the configuration described below.

[0013] [Hardware configuration example] The skin condition estimating method according to this embodiment (hereinafter referred to as this method) is executed by one or more processors included in one or more information processing devices. FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device 10 capable of executing the present method. The information processing device 10 is a so-called computer, and includes a CPU 11, a memory 12, an input / output interface (I / F) 13, a communication unit 14, etc. The information processing device 10 may be a stationary PC (Personal Computer), or may be a portable terminal such as a portable PC, a smartphone, or a tablet.

[0014] The CPU 11 is a so-called microprocessor. The memory 12 is a random access memory (RAM), a read only memory (ROM), or an auxiliary storage device (such as a hard disk). The input / output I / F 13 can be connected to user interface devices such as a display device 15 and an input device 16. The display device 15 is a device such as an LCD (Liquid Crystal Display) that displays a screen corresponding to drawing data prepared by the CPU 11 or the like. The input device 16 is a device such as a keyboard, a mouse, etc. that accepts input of user operations. The display device 15 and the input device 16 may be integrated and realized as a touch panel. The communication unit 14 communicates with other computers via a communication network and exchanges signals with other devices such as a printer, etc. A portable recording medium, etc. can also be connected to the communication unit 14.

[0015] The hardware configuration of the information processing device 10 is not limited to the example in Fig. 1. The information processing device 10 may include other hardware elements not shown. Furthermore, the number of each hardware element is not limited to the example in Fig. 1. For example, the information processing device 10 may have multiple CPUs 11. In particular, to realize a convolutional neural network that estimates a skin condition, the speed of calculations can be increased by using a GPU (Graphics Processing Unit) designed for vector calculations instead of a general CPU. The present invention may also be realized as a so-called system using a plurality of computers formed by combining a plurality of information processing devices 10. When the present invention is realized using a plurality of computers, it is preferable to operate the present invention as a system in which the functions are shared among the plurality of computers and a server and a terminal connected by communication via a network such as the Internet are combined, as described below. In the following description, an example in which the present invention is realized by the information processing device 10 is given.

[0016] The information processing device 10 executes the present method by executing a computer program stored in the memory 12 by the CPU 11. The present method is executed by storing the computer program in the memory 12 after it has been installed once from a portable recording medium such as a CD (Compact Disc) or a memory card, or from another computer on a network, via the input / output I / F 13 or the communication unit 14, or by directly storing the program in the memory 12 and executing the program, and the present device or system is realized as a combination of the various means described below.

[0017] The information processing device 10 (CPU 11) can use a trained model obtained by machine learning using training data. The "trained model" here is a model obtained by machine learning using training data, i.e., supervised learning, and can be expressed as an AI (Artificial Intelligence) model, Machine Learning (ML) model, etc. The trained model used in this embodiment may be a regression equation obtained by regression analysis or a neural network model obtained by deep learning, etc., and the data structure, learning algorithm, etc. of the model are not limited. In this embodiment, it is preferable to use a neural network (CNN) model including a convolution operation as the trained model. The trained model may be stored in memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication.

[0018] In this manner, the information processing device 10 in this embodiment is a skin condition estimating device that can use a trained model and executes a skin condition estimating method. Hereinafter, the trained model used in this method will be referred to as an AI model or an estimated model. In addition, the face images used as training data in the training of this AI model are normalized so that the coordinates of corresponding positions match according to a common rule, as described later, and this rule will be referred to as an input rule of the AI ​​model. In the following description, the subject executing this method will be described as the CPU 11.

[0019] [Skin condition estimation method and device] Details of this embodiment will be described below with reference to Figures 2, 3 and 4. Figure 2 is a block diagram for explaining the function of each unit of the skin condition estimating device according to this embodiment (sometimes simply referred to as this device). Figure 3 is a flowchart showing the skin condition estimating method according to this embodiment, and Figure 4 is a diagram showing the flow of facial image conversion in the skin condition estimating method according to this embodiment.

[0020] To explain each step of the method as a functional block, as shown in Figure 2, the device comprises an image acquisition means for acquiring a facial image of a subject, a position identification means for identifying the position coordinates of multiple predetermined points in the facial area of ​​the human face reflected in the facial image acquired by the image acquisition means, a conversion means for converting the facial image so that multiple points on the contour of the facial area of ​​the human face reflected in the acquired facial image are located at the outer edge of the image to comply with the input rules of the AI ​​model, thereby obtaining a converted facial image, an information acquisition means for inputting the converted facial image as an input image into an AI model formed by a previously prepared convolutional neural network to acquire skin condition information of the subject's face, and an information display means for generating image information to present the acquired skin condition information. Unless otherwise specified, the explanation of each step below also applies to the corresponding means, so please refer to those means for the operation of each means. Each of the means is realized by appropriately combining the CPU 11 of the computer, peripheral circuits, and computer programs and logic circuits for realizing each function. For example, the means for performing skin condition estimation using an AI model is preferably realized using a GPU, but can also be realized by combining a general CPU and a program. In other words, it is possible to provide a program that causes a computer to function as all or part of the above-mentioned skin condition estimation system, and it is also possible to provide a computer-readable recording medium having such a program recorded thereon.

[0021] In this embodiment, the AI ​​model used in the information acquisition means and information acquisition process is a trained model that estimates the skin condition of a human face captured in an input face image of a predetermined image size, i.e., a rectangular shape with a predetermined number of pixels in length and width (referred to as a predetermined rectangular shape in this specification). For this reason, the AI ​​model used in this embodiment is also referred to as a skin condition estimation model. "Predetermined image size" means a predetermined size of an image, and is specified by the number of pixels. The image size is predetermined, for example, 25 x 20 (500) pixels, 20 x 20 (400) pixels, 20 x 11 (220) pixels, etc. Note that the image sizes shown here are just examples. The "predetermined rectangular shape" refers to a predetermined rectangular shape of an image, and is a rectangle or a square having long and short sides whose lengths correspond to the above-mentioned "predetermined image size." Hereinafter, when the term "predetermined rectangular shape" is used in relation to a face image, it refers to a rectangular shape with a predetermined number of pixels in both length and width for the input face image of the AI ​​model. In this embodiment, the predetermined shape is shown as an example, but the shape of the input image to the AI ​​model that can be used, i.e., the predetermined shape, can be a circle, an ellipse, etc. In any shape, by converting so that the coordinates of corresponding positions match according to a common rule, the same effect as when the predetermined shape is a rectangle can be obtained.

[0022] The information estimated by the AI ​​model may be information indicating the condition of the skin of a human face, and the specific estimation items are not limited. As examples of the estimation items of the skin condition, for example, as given in the examples, it is possible to estimate makeup feel, skin age, masculine / feminine skin, powdery, glossy, wetness, moistness, dryness, smoothness, texture, luster, firmness, transparency, skin whiteness, yellowness, dullness, etc. Makeup feel indicates a feeling that is not like bare skin, powdery indicates powdery, and glossy indicates the degree of surface gloss. The estimation items of the AI ​​model are not limited to such items, and may be other items such as an overall evaluation of beauty.

[0023] The AI ​​model may be formed so as to be capable of estimating multiple items related to the skin condition of a human face, or may be formed so as to be capable of estimating any one of the items. In the former case, for example, the AI ​​model may be constructed with multiple neural networks each capable of outputting estimated information of a different skin condition item. In the latter case, one neural network may be constructed so as to output estimated information of multiple skin condition items. The learning method of the AI ​​model will be described later.

[0024] As shown in FIG. 3, this method includes an image acquisition step (S21), a position identification step (S22), a conversion step (S23), and information acquisition steps (S24) and (S25). In the image acquisition step (S21), the CPU 11 acquires a facial image of a subject. In the example of Fig. 4, the reference symbol G1 indicates the acquired facial image. The acquired face image may include the subject's face to an extent that the skin condition of the estimation item can be analyzed, and may include the whole or part of the subject's face. The acquired face image may include parts other than the subject's face, such as hair or neck, or the background. In addition, the angle and position of the subject reflected in the face image are not particularly limited. For example, the acquired face image may include the front face of the subject, or the left or right oblique face of the subject. In addition, the face image may be captured by capturing the face of the subject from below or above.

[0025] In the image acquisition process, i.e., the facial image acquired by the image acquisition means, the facial image is acquired as an image file in, for example, JPEG (Joint Photographic Experts Group) format, BMP (Bitmap image) format, TIFF (Tagged Image File Format), GIF (Graphic Interchange Format) format, etc. Furthermore, the acquired face image may be a color image or a grayscale image, so long as it is possible to estimate the skin condition of the human face captured in the face image. The CPU 11 acquires a face image of the target person from a camera that captures the face of the target person, another computer, or the like, via a communication line or a portable recording medium. Hereinafter, the face image acquired in step (S21) may be referred to as an original face image. The acquired original face image does not need to be the same size as the predetermined image size of the predetermined rectangular shape, but it is preferable that the total number of pixels is greater than the predetermined image size of the predetermined rectangular shape.

[0026] The location specification step (S22) will now be described in detail. In the position specifying step (S22), first, points on the contour of the entire face region are selected as first contour points, and then the inside of the first contour point is divided into triangular meshes as follows. Then, vertices of a plurality of meshes are selected as third contour points so that a figure formed by connecting them includes the predetermined local part region, and then, a second contour point is selected between the first contour point and the third contour point. More specifically, after dividing the face region into a number of meshes, contour points are selected from the intersection points (mesh points) of the sides of each mesh. The first contour point is a point located on the contour of the entire face, that is, on the boundary between the face and parts other than the face including the hair and neck, preferably a point located slightly inside the boundary. The third contour point is a mesh point when the sides of the mesh are connected so as to include the entire predetermined local part region inside, and is selected as a point that has the same shortest number of meshes as the first contour point (hereinafter, the minimum number of mesh sides connecting two mesh points is called the mesh distance). Then, a mesh point between the first contour point and the third contour point is selected as the second contour point. After the device automatically generates a mesh and determines the position of the first contour point on the image, the position of the third contour point is determined on the image so that the mesh distance from the first contour point is equal, and then the position of the second contour point on the image is determined from the generated mesh. If it is not possible to set the first to third contour points according to the above criteria, for example if the mesh distance between the first and third contour points is 1 or less, or if all of the third contour points are not at an equal mesh distance from the first contour point, then the mesh is made finer (the total number of meshes is increased) and recalculation is performed, and the position of each mesh point on the image is changed so that the first to third contour points can be selected.

[0027] In the position specifying step (S22), the CPU 11 specifies the position coordinates (predetermined position coordinates) of a plurality of predetermined points in the face region of the human face captured in the original face image. In the example of Fig. 4, points within a rectangle indicated by the reference symbol G2 indicate the position coordinates of the specified plurality of points. The position coordinates specified in step (S22) include at least the position coordinates of the shape feature points of the face region. The shape feature points of the face region indicate points at positions having shape features of a human face, such as points on the face outline, points on the eye outline (edges of the upper eyelid and the lower eyelid), points on the eyebrow outline, points on the lip outline (upper edge of the upper lip and lower edge of the lower lip), the tip of the nose (point of the tip of the nose), points on the bridge of the nose (bridge of the nose), points on the root of the nose, etc. However, the position coordinates specified in step (S22) are not limited to only the position coordinates of such shape feature points, and may be position coordinates of predetermined points in the face region derived from such shape feature points, or may include position coordinates of shape feature points and position coordinates of other points.

[0028] The position specifying step (S22) can be executed by the CPU 11, or the user can specify the original face images displayed on the display device in sequence according to instructions. In this case, the position may be specified by a pointing device such as a mouse, but from the viewpoint of operability, it is preferable to perform the process using a touch screen, and in the case of a so-called smartphone terminal, a touch screen is used. Since it is cumbersome for the user to perform the process, it goes without saying that it is preferable to perform the process automatically using a computer.

[0029] The predetermined position coordinates identified in step (S22) preferably include position coordinates of a plurality of perimeter points surrounding the predetermined local site region within the face region. The "predetermined local area regions" are areas that do not contribute to estimating skin condition and are therefore preferably excluded from the facial area in the facial image input to the AI ​​model, such as the eyes, eyebrows, lips, nostrils, moustache, and beard. In this embodiment, as a method for removing pixels of a predetermined local part region that is preferably removed from a face region in a face image to be input to an AI model, the skin pixels surrounding the predetermined local part region are arranged on a straight line or curve in the face model, and as a result, the pixels in the predetermined local part region are not projected onto the face model. In this case, the positions of the opposing perimeter points of the predetermined local part region coincide on the face model. In this embodiment, the "predetermined local area region" is an eye region, an eyebrow region, and a lip region. The lip region is an upper lip region and a lower lip region, but in addition, when a subject in the face image has his / her mouth open, it also includes an inner mouth region.

[0030] The multiple outer perimeter points may be points located so as to surround a specified local part area within the face area, and may be points on the contour of the specified local part area or points outside the contour of the specified local part area. In this way, by identifying the position coordinates of a plurality of perimeter points surrounding a predetermined local part region within the face region in step (S22), the predetermined local part region can be removed in a conversion step (S23) described below. It is preferable to completely remove the predetermined local part area, but even if it is reduced, it can contribute to improving the estimation accuracy. For example, when the points on the outline of the predetermined local part area or points slightly inside the outline are specified as the multiple outer periphery points, and the specified outer periphery points are arranged on a straight line or a curved line in the face model, the predetermined local part area is reduced and remains. Even in such a case, it is possible to reduce the area that does not contribute to the estimation of the skin condition, and therefore it is possible to improve the estimation accuracy of the skin condition by the AI ​​model.

[0031] In addition, it is preferable that the specified position coordinates identified in step (S22) include the position coordinates of a plurality of points on the contour of the face region, i.e., on the boundary between the face region and other regions, the tip of the nose within the face region, and a plurality of other points within the face region. In a conversion step (S23) described later, four first contour points among the points on the contour of the face region are arranged at the vertices of a predetermined rectangular shape of the face image, and other points on the contour of the face region are arranged on the sides of the predetermined rectangular shape of the face image. Therefore, the four first contour points are set to two points on the upper left and right sides of the contour of the face region and two points on the lower left and right sides of the contour. In the example of Fig. 4, two points on the contour adjacent to the ends of the left and right eyebrows (G21 and G24) and two points on the contour adjacent to the left and right corners of the mouth (G22 and G23) correspond to the first contour points, and the nose tip is indicated by the symbol G25. However, the first contour points may be any combination of two upper left and right points on the contour of the face region and two lower left and right points on the contour, and are not limited to the example of Fig. 4.

[0032] The method of identifying the position coordinates in step (S22) can be face recognition processing used in various situations. In the face recognition processing, for example, an AI model capable of detecting the positions of multiple nodes of a human face, such as Facemark of OpenCV (Open Source Computer Vision Library) or FaceMesh of Google (registered trademark), is used. In this embodiment, the method of identifying the position coordinates in step (S22) is not limited in any way. For example, in step (S22), the CPU 11 can identify the position coordinates of some or all of the landmarks detected using such a face recognition AI model as the position coordinates of multiple points in the face area.

[0033] In the conversion step (S23), the CPU 11 performs a so-called mapping operation to convert the position of each pixel of the original face image acquired in step (S21) so that the face area becomes a predetermined rectangular shape by converting the position coordinates of the predetermined multiple reference points identified in step (S22) and the position of each pixel of the original face image becomes a position corresponding to the input rule of the AI ​​model. That is, the conversion step (S23) can also be called a coordinate conversion step or a mapping step. Similarly, the conversion means described later can also be called a coordinate conversion means or a mapping means. In the example of FIG. 4, the position coordinates after the coordinate transformation are shown in the rectangle indicated by the code G3, and the code G4 indicates the face image after the transformation. In other words, each pixel maintains its value and only its position is transformed. When the position of each pixel changes, the spatial distance between the pixels changes, so when the distance increases, the value of the pixel that fills the gap is obtained by interpolation, and when the distance decreases, a calculation is performed to obtain the value of a new pixel, for example, using an average value, but the specific method can be achieved by applying well-known techniques such as morphing techniques, so details are omitted. Note that there may be cases where the position coordinates of a predetermined point that serves as the reference for the original face image coincide with a position corresponding to the input rule of the AI ​​model, and in that case, it will be naturally understood by those skilled in the art that this is also included in the mapping operation or position transformation. As long as the coordinate transformation is performed so that the face area of ​​the original face image has a predetermined rectangular shape, the method is not limited. The position specifying step (S22) and the conversion step (S23) can be realized as one conversion step. In other words, if it is determined which position in the reference image a pixel of the original face image corresponds to, the coordinates of the conversion destination (projection destination) can be specified immediately, so it is not necessarily necessary to perform the process in two stages. The same can be said about the position specifying means and the conversion means, and in this device, both can be configured as a conversion means integrated with each other.

[0034] In this way, by converting the original face image into a face image of a predetermined rectangular shape in which the face area occupies the entire face image, it is possible to eliminate areas other than the face area and enlarge the face area in the face image input to the AI ​​model. This makes it possible to prevent a decrease in the resolution of the face area in the face image input to the AI ​​model, and improve the estimation accuracy of the AI ​​model. Such an effect is prominent when the AI ​​model is realized by a neural network model including a convolution operation. In such an AI model, the convolution operation is repeated while information other than the face area of ​​the input image is preserved, so the influence of the areas (areas other than skin areas) that are not the subject of analysis included in the input image remains even in the depths of the AI ​​model. As a result, unnecessary information remains in the AI ​​model. Furthermore, as a result of the inclusion of information other than the face area, the information to be analyzed is reduced. These factors lead to a decrease in the estimation accuracy of the AI ​​model.

[0035] In the case where the position coordinates of the multiple points on the contour of the face region, the nose tip of the face region, and multiple other points in the face region are specified in step (S22), the coordinate conversion in step (S23) is performed as follows: That is, as shown in the example of Fig. 4, the coordinate conversion is performed such that each of the four first contour points (G21, G22, G23, and G24) included in the multiple points on the contour of the face region of the original face image is located at the vertices of a predetermined rectangular shape, the nose tip (G25) is located at the center of the predetermined rectangular shape, and the other multiple points in the face region move in response to the movement of the first contour point and the nose tip.

[0036] By performing coordinate transformation in this way, the face region can be made to occupy the entirety of the face image of a predetermined rectangular shape after transformation. In addition, this coordinate transformation can eliminate and standardize the differences in the shape and size of the face contours of each individual reflected in the original face image. As a result, the positions of the nose, mouth, eyes, etc. within the face region in the face image input to the AI ​​model can be roughly standardized, which can improve the learning efficiency of the AI ​​model and also lead to improved estimation accuracy of the AI ​​model.

[0037] In this embodiment, since the position coordinates of a plurality of perimeter points surrounding a predetermined local part area in the face area are specified in step (S22), the image conversion in step (S23) is performed as follows: That is, in the coordinate conversion, CPU 11 matches the position coordinates of perimeter points facing each other via the predetermined local part area, in other words, connects the perimeter points to remove the predetermined local part area from the face area of ​​the original face image. Here, the "connection of perimeter points" is realized by converting two or more perimeter points that face each other via the specified local site region into the same position coordinates, or may be realized by moving the two or more perimeter points to adjacent positions so that the local site region disappears or is reduced.

[0038] This makes it possible to enlarge the skin area within the face image input to the AI ​​model while excluding local area areas unrelated to the skin condition of the human face from the face image. As a result, it is possible to prevent a decrease in the resolution of the face area, which occurs when the number of pixels in the face area of ​​the original face image is smaller than the specified rectangular shape of the input image of the AI ​​model, more effectively than with conventional methods, and ultimately improve the estimation accuracy of the AI ​​model. For this reason, the specified local area is preferably one of the eye area, eyebrow area, or lip area, which are unrelated to the skin condition, more preferably a combination of any of these, and even more preferably all of these.

[0039] In the information acquisition steps (S24) (S25), the CPU 11 inputs the face image converted in the step (S23) to the AI ​​model (S24), and acquires facial skin condition information of the subject (S25). When a program or the like realizing an AI model is stored in memory 12 in information processing device 10, CPU 11 can obtain facial skin condition information of the subject as the output of the AI ​​model by inputting the facial image converted in step (S23) into the AI ​​model.

[0040] As described above, the skin condition information acquired in step (S25) may be information indicating the condition of the subject's face skin, and the specific estimation items are not limited. If the AI ​​model is formed to be capable of estimating multiple items related to the skin condition of a human face, the acquired skin condition information may be information on multiple skin condition items or information on one skin condition item.

[0041] In addition, in recent years, various techniques have been developed to show the reasoning behind decisions made by AI models, such as GradCam for CNN AI models, which shows which areas contribute to a conclusion. Therefore, in step (S23), coordinate conversion may be performed using such a technique so that the parts of the face area that contribute to the estimation result of the AI ​​model are given more importance. Alternatively, in step (S25), CPU 11 may obtain information on the skin condition information of the subject's face and the position of the part of the face area corresponding to the skin condition information. At this time, CPU 11 may output, for example, a face image in which the part in the face area of ​​the original face image is displayed in an identifiable manner. By providing a step of displaying the parts that contribute to the skin condition on a screen in this way, it is possible to display an image that allows easy understanding of which part of the subject should be treated or a product applied, which is preferable.

[0042] Furthermore, the CPU 11 may further execute a step of outputting recommended information of one or both of a beauty treatment method and a beauty product corresponding to the facial skin condition of the subject based on the skin condition information acquired in step (S25). For example, the output recommended information includes information on a beauty treatment method and a beauty product such as a cosmetic agent or supplement that is recommended to improve the skin condition of the subject or to achieve the skin age desired by the subject. In this case, when it is possible to acquire part information of the face region corresponding to the skin condition information as described above, the output recommended information may be configured to present different beauty products or beauty treatment methods for each part of the face region. In addition, when the AI ​​model is formed to be capable of estimating multiple skin condition items, the output recommended information may be configured to present different beauty products or beauty treatment methods for each skin condition item.

[0043] Each of the multiple beauty treatment methods and multiple beauty products is associated with each skin condition (each skin condition for each skin condition item) that can be estimated by the AI ​​model and stored in memory 12, and CPU 11 can identify one or more beauty treatment methods or one or more beauty products that correspond to the skin condition information acquired in step (S25) based on the information stored in memory 12, and generate recommendation information including one or both of the identified methods or products. The association information between the method / product and the skin condition may be stored in the storage device of another computer and configured to be referenced via communication. The output form of the recommendation information is not particularly limited. For example, the CPU 11 may display the recommendation information on the display device 15, print it on a printer, transmit it to another computer, or store it as an electronic file on a portable recording medium or the like.

[0044] [AI model (skin condition estimation model) learning method] The learning method of the AI ​​model (skin condition estimation model) used in the above-mentioned method will be described with reference to FIG. 5 is a flowchart showing a method for training a skin condition estimation model according to this embodiment (hereinafter, may be referred to as this training method). This training method can be executed by an information processing device 10 (CPU 11) as exemplified in FIG. This learning method generally includes a step of acquiring training data (S41), a conversion step (S42) (S43) of converting each face image included in the training data, and a step (S44) of training an AI model using the training data including the converted face images. Each step of this learning method will be described in detail below.

[0045] In step (S41), the CPU 11 acquires training data having combinations of face images and answer information indicating skin conditions of the human faces appearing in the face images, for a plurality of people. Each face image included in the teacher data is as described above with respect to the face image acquired in step (S21) of the present method, and it is sufficient that the face of the subject is included in each of the face images to such an extent that the skin condition of the estimation item can be analyzed. Each face image included in the teacher data may be normalized to a predetermined rectangular shape, or may include a face image of an image size different from the predetermined image size of the predetermined rectangular shape or a shape different from the predetermined rectangular shape. The correct answer information included in the training data is information indicating the skin condition of a human face obtained by an evaluator evaluating the corresponding face image or the subject of the face image itself by visual inspection or using a measuring device, etc. When the AI ​​model is formed to be able to estimate multiple items related to the skin condition of a human face, the correct answer information is information on the multiple skin condition items.

[0046] Step (S42) and step (S43) are executed for each face image included in the training data obtained in step (S41). In step (S42), CPU 11 identifies position coordinates of a plurality of predetermined locations (predetermined position coordinates) in a face region of a human face captured in a target face image. In step (S43), CPU 11 performs coordinate conversion on the target face image so that the face area has a predetermined rectangular shape based on the position coordinates of the plurality of predetermined locations identified in step (S42). The process contents of steps (S42) and (S43) are similar to those of steps (S22) and (S23) described above.

[0047] When the steps (S42) and (S43) are performed for all face images included in the training data, in step (S44), the CPU 11 trains the AI ​​model using the training data including the converted face images. In this embodiment, the AI ​​model is trained by deep learning. However, the specific learning algorithm for the AI ​​model is not limited in any way.

[0048] In this way, in the learning method, the AI ​​model is trained using as training data face images in which the face region occupies the entire face image. That is, the face images used for training the AI ​​model are images in which at least multiple points on the contour of the face region are located on the outer edge of the image, and more preferably, all points on the contour of the face region are located on the outer edge of the image. This allows face images from which non-face regions have been removed to be used as training data, eliminating the need for a learning process that involves learning not to use non-face regions in estimation, and thus making the learning process of the AI ​​model more efficient. In addition, in face images used as training data in this learning method, the face region is expanded by the number of pixels in the non-face region that has been removed, making it possible to suppress a decrease in the resolution of the face region to be estimated, and ultimately building a highly accurate AI model. Furthermore, it is more preferable that the facial image used for training the AI ​​model is an image in which a specific local area within the facial region has been removed or reduced, and it is more preferable that the facial image is one in which multiple points on the contour of the facial region (including the first contour point), the tip of the nose within the facial region, and multiple other points within the facial region (including the second contour point and the third contour point) have been coordinate-transformed as described in this method. Facial images used to train AI models are coordinate-transformed (normalized) according to these common rules so that the coordinates of corresponding positions match.

[0049] [Variations] The contents of the above-described embodiment can be modified as appropriate without hindrance. For example, in the above-mentioned example of the AI ​​model, a face image of a predetermined rectangular shape was used as training data for learning, but a face image of a non-rectangular shape such as a circle, an ellipse, etc. may be used as training data for learning. Even in this case, in the conversion step (S23), CPU 11 may convert the original face image so that a plurality of points on the contour of the face area of ​​the human face reflected in the original face image are positioned on the outer edge of the image in order to conform to the input rule in the AI ​​model, that is, so that the face image has a predetermined shape other than a rectangle and the face area occupies the entirety of the face image, thereby obtaining a converted face image. Furthermore, in the above embodiment, as a preferred aspect, processing for removing or reducing a predetermined local part area within the face area is performed in the conversion step (S23), but this processing may be omitted.

[0050] When the AI ​​model is configured to be capable of estimating multiple items of skin condition, the above-described embodiment can be modified as shown in FIGS. FIG. 6 is a diagram showing a flow of facial image conversion in a skin condition estimating method according to a modified example (hereinafter referred to as this modified method), and FIG. 7 is a flowchart showing the skin condition estimating method according to the modified example.

[0051] The AI ​​model used in this transformation method is formed, for example, by a plurality of neural networks each capable of outputting estimation information of one or more different skin condition items. Each neural network is trained using training data including a combination of face images transformed to match input rules corresponding to one or more skin condition items to be estimated and correct answer information for the skin condition items. For this reason, it can be said that the AI ​​model used in this transformation method has a plurality of input rules. The CPU 11 is capable of using a plurality of mutually different coordinate transformation rules corresponding to a plurality of estimation items in order to obtain a face image that conforms to the input rules of the AI ​​model. The coordinate transformation rules are used in the coordinate transformation in the step (S65) described below, and stipulate to which positions in the transformed facial image the position coordinates of multiple specified locations identified in the step (S64) described below should be moved.

[0052] As described in the above embodiment, the coordinate transformation rule may be any rule that can transform the original face image so that the face region occupies the entire face image after transformation, and the specific content is not limited. For example, a coordinate transformation rule is set so that a part region corresponding to a skin condition item to be estimated in the face skin region is enlarged and a part region having little relation to the skin condition item is reduced. Specifically, the coordinate transformation rule corresponding to the skin condition item of skin age is prescribed to enlarge the area of ​​the corner of the eye or the area of ​​the nasolabial fold. In addition, the coordinate transformation rule corresponding to the skin condition item of dryness is prescribed to enlarge the area around the eyes and the area around the mouth, which are known to be prone to dryness in the face skin, and the coordinate transformation rule corresponding to the skin condition item of glossy is prescribed to enlarge the area called the T-zone, which is known to have a large amount of sebum.

[0053] The coordinate transformation rules may be provided in the same number as the number of items that the AI ​​model can estimate, or may be provided in a number less than the number of items that the AI ​​model can estimate. In the former case, one coordinate transformation rule is provided for one estimation item, and a neural network is formed for each coordinate transformation rule, in other words, for each estimation item. In the latter case, a coordinate transformation rule provided for multiple estimation items and a coordinate transformation rule provided for one estimation item may be mixed, or multiple coordinate transformation rules provided for multiple estimation items may be provided. In this case, a neural network is formed for each coordinate transformation rule, but not for each estimation item.

[0054] As shown in FIG. 7, this modified method includes steps (S61) to (S67). Step (S61) is similar to step (S21) described above. In step (S62), the CPU 11 acquires estimation target item information. The estimation target item information is information indicating one or more required skin condition items among a plurality of skin condition items that can be estimated by the AI ​​model. The CPU 11 can acquire the estimation target item information in response to a user's input using the input device 16, can acquire preset estimation target item information from the memory 12, or can acquire the estimation target item information via communication from another computer.

[0055] In step (S63), the CPU 11 selects a coordinate transformation rule corresponding to the estimation target item indicated by the estimation target item information acquired in step (S62). At this time, when multiple items are designated as the estimation target items, one or more coordinate transformation rules corresponding to the multiple items are selected. Step (S64) is similar to step (S22) described above.

[0056] In step (S65), the CPU 11 converts the original face image by converting the position coordinates of the multiple locations identified in step (S64) based on the coordinate conversion rule selected in step (S63), and generates a converted face image corresponding to the estimation target item from the original face image. The method of converting the face image in step (S65) is the same as that in step (S23) described above, except that a coordinate conversion rule selected from the multiple coordinate conversion rules is used. However, when multiple coordinate transformation rules are selected in step (S63), multiple transformed face images corresponding to multiple estimation target items are generated in step (S65). In the example of Fig. 6, N position coordinates (G3-1, G3-2, ..., G3-N) are shown as position coordinates after coordinate transformation based on N coordinate transformation rules, and N transformed face images (G4-1, G4-2, ..., G4-N) are generated.

[0057] Step (S66) is similar to the above-mentioned step (S24), and step (S67) is similar to the above-mentioned step (S25). However, when multiple items are designated as the items to be estimated, skin condition information on the multiple items of the subject's face is acquired in step (S67). Also, when multiple coordinate transformation rules are selected in step (S63) and multiple transformed face images are generated in step (S65), the multiple transformed face images are input to the AI ​​model in step (S66), and skin condition information on the multiple items of the subject's face is acquired in step (S67).

[0058] According to this modified method, it is possible to obtain skin condition information of a subject regarding a plurality of items, and also to obtain skin condition information of a subject regarding a desired item among the plurality of items. Furthermore, since the AI ​​model makes estimations based on face images transformed using coordinate transformation rules corresponding to each skin condition item to be estimated, the estimation accuracy for each skin condition item can be improved. This modified method can be further modified. For example, the step of acquiring information on items to be estimated (S62) can be omitted from this modified method, and the method can be modified so that skin condition information on all items that can be estimated by the AI ​​model is acquired.

[0059] [Other embodiments] The configuration of another embodiment of a skin condition estimating system according to the present invention (hereinafter, the present embodiment) is shown in Fig. 8. In this embodiment, a system is configured by connecting a user terminal 30 such as a smartphone and a server device 20 via a network. In this embodiment, the server device 20 includes an AI model 7 and information acquisition means 38. The user terminal 30 includes a camera 31 which is an image acquisition means, a position identification means 32, a conversion means 33, a transmission means 34, an information display means 35 which is a display device, a skin condition output means 36 which converts and generates skin information presented on the information display means 35 into image information that can be read by the user, and a reception means 37. The user terminal 30 and the server device 20 are connected via a communication line 5 such as the Internet, and together form a skin condition estimating system. Note that there may be multiple user terminals 30, and in that case, it will be easily understood by those skilled in the art that each combination of the user terminal 30 and the server device 20 constitutes a skin condition estimating system.

[0060] The converted facial image sent to the server device 20 by the transmitting means 34 of the user terminal 30 is received by the receiving means (not shown) of the server device 20 and input to the AI ​​model 7 via the information acquiring means 38. In this embodiment, the skin condition information output from the AI ​​model 7 and acquired by the information acquiring means 38 is sent from the transmitting means (not shown) of the server device 20 to the user terminal 30 via the communication line 5. The skin condition information received by the receiving means 37 of the user terminal 30 is converted into data constituting a display screen by the skin condition output means 36 and displayed on the display device 35 (screen) of the user terminal 30.

[0061] In this way, by distributing the AI ​​model 7 on the server device 20, the AI ​​model 7 can be replaced or updated without changing the computer program on the user terminal 30, thereby improving accuracy and increasing the frequency of accuracy improvements for the system as a whole. It is preferable that the user terminal 30 is sent a program via a communication medium forming a network, that is, a program for causing the user terminal 30 to function as the position identification means 32, the conversion means 33, the transmission means 34, the information display means 35 which is a display device, the skin condition output means 36 which converts and generates the skin information presented on the information display means 35 into image information readable by the user, and the reception means 37. The program may be stored in a storage medium and provided to the user of the user terminal 30.

[0062] The configuration of still another embodiment (hereinafter, this embodiment) of a skin condition estimating system according to the present invention is shown in Fig. 9. In this embodiment, a system is configured by connecting a user terminal 30 such as a smartphone and a server device 20 via a network, similar to the example shown in Fig. 8. The user terminal 30 includes a camera 31 which is an image acquisition means, a transmission means 34, an information display means 35 which is a display device, a skin condition output means 36, and a reception means 37 which receives the skin condition. The server device 20 includes an AI model 7, a position identification means 32, a conversion means 33, and an information acquisition means 38. 8, the user terminal 30 and the server device 20 are connected via a communication line 5 such as the Internet, and together they constitute a skin condition estimating system. Note that there may be multiple user terminals 30, and in this case too, it will be easily understood by those skilled in the art that each combination of the user terminal 30 and the server device 20 constitutes a skin condition estimating system. It is preferable that a program for causing the user terminal 30 to function as the transmitting means 34, the skin condition output means 36, and the receiving means 37 for receiving the skin condition in this system is sent to each user terminal 30 via a communication medium forming a network. The program may be stored in a storage medium and provided to the user of the user terminal 30.

[0063] The original face image captured by the camera 31 of the user terminal 30 is sent to the server device 20 by the transmission means 34, processed by the position identification means 32 and the conversion means 33 in the server device 20, and then input into the AI ​​model 7 via the information acquisition means 38. In this embodiment, the skin condition information output from the AI ​​model 7 is acquired by information acquisition means 38 of the server device 20, and is sent by transmission means (not shown) to the user terminal 30 via the communication line 5. The skin condition information received by reception means 37 of the user terminal 30 is converted by skin condition output means 36 into data that constitutes a display screen, and is displayed on the display device 35 (screen) of the user terminal 30.

[0064] An example of a display screen that has been processed and created by the skin condition output means 36 and is displayed on the display device 35 of the user terminal 30 is shown in Fig. 10. On this display screen, the original face image is displayed at the top of the screen, the skin age is shown as a number near the center of the screen, and the values ​​for each item are shown in a radar chart below. In this way, by processing information in the skin condition output means 36 so that the original image and a graph showing skin condition information are displayed on the same screen, a screen that makes it easy to grasp the skin condition can be created, or by processing information in the output means 36 so that multiple pieces of skin condition information are displayed on one screen at once, the entire picture of the skin condition can be displayed at once. Although the numerical information sent from the server device 20 can be displayed simply as numbers, it is important as a communication tool with the user that the skin condition output means 36 of the user terminal 30 processes the information and displays it in an easy-to-read format. If the display screen is generated by the server device 20, problems such as part of the screen being missing or characters being difficult to read may occur when the screen is displayed on a different user terminal 30. Therefore, as in this embodiment, it is preferable that the skin condition information sent from the server device 20 is converted for display on the user terminal 30 side. Also, as in this embodiment, it is preferable that each item of the skin condition is processed and displayed in a graph, and the overall evaluation is displayed as a skin age or a numerical value, such as a score on a 5-point scale or a maximum score of 100 points.

[0065] Some or all of the above-described embodiments and modifications may be specified as follows. However, the above-described embodiments and modifications are not limited to the following descriptions.

[0066] <1> A skin condition estimation method using an estimation model that has been trained to output a skin condition of a human face appearing in an input face image, the method comprising: One or more processors An image acquisition step of acquiring a facial image of a subject; a conversion step of converting the acquired face image so that a plurality of points on a contour of a face region of a human face reflected in the acquired face image are positioned on an outer edge of the image in order to conform to an input rule in the estimation model, thereby obtaining a converted face image; an information acquiring step of acquiring facial skin condition information of the subject outputted from the estimation model by inputting the transformed facial image as the input facial image; A skin condition estimation method that performs the above.

[0067] <2> In the transformed face image, a predetermined local part area within a face area of ​​a human face captured in the acquired face image is removed or reduced, In the converting step, positions of outer periphery points that face each other across the predetermined local part region among a plurality of outer periphery points that surround the predetermined local part region within the face region are matched to remove or reduce the predetermined local part region within the face region. <1> The skin condition estimating method according to claim 1, <3> The predetermined local area includes an eye area, an eyebrow area, and a lip area. <2> The skin condition estimating method according to claim 1, <4> the one or more processors: A position specifying step is further performed to specify position coordinates of a plurality of predetermined points in a face area of ​​the human face captured in the acquired face image; In the transformation step, a part or all of the position coordinates of the specified plurality of predetermined locations are transformed to transform the acquired face image so as to conform to the input rule of the estimation model. <2> or <3> The skin condition estimating method according to claim 1, <5> the input face image has a predetermined rectangular shape, the position coordinates of the predetermined plurality of points further include position coordinates of a plurality of points on a contour of the face region, a nose tip of the face region, and other plurality of points of the face region; In the transformation step, the coordinate transformation is performed such that each of four first contour points included in the plurality of points on the contour of the face region is disposed at a vertex of the predetermined rectangular shape, the nose tip is disposed at the center of the predetermined rectangular shape, other points on the contour of the face region are disposed on sides of the predetermined rectangular shape, and the other points in the face region move in response to movements of the plurality of points on the contour of the face region and the nose tip. <4> The skin condition estimating method according to claim 1, <6> the other multiple points in the face region include four third contour points located closer to the contour of the face region than the predetermined local site region and four second contour points located even closer to the contour of the face region than the four third contour points, the transformation step performs the coordinate transformation such that the four second contour points and the four third contour points are located on a diagonal line connecting the four first contour points and the nose tip, and a difference between a distance of the first contour point from the nose tip and a distance of the second contour point from the nose tip is smaller than a difference between a distance of the second contour point from the nose tip and a distance of the third contour point from the nose tip. <5> The skin condition estimating method according to claim 1, <7> The estimation model is capable of estimating a plurality of items of skin condition, A plurality of input rules for the estimation model are provided, In the converting step, the acquired face image is converted so as to correspond to an input rule corresponding to an item to be estimated among the plurality of mutually different input rules corresponding to the plurality of items, thereby acquiring a converted face image corresponding to the item to be estimated; In the information acquisition step, the transformed face image is input to the estimation model to acquire skin condition information of the subject related to the estimation target item. <1> from <6> 13. The skin condition estimating method according to claim 12, <8> An estimation model that has been trained to output the skin condition of a human face captured in an input face image from the input face image; An image acquisition means for acquiring a facial image of a subject; a conversion means for converting the acquired face image so that a plurality of points on a contour of a face region of a human face reflected in the acquired face image are positioned on an outer edge of the image in order to conform to an input rule in the estimation model, thereby obtaining a converted face image; an information acquisition means for inputting the transformed face image as the input face image to the estimation model and acquiring facial skin condition information of the subject output from the estimation model; a skin condition output means for generating image information for presenting the acquired skin condition information; an information display means for displaying the image information; A skin condition estimation system comprising: <9> In the transformed face image, a predetermined local part area within a face area of ​​a human face captured in the acquired face image is removed or reduced, the conversion means matches positions of outer periphery points that face each other across the predetermined local part region among a plurality of outer periphery points that surround the predetermined local part region within the face region, thereby removing or reducing the predetermined local part region within the face region. <8> The skin condition estimation system according to claim 1, <10> The predetermined local area includes an eye area, an eyebrow area, and a lip area. <9> The skin condition estimation system according to claim 1, <11> a position specifying unit that specifies position coordinates of a plurality of predetermined points in a face area of ​​the human face captured in the acquired face image, the conversion means converts a part or all of the position coordinates of the specified plurality of predetermined locations into a coordinate conversion, and converts the acquired face image so as to conform to the input rule of the estimation model. <9> or <10> The skin condition estimation system according to claim 1, <12> the input face image has a predetermined rectangular shape, the position coordinates of the predetermined plurality of points further include position coordinates of a plurality of points on a contour of the face region, a nose tip of the face region, and other plurality of points of the face region; the transformation means performs the coordinate transformation so that each of four first contour points included in the plurality of points on the contour of the face region is disposed at a vertex of the predetermined rectangular shape, the nose tip is disposed at the center of the predetermined rectangular shape, other points on the contour of the face region are disposed on sides of the predetermined rectangular shape, and the other points in the face region move in response to movements of the plurality of points on the contour of the face region and the nose tip. <11> The skin condition estimation system according to claim 1, <13> the other multiple points in the face region include four third contour points located closer to the contour of the face region than the predetermined local site region and four second contour points located even closer to the contour of the face region than the four third contour points, the transformation means performs the coordinate transformation so that the four second contour points and the four third contour points are located on a diagonal line connecting the four first contour points and the nose tip, and a difference between a distance of the first contour point from the nose tip and a distance of the second contour point from the nose tip is smaller than a difference between a distance of the second contour point from the nose tip and a distance of the third contour point from the nose tip. <12> The skin condition estimation system according to claim 1, <14> The estimation model is capable of estimating a plurality of items of skin condition, A plurality of input rules for the estimation model are provided, the conversion means converts the acquired face image so as to correspond to an input rule corresponding to an item to be estimated among the plurality of mutually different input rules corresponding to the plurality of items, thereby acquiring a converted face image corresponding to the item to be estimated; the information acquisition means inputs the transformed face image into the estimation model to acquire skin condition information of the subject related to the estimation target item; <8> from <13> 13. A skin condition estimating system according to claim 12, <15> The skin condition estimating system includes a user terminal, the user terminal includes at least the image acquisition means, the skin condition output means, and the information display means; <8> from <14> 13. A skin condition estimating system according to claim 12, <16> The skin condition estimating system includes a user terminal, the user terminal includes at least the image acquisition means, the skin condition output means, the information display means, and the position identification means; <11> from <13> 13. A skin condition estimating system according to claim 12, <17> <8> from <14> A user terminal for the skin condition estimating system according to any one of claims 1 to 4, The skin condition display device includes at least the image acquisition device, the skin condition output device, and the information display device. User terminal for skin condition estimation system. <18> <11> from <13> A user terminal for the skin condition estimating system according to any one of claims 1 to 4, the image acquisition means, the skin condition output means, the information display means, and the position identification means, User terminal for skin condition estimation system. <19> The conversion means further comprises: <17> or <18> A user terminal for the skin condition estimating system described in claim 1.

[0068] The above content will be described in more detail below with reference to examples, but the description of the following examples does not limit the above content in any way. EXAMPLES

[0069] In this example, 2207 face images (hereinafter referred to as face image samples) were obtained by photographing 353 Japanese women in multiple shooting environments using multiple types of photographing devices, and the accuracy of the skin condition estimation method according to the above-mentioned embodiment (this method) was verified using these face image samples. The multiple types of photographing devices include a single-lens reflex digital camera, a smartphone, and a tablet terminal.

[0070] The accuracy of this method was verified by comparing the accuracy of an existing general method for estimating the skin condition of a human face captured in a facial image (hereinafter referred to as the conventional method) with the method described in the above-mentioned Patent Document 1 (hereinafter referred to as the prior patent method). Specifically, an AI model was constructed and trained for each of this method, the conventional method, and the prior patent method, and the accuracy of the skin condition estimated by each AI model was compared.

[0071] Each AI model was constructed by fine-tuning VGG16 based on a trained model of a convolutional neural network (CNN) for ImageNet. The only difference in the learning conditions for each AI model was the face images used as training data; all other conditions were the same. The learning method used was Cosine Annealing, with a learning rate (lr) of 0.001 to 0.0001, with 20 epochs as one cycle, and a total of 1,000 epochs of learning. Additionally, Mean Squared Error was used as the loss function, and Stochastic Gradient Descent (SGD) was used as the optimization algorithm.

[0072] Each AI model was formed using four CNNs to enable estimation of 18 skin conditions. The items used to estimate skin condition are makeup appearance, skin age, masculine / feminine skin, makeup smudge, powdery, glossy, wet, overall evaluation, moist, dry, smooth, texture, luster, firmness, transparency, whiteness, yellowness, and dullness. The estimated items output by one CNN were determined based on the collinearity between estimated items and the learning efficiency when combined and trained. Specifically, an AI model was created so that a combination of items that are assumed to have high mutual correlation or collinearity, such as texture, smoothness, and transparency, would be output by one CNN, and by training this AI model, combinations of items that would increase the estimation accuracy were explored. As a result, an AI model was constructed that uses four CNNs to estimate 18 skin condition items. The correct answer information for the training data for each AI model was obtained by having expert evaluators visually evaluate the skin condition of the subject's face, or by applying specific analysis methods to facial images of the subject taken with a dedicated skin image measurement device.

[0073] Each original face image included in the face image sample was converted by the present method, the conventional method, and the prior patent method, respectively. That is, a group of 2207 face images converted by the present method, a group of 2207 face images converted by the conventional method, and a group of 2207 face images converted by the prior patent method were prepared. The face image groups prepared for each method were divided into 5:3:2 groups for learning, validation, and testing, with 50% of the face image groups for learning being used as training data for the AI ​​models, 30% of the face image groups for validation being used to select the trained AI model, and the estimation accuracy of each AI model was evaluated using the 20% of the face image groups for testing. In addition, the face image groups for learning were made 10 times larger by randomly rotating each face image by up to ±5 degrees. In each AI model, the image size of the input image was specified to be 224 pixels (the specified image size mentioned above), and the shape of the input image was specified to be a square (the specified rectangular shape mentioned above).

[0074] The facial image conversion methods used in each method are as follows. In the conventional method, the original face image of the face image sample was normalized in terms of image size and shape, and a face image (image size 224 pixels, image shape square) was used in which the face was only aligned to the specified position using face recognition.In contrast, in the prior patent method, in addition to the normalization and alignment of the face using face recognition as in the conventional method, the original face image was processed to fill in areas other than the face (background and hair) as well as the areas of the eyes, eyebrows, and lips with a skin color.

[0075] In this embodiment, the method converts the face image as shown in Fig. 11. Fig. 11 is a diagram showing the flow of converting a face image in the skin condition estimating method according to this embodiment.

[0076] In the position identification step (S22), Google (registered trademark) FaceMesh was used to identify the position coordinates of 468 landmarks in the face region of the original face image. In the example of Fig. 11, a mesh formed by connecting the points of each landmark with line segments is shown in the reference symbol E2.

[0077] The detected landmarks include points indicating the facial contour, the tip of the nose, and points indicating the contours of the eyes, eyebrows, and lips, and two points adjacent to the ends of the left and right eyebrows on the facial contour are indicated by the symbols E21 and E24, two points adjacent to the left and right corners of the mouth on the facial contour are indicated by the symbols E22 and E23, and the tip of the nose is indicated by the symbol E25. The landmarks indicated by the symbols E21 to E24 correspond to the above-mentioned first contour points. Moreover, four points adjacent to the inside of the first contour points (E21 to E24) are indicated by the reference characters E211, E221, E231, and E241, and these are referred to as second contour points. Four points adjacent to the second contour points E211, E221, E231 and E241 on the further inner side are indicated by the reference characters E212, E222, E232 and E242, and these are referred to as third contour points.

[0078] In the transformation step (S23), the position coordinates of the landmarks identified in the step (S22) are transformed as shown by the reference symbol E3 in FIG. Specifically, the four first contour points E21, E22, E23 and E24 are each located at the vertices of a square in the transformed face image, the nose vertex E25 is located at the center of the transformed face image, and the other points on the facial contour are located on the sides of the transformed face image. Furthermore, the four second contour points E211, E221, E231 and E241 and the four third contour points E212, E222, E232 and E242 are each arranged on a diagonal line connecting the four first contour points E21, E22, E23 and E24 and the nose apex E25, and are arranged so that the difference between the distance of the first contour point from the nose apex and the distance of the second contour point from the nose apex is smaller than the difference between the distance of the second contour point from the nose apex and the distance of the third contour point from the nose apex. The other landmarks are each moved linearly in accordance with the movement of the points on the contour of the face region and the nose tip, while maintaining their relative positions as much as possible.

[0079] In this embodiment, the original face image is transformed by such coordinate transformation so that the face region occupies the entire square face image after transformation, as shown by reference symbol E4 in FIG. Furthermore, the original face image is transformed by the coordinate transformation of the second and third contour points so that in the transformed face image, the area closer to the face contour is not enlarged and the area closer to the center of the face is enlarged. As a result, the facial area is enlarged with a focus on the area that represents a high level of skin condition on the face, so that even when the image size is compressed to normalize the facial image, the reduction in resolution of the area that represents a high level of skin condition is sufficiently suppressed, thereby improving the estimation accuracy of the AI ​​model.

[0080] 11, the cheek region of the face region is enlarged while maintaining the mesh shape (triangle) as much as possible, because the cheek region is more representative of the skin condition of a human face. 11, there are black areas on the left, right and top edges of the square that are not considered to be face areas because the position coordinates of the landmarks specified in step (S22) are specified slightly outside the outline of the face area in the face image. Thus, the present invention can achieve the above-mentioned effects while allowing for slight deviations in the specification of the position coordinates of the face area and the arrangement of the face area in the transformed face image.

[0081] Furthermore, in the transformed face image, the eye regions, eyebrow regions, and lip regions are collapsed as shown by reference symbol E4 in Fig. 11. In this embodiment, among the landmarks on the contour of the lip region, each point on the upper contour of the upper lip region and each point on the lower contour of the lower lip region, which are located at positions opposite each other vertically, are coordinate-converted to the same position coordinates, thereby collapsing the lip region in the transformed face image. Also, among the landmarks on the contour of the eyebrow region, each point on the upper contour of the eyebrow region and each point on the lower contour of the eyebrow region, which are located at positions opposite each other vertically, are coordinate-converted to the same position coordinates, thereby collapsing the eyebrow region in the transformed face image. On the other hand, for the eye region, a landmark (hereinafter referred to as an eye region outer periphery point) that surrounds the eye region one point outside the points on the outline of the eye region is used. That is, the eye region of the transformed face image is collapsed by converting the eye region outer periphery point of the upper eyelid and the eye region outer periphery point of the lower eyelid, which exist in positions opposite each other vertically, to the same position coordinates.

[0082] In this way, it is possible to eliminate regions unrelated to the skin condition of a human face, such as eyebrows and lips, from the input face image to the AI ​​model and the face image of the teacher data, while enlarging the skin region within the face image. As a result, it is possible to prevent a decrease in the resolution of the face region when inputting it to the AI ​​model, and ultimately to improve the estimation accuracy of the AI ​​model.

[0083] Fig. 12 is a graph showing the difference in prediction error between the skin condition estimation method according to the present embodiment (this method), the conventional method, and the prior patent method (WO 2018 / 222808) based on the sensory evaluation by experts. Fig. 12 shows the mean square error. For this comparison, a dataset consisting of facial images and impression evaluation items associated with the images was used. A total of 2,207 facial images were used, including images taken under multiple shooting conditions, such as images taken in a specified shooting environment and images taken by the subject in a random environment using their own smartphone. For all of these images, visual evaluations by experienced expert evaluators or a pre-trained machine learning model and measuring device that can predict the visual evaluation values ​​were used to link the images to corresponding impression evaluation scores. For the three methods, the conventional method, the prior patent method, and this method, the AI ​​model was trained to minimize the prediction error of the impression evaluation items compared to the sensory evaluation by experts, and the mean square error between the impression evaluation score and the predicted value was calculated. According to FIG. 12, the present method shows the smallest error, followed by the prior patent method, and then the conventional method shows the largest error.

[0084] Table 1 shows the results of comparing the mean square error for each of the following items: firmness, luster, smoothness of the skin surface, skin color, yellowness of the skin, and dullness, with the mean square error of the conventional method set to 100%: the conventional method, the prior patent method, and our method. Among these, the results were favorable for the items of luster, smoothness of the skin surface, skin color, and yellowness of the skin. [Table 1]

[0085] Therefore, according to this embodiment, it was demonstrated that the accuracy of skin condition estimation by an AI model can be improved by using a facial image converted by this method.

[0086] In this embodiment, a case was also considered in which the original face images of the above-mentioned face image samples were transformed using a coordinate transformation rule different from the coordinate transformation rule exemplified in FIG. FIG. 13 is a diagram showing an example of facial image transformation using a coordinate transformation rule in which several meshes around the corners of the eyes, just inside E212 and E242, are enlarged more than other parts in order to estimate the wrinkles at the corners of the eyes in more detail, in other words, the magnification rate is increased. FIG. 14 is a diagram showing an example of facial image transformation using a coordinate transformation rule in which the area around the nasolabial folds, specifically the area below E25, is enlarged more than other parts in order to estimate the state of the nasolabial folds in more detail, in other words, the magnification rate is increased. Even if the facial images E41 and E42 transformed using such coordinate transformation rules are used, it is possible to estimate the skin condition of the human face, as in the above-described embodiment. In addition, when facial images E41 and E42 are used, the wrinkle area at the corner of the eyes or the nasolabial fold area is enlarged, so that the skin condition presented in that area, for example, the prominence of the nasolabial folds or wrinkles at the corner of the eyes, can be more accurately evaluated. [Explanation of symbols]

[0087] 5. Communication lines 7 AI Model 10 Information processing device (skin condition estimation device) (skin condition estimation model learning device) 11 CPU 12. Memory 13 Input / Output Interface 14 Communication unit 15 Display device 16 Input Devices 20 Server device 30 User terminals 31 Image acquisition means (camera) 32 Location identification means 33 Conversion methods 34 Information acquisition means 35 Information display means (display device)

Claims

1. A computer program for causing a computer having one or more processors to execute a skin condition estimation method using an estimation model that has been trained to output, from an input face image, a skin condition of a human face appearing in the input face image, the skin condition estimating method is executed by the one or more processors, An image acquisition step of acquiring a facial image of a subject; a conversion step of converting the acquired face image so that a plurality of points on a contour of a face region of a human face reflected in the acquired face image are positioned on an outer edge of the image in order to conform to an input rule in the estimation model, thereby obtaining a converted face image; an information acquiring step of acquiring facial skin condition information of the subject outputted from the estimation model by inputting the transformed facial image as the input facial image; A computer program comprising:

2. An estimation model that has been trained to output the skin condition of a human face captured in an input face image from the input face image; An image acquisition means for acquiring a facial image of a subject; a conversion means for converting the acquired face image so that a plurality of points on a contour of a face region of a human face reflected in the acquired face image are positioned on an outer edge of the image in order to conform to an input rule in the estimation model, thereby obtaining a converted face image; an information acquisition means for inputting the transformed face image as the input face image to the estimation model and acquiring facial skin condition information of the subject output from the estimation model; a skin condition output means for generating image information for presenting the acquired skin condition information; an information display means for displaying the image information; A computer program on a user terminal constituting a skin condition estimating system together with a server device, comprising: causing the user terminal to function as the image acquisition means, the skin condition output means, the information display means, and the conversion means; Computer program.

Images