Skin state estimation method

Abstract

In the skin state estimation method, one or more processors capable of using a learned model that estimates, from an input face image having a given image size and a given rectangular shape, the skin state of a human face reflected in the input face image. The steps are as follows: acquiring a face image of a subject; determining position coordinates of a plurality of parts in a face area of the human face reflected in the acquired face image; converting the face image so that a part or all of the position coordinates of the specified plurality of parts are subjected to coordinate conversion, and the face area is formed into a given image size and a given rectangular shape; and inputting the converted face image into the learned model to acquire skin state information of the face of the subject.

Description

Technical Field

[0001] The present invention relates to a technology for estimating the skin condition of a face of a person reflected in an input facial image, using a learned estimation model. Background Art

[0002] Patent document 1 listed below discloses a system that determines a person's apparent skin age by analyzing an image using a convolutional neural network after masking macro features of the person's face in the image, and displays the determined apparent skin age on a display device that can be visually recognized by the user.

[0003] Prior Art Literature

[0004] Patent Document 1: International Publication No. 2018 / 222808 Summary of the invention

[0005] According to the present invention, there is provided a skin condition estimation method, which uses an estimation model that has been learned so that the skin condition of a face projected in an input facial image is outputted based on an input facial image, and is executed by one or more processors: an image acquisition step of acquiring a facial image of a subject; a conversion step of converting the facial image to conform to an input rule of the estimation model so that multiple points on the contour of the facial area of ​​the face projected in the acquired facial image are located at the outer edge of the image, thereby obtaining a converted facial image; and an information acquisition step of acquiring skin condition information of the face of the subject that is outputted from the estimation model by inputting the converted facial image as the input facial image.

[0006] In addition, according to the present invention, a skin condition estimation system is provided, which comprises: an estimation model, which implements learning so that the skin condition of a face reflected in the input facial image is outputted based on the input facial image; an image acquisition unit, which acquires the facial image of a subject; a conversion unit, which converts the facial image to conform to the input rule of the estimation model so that multiple points on the contour of the facial area of ​​the face reflected in the acquired facial image are located at the outer edge of the image, thereby obtaining a converted facial image; an information acquisition unit, which inputs the converted facial image as the input facial image into the estimation model and acquires the skin condition information of the face of the subject outputted from the estimation model; a skin condition output unit, which generates image information for prompting the acquired skin condition information; and an information display unit, which displays the image information.

[0007] Furthermore, according to the present invention, it is possible to provide a user terminal for use in the above-mentioned skin condition estimating system.

[0008] Furthermore, a program for causing a computer to execute the above-mentioned skin condition estimation method, that is, causing a computer to function as all or part of the above-mentioned skin condition estimation system, and a computer-readable recording medium having such a program recorded thereon can also be provided. The recording medium includes a non-transitory tangible medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a diagram conceptually showing an example of the hardware configuration of an information processing device capable of executing the skin condition estimating method according to the present embodiment.

[0010] Figure 2 This is a block diagram for explaining the functions of each unit of the skin condition estimating device according to the present embodiment.

[0011] Figure 3 : is a flowchart showing the skin condition estimating method involved in this embodiment.

[0012] Figure 4 1 is a diagram showing a conversion flow of a facial image in the skin condition estimating method according to the present embodiment.

[0013] Figure 5 This is a flowchart showing a method for learning a skin condition estimation model according to the present embodiment.

[0014] Figure 6 1 is a diagram showing a conversion flow of a facial image in a skin condition estimating method according to a modification.

[0015] Figure 7 This is a flowchart showing a skin condition estimation method according to a modification.

[0016] Figure 8 This is a diagram functionally showing an embodiment in which a skin condition estimation system is configured as a combination of a user terminal and a server.

[0017] Fig. 9 This is a diagram functionally showing another embodiment of a skin condition estimation system configured as a combination of a user terminal and a server.

[0018] Fig.10 2 is a diagram showing an example of a display screen of skin condition information on a user terminal.

[0019] Fig.11 It is a diagram showing the conversion flow of a facial image in the skin condition estimating method according to the embodiment.

[0020] Fig.12This is a graph showing the difference in prediction error among the skin condition estimating method according to the present embodiment (present method), the conventional method, and the prior patent method.

[0021] Fig.13 FIG. 1 is a diagram showing an example of facial image conversion using a coordinate conversion rule that makes the magnification around the outer corner of the eye larger than that of other parts in order to estimate the wrinkles at the outer corner of the eye in more detail.

[0022] Fig.14 1 is a diagram showing an example of facial image conversion using a coordinate conversion rule that makes the magnification of the area around the nasolabial folds larger than that of other parts in order to estimate the state of the nasolabial folds in more detail. DETAILED DESCRIPTION

[0023] In a conventional method for estimating the skin condition of a face of a person in a face image using a learned estimation model as described in the system of the background art, the face image input to the estimation model is normalized into a predetermined image size and rectangular shape, and the input face image is analyzed. On the other hand, the face image input to the estimation model includes not only the face area but also areas other than the face such as the background and hair.

[0024] Therefore, in its learning process, the inference model must not only learn the skin condition of the face area, but also learn not to use areas that do not look like skin, and must maintain the knowledge resources learned in this way.

[0025] However, for the estimation model, in order to estimate the skin condition of the face, calculations or information related to areas other than the face are redundant. In addition, when the face image is normalized, the image size is compressed to include areas other than the face, and the resolution of the face area is reduced, so the estimation accuracy of the learning model may also be reduced.

[0026] The present invention takes the above-mentioned viewpoint into consideration and provides a technique for improving the estimation accuracy of an estimation model for estimating the skin condition of a face reflected in an input facial image based on the input facial image.

[0027] According to the present invention, it is possible to provide a technique for improving the estimation accuracy of an estimation model for estimating, from an input facial image, the skin condition of a face reflected in the input facial image.

[0028] Hereinafter, an example of a preferred embodiment of the present invention (hereinafter referred to as the present embodiment) will be described. In addition, the present embodiment described below is for illustration only, and the present invention is not limited to the configuration exemplified below.

[0029] (Hardware configuration example)

[0030] The skin condition estimating method according to the present embodiment (hereinafter referred to as the present method) is executed by one or more processors included in one or more information processing devices.

[0031] Figure 1 This is a diagram conceptually showing a hardware configuration example of an information processing device 10 that can execute the present method.

[0032] The information processing device 10 is a so-called computer, and includes a CPU (Central Processing Unit) 11, a memory 12, an input / output interface (I / F) 13, a communication unit 14, etc. The information processing device 10 may be a fixed-type PC (Personal Computer) or a mobile terminal such as a portable PC, a smartphone, or a tablet computer.

[0033] The CPU 11 is a so-called microprocessor. The memory 12 is a RAM (Random Access Memory), a ROM (Read Only Memory), or an auxiliary storage device (a hard disk or the like).

[0034] The input / output I / F 13 can be connected to a user interface device such as a display device 15 and an input device 16. The display device 15 is a device such as an LCD (Liquid Crystal Display) that displays a screen corresponding to the drawing data prepared by the CPU 11, etc. The input device 16 is a device such as a keyboard and a mouse that accepts input from a user operation. The display device 15 and the input device 16 can also be integrated and implemented in the form of a touch panel.

[0035] The communication unit 14 communicates with other computers via a communication network, exchanges signals with other devices such as a printer, etc. A portable recording medium or the like can also be connected to the communication unit 14 .

[0036] The hardware structure of the information processing device 10 is not limited to Figure 1 The information processing device 10 may also include other hardware components not shown in the figure. In addition, the number of each hardware component is not limited to Figure 1 For example, the information processing device 10 may also have a plurality of CPUs 11. In particular, when a GPU (Graphics Processing Unit) designed for vector operations is used instead of a normal CPU to implement a convolutional neural network for estimating skin conditions, the operation can be accelerated.

[0037] In addition, the present invention can also be implemented in the form of a so-called system by a plurality of computers composed of a plurality of information processing devices 10. When the present invention is implemented by a plurality of computers, it is preferred to share the functions of the plurality of computers as described below, and to operate as a system that functions by combining a server and a terminal connected via communication via a network such as the Internet. In addition, in the following description, the form of implementing the present invention by using the information processing device 10 is taken as an example.

[0038] The information processing device 10 executes the present method by executing a computer program stored in a memory 12 through a CPU 11. The computer program is temporarily installed from a removable recording medium such as a CD (Compact Disc) or a memory card or other computer on a network via an input / output I / F 13 or a communication unit 14 and then executed, or is directly stored in the memory 12 and executed, thereby executing the present method. In addition, the present device or the present system is implemented in the form of a combination of the following means.

[0039] The information processing device 10 (CPU 11 ) can use a learned model obtained by machine learning using teaching data.

[0040] The "learned model" here is a model obtained by machine learning using teaching data, that is, supervised learning, and can be recorded as an AI (Artificial Intelligence) model, a machine learning (ML) model, etc.

[0041] The learned model used in this embodiment can be a regression equation obtained by regression analysis, or a neural network model obtained by deep learning, etc. The data structure and learning algorithm of the model are not limited. In this embodiment, as the learned model, a neural network (CNN) model including convolution operation is preferably used.

[0042] The learned model may be stored in the memory 12 in the information processing device 10 or in the memory of another computer to which the information processing device 10 can access through communication.

[0043] As described above, the information processing device 10 according to the present embodiment is a device that can utilize a learned model and is a skin condition estimating device that can execute a skin condition estimating method.

[0044] Hereinafter, the learned model used in the present method is recorded as an AI model or an estimated model. In addition, the facial image group used as teaching data in the learning of the AI ​​model is normalized according to a common rule as described below so that the coordinates of the corresponding positions are consistent, and the rule is recorded as an input rule of the AI ​​model. In addition, in the following description, the execution subject of the present method is set to CPU11 for description.

[0045] (Skin condition estimation method and device)

[0046] Below, use Figure 2 , Figure 3 and Figure 4 The details involved in this embodiment will be described. Figure 2 This is a block diagram for explaining the functions of each unit of the skin condition estimating device (sometimes simply referred to as the present device) according to the present embodiment. Figure 3 is a flowchart showing a skin condition estimation method according to the present embodiment. Figure 4 1 is a diagram showing a conversion flow of a facial image in the skin condition estimating method according to the present embodiment.

[0047] If the steps of the method are described in the form of functional blocks, the device is as follows: Figure 2 As shown, it comprises: an image acquisition unit, which acquires a facial image of a subject; a position determination unit, which determines the position coordinates of a given plurality of parts in a facial area of ​​a face projected in the facial image acquired by the image acquisition unit; a conversion unit, which converts the facial image in order to conform to an input rule of an AI model so that a plurality of points on the contour of the facial area of ​​the face projected in the acquired facial image are located at an outer edge of the image, thereby obtaining a converted facial image; an information acquisition unit, which inputs the converted facial image as an input image into a pre-prepared AI model formed by a convolutional neural network, thereby obtaining skin condition information of the face of the subject; and an information display unit, which generates image information for prompting the acquired skin condition information.

[0048] The description of each step below is also applicable to the corresponding units unless otherwise specified, so for the operation of each unit, please refer to the description of each step above.

[0049] In addition, each unit is implemented by appropriately combining the CPU 11 of the computer with peripheral circuits and computer programs or logic circuits for implementing each function. For example, the unit for performing skin condition estimation using an AI model is preferably implemented using a GPU, but can also be implemented by combining a program with a normal CPU. In other words, it is possible to provide a program that enables a computer to function as all or part of the above-mentioned skin condition estimation system, or to provide a computer-readable recording medium that records such a program.

[0050] In this embodiment, the AI ​​model used in the information acquisition unit and the information acquisition step is a learned model that estimates the skin condition of the face reflected in the input facial image based on an input facial image of a given image size, that is, a rectangular shape with a given number of vertical and horizontal pixels (referred to as a given rectangular shape in this specification). Therefore, the AI ​​model used in this embodiment is also recorded as a skin condition estimation model.

[0051] The “given image size” refers to a predetermined size of an image, which is specified by the number of pixels. The image size is predetermined to be, for example, 25×20 (500) pixels, 20×20 (400) pixels, 20×11 (220) pixels, etc. The image sizes shown here are just examples.

[0052] The “predetermined rectangular shape” means a predetermined rectangular shape of an image, and is a rectangle or a square having long and short side lengths corresponding to the above-mentioned “predetermined image size”.

[0053] When a facial image is described as a “given rectangular shape” below, it means a rectangular shape having a predetermined number of vertical and horizontal pixels for an input facial image of an AI model.

[0054] In addition, the given rectangular shape is shown as an example in this embodiment, but the shape of the input image to the AI ​​model that can be used, that is, the given shape, includes a circle, an ellipse, etc., in addition to a rectangle. Regardless of the shape, by converting according to a common rule so that the coordinates of the corresponding positions are consistent, the same effect as when the given shape is a rectangle can be obtained.

[0055] The information inferred by the AI ​​model can be any information indicating the skin condition of the face, and the specific inferred items are not limited. As inferred items of skin condition, for example, as exemplified in the embodiments, makeup feeling, skin age, male / female skin characteristics, powdery feeling, glossy degree, wetness, moisturization, dryness, smoothness, texture fineness, moisturization, firmness, transparency, skin whiteness, yellowness, dullness, etc. can be inferred. The makeup feeling indicates a non-bare face feeling, the powdery feeling indicates a powdery feeling, and the glossiness indicates the degree of surface gloss. The inferred items of the AI ​​model are not limited to such items, but can also be other items such as comprehensive aesthetic evaluation.

[0056] The AI ​​model can be formed to be able to estimate multiple items related to the skin condition of the face, or it can be formed to be able to estimate any one item. In the former case, for example, the AI ​​model can also be constructed using multiple neural networks that can output estimated information of different skin condition items. In the latter case, a neural network can also be constructed in a way that outputs estimated information of multiple skin condition items.

[0057] In addition, the learning method of the AI ​​model will be described below.

[0058] like Figure 3 As shown, the method includes an image acquisition step (S21), a position determination step (S22), a conversion step (S23) and an information acquisition step (S24) (S25).

[0059] In the image acquisition step (S21), the CPU 11 acquires a facial image of a subject. Figure 4 In the example of , reference symbol G1 represents an acquired facial image.

[0060] In the acquired facial image, the subject's face may be included to the extent that the skin condition of the estimated item can be analyzed, and the entire or a part of the subject's face may be included. In the acquired facial image, parts other than the subject's face such as hair, neck, or background may also be included.

[0061] In addition, there is no particular limitation on the angle and position of the subject reflected in the facial image. For example, the acquired facial image may include the front face of the subject, or the left side face or the right side face of the subject. In addition, the facial image may be acquired by photographing the subject's face from below or above.

[0062] The facial image acquired by the image acquisition step, i.e., the image acquisition unit, is acquired as an image file in, for example, a JPEG (Joint Photographic Experts Group) format, a BMP (Bitmap image) format, a TIFF (Tagged Image File Format) format, a GIF (Graphic Interchange Format) format, or the like.

[0063] Furthermore, the acquired facial image may be a color image or a grayscale image as long as the skin condition of the face of the person reflected in the facial image can be estimated.

[0064] The CPU 11 acquires a facial image from a camera that has captured the face of a subject, another computer, or the like, via a communication line or a portable recording medium.

[0065] Hereinafter, the facial image acquired in step ( S21 ) may also be described as an original facial image.

[0066] In addition, the acquired original facial image does not need to have the same size as the given image size of the given rectangular shape, but the total number of pixels is preferably greater than the given image size of the given rectangular shape.

[0067] The position specifying step ( S22 ) will be described in detail.

[0068] In the position determination step (S22), first, as described below, after selecting a point on the contour of the entire face region as a first contour point, the interior of the first contour point is divided into triangular meshes. Thereafter, the vertices of a plurality of meshes are selected as third contour points in such a manner that a figure obtained by connecting them includes a given local part region, and then a second contour point is selected between the first contour point and the third contour point. More specifically, after the face region is divided into a plurality of meshes, contour points are selected from points (mesh points) where the edges of each mesh intersect. The first contour point is a point located at the boundary of the contour of the entire face, that is, the face and the part of the face other than the hair and the neck, preferably a point located slightly inside the boundary. The third contour point is selected as a mesh point when the edges of the meshes are connected in such a manner that the given local part region is completely included inside thereof, and is equal to the shortest mesh number (hereinafter, the minimum number of mesh edges connecting two mesh points is referred to as mesh distance) of the first contour point. Then, the second contour point is selected as a mesh point between the first contour point and the third contour point.

[0069] After the device automatically generates a grid and finds the position of the first contour point on the image, the position of the third contour point is determined on the image so that the grid distance from the first contour point is equal. Thereafter, the position of the second contour point on the image is found based on the generated grid.

[0070] In addition, when the first to third contour points cannot be set according to the above-mentioned criteria, for example, when the grid distance between the first contour point and the third contour point is less than 1, or when not all third contour points are at the same grid distance from the first contour point, the grid is refined (increasing the total number of grids) and the position of each grid point on the image is changed so that the first to third contour points can be selected.

[0071] In the position determination step (S22), the CPU 11 determines the position coordinates of a plurality of predetermined parts (predetermined position coordinates) in the facial region of the face of the person reflected in the original facial image. Figure 4In the example of FIG. 1 , the position coordinates of a plurality of points within the rectangle of the reference numeral G2 are identified.

[0072] The position coordinates determined in step (S22) at least include the position coordinates of the shape feature points of the facial area. The shape feature points of the facial area represent points of positions having the shape features of a human face, such as points on the contour of the face, points on the contour of the eyes (upper eyelid margin and lower eyelid margin), points on the contour of the eyebrows, points on the contour of the lips (upper edge of the upper lip and lower edge of the lower lip), the apex of the nose (points on the tip of the nose), points on the bridge of the nose, points on the root of the nose, etc. However, the position coordinates determined in step (S22) are not limited to the position coordinates of such shape feature points, but may also be the position coordinates of a given part in the facial area derived from such shape feature points, and may also include the position coordinates of the shape feature points and the position coordinates of other parts.

[0073] In addition, the position determination step (S22) is executed by the CPU 11, and can also be determined in sequence by the user in accordance with instructions in the original face image displayed on the display device. In this case, although the position can also be determined using a pointing device such as a mouse, it is preferable to perform the operation using a touch screen in terms of operability, that is, using a touch screen on a so-called smart phone terminal. In addition, since the operation performed by the user is relatively complicated, it is undoubtedly preferable to use a computer to automatically perform the operation.

[0074] The predetermined position coordinates determined in step ( S22 ) are preferably position coordinates including a plurality of peripheral points surrounding a predetermined local part region in the face region.

[0075] The "given local area" is a part that is preferably removed from the facial area within the facial image input to the AI ​​model because it does not contribute to the estimation of the skin condition, such as the eyes, eyebrows, lips, nostrils, mustache, beard, etc.

[0076] As a method of removing pixels of a given local region that is preferably removed from the face region in the face image input to the AI ​​model, in this embodiment, the pixels of the skin surrounding the given local region are arranged on a straight line or a curve of the face model, so that the pixels in the given local region are not projected onto the face model. At this time, the positions of the mutually opposing peripheral points of the given local region become consistent on the face model.

[0077] In this embodiment, the "given local area" is the eye area, eyebrow area and lip area. The lip area is the upper lip area and the lower lip area, but in addition, when the subject reflected in the face image is in a state of opening the mouth, it also includes the intraoral area.

[0078] The plurality of peripheral points may be points positioned so as to surround a given local part region in the face region, and may be points on the contour of the given local part region or points located outside the contour of the given local part region.

[0079] In this way, the position coordinates of a plurality of peripheral points surrounding a given local part region in the face region are determined in step ( S22 ), and thus the given local part region can be removed in the conversion step ( S23 ) described below.

[0080] The given local area is preferably completely removed, but even in a reduced form, it can help improve the estimation accuracy. For example, when points on the contour of the given local area or points slightly inside the contour are determined as the multiple peripheral points, and the determined peripheral points are arranged on a straight line or curve in the face model, the given local area will be reduced and remain. Even in such a situation, since the area that does not contribute to the estimation of the skin condition can be reduced, the estimation accuracy of the skin condition using the AI ​​model can be improved.

[0081] Furthermore, the given position coordinates determined in step ( S22 ) preferably include the position coordinates of a plurality of points on the contour of the face region, i.e., on the boundary between the face region and other regions, the nose vertex inside the face region, and a plurality of other points inside the face region.

[0082] In the following conversion step (S23), four first contour points among the points on the contour of the face region are arranged at the vertices of a given rectangular shape of the face image, and the other points on the contour of the face region are arranged on the sides of the given rectangular shape of the face image. Therefore, the four first contour points are set as two points on the upper left and right sides of the contour of the face region and two points on the lower left and right sides of the contour.

[0083] exist Figure 4 In the example, two points on the contour adjacent to the left and right eyebrows (G21 and G24) and two points on the contour adjacent to the left and right corners of the mouth (G22 and G23) correspond to the first contour point, and the nose vertex is shown as G25. However, the first contour point can be any combination of two points on the upper left and right sides of the contour of the face area and two points on the lower left and right sides of the contour, and is not limited to Figure 4 Example.

[0084] In the method for determining the position coordinates in step (S22), facial recognition processing used in various scenarios can be used. In facial recognition processing, for example, an AI model that can detect the positions of multiple nodes of a face, such as Facemark of OpenCV (Open Source Computer Vision Library) or FaceMesh of Google (registered trademark), is used. In this embodiment, the method for determining the position coordinates in step (S22) is not limited in any way. For example, in step (S22), CPU11 can determine the position coordinates of part or all of the marks detected using such a facial recognition AI model as the position coordinates of multiple parts in the face area.

[0085] In the conversion step (S23), the CPU 11 performs a so-called mapping operation to convert the position of each pixel of the original facial image obtained in step (S21) so that the position coordinates of the given multiple parts determined as the reference in step (S22) are converted to make the facial area a given rectangular shape, and the position of each pixel of the original facial image becomes a position corresponding to the input rule of the AI ​​model. That is, the conversion step (S23) can also be called a coordinate conversion step or a mapping step. Similarly, the conversion unit described below can also be called a coordinate conversion unit or a mapping mechanism.

[0086] exist Figure 4 In the example, the position coordinates after the coordinate conversion are shown in the rectangle shown by the reference numeral G3, and the reference numeral G4 represents the converted facial image. In other words, while each pixel maintains the value, only its position is converted. When the position of each pixel changes, the spatial distance between the pixels will change. Therefore, when the distance is extended, interpolation is used to find the value of the pixel filling in between, and when the distance is shortened, operations such as using the average value are performed to find the value of the new pixel. Regarding the specific method, it is sufficient to use well-known techniques such as gradient technology, so the details are omitted. In addition, there are occasional situations where the position coordinates of a given part of the original facial image that serves as a reference are consistent with the position corresponding to the input rule of the AI ​​model. Those skilled in the art will of course understand that this situation is also included in the mapping operation or position conversion.

[0087] As long as the coordinate conversion is performed so that the face region of the original face image becomes a given rectangular shape, the method is not limited.

[0088] In addition, the position determination step (S22) and the conversion step (S23) can also be implemented as one conversion step. That is, as long as it is determined that a certain pixel of the original facial image corresponds to a certain position of the reference image, the coordinates of the conversion target (projection target) can be immediately determined, so it is not necessarily necessary to use a two-stage step. This also applies to the position determination unit and the conversion unit, and the present device can also be configured as a conversion unit that is composed of the two.

[0089] In this way, by converting the original facial image into a facial image, it is possible to eliminate areas other than the facial area in the facial image input to the AI ​​model, and to enlarge the facial area, wherein the facial image is a facial image of a given rectangular shape and the facial area occupies the entirety of the facial image. Thus, it is possible to prevent the resolution of the facial area in the facial image input to the AI ​​model from being reduced, and to improve the estimation accuracy of the AI ​​model.

[0090] This effect is more significant when an AI model is implemented using a neural network model that includes convolution operations. In such an AI model, convolution operations are repeatedly performed while information other than the face area of ​​the input image is stored, so the influence of the non-analysis object area (area other than the skin area) contained in the input image also remains in the deep layer of the AI ​​model. As a result, useless information will remain in the AI ​​model. Furthermore, the inclusion of information other than the face area results in a reduction in the information that is the object of analysis. These factors lead to a decrease in the estimation accuracy of the AI ​​model.

[0091] When the position coordinates of the multiple points on the contour of the face area, the nose vertex of the face area, and other multiple points of the face area are determined in step (S22), the coordinate conversion in step (S23) is performed as follows. Figure 4 As shown in the example, the coordinate transformation is performed so that four first contour points (G21, G22, G23 and G24) included in the multiple points on the contour of the face area of ​​the original face image are respectively arranged at the vertices of a given rectangular shape, and the nose vertex (G25) is arranged at the center of the given rectangular shape, and the other multiple points in the face area are moved corresponding to the movement of the first contour points and the nose vertex.

[0092] By performing the coordinate conversion in this way, the face area can occupy the entire face image of a given rectangular shape after the conversion. In addition, according to the coordinate conversion, the difference in the shape or size of the contours of the faces of each person reflected in the original face image can be eliminated and fixed. As a result, the positions of the nose, mouth, eyes, etc. in the face area of ​​the face image input to the AI ​​model can be roughly standardized, so that the learning efficiency of the AI ​​model can be made more efficient, and the estimation accuracy of the AI ​​model is also improved.

[0093] In this embodiment, in step (S22), the position coordinates of a plurality of peripheral points surrounding a given local part region in the face region are determined, and therefore the image conversion in step (S23) is performed as follows. That is, the CPU 11 makes the position coordinates of the peripheral points that are mutually opposed across the given local part region consistent during the coordinate conversion, in other words, the CPU 11 connects the peripheral points and removes the given local part region in the face region of the original face image.

[0094] The "connection of peripheral points" here can be achieved not only by converting the coordinates of two or more peripheral points facing each other across the given local area into the same position coordinates, but also by moving the two or more peripheral points to close positions so that the local area is eliminated or reduced.

[0095] Thus, it is possible to exclude local areas irrelevant to the skin condition of the face from the face image input to the AI ​​model, and to enlarge the skin area in the face image. As a result, compared with the existing method, it is possible to prevent the resolution of the face area from being reduced when the number of pixels in the face area of ​​the original face image is less than the given rectangular shape of the input image of the AI ​​model, thereby improving the estimation accuracy of the AI ​​model.

[0096] Therefore, the predetermined local area is preferably any one of the eye area, eyebrow area, or lip area that is not related to the skin condition, more preferably any one or more of these, and even more preferably all of these.

[0097] In the information acquisition steps ( S24 ) and ( S25 ), the CPU 11 inputs the facial image converted in step ( S23 ) into the AI ​​model ( S24 ), thereby acquiring skin condition information of the face of the subject ( S25 ).

[0098] When a program implementing the AI ​​model is stored in the memory 12 in the information processing device 10 , the CPU 11 can obtain the skin condition information of the subject's face as the output of the AI ​​model by inputting the facial image converted in step ( S23 ) to the AI ​​model.

[0099] As described above, the skin condition information obtained in step (S25) only needs to be information indicating the skin condition of the subject's face, and the specific estimated items are not limited. When the AI ​​model is formed to be able to estimate multiple items related to the skin condition of the face, the skin condition information obtained can be information on multiple skin condition items or information on one skin condition item.

[0100] In recent years, there are various technologies that show the basis for the judgment of AI models. For example, there is a technology called GradCam for CNN AI models that allows people to know which area contributes to the conclusion.

[0101] Therefore, in step (S23), coordinate transformation may be performed using such a technique so that more emphasis is placed on the part of the facial area that contributes to the estimated result of the AI ​​model. Alternatively, in step (S25), CPU 11 may also obtain information about the skin condition of the face of the subject and information about the position of the part of the facial area corresponding to the skin condition information. At this time, CPU 11 may also output a facial image in which the part of the facial area of ​​the original facial image is displayed in a recognizable manner. By setting the step of displaying a part that contributes to identifying the skin condition on the screen, an image that allows people to easily understand which part of the subject is suitable for surgery or application of the product can be displayed, which is preferred.

[0102] Furthermore, CPU 11 may further perform the following step, that is, based on the skin condition information obtained in step (S25), output recommendation information of one or both of the cosmetic surgery method or cosmetic products corresponding to the skin condition of the subject's face. For example, the output recommendation information includes information on cosmetic surgery methods or cosmetic products such as makeup costs and supplements recommended to improve the subject's skin condition or achieve the subject's desired skin age. At this time, when it is possible to obtain the part information of the facial area corresponding to the skin condition information as described above, the output recommendation information can also prompt different cosmetic products or cosmetic surgery methods for each part of the facial area. In addition, when the AI ​​model is formed to be able to infer multiple skin condition items, the output recommendation information can also prompt different cosmetic products or cosmetic surgery methods for each skin condition item.

[0103] A plurality of cosmetic surgery methods and a plurality of cosmetic products are associated with each skin condition (each skin condition of each skin condition item) that can be estimated by the AI ​​model and stored in the memory 12. The CPU 11 can determine one or more cosmetic surgery methods or one or more cosmetic products corresponding to the skin condition information obtained in step (S25) based on the information stored in the memory 12, and can generate recommendation information including one or both of the determined methods or products. It can also be configured to store the association information of the method / product and the skin condition in the storage device of another computer and refer to it via communication.

[0104] The output form of the recommendation information is not particularly limited. For example, the CPU 11 may display the recommendation information on the display device 15, print it on a printer, send it to another computer, or store it in a portable recording medium in the form of an electronic file.

[0105] (Learning method of AI model (skin condition estimation model))

[0106] use Figure 5 The learning method of the AI ​​model (skin condition estimation model) used in the above-mentioned present method will be described.

[0107] Figure 5 is a flowchart showing a learning method for a skin condition estimation model according to the present embodiment (hereinafter sometimes referred to as the present learning method). The present learning method can be performed by Figure 1 The information processing device 10 (CPU 11 ) shown as an example is executed.

[0108] The learning method generally includes: a step of obtaining teaching data (S41), a step of converting each facial image included in the teaching data (S42)(S43); and a step of using the teaching data including the converted facial image to make the AI ​​model learn (S44). The following is a detailed description of each step of the learning method.

[0109] In step ( S41 ), the CPU 11 acquires teaching data including a combination of facial images of a plurality of persons and correct answer information indicating the skin conditions of the faces of the persons reflected in the facial images.

[0110] Each facial image included in the teaching data may be, as described above with respect to the facial image obtained in step (S21) of the present method, a facial image of the subject as long as the face is reflected to a degree that allows the skin condition of the estimated item to be analyzed. Each facial image included in the teaching data may be normalized into a given rectangular shape, or may include a facial image having an image size different from a given image size of the given rectangular shape or a shape different from the given rectangular shape.

[0111] The correct answer information included in the teaching data is information indicating the skin condition of the face obtained by the evaluator by visually evaluating the corresponding facial image or the subject of the facial image itself or by using a measuring device. When the AI ​​model is formed to be able to infer multiple items related to the skin condition of the face, the correct answer information is used as information on multiple skin condition items.

[0112] Step ( S42 ) and step ( S43 ) are performed for each facial image included in the teaching data obtained in step ( S41 ).

[0113] In step ( S42 ), the CPU 11 specifies position coordinates (given position coordinates) of a plurality of given parts in the facial region of the face of the person reflected in the target facial image.

[0114] In step ( S43 ), the CPU 11 performs coordinate conversion on the target facial image so that the facial region becomes a predetermined rectangular shape based on the position coordinates of the plurality of predetermined parts determined in step ( S42 ).

[0115] The processing contents of step ( S42 ) and step ( S43 ) are the same as those of the above-mentioned steps ( S22 ) and ( S23 ).

[0116] When steps (S42) and (S43) are executed for all facial images included in the teaching data, in step (S44), CPU 11 uses the teaching data including the converted facial images to learn the AI ​​model. In this embodiment, the AI ​​model is learned by deep learning. However, the specific learning algorithm for the AI ​​model is not limited in any way.

[0117] Thus, in the present learning method, a face image in which the face region occupies the entire face image is used as teaching data to make the AI ​​model learn. That is, the face image used for learning the AI ​​model is an image in which at least a plurality of points on the contour of the face region are located at the outer edge of the image, and an image in which all points on the contour of the face region are located at the outer edge of the image is more preferred.

[0118] Thus, the learning process of using the facial image after excluding the image area other than the facial area as teaching data to learn that the image area other than the facial area is not used for estimation can be omitted, so the learning process of the AI ​​model can be made more efficient. In addition, in the facial image used as teaching data in the present learning method, the facial area is enlarged by the amount of the number of pixels of the excluded image area other than the facial area, so that the resolution of the facial area of ​​the estimation object can be suppressed from being reduced, and thus a high-precision AI model can be achieved.

[0119] Furthermore, the facial image used in learning the AI ​​model is preferably an image in which a given local area within the facial region is removed or reduced, and more preferably a facial image in which multiple points on the contour of the facial region (including the first contour point), the nose vertex inside the facial region, and other multiple points inside the facial region (including the second contour point and the third contour point) have been coordinate-transformed according to the method described in the present method.

[0120] The facial images used in learning the AI ​​model are subjected to coordinate transformation (normalization) according to such a common rule so that the coordinates of corresponding positions are consistent.

[0121] [Modifications]

[0122] The contents of the above-described embodiments can be appropriately changed within a range not causing any adverse effects.

[0123] For example, in the above AI model example, a facial image of a given rectangular shape is used as teaching data for learning, but a facial image of a non-rectangular shape such as a circle or an ellipse may also be used as teaching data for learning. Even in this case, in the conversion step (S23), the CPU 11 converts the original facial image so that multiple points on the contour of the facial area of ​​the face reflected in the original facial image are located at the outer edge of the image, that is, the facial image is in a given shape other than a rectangle and the facial area occupies the entirety of the facial image, thereby obtaining a converted facial image.

[0124] Furthermore, in the above-mentioned embodiment, as a preferred mode, the process of removing or reducing the predetermined local part region in the face region is performed in the conversion step ( S23 ), but this process may be omitted.

[0125] When the AI ​​model is formed into a plurality of items capable of estimating skin condition, the above-mentioned embodiment can be used as follows: Figure 6 and Figure 7 Changes as shown.

[0126] Figure 6 is a diagram showing a conversion flow of a facial image in a skin condition estimating method according to a modified example (hereinafter referred to as the present modified method). Figure 7 This is a flowchart showing a skin condition estimation method according to a modification.

[0127] The AI ​​model used in this variant method is formed, for example, by using a plurality of neural networks that can respectively output estimated information of one or more skin condition items that are different from each other. Each neural network is learned using the following teaching data, which includes: a combination of a facial image that is transformed into an input rule corresponding to one or more skin condition items that are the objects of estimation and the correct answer information of the skin condition items. Therefore, it can be said that the input rules in the AI ​​model used in this variant method are set in plurality.

[0128] The CPU 11 can use a plurality of different coordinate conversion rules corresponding to a plurality of estimation items in order to obtain a facial image that fits the input rule of the AI ​​model.

[0129] The coordinate conversion rule is used in the coordinate conversion in the following step ( S65 ), and specifies to which position in the converted facial image the position coordinates of the plurality of predetermined parts identified in the following step ( S64 ) are to be moved.

[0130] The specific content of the coordinate conversion rule is not limited as long as it can convert the original facial image as described in the above-mentioned embodiment so that the facial area occupies the entire face after conversion. For example, the following coordinate conversion rule is set, that is, in the skin area of ​​the face, the area corresponding to the skin condition item that is the object of estimation is enlarged, and the area with a weak correlation with the skin condition item is reduced. Specifically, the coordinate conversion rule corresponding to the skin condition item of skin age is stipulated to enlarge the outer corner of the eye area or the nasolabial fold area. In addition, the coordinate conversion rule corresponding to the skin condition item of dryness is stipulated to enlarge the eye corner area or the mouth corner area of ​​the facial skin that is known to be prone to dryness, and the coordinate conversion rule corresponding to the skin condition item of glossiness is stipulated to enlarge the area called the T zone, which is known to be a part with a large amount of sebum.

[0131] The coordinate conversion rules can be set to the same number as the number of items that the AI ​​model can infer, or to a number less than the number of items that the AI ​​model can infer. In the former case, a coordinate conversion rule is set for one estimated item, and a neural network is formed for each coordinate conversion rule, in other words, for each estimated item. In the latter case, there can be a mixture of one coordinate conversion rule set for multiple estimated items and one coordinate conversion rule set for one estimated item, or multiple coordinate conversion rules can be set, each of which is set for multiple estimated items. In this case, a neural network is also formed for each coordinate conversion rule, but not for each estimated item.

[0132] like Figure 7 As shown, this modified method includes steps (S61) to (S67).

[0133] Step (S61) is the same as the above-mentioned step (S21).

[0134] In step (S62), the CPU 11 obtains the estimated object item information. The estimated object item information is information indicating one or more requested skin condition items among a plurality of skin condition items that can be estimated by the AI ​​model. The CPU 11 can obtain the estimated object item information based on the input of the user using the input device 16, obtain the pre-set estimated object item information from the memory 12, or obtain the estimated object item information via communication from other computers.

[0135] In step ( S63 ), the CPU 11 selects a coordinate conversion rule corresponding to the estimation target item indicated by the estimation target item information acquired in step ( S62 ). At this time, when a plurality of items are designated as estimation target items, one or more coordinate conversion rules corresponding to the plurality of items are selected.

[0136] Step (S64) is the same as the above-mentioned step (S22).

[0137] In step (S65), CPU 11 transforms the original facial image by transforming the position coordinates of the plurality of parts determined in step (S64) based on the coordinate transformation rule selected in step (S63), thereby generating a transformed facial image corresponding to the estimation target item from the original facial image. The method for transforming the facial image in step (S65) is the same as that in step (S23) above, except that a coordinate transformation rule selected from a plurality of coordinate transformation rules is used.

[0138] However, when a plurality of coordinate conversion rules are selected in step (S63), a plurality of converted facial images corresponding to the plurality of estimation target items are generated in step (S65). Figure 6 In the example, N types (G3-1, G3-2, ..., G3-N) of position coordinates are shown as position coordinates after coordinate conversion based on N coordinate conversion rules, and N converted facial images (G4-1, G4-2, ..., G4-N) are generated.

[0139] Step (S66) is the same as the above step (S24), and step (S67) is the same as the above step (S25).

[0140] However, when multiple items are designated as the estimated target items, skin condition information related to multiple items of the subject's face is obtained in step (S67). In addition, multiple coordinate transformation rules are selected in step (S63), and multiple transformed facial images are generated in step (S65). In such a case, the multiple transformed facial images are input into the AI ​​model in step (S66), and skin condition information related to multiple items of the subject's face is obtained in step (S67).

[0141] According to this modified method, it is possible to obtain skin condition information of a subject related to a plurality of items. In addition, it is possible to obtain skin condition information of a subject related to a desired item among the plurality of items.

[0142] In other words, for each skin condition item to be estimated, the AI ​​model makes an estimate based on the facial image converted using the coordinate conversion rule corresponding to the skin condition item, thereby being able to improve the estimation accuracy of each estimated skin condition item.

[0143] This modified method can be further modified. For example, the modified method can be modified as follows: the step of obtaining the estimation target item information (S62) is omitted from this modified method, and skin condition information related to all estimation items that can be estimated by the AI ​​model is obtained.

[0144] [Another embodiment]

[0145] The structure of another embodiment of the skin condition estimation system according to the present invention (hereinafter referred to as this embodiment) is shown in Figure 8 In this embodiment, a user terminal 30 such as a smartphone is connected to a server device 20 via a network to form a system.

[0146] In this embodiment, the server device 20 includes an AI model 7 and an information acquisition unit 38. The user terminal 30 includes a camera 31 as an image acquisition unit, a position determination unit 32, a conversion unit 33, a transmission unit 34, an information display unit 35 as a display device, a skin condition output unit 36 ​​that converts the skin information presented by the information display unit 35 to generate image information that can be read by the user, and a receiving unit 37.

[0147] The user terminal 30 and the server device 20 are connected via a communication line 5 such as the Internet, and together constitute a skin condition estimation system. In addition, those skilled in the art can easily understand that there can be multiple user terminals 30. In this case, each combination of the user terminal 30 and the server device 20 constitutes a skin condition estimation system.

[0148] The converted facial image transmitted to the server device 20 by the transmitting unit 34 of the user terminal 30 is received by the receiving unit (not shown) of the server device 20, and is input to the AI ​​model 7 via the information acquisition unit 38. In the present embodiment, the skin condition information output from the AI ​​model 7 and acquired by the information acquisition unit 38 is transmitted from the transmitting unit (not shown) of the server device 20 to the user terminal 30 via the communication line 5. The skin condition information received by the receiving unit 37 of the user terminal 30 is converted into data constituting a display screen by the skin condition output unit 36, and is displayed on the display device 35 (screen) of the user terminal 30.

[0149] In this way, by configuring the AI ​​model 7 on the server device 20, the AI ​​model 7 can be replaced or updated without changing the computer program on the user terminal 30, thereby achieving improved accuracy. As a whole, the frequency of improved accuracy can be increased.

[0150] In the user terminal 30, it is preferred to transmit a program via a communication medium forming a network, that is, a program for causing the user terminal 30 to function as a position determination unit 32, a conversion unit 33, a transmission unit 34, an information display unit 35 as a display device, a skin condition output unit 36 ​​that converts the skin information presented by the information display unit 35 to generate image information readable by the user, and a receiving unit 37 in the present system. The program can be stored in a storage medium and provided to the user of the user terminal 30.

[0151] The structure of another embodiment of the skin condition estimation system according to the present invention (hereinafter referred to as this embodiment) is shown in Fig. 9 This embodiment is similar to Figure 8 In the illustrated example, a user terminal 30 such as a smartphone is connected to a server device 20 via a network to form a system.

[0152] The user terminal 30 includes a camera 31 as an image acquisition unit, a transmission unit 34, an information display unit 35 as a display device, a skin condition output unit 36, and a skin condition receiving unit 37. The server device 20 includes an AI model 7, a position determination unit 32, a conversion unit 33, and an information acquisition unit 38.

[0153] and Figure 8 Similarly, in the illustrated skin condition estimation system, the user terminal 30 and the server device 20 are connected via a communication line 5 such as the Internet, and the skin condition estimation system is integrally constituted. In addition, it is of course easy to understand for those skilled in the art that there may be multiple user terminals 30, and in this case, each combination of the user terminal 30 and the server device 20 also constitutes a skin condition estimation system.

[0154] In each user terminal 30, a program for causing the user terminal 30 to function as a transmitting unit 34, a skin condition output unit 36, and a skin condition receiving unit 37 in the present system is preferably transmitted via a communication medium forming a network. The program may be stored in a storage medium and provided to the user of the user terminal 30.

[0155] The original facial image captured by the camera 31 of the user terminal 30 is sent to the server device 20 by the sending unit 34, processed by the position determination unit 32 and the conversion unit 33 in the server device 20, and then input into the AI ​​model 7 via the information acquisition unit 38.

[0156] In this embodiment, the skin condition information output from the AI ​​model 7 is acquired by the information acquisition unit 38 of the server device 20, and is sent to the user terminal 30 by the sending unit (not shown) via the communication line 5. The skin condition information received by the receiving unit 37 of the user terminal 30 is converted by the skin condition output unit 36 ​​into data constituting a display screen, and is displayed on the display device 35 (screen) of the user terminal 30.

[0157] An example of a display screen processed and created by the skin condition output unit 36 ​​and displayed on the display device 35 of the user terminal 30 is shown in FIG. Fig.10 In this display screen, the original face image is displayed at the top of the screen, the skin age is indicated by numbers near the center of the screen, and the values ​​of each item are indicated by a radar graph at the bottom.

[0158] In this way, information processing is performed using the skin condition output unit 36 ​​so that a chart representing the original image and skin condition information is displayed on the same screen, thereby forming a screen that is easy to understand the skin condition, or information processing is performed using the output unit 36 ​​so that multiple skin condition information are displayed at once in one screen, thereby displaying an overall image of the skin condition at once.

[0159] Although the numerical information sent from the server device 20 can also be displayed in simple numbers, it is more important to display it in an easily visible manner after processing it using the skin condition output unit 36 ​​of the user terminal 30 as a communication tool with the user. If the display screen is generated by the server device 20, there may be disadvantages such as a part of the screen being missing or the text being difficult to read in the case of different user terminals 30. Therefore, it is preferable to convert the skin condition information sent from the server device 20 into a display on the user terminal 30 side as in the present embodiment. In addition, it is preferable to process each item of the skin condition into a graph and display it as in the present embodiment, and the comprehensive evaluation is expressed as a numerical value such as skin age or a score of a 5-level or 100-point full score standard.

[0160] A part or all of the above-mentioned embodiment and modification examples can also be specified as follows. However, the above-mentioned embodiment and modification examples are not limited to the following description.

[0161] <1>

[0162] A skin condition estimation method uses an estimation model, wherein the estimation model implements learning so as to output the skin condition of a face reflected in an input facial image based on the input facial image, and the skin condition estimation method is executed by one or more processors: an image acquisition step of acquiring a facial image of a subject; a conversion step of converting the acquired facial image to conform to an input rule in the estimation model so that multiple points on the contour of the facial area of ​​the face reflected in the facial image are located at the outer edge of the image, thereby obtaining a converted facial image; and an information acquisition step of acquiring skin condition information of the face of the subject output from the estimation model by inputting the converted facial image as the input facial image.

[0163] <2>

[0164] A skin condition estimating method as described in <1>, wherein, in the converted facial image, a given local area within the facial region of the human face projected in the acquired facial image is removed or reduced, and in the conversion step, the positions of peripheral points of a plurality of peripheral points surrounding the given local area in the facial region that are opposite to each other across the given local area are made consistent to remove or reduce the given local area in the facial region.

[0165] <3>

[0166] The skin condition estimating method as described in <2>, wherein the given local part area includes an eye area, an eyebrow area, and a lip area.

[0167] <4>

[0168] A skin condition estimation method as described in <2> or <3>, wherein the one or more processors further execute: a position determination step of determining the position coordinates of a given plurality of parts in the facial area of ​​the human face projected in the acquired facial image; and in the conversion step, converting the acquired facial image so that a part or all of the position coordinates of the determined given plurality of parts are converted to conform to the input rule of the estimation model.

[0169] <5>

[0170] A skin condition estimation method as described in <4>, wherein the input facial image has a given rectangular shape, the position coordinates of the given multiple parts also include the position coordinates of multiple points on the contour of the facial area, the nose vertex of the facial area and other multiple points of the facial area, and in the conversion step, the coordinate conversion is performed so that the four first contour points included in the multiple points on the contour of the facial area are respectively arranged at the vertices of the given rectangular shape, the nose vertex is arranged at the center of the given rectangular shape, and the other points on the contour of the facial area are arranged on the sides of the given rectangular shape, and the other multiple points of the facial area move corresponding to the movement of the multiple points on the contour of the facial area and the nose vertex.

[0171] <6>

[0172] A skin condition estimating method as described in <5>, wherein the other multiple points of the facial area include four third contour points located closer to the contour of the facial area than the given local area and four second contour points located closer to the contour of the facial area than the four third contour points, and in the conversion step, the coordinate conversion is performed so that the four second contour points and the four third contour points are respectively arranged on the diagonal lines connecting the four first contour points and the nose vertex, and the difference between the distance of the first contour point from the nose vertex and the distance of the second contour point from the nose vertex is smaller than the difference between the distance of the second contour point from the nose vertex and the distance of the third contour point from the nose vertex.

[0173] <7>

[0174] A skin condition estimating method as described in any one of <1> to <6>, wherein the estimation model is capable of estimating multiple items of skin condition, and the estimation model has multiple input rules. In the conversion step, the acquired facial image is converted so as to correspond to an input rule corresponding to an estimation object item among a plurality of different input rules corresponding to the multiple items, thereby acquiring a converted facial image corresponding to the estimation object item. In the information acquisition step, the converted facial image is input into the estimation model to acquire skin condition information of the subject related to the estimation object item.

[0175] <8>

[0176] A skin condition estimation system comprises: an estimation model which implements learning so as to output the skin condition of a face reflected in an input facial image based on the input facial image; an image acquisition unit which acquires a facial image of a subject; a conversion unit which converts the acquired facial image so as to conform to an input rule in the estimation model so that multiple points on the contour of the facial area of ​​the face reflected in the facial image are located at the outer edge of the image, thereby obtaining a converted facial image; an information acquisition unit which inputs the converted facial image as the input facial image into the estimation model and acquires skin condition information of the face of the subject output from the estimation model; a skin condition output unit which generates image information for prompting the acquired skin condition information; and an information display unit which displays the image information.

[0177] <9>

[0178] A skin condition estimation system as described in <8>, wherein, in the converted facial image, a given local area within the facial region of the human face projected in the acquired facial image is removed or reduced, and the conversion unit makes the positions of peripheral points of a plurality of peripheral points surrounding the given local area in the facial region that are opposite to each other with the given local area separated by the given local area consistent, thereby removing or reducing the given local area within the facial region.

[0179] <10>

[0180] The skin condition estimating system as described in <9>, wherein the given local area includes an eye area, an eyebrow area, and a lip area.

[0181] <11>

[0182] The skin condition estimation system as described in <9> or <10>, further comprising: a position determination unit, which determines the position coordinates of a given plurality of parts in the facial area of ​​the human face reflected in the acquired facial image, and the conversion unit transforms the acquired facial image so as to perform coordinate transformation on a part or all of the position coordinates of the determined given plurality of parts so as to conform to the input rule of the estimation model.

[0183] <12>

[0184] A skin condition estimation system as described in <11>, wherein the input facial image has a given rectangular shape, the position coordinates of the given multiple parts also include the position coordinates of multiple points on the contour of the facial area, the nose vertex of the facial area and other multiple points of the facial area, and the conversion unit performs the coordinate conversion so that the four first contour points included in the multiple points on the contour of the facial area are respectively arranged at the vertices of the given rectangular shape, the nose vertex is arranged at the center of the given rectangular shape, and the other points on the contour of the facial area are arranged on the sides of the given rectangular shape, and the other multiple points of the facial area move corresponding to the movement of the multiple points on the contour of the facial area and the nose vertex.

[0185] <13>

[0186] A skin condition estimation system as described in <12>, wherein the other multiple points of the facial area include 4 third contour points located closer to the contour of the facial area than the given local area and 4 second contour points located closer to the contour of the facial area than the 4 third contour points, and the conversion unit performs the coordinate conversion so that the 4 second contour points and the 4 third contour points are respectively arranged on the diagonal lines connecting the 4 first contour points and the nose vertex, and the difference between the distance of the first contour point from the nose vertex and the distance of the second contour point from the nose vertex is smaller than the difference between the distance of the second contour point from the nose vertex and the distance of the third contour point from the nose vertex.

[0187] <14>

[0188] A skin condition estimation system as described in any one of <8> to <13>, wherein the estimation model is capable of estimating multiple items of skin condition, the estimation model is provided with multiple input rules, the conversion unit transforms the acquired facial image so as to correspond to an input rule corresponding to an estimation object item among a plurality of different input rules corresponding to the multiple items, thereby acquiring a converted facial image corresponding to the estimation object item, and the information acquisition unit inputs the converted facial image into the estimation model to acquire skin condition information of the subject person related to the estimation object item.

[0189] <15>

[0190] The skin condition estimating system according to any one of <8> to <14>, wherein the skin condition estimating system comprises a user terminal, and the user terminal includes at least the image acquisition unit, the skin condition output unit, and the information display unit.

[0191] <16>

[0192] The skin condition estimating system according to any one of <11> to <13>, wherein the skin condition estimating system comprises a user terminal, and the user terminal includes at least the image acquisition unit, the skin condition output unit, the information display unit, and the position determination unit.

[0193] <17>

[0194] A user terminal for a skin condition estimation system is the user terminal for a skin condition estimation system according to any one of <8> to <14>, and includes at least the image acquisition unit, the skin condition output unit, and the information display unit.

[0195] <18>

[0196] A user terminal for a skin condition estimation system is the user terminal for a skin condition estimation system according to any one of <11> to <13>, comprising the image acquisition unit, the skin condition output unit, the information display unit, and the position determination unit.

[0197] <19>

[0198] The user terminal for the skin condition estimation system as described in <17> or <18> further includes the conversion unit.

[0199] The above contents are described in more detail with reference to the following examples, but the following examples do not limit the above contents in any way.

[0200] [Example]

[0201] In this embodiment, 2207 facial images (hereinafter referred to as facial image samples) obtained by photographing 353 Japanese women under various shooting environments using various shooting devices are prepared to test the accuracy of the skin condition estimation method (this method) involved in the above-mentioned embodiment. The various shooting devices include single-lens reflex digital cameras, smart phones, and tablet terminals.

[0202] The accuracy of the present method is verified by comparing the accuracy of an existing common method for estimating the skin condition of a face reflected in a face image (hereinafter referred to as the existing method) with the accuracy of the method described in the above-mentioned Patent Document 1 (referred to as the prior patent method). Specifically, for each of the present method, the existing method, and the prior patent method, an AI model is constructed and the AI ​​model is made to learn, and the accuracy of the skin condition estimated by each AI model is compared.

[0203] Each AI model is constructed by fine-tuning VGG16 based on the learned model of the convolutional neural network (CNN) used for ImageNet. The learning conditions of each AI model are the same, except that the facial images used as teaching data are different. As a learning method, cosine annealing with a learning rate (lr) of 0.001 to 0.0001 and 20 epochs as 1 cycle is used, and the total number of learning epochs is 1000. In addition, the mean squared error is used in the loss function, and the stochastic gradient descent method (SGD (Stochastic Gradient Descent) method) is used as the optimization algorithm.

[0204] Each AI model is formed by 4 CNNs and can estimate 18 items of skin condition.

[0205] The estimated items for skin condition are makeup application feel, skin age, male / female skin characteristics, makeup removal, powdery feel, glossiness, moisture, comprehensive evaluation, moisture content, dryness, smoothness, texture fineness, moisturization, firmness, transparency, skin whiteness, yellowness and dullness.

[0206] The estimated items output by a CNN are determined based on the collinearity between the estimated items or the learning efficiency when combined and learned. Specifically, an AI model is formed so that a combination of items such as texture, smoothness, and transparency that are estimated to be highly interconnected or collinear is used as the output of a CNN. By learning the AI ​​model, a combination of items with higher estimation accuracy is searched. As a result, an AI model that estimates 18 items of skin condition using 4 CNNs is constructed.

[0207] The correct answer information of the teaching data of each AI model is obtained by visually evaluating the skin condition of the subject's face by a professional evaluator, or by applying a specific analysis method to the subject's facial image taken by a dedicated skin image measuring device.

[0208] Each original facial image included in the facial image sample is converted using the present method, the prior art method, and the prior patent method. That is, a group of 2207 facial images converted using the present method, a group of 2207 facial images converted using the prior art method, and a group of 2207 facial images converted using the prior patent method are prepared.

[0209] The facial image groups prepared for each method were divided into 5:3:2 for learning, verification and testing. 50% of the facial image group for learning was used as teaching data for the AI ​​model, 30% of the facial image group for verification was used to select the AI ​​model after learning, and 20% of the facial image group for testing was used to evaluate the estimation accuracy of each AI model. In addition, the facial image group for learning was multiplied by 10 by performing a random rotation of up to ±5 degrees on each facial image.

[0210] In each AI model, the image size of the input image is set to 224 pixels (the above-given image size), and the shape of the input image is set to a square (the above-given rectangular shape).

[0211] The facial image conversion method used in each method is as follows.

[0212] In the conventional method, an original facial image of a facial image sample is normalized only in terms of image size and shape and is aligned using facial recognition so that the face is at a given position (image size 224 pixels, image shape is square). In the prior patent method, in addition to the normalization of the conventional method and the alignment of the face based on facial recognition, the original facial image is also processed by painting the entire area other than the facial area (background or hair) and the areas of the eyes, eyebrows and lips with skin color.

[0213] In this embodiment, the method is as follows Fig.11 The facial image is transformed as shown. Fig.11 1 is a diagram showing the conversion flow of a facial image in the skin condition estimating method according to the present embodiment.

[0214] In the position determination step (S22), FaceMesh by Google (registered trademark) is used to determine the position coordinates of 468 markers in the face area of ​​the original face image. Fig.11 In the example of FIG. 1 , a grid formed by connecting the points of each mark with line segments is shown in the reference numeral E2.

[0215] The detected marks include points indicating the contour of the face, points indicating the apex of the nose, points indicating the contours of the eyes, eyebrows, and lips, etc. Two points on the contour of the face adjacent to the left and right eyebrows are indicated by reference numerals E21 and E24, two points on the contour of the face adjacent to the left and right corners of the mouth are indicated by reference numerals E22 and E23, and the apex of the nose is indicated by reference numeral E25. The marks indicated by reference numerals E21 to E24 correspond to the first contour points described above.

[0216] Furthermore, four points adjacent to the inner sides of the first outline points ( E21 to E24 ) are denoted by reference numerals E211 , E221 , E231 , and E241 , and these points are described as second outline points.

[0217] Four points adjacent to the second contour points E211 , E221 , E231 , and E241 that are located further inside thereof are denoted by reference numerals E212 , E222 , E232 , and E242 , and these points are described as third contour points.

[0218] In the conversion step (S23), the position coordinates of the marker determined in step (S22) are converted into Fig.11 The coordinate transformation is performed as shown in the figure mark E3.

[0219] Specifically, the four first contour points E21, E22, E23 and E24 are respectively arranged at the vertices of the square in the converted facial image, the nose vertex E25 is arranged at the center of the converted facial image, and the other points on the facial contour are arranged on the edges of the converted facial image.

[0220] Furthermore, the four second contour points E211, E221, E231 and E241 and the four third contour points E212, E222, E232 and E242 are respectively arranged on the diagonal line connecting the four first contour points E21, E22, E23 and E24 and the nose vertex E25, and are arranged in such a way that the difference between the distance from the first contour point to the nose vertex and the distance from the second contour point to the nose vertex is smaller than the difference between the distance from the second contour point to the nose vertex and the distance from the third contour point to the nose vertex.

[0221] In addition, other markers are linearly moved respectively while maintaining their mutual positional relationships as much as possible in accordance with the movement of the points on the contour of the face region and the nose vertex.

[0222] In this embodiment, through such coordinate transformation, Fig.11 As shown in the figure mark E4, the original facial image is converted so that the facial area occupies the entire square facial image after conversion.

[0223] Furthermore, the original facial image is transformed so that, through the coordinate transformation of the second and third contour points, the area closer to the center of the face in the transformed facial image is more enlarged, while the area closer to the contour of the face is less enlarged.

[0224] Thus, the facial area is enlarged with the area with a higher degree of suggestion of the skin condition of the face as the center. Therefore, even if the image size is compressed each time the facial image is normalized, the resolution reduction in the area with a higher degree of suggestion of the skin condition can be fully suppressed, thereby improving the estimation accuracy of the AI ​​model.

[0225] exist Fig.11 In the example of , the cheek area is also kept in the mesh shape (triangle) as much as possible in the face area and further enlarged. The reason is that the cheek area has a higher degree of presentation of the skin condition of the face.

[0226] In addition, Fig.11The reason why black areas that are not considered to be face areas exist on the left and right sides and the upper edge of the square in the converted face image shown by the reference numeral E4 is that the position coordinates of the marker determined in step (S22) are determined slightly outside the contour of the face area in the face image. The present invention can achieve the above-mentioned effect while allowing a slight deviation in the determination of the position coordinates of the face area or the arrangement of the face area in the converted face image.

[0227] Furthermore, in the converted face image, Fig.11 As shown in the figure mark E4, the eye area, eyebrow area and lip area are flattened. In this embodiment, the coordinates of each point on the upper contour of the upper lip area and each point on the lower contour of the lower lip area, which are located in positions opposite to each other in the marks on the contour of the lip area, are respectively converted into the same position coordinates, thereby flattening the lip area of ​​the converted facial image. In addition, the coordinates of each point on the upper contour of the eyebrow area and each point on the lower contour of the eyebrow area, which are located in positions opposite to each other in the marks on the contour of the eyebrow area, are respectively converted into the same position coordinates, thereby flattening the eyebrow area of ​​the converted facial image.

[0228] On the other hand, regarding the eye region, a mark surrounding the periphery of the eye region outside the point on the contour of the eye region (hereinafter referred to as an eye region periphery point) is used. That is, the eye region periphery point of the upper eyelid and the eye region periphery point of the lower eyelid, which exist at positions vertically opposed to each other, are coordinate-converted to the same position coordinates, and the eye region of the converted facial image is flattened.

[0229] In this way, it is possible to exclude areas irrelevant to the skin condition of the human face, the eyebrow area, and the lip area from the input facial image to the AI ​​model and the facial image of the teaching data, and to enlarge the skin area in the facial image. As a result, it is possible to prevent the resolution of the facial area from being reduced when input to the AI ​​model, and further, to improve the estimation accuracy of the AI ​​model.

[0230] Fig.12 This is a graph showing the difference in prediction error based on sensory evaluation by experts regarding the skin condition estimation method involved in the present embodiment (present method), the conventional method, and the prior patent method (International Publication No. 2018 / 222808). Fig.12 The mean square error is shown in .

[0231] In this comparison, facial images and data sets containing impression evaluation items associated with the images were used. The facial images include images taken under a given shooting environment, or images taken by the subject himself using his own smartphone in any environment, totaling 2,207 images. For all these images, the impression evaluation score corresponding to the image is associated using the visual evaluation of experienced professional evaluators, or using a machine learning model and measurement device that have been learned in advance to be able to predict the visual evaluation value. Regarding the three methods of the existing method, the prior patent method, and the present method, the AI ​​model is learned so that the prediction error of the impression evaluation item is minimized relative to the expert's sensory evaluation, and the mean square error between the impression evaluation score and the predicted value is calculated.

[0232] according to Fig.12 , this method shows the smallest error, followed by the smaller error of the prior patented method, and the largest error of the existing method.

[0233] Table 1 shows the differences in the mean square error between the conventional method, the prior patent method, and the present method, with the mean square error in the conventional method set to 100% for each of the items of firmness, luster, smoothness of the skin surface, skin color, yellowness of the skin, and dullness. Among them, the results for each of the items of luster, smoothness of the skin surface, skin color, and yellowness of the skin are preferred.

[0234] [Table 1]

[0235]

[0236] Therefore, according to this embodiment, it was confirmed that the estimation accuracy of the skin condition of the AI ​​model can be improved by using the facial image converted by this method.

[0237] In addition, in this embodiment, the use of Fig.11 The illustrated coordinate transformation rules are different from the coordinate transformation rules in that the original facial image of the facial image sample is transformed.

[0238] Fig.13 FIG. 1 is a diagram showing an example of facial image conversion. In order to estimate the wrinkles at the outer corners of the eyes in more detail, the facial image conversion uses a coordinate conversion rule that magnifies the several grids around the outer corners of the eyes and immediately inside E212 and E242 more than other parts, in other words, increases the magnification. Fig.14 This is a diagram showing an example of facial image conversion, in which the coordinate conversion rule is used to enlarge the area around the nasolabial folds, specifically the area below E25, more than other areas, in other words, to increase the magnification ratio in order to estimate the state of the nasolabial folds in more detail.

[0239] Even when the facial images E41 and E42 converted using such a coordinate conversion rule are used, the skin condition of the face can be estimated in the same manner as in the above-described embodiment.

[0240] Furthermore, when using the facial images E41 and E42, the wrinkle region at the outer corner of the eye or the nasolabial fold region is magnified, so that the skin condition presented in the region can more accurately evaluate the conspicuity of, for example, the nasolabial folds or wrinkles at the outer corner of the eye.

[0241] This application claims priority based on Japanese Patent Application No. 2022-124831 filed in Japan on August 4, 2022, and Japanese Patent Application No. 2023-52673 filed in Japan on March 29, 2023, and all the disclosures of these are incorporated herein by reference.

[0242] Description of Reference Numerals

[0243] 5: Communication lines

[0244] 7: AI Model

[0245] 10: Information processing device (skin condition estimation device) (skin condition estimation model learning device)

[0246] 11: CPU

[0247] 12: Memory

[0248] 13: Input and output I / F

[0249] 14: Communication unit

[0250] 15: Display device

[0251] 16: Input Device

[0252] 20: Server device

[0253] 30: User terminal

[0254] 31: Image acquisition unit (camera)

[0255] 32: Position determination unit

[0256] 33: Conversion unit

[0257] 34: Information acquisition unit

[0258] 35: Information display unit (display device).

Claims

1. A skin condition estimation method, using an estimation model, wherein the estimation model is trained so as to output the skin condition of a face reflected in the input facial image based on the input facial image, wherein the skin condition estimation method is characterized in that: Executed by more than one processor: An image acquisition step of acquiring a facial image of a subject; a conversion step, in order to conform to the input rule of the estimation model, converting the acquired facial image so that multiple points on the contour of the facial region of the face reflected in the facial image are located at the outer edge of the image, thereby obtaining a converted facial image; and The information acquisition step acquires skin condition information of the face of the subject output from the estimation model by inputting the converted facial image as the input facial image.

2. The skin condition estimation method according to claim 1, wherein: In the converted facial image, a predetermined local area within the facial area of ​​the human face reflected in the acquired facial image is removed or reduced. In the conversion step, positions of peripheral points that are opposed to each other across the given local part region among a plurality of peripheral points surrounding the given local part region in the face region are aligned to remove or reduce the given local part region in the face region.

3. The skin condition estimation method according to claim 2, wherein: The given local part area includes an eye area, an eyebrow area, and a lip area.

4. The skin condition estimation method according to claim 2 or 3, wherein: The one or more processors further perform: a position determination step of determining position coordinates of a plurality of given parts in the facial region of the face of the person reflected in the acquired facial image, In the conversion step, the acquired facial image is converted so that part or all of the position coordinates of the specified plurality of parts are converted to conform to the input rule of the estimation model.

5. The skin condition estimation method according to claim 4, wherein: The input face image has a given rectangular shape, The position coordinates of the given multiple parts also include the position coordinates of multiple points on the contour of the facial region, the nose vertex of the facial region, and other multiple points of the facial region. In the conversion step, the coordinate conversion is performed so that four first contour points included in the multiple points on the contour of the facial area are respectively arranged at vertices of the given rectangular shape, the nose vertex is arranged at the center of the given rectangular shape, and other points on the contour of the facial area are arranged on sides of the given rectangular shape, and the other multiple points of the facial area move corresponding to the movement of the multiple points on the contour of the facial area and the nose vertex.

6. The skin condition estimation method according to claim 5, wherein: The other multiple points of the face region include four third contour points located closer to the contour of the face region than the given local part region, and four second contour points located closer to the contour of the face region than the four third contour points. In the conversion step, the coordinate conversion is performed so that the four second contour points and the four third contour points are respectively arranged on the diagonal lines connecting the four first contour points and the nose vertex, and the difference between the distance between the first contour point and the nose vertex and the distance between the second contour point and the nose vertex is smaller than the difference between the distance between the second contour point and the nose vertex and the distance between the third contour point and the nose vertex.

7. The skin condition estimating method according to any one of claims 1 to 6, wherein: The estimation model can estimate multiple items of skin condition. The input rules of the inference model are set to be multiple, In the conversion step, the acquired facial image is converted so as to correspond to an input rule corresponding to the estimation target item among the plurality of input rules that are different from each other and correspond to the plurality of items, thereby acquiring a converted facial image corresponding to the estimation target item. In the information acquisition step, the converted facial image is input to the estimation model to acquire skin condition information of the subject person related to the estimation target item.

8. A skin condition estimation system, characterized in that: have: an estimation model that is trained so as to output, based on an input facial image, a skin condition of a face of a person reflected in the input facial image; an image acquisition unit that acquires a facial image of a subject; a conversion unit, which converts the acquired facial image so as to conform to the input rule of the estimation model, so that a plurality of points on the contour of the facial region of the face reflected in the facial image are located at the outer edge of the image, thereby obtaining a converted facial image; an information acquisition unit that inputs the converted facial image as the input facial image into the estimation model, and acquires the skin condition information of the face of the subject output from the estimation model; a skin condition output unit for generating image information for presenting the acquired skin condition information; and An information display unit displays the image information.

9. The skin condition estimating system according to claim 8, wherein: In the converted facial image, a predetermined local area within the facial area of ​​the human face reflected in the acquired facial image is removed or reduced. The conversion unit makes positions of peripheral points that are opposed to each other across the given local part region among a plurality of peripheral points surrounding the given local part region in the face region coincide with each other, thereby removing or reducing the given local part region in the face region.

10. The skin condition estimating system according to claim 9, wherein: The given local part area includes an eye area, an eyebrow area, and a lip area.

11. The skin condition estimating system according to claim 9 or 10, wherein: The skin condition estimation system further comprises: a position specifying unit that specifies position coordinates of a plurality of predetermined locations in a facial region of the face of the person reflected in the acquired facial image; The conversion unit converts the acquired facial image so that a part or all of the position coordinates of the specified plurality of parts are converted to conform to the input rule of the estimation model.

12. The skin condition estimating system according to claim 11, wherein: The input face image has a given rectangular shape, The position coordinates of the given multiple parts also include the position coordinates of multiple points on the contour of the facial region, the nose vertex of the facial region, and other multiple points of the facial region. The conversion unit performs the coordinate conversion so that four first contour points included in the multiple points on the contour of the facial area are respectively arranged at vertices of the given rectangular shape, the nose vertex is arranged at the center of the given rectangular shape, and other points on the contour of the facial area are arranged on sides of the given rectangular shape, and the other multiple points of the facial area move corresponding to the movement of the multiple points on the contour of the facial area and the nose vertex.

13. The skin condition estimating system according to claim 12, wherein: The other multiple points of the face region include four third contour points located closer to the contour of the face region than the given local part region, and four second contour points located closer to the contour of the face region than the four third contour points. The conversion unit performs the coordinate conversion so that the four second contour points and the four third contour points are respectively arranged on the diagonal lines connecting the four first contour points and the nose vertex, and the difference between the distance between the first contour point and the nose vertex and the distance between the second contour point and the nose vertex is smaller than the difference between the distance between the second contour point and the nose vertex and the distance between the third contour point and the nose vertex.

14. The skin condition estimating system according to any one of claims 8 to 13, wherein: The estimation model can estimate multiple items of skin condition. The input rules of the inference model are set to be multiple, The conversion unit converts the acquired facial image so as to correspond to an input rule corresponding to the estimation target item among the plurality of input rules that are different from each other and correspond to the plurality of items, thereby acquiring a converted facial image corresponding to the estimation target item. The information acquisition unit inputs the converted facial image into the estimation model to acquire skin condition information of the subject person related to the estimation target item.

15. The skin condition estimating system according to any one of claims 8 to 14, wherein: The skin condition estimation system comprises a user terminal, The user terminal includes at least the image acquisition unit, the skin condition output unit, and the information display unit.

16. The skin condition estimating system according to any one of claims 11 to 13, wherein: The skin condition estimation system comprises a user terminal, The user terminal includes at least the image acquisition unit, the skin condition output unit, the information display unit, and the position determination unit.

17. A user terminal for a skin condition estimation system, characterized in that: A user terminal for use in the skin condition estimation system according to any one of claims 8 to 14, The skin condition estimation system user terminal includes at least the image acquisition unit, the skin condition output unit, and the information display unit.

18. A user terminal for a skin condition estimation system, the user terminal for a skin condition estimation system according to any one of claims 11 to 13, The skin condition estimation system user terminal includes the image acquisition unit, the skin condition output unit, the information display unit, and the position identification unit.

19. The user terminal for the skin condition estimation system according to claim 17 or 18, wherein: The skin condition estimation system user terminal further includes the conversion unit.

Images