Skin condition estimation method, device, program, system, trained model generation method, and trained model

Abstract

The present invention makes it possible to easily obtain the state of a skin. A method according to one embodiment of the present invention comprises: a step for identifying a feature of the nose of a user; and a step for inferring the state of a skin of the user on the basis of the feature of the nose of the user.

Description

[Technical Field]

[0001] The present invention relates to a skin condition estimation method, an apparatus, a program, a system, a trained model generation method, and a trained model. [Background technology]

[0002] Conventionally, there are known techniques for predicting skin conditions to facilitate appropriate skin care, etc. For example, Patent Document 1 predicts the future formation and level of wrinkles around the eyes and mouth from ultrasound images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-200284 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 requires an ultrasonic diagnostic device, and it is not easy to simply predict skin conditions that are likely to occur in the future.

[0005] Therefore, the present invention aims to easily achieve a good skin condition. [Means for solving the problem]

[0006] A method according to one embodiment of the present invention includes identifying a user's nose features and estimating the user's skin condition based on the user's nose features. [Effects of the Invention]

[0007] In the present invention, the skin condition can be easily estimated from the nose features. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overall configuration according to an embodiment of the present invention; [Figure 2] 1 is a diagram showing functional blocks of a skin condition estimating device according to an embodiment of the present invention. FIG. [Figure 3] 10 is a flowchart showing a flow of a skin condition estimation process according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram illustrating features of a nose according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating extraction of a nose region according to an embodiment of the present invention. [Figure 6] 10A and 10B are diagrams illustrating calculation of nose features according to an embodiment of the present invention. [Figure 7] 10 is an example of nose features for each face type according to an embodiment of the present invention. [Figure 8] 1 is an example of a face estimated from nose features according to an embodiment of the present invention. [Figure 9] 1 is a diagram illustrating a hardware configuration of a skin condition estimating device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0010] <Terminology> "Skin condition" refers to at least one of wrinkles, age spots, sagging, dark circles, smile lines, dullness, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, pores, and skin color. For example, "skin condition" refers to the presence or absence and degree of factors that make up the skin condition, such as wrinkles, age spots, sagging, dark circles, smile lines, dullness, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, pores, and skin color. Furthermore, "skin condition" refers to the skin condition of one part of the face, the entire face, or multiple parts of the face. Note that "skin condition" may refer to the user's future skin condition or the user's current skin condition. In the present invention, the skin condition is estimated from nose features based on the correlation between the nose features and the skin condition.

[0011] <Overall structure> FIG. 1 is a diagram showing the overall configuration of an embodiment of the present invention. A skin condition estimation device 10 estimates the skin condition of a user 20 based on the characteristics of the user's nose. For example, the skin condition estimation device 10 is a smartphone or the like with a camera function. The skin condition estimation device 10 will be described in detail later with reference to FIG. 2.

[0012] Although the present specification describes a case where skin condition estimation device 10 is a single device (for example, a smartphone with a camera function), skin condition estimation device 10 may also be composed of multiple devices (for example, a device without a camera function and a digital camera). The camera function may be a function for capturing images of skin in three dimensions or two dimensions. Furthermore, a device other than skin condition estimation device 10 (such as a server) may perform some of the processes performed by skin condition estimation device 10 described in the present specification.

[0013] <Functional blocks of the skin condition estimation device 10> 2 is a diagram showing functional blocks of a skin condition estimation device 10 according to one embodiment of the present invention. The skin condition estimation device 10 can include an image acquisition unit 101, a nose feature identification unit 102, a skin condition estimation unit 103, a bone structure estimation unit 104, and an output unit 105. By executing a program, the skin condition estimation device 10 can function as the image acquisition unit 101, the nose feature identification unit 102, the skin condition estimation unit 103, the bone structure estimation unit 104, and the output unit 105. Each of these units will be described below.

[0014] Image acquisition unit 101 acquires an image including the nose of user 20. Note that the image including the nose may be an image capturing both the nose and other parts (for example, an image capturing the entire face), or an image capturing only the nose (for example, an image captured so that the nose area of ​​user 20 fits within a predetermined area displayed on the display device of skin condition estimation device 10). Note that if the features of the nose can be identified from something other than the image, image acquisition unit 101 is not required.

[0015] The nose feature identification unit 102 identifies the nose features of the user 20. For example, the nose feature identification unit 102 identifies the nose features of the user 20 from image information (for example, pixel values ​​of the image) of an image including the nose of the user 20 acquired by the image acquisition unit 101.

[0016] The skin condition estimation unit 103 estimates the skin condition of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102. For example, the skin condition estimation unit 103 classifies the skin condition of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102.

[0017] In addition, the skin condition estimation unit 103 can also estimate the skin condition of the user 20 (for example, the skin condition caused by the shape of the facial skeleton) based on the shape related to the facial skeleton of the user 20 estimated by the skeleton estimation unit 104.

[0018] The skeleton estimation unit 104 estimates a shape related to the skeleton of the face of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102. For example, the skeleton estimation unit 104 classifies a shape related to the skeleton of the face of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102.

[0019] The output unit 105 outputs (for example, displays) information on the skin condition of the user 20 estimated by the skin condition estimation unit 103.

[0020] <Skin condition> Here, the skin condition will be described. For example, the skin condition may be at least one of wrinkles, blemishes, sagging, dark circles, nasolabial folds, dullness, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, texture, pores, and skin color. More specifically, the skin condition may include, for example, wrinkles at the corners of the eyes, wrinkles under the eyes, wrinkles on the forehead, wrinkles around the eye sockets, sagging eye bags, dark circles under the eyes, nasolabial folds (nasolabial folds and around the mouth), depth of the nasolabial folds, sagging marionette lines, sagging chin, HbSO2 Index (blood oxygen saturation index), Hb Index (amount of hemoglobin), HbO2 (amount of oxyhemoglobin), skin color, skin brightness, moisture retention capacity (TEWL), number of skin ridges, skin viscoelasticity, amount of oxygen in the blood, blood vessel density, number of capillaries, number of blood vessel branches, distance between blood vessels and the epidermis, epidermal thickness, HDL cholesterol, sebum, moisture content, melanin index (an index of melanin), pores, transparency, uneven skin tone (brownishness, reddishness), pH, etc. The skin condition estimation unit 103 estimates the skin condition from the nose features based on the correlation between the nose features and the skin condition.

[0021] <Correspondence between nose features and skin condition> Here, the correspondence between nose features and skin conditions will be described. Skin condition estimation unit 103 estimates the skin condition based on the correspondence between nose features and skin conditions stored in advance in skin condition estimation device 10, etc. Note that the skin condition may be estimated based not only on nose features but also on nose features and some of facial features.

[0022] The correspondence may be a predetermined database or a trained model generated by machine learning. The database associates nose features (which may be nose features and some facial features) with skin conditions based on the results of experiments conducted on subjects. The trained model is a prediction model that outputs information on skin conditions when information on nose features (which may be nose features and some facial features) is input.

[0023] <<Generating a trained model>> In one embodiment of the present invention, a computer such as the skin condition estimation device 10 can generate a trained model. Specifically, the computer such as the skin condition estimation device 10 can acquire training data in which input data is nose features (which may be nose features and part of facial features) and output data is skin condition, and perform machine learning using the training data to generate a trained model that outputs a skin condition when nose features (which may be nose features and part of facial features) are input. In this way, by performing machine learning using training data in which input data is nose features (which may be nose features and part of facial features) and output data is skin condition, a trained model is generated that outputs a skin condition when nose features (which may be nose features and part of facial features) are input.

[0024] <The relationship between facial bone structure and skin condition> Here, the correspondence relationship between the shape of the facial skeleton and the skin condition will be described. As described above, the skin condition estimation unit 103 can also estimate the skin condition based on the correspondence relationship between the shape of the facial skeleton and the skin condition that is stored in advance in the skin condition estimation device 10 or the like.

[0025] The correspondence may be a predetermined database or a trained model generated by machine learning. The database associates the shape of the facial skeleton with the skin condition based on the results of experiments conducted on subjects. The trained model is a prediction model that outputs information on the skin condition when information on the shape of the facial skeleton is input.

[0026] <<Generating a trained model>> In one embodiment of the present invention, a computer such as the skin condition estimation device 10 can generate a trained model. Specifically, the computer such as the skin condition estimation device 10 can acquire training data in which input data is a shape related to the facial skeleton and output data is a skin condition, and perform machine learning using the training data to generate a trained model that outputs a skin condition when a shape related to the facial skeleton is input. In this way, by performing machine learning using training data in which input data is a shape related to the facial skeleton and output data is a skin condition, a trained model is generated that outputs a skin condition when a shape related to the facial skeleton is input.

[0027] <Future skin condition and current skin condition> The estimated skin condition may be the future skin condition of the user 20 or the current skin condition of the user 20. When the correspondence between the nose features (or the shape related to the facial bones estimated from the nose features) and the skin condition is created based on data of a person older than the actual age of the user 20 (for example, the age of the subject of an experiment or the age of the person who served as learning data for machine learning is older than the actual age of the user 20), the future skin of the user 20 is estimated. On the other hand, when the correspondence between the nose features (or the shape related to the facial bones estimated from the nose features) and the skin condition is created based on data of a person who is the same age as the actual age of the user 20 (for example, the age of the subject of an experiment or the age of the person who served as learning data for machine learning is the same as the actual age of the user 20), the current skin of the user 20 is estimated. Note that the skin condition may be estimated based not only on the nose features but also on some of the nose features and facial features.

[0028] An example of estimation based on the correspondence between nose features (or the shape related to the facial skeleton estimated from nose features) and skin conditions will be described below.

[0029] <<Skin condition estimation example 1>> For example, the skin condition estimation unit 103 can estimate that wrinkles are more likely to appear at the corners of the eyes when the root and bridge of the nose are high. Also, for example, if the cheek shape has a high cheekbone at the top, the skin condition estimation unit 103 can estimate (determine ON / OFF) that there are wrinkles at the corners of the eyes or that there is a possibility that wrinkles will appear in the future.

[0030] <<Skin condition estimation example 2>> For example, the skin condition estimation unit 103 can estimate that the more rounded the nostrils are, or the larger the eyes are, the more likely wrinkles are to form under the eyes.

[0031] For example, the skin condition estimation unit 103 can estimate that, although eye sockets have shape characteristics such as being horizontally long and small, large eye sockets with similar vertical and horizontal widths have more wrinkles under the eyes. Furthermore, for example, the skin condition estimation unit 103 can estimate wrinkles under the eyes based on the facial contour. Furthermore, for example, the skin condition estimation unit 103 can estimate that the greater the distance between the eyes, the fewer wrinkles there are under the eyes.

[0032] <<Skin condition estimation example 3>> For example, the skin condition estimation unit 103 can estimate the sagging of the eye bags based on the roundness of the nostrils and the height of the nose bridge. Specifically, the skin condition estimation unit 103 can estimate that the larger the sum of the roundness of the nostrils and the height of the nose bridge, the more sagging the eye bags are.

[0033] For example, the skin condition estimation unit 103 can estimate that the eye bags are likely to sag when the face contour is oval and long.

[0034] <<Skin condition estimation example 4>> For example, the skin condition estimation unit 103 can estimate HbCO2 (reduced hemoglobin) based on the lowness of the nose bridge and the roundness of the nostrils.

[0035] For example, the skin condition estimation unit 103 can estimate HbSO2 (oxygen saturation) based on the facial contour.

[0036] <<Skin condition estimation example 5>> For example, the skin condition estimation unit 103 can estimate that the lower the nose bridge, the rounder the nostrils, or the greater the distance between the eyes, the lower the moisture content.

[0037] For example, the skin condition estimation unit 103 can estimate the skin moisture content based on the height of the skull index and the aspect ratio of the face.

[0038] <<Example 6 of skin condition estimation>> For example, the skin condition estimation unit 103 can estimate sebum based on the roundness of the ala of the nose.

[0039] For example, the skin condition estimation unit 103 can estimate sebum based on the facial contour.

[0040] <<Example 7 of skin condition estimation>> For example, the skin condition estimation unit 103 determines that the melanin index is higher when the nostrils are rounder and the bridge of the nose is higher. It can be estimated that the higher the amount of melanin, the lower the nose bridge, and the narrower the distance between the eyes, the lower the melanin index.

[0041] For example, the skin condition estimation unit 103 can estimate that the thicker the upper and lower lips are, the higher the melanin index and the greater the amount of melanin. Also, for example, the skin condition estimation unit 103 can estimate that the thinner the upper and lower lips are, the lower the melanin index.

[0042] <<Example 8 of skin condition estimation>> For example, the skin condition estimation unit 103 can estimate that dark circles under the eyes are more likely to appear when the nostrils are round.

[0043] <<Example 9 of skin condition estimation>> For example, the skin condition estimation unit 103 can estimate that the face line is prone to sagging when the nose bridge is low and the distance between the eyes is wide, or when the angle of the chin is rounded.

[0044] <<Example 10 of skin condition estimation>> For example, the skin condition estimation unit 103 can estimate that the higher the bridge of the nose, the higher the amount of oxygen in the blood.

[0045] <<Skin condition estimation example 11>> For example, the skin condition estimation unit 103 can estimate the blood vessel density from the size of the nostrils or the position of change in the height of the nose root, and the larger the nostrils, the higher the blood vessel density.

[0046] <<Skin condition estimation example 12>> For example, the skin condition estimation unit 103 can estimate the epidermal thickness from the size of the ala of the nose.

[0047] <<Skin condition estimation example 13>> For example, the skin condition estimation unit 103 can estimate the number of blood vessel branches from the position where the height of the bridge of the nose changes.

[0048] <<Comprehensive skin condition estimation>> In one embodiment of the present invention, the skin condition estimation unit 103 can comprehensively express the skin condition as wrinkles, age spots, sagging, dark circles under the eyes, nasolabial folds, dullness, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, pores, and skin color from the values ​​estimated in the above estimation examples 1 to 9. An example is shown below. Wrinkles: This refers to one or more of the following: wrinkles at the corners of the eyes, under the eyes, forehead, and eyelids. · Spots: Described by one or more of the following: brown discoloration, red discoloration, and melanin. Sagging: This can be expressed as one or more of the following: eye bags, jowls, and marionette lines. · Dark circles: Represented by one or two items: brown or blue dark circles under the eyes. · Nasolabial folds: This is expressed by one or two items: nasolabial folds in the nasolabial folds and nasolabial folds around the mouth. Dullness: Expressed from one or more of the following items: transparency, melanin, uneven skin tone, skin color, oxygen saturation, moisture, and number of skin ridges. · Firmness: Expressed by one or more of the following items: moisture, sebum, sagging, and skin viscoelasticity. Moisture: Expressed by one or two of the following items: moisture content, water retention capacity (TEWL), number of skin ridges, and pH. Texture: expressed by one or more of the following items: number of skin ridges, moisture. Skin color: Expressed by one or more of the following items: skin color, skin brightness, melanin, amount of oxygen in the blood, and HbO2 (oxygenated hemoglobin). Sebum: Expressed by one or two items: amount of sebum and pores. Skin may be classified into normal skin, dry skin, oily skin, and combination skin based on moisture and sebum. Melanin: Expressed by one or two of the following items: melanin index, melanin amount, and uneven skin tone. Blood flow: Expressed by at least one or two of the following items: HbSO2 Index (blood oxygen saturation index), Hb Index (hemoglobin content), HbO2 (oxyhemoglobin content), blood oxygen content, and skin color. Blood vessels: This is expressed by one or more of the following items: blood vessel density, number of capillaries, number of blood vessel branches, distance between blood vessels and the epidermis, and epidermal thickness. Blood: HDL cholesterol

[0049] In one embodiment of the present invention, the skin condition estimation unit 103 can represent skin features such as skin strengths and skin weaknesses from nose features. For example, if the nose features are Type 1, the evaluation value of wrinkles at the corners of the eyes is lower than the average evaluation value, and therefore the skin is represented as strong. If the nose features are Type 2, the evaluation value of wrinkles at the corners of the eyes is higher than the average evaluation value, and therefore the skin is represented as weak. Skin strengths and weaknesses can be represented for each part of the face. For Type 1, the skin strengths are wrinkles at the corners of the eyes and forehead, and age spots, while the skin weaknesses are dark circles, nasolabial folds, sagging around the mouth, and moisture retention. The skin condition estimation unit 103 can estimate a comprehensive skin index (in this case, sagging skin) from these skin conditions. For Type 2, the skin strengths are sagging cheeks, moisture retention, blood circulation, and age spots, and the skin weaknesses are wrinkles at the corners of the eyes and forehead, and age spots. The skin condition estimation unit 103 can estimate a comprehensive index of the skin (in this case, wrinkled skin type) from these skin conditions.

[0050] <Facial bone structure> Here, the shape related to the facial skeleton will be explained. "Shape related to the facial skeleton" refers to at least one of the shape of the facial skeleton itself and the shape of the face resulting from the skeleton. Skeleton estimation unit 104 estimates the shape related to the facial skeleton from the nose features based on the correlation between the nose features and the shape related to the facial skeleton.

[0051] For example, the shape of the facial skeleton includes the characteristics of the shape of each bone, the positional relationship of the skeleton, angles, etc., of at least one of the eye sockets, cheekbones, nasal bones, piriform aperture (the opening of the nasal cavity toward the face), skull index, maxilla, mandible, lips, corners of the mouth, eyes, epicanthic folds (skin folds in the area where the upper eyelid covers the inner corner of the eye), facial contour, and the positional relationship between the eyes and eyebrows (for example, whether the eyes and eyebrows are far apart or close together). An example of the shape of the facial skeleton is shown below. Note that the information in parentheses is an example of the specific content that is estimated. Eye sockets (horizontal, square, rounded) Cheekbones, cheeks (peak position, roundness) ·Nose bone (width, shape) Piriform mouth (shape) Skull Index (skull width / depth = 70, 75, 80, 85, 90) Maxilla, upper jaw (position relative to the eye socket, nasolabial angle) Mandible, lower jaw (depth, depth angle, forward angle, contour shape (jaw)) - Forehead (roundness of the forehead, shape of the forehead) Eyebrows (distance between eyes and eyebrows, eyebrow shape, eyebrow thickness) Lips (thick upper and lower lips, thick lower lip, thin upper and lower lips, large or small lips) Mouth corners (upward, downward, normal) Eyes (area, angle, distance between eyebrows and eyes, distance between eyes) Mongolian folds (yes, no) Face contour (Rectangle, Round, Obal, Heart, Square, Average, Natural, Long)

[0052] <Correspondence between nose features and facial bone structure> Here, the correspondence between nose features and facial skeletal shapes will be described. Skeletal shape estimation unit 104 estimates the facial skeletal shape based on the correspondence between nose features and facial skeletal shapes stored in advance in skin condition estimation device 10, etc. Note that the facial skeletal shape may be estimated based not only on nose features but also on nose features and part of facial features.

[0053] The correspondence may be a predetermined database or a trained model generated by machine learning. The database associates nose features (which may be nose features and some facial features) with shapes related to the facial skeleton based on the results of experiments conducted on subjects, etc. The trained model is a predictive model that outputs information on the shape of the facial skeleton when information on nose features (which may be nose features and some facial features) is input. The correspondence between nose features and shapes related to the facial skeleton may be created for each group (e.g., Caucasoid, Mongoloid, Negroid, Australoid, etc.) classified based on factors that may affect the skeleton.

[0054] <<Generating a trained model>> In one embodiment of the present invention, a computer such as the skin condition estimation device 10 can generate a trained model. Specifically, the computer such as the skin condition estimation device 10 acquires training data in which input data are nose features (which may be nose features and part of facial features) and output data are shapes related to the facial skeleton, and performs machine learning using the training data to generate a trained model that outputs a shape related to the facial skeleton when nose features (which may be nose features and part of facial features) are input. In this way, by performing machine learning using training data in which input data are nose features (which may be nose features and part of facial features) and output data are shapes related to the facial skeleton, a trained model is generated that outputs a shape related to the facial skeleton when nose features (which may be nose features and part of facial features) are input.

[0055] An example of estimation based on the correspondence between nose features and facial skeletal shapes will be described below.

[0056] <<Example 1 of estimating the shape of the facial skeleton>> For example, the skeleton estimation unit 104 can estimate the skull index based on the height or lowness of the nose bridge or the change position of the nose bridge height and the height or lowness of the nose bridge. Specifically, the skeleton estimation unit 104 estimates that the higher at least one of the nose bridge and the nose bridge is, the lower the skull index is.

[0057] <<Example 2 of estimating the shape of the facial skeleton>> For example, the bone structure estimation unit 104 can estimate whether the corners of the mouth are tilted upward or downward based on the width of the bridge of the nose. Specifically, the bone structure estimation unit 104 estimates that the wider the bridge of the nose, the more the corners of the mouth are tilted downward.

[0058] <<Example 3 of estimating the shape of the facial skeleton>> For example, the skeleton estimation unit 104 can estimate the size and thickness of the lips (1. both upper and lower lips are large and thick, 2. the lower lip is thick, 3. both upper and lower lips are thin and small) based on the roundness of the nostrils and the sharpness of the nasal tip.

[0059] <<Example 4 of estimating the shape of the facial skeleton>> For example, the bone structure estimation unit 104 can estimate the presence or absence of epicanthic folds based on the bridge of the nose. Specifically, the bone structure estimation unit 104 estimates that epicanthic folds are present when the bridge of the nose is determined to be low.

[0060] <<Example 5 of estimating the shape of the facial skeleton>> For example, the skeleton estimation unit 104 can classify the shape of the lower jaw (for example, into three categories) based on the height or lowness of the nasal bridge, the height of the nasal root, and the roundness and size of the nostrils.

[0061] <<Example 6 of estimating the shape of the facial skeleton>> For example, the skeleton estimation unit 104 can estimate a piriform mouth based on the height of the bridge of the nose.

[0062] <<Example 7 of estimating the shape of the facial skeleton>> For example, the bone structure estimation unit 104 can estimate the distance between the eyes based on the lowness of the nose bridge. Specifically, the bone structure estimation unit 104 estimates that the lower the nose bridge, the greater the distance between the eyes.

[0063] <<Example 8 of estimating the shape of the facial skeleton>> For example, the bone structure estimation unit 104 can estimate the roundness of the forehead based on the height of the root of the nose and the height of the bridge of the nose.

[0064] <<Example 9 of estimating the shape of the facial skeleton>> For example, the skeleton estimation unit 104 can estimate the distance between the eyes and the eyebrows and the shape of the eyebrows based on the height and lowness of the bridge of the nose, the size of the nostrils, and the position of change in the height of the root of the nose.

[0065] <Processing method> FIG. 3 is a flowchart showing the flow of a skin condition estimation process according to one embodiment of the present invention.

[0066] In step 1 (S1), the nose feature specifying unit 102 extracts feature points (for example, feature points of the inner corners of the eyebrows, the inner corners of the eyes, and the tip of the nose) from an image including the nose.

[0067] In step 2 (S2), the nose feature identification unit 102 extracts the nose region based on the feature points extracted in S1.

[0068] In addition, if the image including the nose is an image in which only the nose is captured (for example, an image captured so that the nose area of ​​user 20 fits within a specified area displayed on the display device of skin condition estimating device 10), the image in which only the nose is captured is used as is (i.e., S1 can be omitted).

[0069] In step 3 (S3), the nose feature identification unit 102 reduces the number of gradations of the image of the nose region extracted in S2 (for example, binarizes it). For example, the nose feature identification unit 102 reduces the number of gradations of the image of the nose region using at least one of lightness, luminance, RGB blue, and RGB green. Note that S3 may be omitted.

[0070] In step 4 (S4), the nose feature identification unit 102 identifies nose features (nose bone structure). Specifically, the nose feature identification unit 102 calculates nose feature amounts based on image information of the image of the nose area (for example, pixel values ​​of the image). For example, the nose feature identification unit 102 calculates the average pixel value of the nose area, the number of pixels equal to or less than a predetermined value or equal to or greater than a predetermined value, the cumulative pixel value, the amount of change in pixel value, etc. as nose feature amounts.

[0071] In step 5 (S5), the skeleton estimation unit 104 estimates the shape of the facial skeleton. Note that S5 can be omitted.

[0072] In step 6 (S6), the skin condition estimation unit 103 estimates the skin condition (for example, future skin concerns) based on the nose features identified in S4 (or the shape related to the facial skeleton estimated in S5).

[0073] <Nose features> Here, the features of the nose will be described. For example, the features of the nose include at least one of the root of the nose, the bridge of the nose, the tip of the nose, and the ala of the nose.

[0074] Fig. 4 is a diagram for explaining features of a nose according to one embodiment of the present invention, showing the positions of the root of the nose, the bridge, the tip of the nose, and the ala of the nose.

[0075] <<Root of nose>> The nasal root is the base of the nose. For example, the nasal features include at least one of the height of the nasal root, the lowness of the nasal root, the width of the nasal root, and the change in the height of the nasal root or the change in the position of the nasal root.

[0076] <<bridge of nose>> The bridge of the nose is the area between the eyebrows and the tip of the nose. For example, the nose features at least one of the height of the bridge of the nose, the width of the bridge of the nose, and the width of the bridge of the nose.

[0077] <<Nose tip>> The tip of the nose is the tip of the nose. For example, the nose features include at least one of the roundness or sharpness of the tip of the nose and the direction of the tip of the nose.

[0078] <<Nose wings>> The nasal alae are the bulging parts on either side of the tip of the nose. For example, nasal characteristics include at least one of the roundness or sharpness of the alae and the size of the alae.

[0079] <Extracting the nose area> 5 is a diagram illustrating extraction of a nose region according to an embodiment of the present invention. The nose feature identification unit 102 extracts the nose region from an image including a nose. For example, the nose region may be the entire nose as shown in FIG. 5(a), or may be a portion of the nose (e.g., the right or left half) as shown in FIG. 5(b).

[0080] <Calculation of nose features> FIG. 6 is a diagram for explaining calculation of nose feature amounts according to one embodiment of the present invention.

[0081] In step 11 (S11), a nose region is extracted from an image containing a nose.

[0082] In step 12 (S12), the number of gradations of the image of the nose area extracted in S11 is reduced (for example, binarized). Note that S12 can be omitted.

[0083] In step 13 (S13), nose feature values ​​are calculated. Note that in FIG. 6, pixel cumulative values ​​are expressed by setting the high-brightness side of the image to 0 and the low-brightness side to 255. For example, the nose feature identification unit 102 performs normalization for each of a plurality of regions (e.g., the regions divided in S12). Next, for each region, the nose feature identification unit 102 calculates, as nose feature values, the average pixel value, the number of pixels equal to or less than a predetermined value, the pixel cumulative value in at least one of the X and Y directions, and the amount of change in pixel value in at least one of the X and Y directions (e.g., using data on the low-brightness or high-brightness side of the image). In S13 of FIG. 6, the pixel cumulative value in the X direction at each position in the Y direction is calculated.

[0084] The method for calculating each feature amount will be described below.

[0085] For example, the feature value of the root of the nose is a feature value of the upper region (closer to the eyes) of the divided region of S12, the feature value of the bridge of the nose is a feature value of the upper or central region of the divided region of S12, and the feature values ​​of the tip of the nose and the wings of the nose are a feature value of the lower region (closer to the mouth) of the divided region of S12. These nose features are normalized by the distance between the eyes.

[0086] Nose root height: The height or lowness is determined from the change in pixel value in the Y direction in the area above the nose. Note that the height or lowness may be calculated as a numerical value or may be classified as high or low. In S13, the value of nose 2 changes immediately in the Y direction, and it can be seen that the change in the height of the nose root is located at the top. Nose bridge width: The area above the nose is divided into multiple parts (2 to 4, etc.) in the X direction, and the width is determined from the pattern of the average pixel values ​​of each part. Nose bridge height: The height or shortness is determined from the average cumulative pixel value of the central area of ​​the nose. Note that the height or shortness may be calculated as a numerical value or may be classified as high or low. Nose bridge width: The central area of ​​the nose is divided into multiple areas (2 to 4, etc.) in the X direction, and the width is determined from the pattern of the average pixel values ​​of each area. Roundness or sharpness of the nasal tip: This is determined from other nasal features (height of the nasal bridge, roundness or sharpness of the nostrils), and the lower the nasal bridge and the rounder the nostrils, the rounder the nasal tip. · Direction of nose tip: In the central area of ​​the nose, it is calculated from the width from the lowest point of the nose at a position that is a specified percentage of the maximum cumulative pixel value in the X direction. The wider the width, the more upward-facing the nose tip is. Roundness or kurtosis of the nasal wings: The roundness or kurtosis is determined by the change in the Y value in the lower area of ​​the nose. Nose size: Determined by the percentage of pixels in the center of the lower area that are below a certain value. The more pixels there are, the larger the nostrils.

[0087] <<Face type>> As described above, the "shape related to the facial bone structure" refers to at least one of the "shape of the facial bone structure itself" and the "shape of the face resulting from the bone structure." The "shape related to the facial bone structure" can include face types.

[0088] In one embodiment of the present invention, it is possible to estimate which of a plurality of face types the user's face belongs to (specifically, face types classified based on at least one of "the shape of the facial bone structure itself" and "the shape of the face resulting from the bone structure") based on the features of the user's nose. Hereinafter, the face types will be described with reference to Figs. 7 and 8.

[0089] Fig. 7 shows an example of nose features for each face type according to one embodiment of the present invention. Fig. 7 shows nose features for each face type (face types A to L). Note that the face type may be estimated using all four of the nose bridge, nasal ala, nose root, and nose tip, or may be estimated using only some of the nose bridge and nasal ala (for example, two of the nose bridge and nasal ala, two of the nose bridge and nose root, only the nose bridge, only the nasal ala, etc.).

[0090] In this way, face types are estimated from nose features. For example, from the nose features of face type A, eye roundness: round, eye tilt: downward, eye size: small, eyebrow shape: arched, eyebrow and eye position: far apart, and facial outline: round. Also, from the nose features of face type L, for example, eye roundness: sharp, eye tilt: fairly upward, eye size: large, eyebrow shape: sharp, eyebrow and eye position: fairly close, and facial outline: rectangular.

[0091] 8 is an example of a face estimated from nose features according to an embodiment of the present invention. In one embodiment of the present invention, it is possible to estimate which of the various face types shown in FIG. 8 the user's face belongs to based on the user's nose features.

[0092] In this way, face types can be classified based on nose features, which are less affected by lifestyle habits and shooting conditions. For example, face types classified based on nose features can be used to present makeup guides and skin characteristics (for example, makeup guides and skin characteristics can be presented based on what kind of facial features a face type has and what kind of impression a face type gives).

[0093] <Effects> In this way, the present invention makes it possible to easily estimate skin conditions from nose features. In one embodiment of the present invention, future skin conditions can be estimated from nose features, and cosmetics that can more effectively alleviate future skin problems can be selected, or beauty treatments such as massages can be determined.

[0094] <Hardware configuration> 9 is a diagram showing the hardware configuration of a skin condition estimating device 10 according to one embodiment of the present invention. The skin condition estimating device 10 has a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003. The CPU 1001, ROM 1002, and RAM 1003 form a so-called computer.

[0095] The skin condition estimating device 10 may also include an auxiliary storage device 1004 , a display device 1005 , an operation device 1006 , an I / F (Interface) device 1007 , and a drive device 1008 .

[0096] The hardware components of the skin condition estimating device 10 are connected to one another via a bus B.

[0097] The CPU 1001 is a computing device that executes various programs installed in the auxiliary storage device 1004 .

[0098] The ROM 1002 is a non-volatile memory. The ROM 1002 functions as a main storage device that stores various programs, data, etc. required for the CPU 1001 to execute various programs installed in the auxiliary storage device 1004. Specifically, the ROM 1002 functions as a main storage device that stores boot programs such as a BIOS (Basic Input / Output System) and an EFI (Extensible Firmware Interface).

[0099] The RAM 1003 is a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 1003 functions as a main storage device that provides a working area in which various programs installed in the auxiliary storage device 1004 are expanded when the CPU 1001 executes them.

[0100] The auxiliary storage device 1004 is an auxiliary storage device that stores various programs and information used when the various programs are executed.

[0101] The display device 1005 is a display device that displays the internal state of the skin condition estimating device 10 and the like.

[0102] The operation device 1006 is an input device through which the person operating the skin condition estimating device 10 inputs various instructions to the skin condition estimating device 10 .

[0103] The I / F device 1007 is a communication device that connects to a network and communicates with other devices.

[0104] Drive device 1008 is a device for loading storage medium 1009. The storage medium 1009 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, and magneto-optical disks. Storage medium 1009 may also include semiconductor memories that record information electrically, such as EPROMs (Erasable Programmable Read Only Memory) and flash memories.

[0105] The various programs to be installed in the auxiliary storage device 1004 are installed, for example, by setting the distributed storage medium 1009 in the drive device 1008 and reading out the various programs recorded on the storage medium 1009 by the drive device 1008. Alternatively, the various programs to be installed in the auxiliary storage device 1004 may be installed by being downloaded from a network via the I / F device 1007.

[0106] The skin condition estimating device 10 includes an image capturing device 1010. The image capturing device 1010 captures an image of the user 20.

[0107] Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims.

[0108] This international application claims priority to Japanese Patent Application No. 2021-021916, filed on February 15, 2021, the entire contents of which are hereby incorporated by reference into this international application. [Explanation of symbols]

[0109] 10 Skin condition estimation device 20 users 101 Image acquisition unit 102 Nose feature identification unit 103 Skin condition estimation unit 104 Skeleton Estimation Department 105 Output section 1001 CPU 1002 ROM 1003 RAM 1004 Auxiliary storage device 1005 Display device 1006 Operating device 1007 I / F device 1008 Drive device 1009 Storage medium 1010 Imaging device

Claims

1. Identifying a user's nose shape; estimating a skin condition of the user based on a shape of the user's nose; A skin condition estimation method comprising:

2. acquiring an image including the user's nose; The skin condition estimating method according to claim 1 , wherein the shape of the user's nose is identified from image information of the image.

3. The skin condition estimating method according to claim 1 , wherein the user's skin condition is a future skin condition of the user.

4. 4. The skin condition estimation method according to claim 1, wherein the skin condition is at least one of wrinkles, age spots, facial sagging, dark circles under the eyes, nasolabial folds, dull skin, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, skin texture, skin pores, and skin color.

5. The skin condition estimating method according to claim 4 , further comprising the step of estimating a comprehensive skin index from the skin condition.

6. The skin condition estimating method according to claim 1 , wherein the skin condition is the skin condition of one of a part of the face, the entire face, and a plurality of parts of the face.

7. The method further includes estimating a shape related to the user's facial bones based on the shape of the user's nose, The skin condition estimating method according to claim 1 , wherein the estimation of the user's skin condition is based on a shape related to the user's facial bone structure.

8. The skin condition estimating method according to claim 7 , wherein the user's skin condition is determined based on a shape related to the user's facial bone structure.

9. The skin condition estimating method according to claim 1 , wherein the shape of the nose is at least one of the shape of a root of the nose, a bridge of the nose, a tip of the nose, and an ala of the nose.

10. The skin condition estimation method according to claim 1 , wherein the user's skin condition is estimated using a trained model that outputs the skin condition when the nose shape is input.

11. an identification unit that identifies the shape of the user's nose; an estimation unit that estimates a skin condition of the user based on a shape of the user's nose; A skin condition estimation device comprising:

12. Computer an identification unit that identifies the shape of the user's nose; an estimation unit that estimates the skin condition of the user based on the shape of the user's nose; A program to function as a

13. A system including a skin condition estimating device and a server, an identification unit that identifies the shape of the user's nose; an estimation unit that estimates a skin condition of the user based on a shape of the user's nose; A system with.

14. A step of acquiring training data in which input data is nose shape and output data is skin condition; a step of performing machine learning using the training data to generate a trained model that outputs the skin condition when the nose shape is input; A method for generating trained models, including:

15. A trained model for causing a computer to function so that when the nose shape is input, it outputs the skin condition, and is generated by machine learning using training data in which the input data is the nose shape and the output data is the skin condition.

Images