Skin state inference method, device, program, system, trained model generation method, and trained model

Abstract

The present invention makes it possible to easily obtain the state of a skin. A method according to one embodiment of the present invention comprises: a step for identifying a feature of the nose of a user; and a step for inferring the state of a skin of the user on the basis of the feature of the nose of the user.

Description

Skin condition estimation method, device, program, system, trained model generation method, and trained model

[0001] The present invention relates to a skin condition estimation method, an apparatus, a program, a system, a trained model generation method, and a trained model.

[0002] Conventionally, there are known techniques for predicting skin conditions in order to provide appropriate skin care, etc. For example, in Patent Document 1, future formation and level of wrinkles around the eyes and mouth are predicted from ultrasound images.

[0003] JP 2011-200284 A

[0004] However, in Patent Document 1, an ultrasonic diagnostic device is required, and it is not easy to simply predict skin conditions that are likely to occur in the future.

[0005] Therefore, an object of the present invention is to easily achieve a desired skin condition.

[0006] A method according to one embodiment of the present invention includes identifying a user's nose features and estimating the user's skin condition based on the user's nose features.

[0007] In the present invention, the skin condition can be easily estimated from the nose features.

[0008] FIG. 1 is a diagram showing an overall configuration according to an embodiment of the present invention. FIG. 2 is a diagram showing functional blocks of a skin condition estimation device according to an embodiment of the present invention. FIG. 3 is a flowchart showing a processing flow of skin condition estimation according to an embodiment of the present invention. FIG. 4 is a diagram for explaining nose features according to an embodiment of the present invention. FIG. 5 is a diagram for explaining extraction of a nose region according to an embodiment of the present invention. FIG. 6 is a diagram for explaining calculation of nose feature amounts according to an embodiment of the present invention. FIG. 7 is an example of nose features for each face type according to an embodiment of the present invention. FIG. 8 is an example of a face estimated from nose features according to an embodiment of the present invention. FIG. 9 is a diagram showing a hardware configuration of a skin condition estimation device according to an embodiment of the present invention.

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0010] <Explanation of Terms> "Skin condition" refers to at least one of wrinkles, age spots, sagging, dark circles, nasolabial folds, dullness, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, pores, and skin color. For example, "skin condition" refers to the presence or absence and degree of elements that make up the skin condition, such as wrinkles, age spots, sagging, dark circles, nasolabial folds, dullness, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, pores, and skin color. Furthermore, "skin condition" refers to the skin condition of one part of the face, the entire face, or multiple parts of the face. Note that "skin condition" may refer to the user's future skin condition or the user's current skin condition. In the present invention, the skin condition is estimated from nose features based on the correlation between nose features and skin condition.

[0011] <Overall Configuration> Fig. 1 is a diagram showing the overall configuration according to one embodiment of the present invention. A skin condition estimation device 10 estimates the skin condition of a user 20 from the characteristics of the user's nose. For example, the skin condition estimation device 10 is a smartphone or the like with a camera function. The skin condition estimation device 10 will be described in detail later with reference to Fig. 2.

[0012] Although the present specification describes a case where skin condition estimation device 10 is a single device (e.g., a smartphone with a camera function), skin condition estimation device 10 may also be composed of multiple devices (e.g., a device without a camera function and a digital camera). Furthermore, the camera function may be a function for capturing images of skin in three dimensions or two dimensions. Furthermore, a device other than skin condition estimation device 10 (e.g., a server) may perform some of the processes performed by skin condition estimation device 10 described herein.

[0013] <Functional Blocks of Skin Condition Estimation Device 10> Figure 2 is a diagram showing functional blocks of the skin condition estimation device 10 according to one embodiment of the present invention. The skin condition estimation device 10 can include an image acquisition unit 101, a nose feature identification unit 102, a skin condition estimation unit 103, a skeleton estimation unit 104, and an output unit 105. Furthermore, by executing a program, the skin condition estimation device 10 can function as the image acquisition unit 101, the nose feature identification unit 102, the skin condition estimation unit 103, the skeleton estimation unit 104, and the output unit 105. Each of these units will be described below.

[0014] The image acquisition unit 101 acquires an image including the nose of the user 20. Note that the image including the nose may be an image capturing both the nose and other parts (for example, an image capturing the entire face), or an image capturing only the nose (for example, an image captured so that the nose area of ​​the user 20 fits within a predetermined area displayed on the display device of the skin condition estimation device 10). Note that if the features of the nose can be identified from something other than the image, the image acquisition unit 101 is not necessary.

[0015] The nose feature identification unit 102 identifies the nose features of the user 20. For example, the nose feature identification unit 102 identifies the nose features of the user 20 from image information (e.g., pixel values ​​of the image) of an image including the nose of the user 20 acquired by the image acquisition unit 101.

[0016] The skin condition estimation unit 103 estimates the skin condition of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102. For example, the skin condition estimation unit 103 classifies the skin condition of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102.

[0017] In addition, the skin condition estimation unit 103 can also estimate the skin condition of the user 20 (e.g., the skin condition resulting from the shape of the facial skeleton) based on the shape of the facial skeleton of the user 20 estimated by the skeleton estimation unit 104.

[0018] The skeleton estimation unit 104 estimates the shape of the skeleton of the face of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102. For example, the skeleton estimation unit 104 classifies the shape of the skeleton of the face of the user 20 based on the nose features of the user 20 identified by the nose feature identification unit 102.

[0019] The output unit 105 outputs (for example, displays) information on the skin condition of the user 20 estimated by the skin condition estimation unit 103 .

[0020] <Skin Condition> Here, the skin condition will be described. For example, the skin condition may be at least one of wrinkles, age spots, sagging, dark circles, nasolabial folds, dullness, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, texture, pores, and skin color. More specifically, the skin condition may include, for example, wrinkles at the corners of the eyes, wrinkles under the eyes, wrinkles on the forehead, wrinkles around the eye sockets, sagging eye bags, dark circles under the eyes, nasolabial folds (nasolabial folds and around the mouth), depth of the nasolabial folds, sagging marionette lines, sagging chin, HbSO2 Index (blood oxygen saturation index), Hb Index (amount of hemoglobin), HbO2 (amount of oxyhemoglobin), skin color, skin brightness, moisture retention capacity (TEWL), number of skin ridges, skin viscoelasticity, amount of oxygen in the blood, blood vessel density, number of capillaries, number of blood vessel branches, distance between blood vessels and the epidermis, epidermal thickness, HDL cholesterol, sebum, moisture content, melanin index (an index of melanin), pores, transparency, uneven skin tone (brownishness, reddishness), pH, etc. The skin condition estimation unit 103 estimates the skin condition from the nose features based on the correlation between the nose features and the skin condition.

[0021] <Correspondence between nose features and skin condition> Here, the correspondence between nose features and skin condition will be described. The skin condition estimation unit 103 estimates the skin condition based on the correspondence between nose features and skin condition stored in advance in the skin condition estimation device 10, etc. Note that the skin condition may be estimated not only based on nose features but also on some of nose features and facial features.

[0022] The correspondence may be a predetermined database or a trained model based on machine learning. The database associates nose features (which may be nose features and some facial features) with skin conditions based on the results of experiments conducted on subjects. The trained model is a prediction model that outputs information on skin conditions when information on nose features (which may be nose features and some facial features) is input.

[0023] <<Generation of Trained Model>> In one embodiment of the present invention, a computer such as the skin condition estimation device 10 can generate a trained model. Specifically, the computer such as the skin condition estimation device 10 acquires training data in which input data is nose features (which may be nose features and part of facial features) and output data is a skin condition, and performs machine learning using the training data to generate a trained model that outputs a skin condition when nose features (which may be nose features and part of facial features) are input. In this way, by performing machine learning using training data in which input data is nose features (which may be nose features and part of facial features) and output data is a skin condition, a trained model is generated that outputs a skin condition when nose features (which may be nose features and part of facial features) are input.

[0024] <Correspondence between facial skeletal shapes and skin conditions> Here, the correspondence between facial skeletal shapes and skin conditions will be described. As described above, the skin condition estimation unit 103 can also estimate the skin condition based on the correspondence between facial skeletal shapes and skin conditions that is stored in advance in the skin condition estimation device 10, etc.

[0025] The correspondence may be a predetermined database or a trained model generated by machine learning. The database associates the shape of the facial skeleton with the skin condition based on the results of experiments conducted on subjects. The trained model is a prediction model that outputs information on the skin condition when information on the shape of the facial skeleton is input.

[0026] <<Generation of Trained Model>> In one embodiment of the present invention, a computer such as the skin condition estimation device 10 can generate a trained model. Specifically, the computer such as the skin condition estimation device 10 can acquire training data in which input data is a shape related to the facial skeleton and output data is a skin condition, and perform machine learning using the training data to generate a trained model that outputs a skin condition when a shape related to the facial skeleton is input. In this way, by performing machine learning using training data in which input data is a shape related to the facial skeleton and output data is a skin condition, a trained model is generated that outputs a skin condition when a shape related to the facial skeleton is input.

[0027] <Future Skin Condition and Current Skin Condition> The estimated skin condition may be the future skin condition of the user 20 or the current skin condition of the user 20. When the correspondence between the nose features (or the shape related to the facial bones estimated from the nose features) and the skin condition is created based on data of a person older than the actual age of the user 20 (for example, the age of the experimental subject or the age of the person used as learning data for machine learning is older than the actual age of the user 20), the future skin of the user 20 is estimated. On the other hand, when the correspondence between the nose features (or the shape related to the facial bones estimated from the nose features) and the skin condition is created based on data of a person of the same age as the actual age of the user 20 (for example, the age of the experimental subject or the age of the person used as learning data for machine learning is the same as the actual age of the user 20), the current skin of the user 20 is estimated. The skin condition may be estimated based not only on the nose features but also on some of the nose features and facial features.

[0028] An example of estimation based on the correspondence between nose features (or the shape of the facial skeleton estimated from nose features) and skin conditions will be described below.

[0029] <<Skin Condition Estimation Example 1>> For example, the skin condition estimation unit 103 can estimate that wrinkles are likely to appear at the corners of the eyes when the root and bridge of the nose are high. Furthermore, for example, if the cheek shape has a high cheekbone at the top, the skin condition estimation unit 103 can estimate (determine ON / OFF) that there are wrinkles at the corners of the eyes or that there is a possibility that wrinkles will appear in the future.

[0030] <<Skin Condition Estimation Example 2>> For example, the skin condition estimation unit 103 can estimate that the more rounded the nostrils are, or the larger the eyes are, the more likely wrinkles are to form under the eyes.

[0031] For example, the skin condition estimation unit 103 can estimate that, although eye sockets have shape characteristics such as being horizontally long and small, large eye sockets with similar vertical and horizontal widths have more wrinkles under the eyes. Furthermore, for example, the skin condition estimation unit 103 can estimate wrinkles under the eyes based on the facial contour. Furthermore, for example, the skin condition estimation unit 103 can estimate that the greater the distance between the eyes, the fewer wrinkles there are under the eyes.

[0032] <<Skin Condition Estimation Example 3>> For example, the skin condition estimation unit 103 can estimate the sagging of the eye bags based on the roundness of the nostrils and the height of the nose bridge. Specifically, the skin condition estimation unit 103 can estimate that the greater the sum of the roundness of the nostrils and the height of the nose bridge, the more sagging the eye bags.

[0033] For example, the skin condition estimation unit 103 can estimate that the eye bags are likely to sag when the face contour is oval and long.

[0034] <<Skin Condition Estimation Example 4>> For example, the skin condition estimation unit 103 can estimate HbCO2 (reduced hemoglobin) based on the lowness of the nasal bridge and the roundness of the nostrils.

[0035] For example, the skin condition estimation unit 103 can estimate HbSO2 (oxygen saturation) based on the facial contour.

[0036] <<Skin Condition Estimation Example 5>> For example, the skin condition estimation unit 103 can estimate that the lower the nose bridge, the rounder the nostrils, or the greater the distance between the eyes, the lower the moisture content.

[0037] For example, the skin condition estimation unit 103 can estimate the skin moisture content based on the height of the skull index and the aspect ratio of the face.

[0038] <<Skin Condition Estimation Example 6>> For example, the skin condition estimation unit 103 can estimate sebum based on the roundness of the ala of the nose.

[0039] For example, the skin condition estimation unit 103 can estimate sebum based on the facial contour.

[0040] <<Skin Condition Estimation Example 7>> For example, the skin condition estimation unit 103 can estimate that the rounder the nostrils and the higher the nasal bridge, the higher the melanin index and the greater the amount of melanin, and that the lower the nasal bridge and the narrower the distance between the eyes, the lower the melanin index.

[0041] For example, the skin condition estimation unit 103 can estimate that the thicker the upper and lower lips are, the higher the melanin index and the greater the amount of melanin. Also, for example, the skin condition estimation unit 103 can estimate that the thinner the upper and lower lips are, the lower the melanin index.

[0042] <<Skin Condition Estimation Example 8>> For example, the skin condition estimation unit 103 can estimate that dark circles under the eyes are more likely to occur when the nostrils are round.

[0043] <<Skin Condition Estimation Example 9>> For example, the skin condition estimation unit 103 can estimate that the facial line is prone to sagging when the nose bridge is low and the distance between the eyes is wide, or when the angle of the chin is rounded.

[0044] <<Skin Condition Estimation Example 10>> For example, the skin condition estimation unit 103 can estimate that the higher the bridge of the nose, the higher the amount of oxygen in the blood.

[0045] <<Skin Condition Estimation Example 11>> For example, the skin condition estimation unit 103 can estimate the blood vessel density from the size of the nostrils or the position of change in the height of the nose root, and the larger the nostrils, the higher the blood vessel density.

[0046] <<Skin Condition Estimation Example 12>> For example, the skin condition estimation unit 103 can estimate the epidermal thickness from the size of the ala of the nose.

[0047] <<Skin Condition Estimation Example 13>> For example, the skin condition estimation unit 103 can estimate the number of blood vessel branches from the position of change in the height of the bridge of the nose.

[0048] <<Estimating Comprehensive Skin Condition>> In one embodiment of the present invention, the skin condition estimation unit 103 can comprehensively represent the skin condition as wrinkles, age spots, sagging, dark circles, nasolabial folds, dullness, firmness, moisture, sebum, melanin, circulation, blood vessels, blood, pores, and skin color from the values ​​estimated in the above estimation examples 1 to 9. Examples are shown below. Wrinkles: Represented by one or more of the following: wrinkles at the corners of the eyes, under the eyes, forehead, and eyelids. Age spots: Represented by one or more of the following: brown discoloration, red discoloration, and melanin. Sagging: Represented by one or more of the following: bags under the eyes, chin, and marionette lines. Dark circles: Represented by one or two of the following: dark circles under the eyes and blue circles under the eyes. Nasolabial folds: Represented by one or two of the following: nasolabial folds in the nasolabial folds and nasolabial folds around the mouth. - Dullness: Expressed by one or more of the following items: transparency, melanin, uneven skin tone, skin color, oxygen saturation, moisture, and number of skin ridges. - Firmness: Expressed by one or more of the following items: moisture, sebum, sagging, and skin viscoelasticity. - Moisture: Expressed by one or two of the following items: moisture content, moisture retention capacity (TEWL), number of skin ridges, and pH. - Texture: Expressed by one or two of the following items: number of skin ridges and moisture. - Skin color: Expressed by one or two of the following items: skin tone, skin brightness, melanin, amount of oxygen in the blood, and HbO2 (oxygenated hemoglobin). - Sebum: Expressed by one or two of the following items: sebum amount and pores. Skin can also be classified into normal, dry, oily, and combination skin types based on moisture and sebum. - Melanin: Expressed by one or two of the following items: melanin index, melanin amount, and uneven skin tone.・Blood circulation: Expressed from at least one or two of the following items: HbSO2 Index (blood oxygen saturation index), Hb Index (hemoglobin amount), HbO2 (oxyhemoglobin amount), blood oxygen amount, and skin color. ・Blood vessels: Expressed from one or more of the following items: blood vessel density, number of capillaries, number of blood vessel branches, distance between blood vessels and epidermis, and epidermal thickness. ・Blood: HDL cholesterol

[0049] In one embodiment of the present invention, the skin condition estimation unit 103 can represent skin features such as skin strengths and skin weaknesses from nose features. For example, if the nose features are Type 1, the evaluation value of wrinkles at the corners of the eyes is lower than the average evaluation value, and therefore the skin is represented as strong. If the nose features are Type 2, the evaluation value of wrinkles at the corners of the eyes is higher than the average evaluation value, and therefore the skin is represented as weak. Skin strengths and weaknesses can be represented for each part of the face. For Type 1, skin strengths include wrinkles at the corners of the eyes and forehead, and age spots, while skin weaknesses include dark circles, nasolabial folds, sagging around the mouth, and moisture retention. The skin condition estimation unit 103 can estimate a comprehensive skin index (in this case, sagging skin) from these skin conditions. For Type 2, skin strengths include cheek sagging, moisture retention, blood circulation, and age spots, and skin weaknesses include wrinkles at the corners of the eyes and forehead, and age spots. The skin condition estimation unit 103 can estimate a comprehensive skin index (in this case, wrinkled skin type) from these skin conditions.

[0050] <Shape Related to the Facial Skeleton> Here, the shape related to the facial skeleton will be described. The "shape related to the facial skeleton" refers to at least one of the shape of the facial skeleton itself and the shape of the face resulting from the skeleton. The skeleton estimation unit 104 estimates the shape related to the facial skeleton from the nose features based on the correlation between the nose features and the shape related to the facial skeleton.

[0051] For example, the shape of the facial skeleton includes the characteristics of the shape of each bone, the positional relationship of the skeleton, angles, etc., of at least one of the eye sockets, cheekbones, nasal bones, piriform aperture (the opening of the nasal cavity toward the face), skull index, maxilla, mandible, lips, corners of the mouth, eyes, epicanthic folds (skin folds in the area where the upper eyelid covers the inner corner of the eye), facial contour, and the positional relationship between the eyes and eyebrows (for example, whether the eyes and eyebrows are far apart or close together). An example of the shape of the facial skeleton is shown below. Note that the information in parentheses is an example of the specific content to be estimated.・Eye sockets (long, square, rounded) ・Cheekbones, cheeks (peak position, roundness) ・Nose bones (width, shape) ・Piriform mouth (shape) ・Skull index (skull width / depth = 70, 75, 80, 85, 90) ・Maxilla, upper jaw (position relative to eye socket, nasolabial angle) ・Mandible bones, lower jaw (depth length, depth angle, forward angle, outline shape (jaw)) ・Forehead (roundness of forehead, forehead shape) ・Eyebrows (distance between eyes and eyebrows, eyebrow shape, eyebrow thickness) ・Lips (thick both upper and lower lips, thick lower lip, thin both upper and lower lips, large horizontally, small) ・Corner of mouth (upward, downward, normal) ・Eyes (area, angle, distance between eyebrows and eyes, distance between eyes) ・Mongolian folds (yes, no) -Facial contour (Rectangle, Round, Obal, Heart, Square, Average, Natural, Long)

[0052] <Correspondence between nose features and facial skeletal shapes> Here, the correspondence between nose features and facial skeletal shapes will be described. The skeleton estimation unit 104 estimates the facial skeletal shape based on the correspondence between nose features and facial skeletal shapes that is stored in advance in the skin condition estimation device 10, etc. Note that the facial skeletal shape may be estimated not only based on nose features but also on some of the nose features and facial features.

[0053] The correspondence may be a predetermined database or a trained model generated by machine learning. The database associates nose features (which may be nose features and some facial features) with shapes related to the facial skeleton based on the results of experiments conducted on subjects. The trained model is a predictive model that outputs information on the shape of the facial skeleton when information on nose features (which may be nose features and some facial features) is input. The correspondence between nose features and shapes related to the facial skeleton may be created for each group (e.g., Caucasoid, Mongoloid, Negroid, Australoid, etc.) classified based on factors that may affect the skeleton.

[0054] <<Generation of Trained Model>> In one embodiment of the present invention, a computer such as the skin condition estimation device 10 can generate a trained model. Specifically, the computer such as the skin condition estimation device 10 acquires training data in which input data are nose features (which may be nose features and part of facial features) and output data are shapes related to the facial skeleton, and performs machine learning using the training data to generate a trained model that outputs a shape related to the facial skeleton when nose features (which may be nose features and part of facial features) are input. In this way, by performing machine learning using training data in which input data are nose features (which may be nose features and part of facial features) and output data are shapes related to the facial skeleton, a trained model is generated that outputs a shape related to the facial skeleton when nose features (which may be nose features and part of facial features) are input.

[0055] An example of estimation based on the correspondence between nose features and facial skeletal shapes will be described below.

[0056] <<First Example of Estimation of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can estimate the skull index based on the height or lowness of the nose root or the change position of the height of the nose root and the height or lowness of the nose bridge. Specifically, the skeleton estimation unit 104 estimates that the higher at least one of the nose root and the nose bridge is, the lower the skull index is.

[0057] <<Estimation Example 2 of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can estimate whether the corners of the mouth are tilted upward or downward based on the width of the bridge of the nose. Specifically, the skeleton estimation unit 104 estimates that the wider the bridge of the nose, the more the corners of the mouth are tilted downward.

[0058] <<Example 3 of estimating shapes related to the facial skeleton>> For example, the skeleton estimation unit 104 can estimate the size and thickness of the lips (1. both the upper and lower lips are large and thick, 2. the lower lip is thick, 3. both the upper and lower lips are thin and small) based on the roundness of the nostrils and the sharpness of the nasal tip.

[0059] <<Fourth Example of Estimation of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can estimate the presence or absence of epicanthic folds based on the nasal bridge. Specifically, the skeleton estimation unit 104 estimates the presence of epicanthic folds when the nasal bridge is determined to be low.

[0060] <<5th Example of Estimation of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can classify the shape of the lower jaw (e.g., into three categories) based on the height or lowness of the nasal bridge, the height of the nasal root, and the roundness and size of the nostrils.

[0061] <<Estimation Example 6 of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can estimate a piriform mouth based on the height of the bridge of the nose.

[0062] <<Estimation Example 7 of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can estimate the distance between the eyes based on the lowness of the nose bridge. Specifically, the skeleton estimation unit 104 estimates that the lower the nose bridge, the wider the distance between the eyes.

[0063] <<Estimation Example 8 of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can estimate the roundness of the forehead based on the height of the root of the nose and the height of the nose bridge.

[0064] <<Example 9 of Estimation of Shape Related to Facial Skeleton>> For example, the skeleton estimation unit 104 can estimate the distance between the eyes and the eyebrows and the shape of the eyebrows based on the height or lowness of the bridge of the nose, the size of the nostrils, and the position of change in the height of the root of the nose.

[0065] <Processing Method> FIG. 3 is a flowchart showing the flow of processing for skin condition estimation according to one embodiment of the present invention.

[0066] In step 1 (S1), the nose feature specifying unit 102 extracts feature points (for example, feature points of the inner corners of the eyebrows, the inner corners of the eyes, and the tip of the nose) from an image including the nose.

[0067] In step 2 (S2), the nose feature specifying unit 102 extracts the nose region based on the feature points extracted in S1.

[0068] In addition, if the image including the nose is an image in which only the nose is captured (for example, an image captured so that the nose area of ​​the user 20 fits within a specified area displayed on the display device of the skin condition estimation device 10), the image in which only the nose is captured is used as is (i.e., S1 can be omitted).

[0069] In step 3 (S3), the nose feature identification unit 102 reduces the number of gradations of the image of the nose region extracted in S2 (for example, binarizes it). For example, the nose feature identification unit 102 reduces the number of gradations of the image of the nose region using at least one of lightness, luminance, RGB blue, and RGB green. Note that S3 may be omitted.

[0070] In step 4 (S4), the nose feature identification unit 102 identifies nose features (nose bone structure). Specifically, the nose feature identification unit 102 calculates nose feature amounts based on image information of the image of the nose region (e.g., pixel values ​​of the image). For example, the nose feature identification unit 102 calculates, as nose feature amounts, the average pixel value of the nose region, the number of pixels equal to or less than a predetermined value or equal to or greater than a predetermined value, the cumulative pixel value, the amount of change in pixel value, etc.

[0071] In step 5 (S5), the skeleton estimation unit 104 estimates the shape of the facial skeleton. Note that S5 can be omitted.

[0072] In step 6 (S6), the skin condition estimation unit 103 estimates the skin condition (e.g., future skin concerns) based on the nose features identified in S4 (or the shape related to the facial skeleton estimated in S5).

[0073] <Nose Features> Here, the nose features will be described. For example, the nose features include at least one of the root of the nose, the bridge of the nose, the tip of the nose, and the ala of the nose.

[0074] 4 is a diagram illustrating the features of a nose according to an embodiment of the present invention, showing the positions of the root, bridge, tip, and ala of the nose.

[0075] <<Root of nose>> The root of the nose is the base of the nose. For example, the characteristics of the nose are at least one of the height of the root of the nose, the lowness of the root of the nose, the width of the root of the nose, and the position where the root of the nose changes to a high level.

[0076] <<Nose Bridge>> The nose bridge is the area between the eyebrows and the tip of the nose. For example, the nose's characteristics include at least one of the height of the nose bridge, the width of the nose bridge, and the depth of the nose bridge.

[0077] <<Nose Tip>> The nose tip is the tip of the nose. For example, nose features include at least one of the roundness or sharpness of the nose tip and the direction of the nose tip.

[0078] <<Nose Alar>> The nose ala is the bulging part on either side of the tip of the nose. For example, the characteristics of the nose are at least one of the roundness or sharpness of the nose ala and the size of the nose ala.

[0079] <Extraction of Nose Region> Fig. 5 is a diagram for explaining extraction of a nose region according to one embodiment of the present invention. The nose feature identification unit 102 extracts the nose region from an image including a nose. For example, the nose region may be the entire nose as shown in Fig. 5(a) or a part of the nose (e.g., the right half or the left half) as shown in Fig. 5(b).

[0080] <Calculation of Nose Feature Amount> FIG. 6 is a diagram for explaining calculation of nose feature amount according to one embodiment of the present invention.

[0081] In step 11 (S11), a nose region is extracted from an image containing a nose.

[0082] In step 12 (S12), the number of gradations of the image of the nose area extracted in S11 is reduced (for example, binarized). Note that S12 can be omitted.

[0083] In step 13 (S13), nose feature values ​​are calculated. Note that in FIG. 6, pixel cumulative values ​​are represented by setting the high-brightness side of the image to 0 and the low-brightness side to 255. For example, the nose feature specification unit 102 performs normalization for each of a plurality of regions (e.g., the regions divided in S12). Next, for each region, the nose feature specification unit 102 calculates, as nose feature values, the average pixel value, the number of pixels equal to or less than a predetermined value, the pixel cumulative value in at least one of the X and Y directions, and the amount of change in pixel value in at least one of the X and Y directions (e.g., using data on the low-brightness or high-brightness side of the image). In S13 of FIG. 6, the pixel cumulative value in the X direction at each position in the Y direction is calculated.

[0084] The method for calculating each feature amount will be described below.

[0085] For example, the feature amount of the root of the nose is the feature amount of the upper region (closer to the eyes) of the divided region S12, the feature amount of the bridge of the nose is the feature amount of the upper or central region of the divided region S12, and the feature amount of the tip of the nose and the wings of the nose are the feature amount of the lower region (closer to the mouth) of the divided region S12. These nose feature amounts are normalized by the distance between the eyes.

[0086] - Nose root height: The height or shortness of the nose is determined from the change in pixel values ​​in the Y direction in the upper region of the nose. The height or shortness may be calculated as a numerical value or classified as high or low. Regarding the change in the height of the nose root, in S13, the value of nose 2 changes immediately in the Y direction, indicating that the change in the height of the nose root is located at the upper part. - Nose root width: The upper region of the nose is divided into multiple regions (2 to 4, etc.) in the X direction, and the width is determined from the pattern of the average pixel values ​​of each region. - Nose bridge height: The height or shortness of the nose is determined from the average cumulative pixel values ​​of the central region of the nose. The height or shortness may be calculated as a numerical value or classified as high or low. - Nose bridge width: The central region of the nose is divided into multiple regions (2 to 4, etc.) in the X direction, and the width is determined from the pattern of the average pixel values ​​of each region. - Nose tip roundness or sharpness: This is determined from other nasal features (nasal bridge height, roundness or sharpness of the nostrils), and the lower the nose bridge and the rounder the nostrils, the more rounded the nose. - Direction of nose tip: In the central area of ​​the nose, it is calculated from the width from the lowest point of the nose at a position that is a specified percentage of the maximum cumulative pixel value in the X direction; the wider the width, the more upward-pointing it is. - Roundness or kurtosis of the nostrils: The roundness or kurtosis is determined from the amount of change in the Y direction value in the lower area of ​​the nose. - Size of the nostrils: It is determined from the percentage of pixels that are below a specified value in the central part of the lower area. The more pixels there are, the larger the nostrils.

[0087] <<Facial Type>> As described above, the "shape related to the facial bone structure" refers to at least one of the "shape of the facial bone structure itself" and the "shape of the face resulting from the bone structure." The "shape related to the facial bone structure" can include the face type.

[0088] In one embodiment of the present invention, it is possible to estimate which of a plurality of face types the user's face belongs to (specifically, face types classified based on at least one of the "shape of the facial bone structure itself" and the "shape of the face resulting from the bone structure") based on the features of the user's nose. Hereinafter, face types will be described with reference to FIGS. 7 and 8.

[0089] 7 shows an example of nose features for each face type according to one embodiment of the present invention. Fig. 7 shows nose features for each face type (face types A to L). Note that a face type may be estimated using all four of the features: the bridge of the nose, the alae of the nose, the root of the nose, and the tip of the nose, or using only some of the features (for example, two of the bridge of the nose and the alae of the nose, two of the bridge of the nose and the root of the nose, only the bridge of the nose, only the alae of the nose, etc.).

[0090] In this way, face types are estimated from nose features. For example, from the nose features of face type A, eye roundness: round, eye tilt: downward, eye size: small, eyebrow shape: arched, eyebrow and eye position: far apart, and facial outline: round. Also, from the nose features of face type L, for example, eye roundness: sharp, eye tilt: significantly upward, eye size: large, eyebrow shape: sharp, eyebrow and eye position: significantly close, and facial outline: rectangular.

[0091] 8 is an example of a face estimated from nose features according to an embodiment of the present invention. In an embodiment of the present invention, it is possible to estimate which of the various face types shown in FIG. 8 the user's face belongs to based on the user's nose features.

[0092] In this way, face types can be classified based on nose features that are less affected by lifestyle habits and shooting conditions. For example, face types classified based on nose features can be used to present makeup guides and skin characteristics (for example, makeup guides and skin characteristics can be presented based on what kind of facial features a face type has and what kind of impression a face type gives).

[0093] <Effects> In this way, the present invention makes it possible to easily estimate skin conditions from nose features. In one embodiment of the present invention, future skin conditions can be estimated from nose features, and cosmetics that can more effectively alleviate future skin problems can be selected, or beauty treatments such as massages can be determined.

[0094] 9 is a diagram showing the hardware configuration of a skin condition estimation device 10 according to an embodiment of the present invention. The skin condition estimation device 10 has a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003. The CPU 1001, ROM 1002, and RAM 1003 form a so-called computer.

[0095] The skin condition estimating device 10 may also include an auxiliary storage device 1004 , a display device 1005 , an operation device 1006 , an I / F (Interface) device 1007 , and a drive device 1008 .

[0096] The hardware components of the skin condition estimating device 10 are connected to one another via a bus B.

[0097] The CPU 1001 is a computing device that executes various programs installed in the auxiliary storage device 1004 .

[0098] The ROM 1002 is a non-volatile memory. The ROM 1002 functions as a main storage device that stores various programs, data, etc. required for the CPU 1001 to execute various programs installed in the auxiliary storage device 1004. Specifically, the ROM 1002 functions as a main storage device that stores boot programs such as a Basic Input / Output System (BIOS) and an Extensible Firmware Interface (EFI).

[0099] The RAM 1003 is a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 1003 functions as a main storage device that provides a working area in which various programs installed in the auxiliary storage device 1004 are expanded when the CPU 1001 executes them.

[0100] The auxiliary storage device 1004 is an auxiliary storage device that stores various programs and information used when the various programs are executed.

[0101] The display device 1005 is a display device that displays the internal state of the skin condition estimating device 10, etc.

[0102] The operation device 1006 is an input device through which the person operating the skin condition estimating device 10 inputs various instructions to the skin condition estimating device 10 .

[0103] The I / F device 1007 is a communication device that connects to a network and communicates with other devices.

[0104] The drive device 1008 is a device for loading a storage medium 1009. The storage medium 1009 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. The storage medium 1009 may also include semiconductor memory that records information electrically, such as an EPROM (Erasable Programmable Read Only Memory) or a flash memory.

[0105] The various programs to be installed in the auxiliary storage device 1004 are installed, for example, by setting the distributed storage medium 1009 in the drive device 1008 and reading the various programs recorded on the storage medium 1009 by the drive device 1008. Alternatively, the various programs to be installed in the auxiliary storage device 1004 may be installed by being downloaded from a network via the I / F device 1007.

[0106] The skin condition estimating device 10 includes an image capturing device 1010. The image capturing device 1010 captures an image of the user 20.

[0107] Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of the present invention as set forth in the claims.

[0108] This international application claims priority to Japanese Patent Application No. 2021-021916, filed on February 15, 2021, the entire contents of which are hereby incorporated by reference into this international application.

[0109] REFERENCE SIGNS LIST 10 Skin condition estimation device 20 User 101 Image acquisition unit 102 Nose feature identification unit 103 Skin condition estimation unit 104 Skeleton estimation unit 105 Output unit 1001 CPU 1002 ROM 1003 RAM 1004 Auxiliary storage device 1005 Display device 1006 Operation device 1007 I / F device 1008 Drive device 1009 Storage medium 1010 Photography device

Claims

1. Identifying a user's nose features; estimating a skin condition of the user based on a nose feature of the user; A skin condition estimation method comprising:

2. acquiring an image including the user's nose; The method of claim 1 , wherein the nose feature of the user is identified from image information of the image.

3. The skin condition estimating method according to claim 1 , wherein the skin condition of the user is a future skin condition of the user.

4. The skin condition estimating method according to claim 1 , wherein the skin condition is at least one of wrinkles, age spots, facial sagging, dark circles under the eyes, nasolabial folds, dull skin, firmness, moisture, sebum, melanin, blood circulation, blood vessels, blood, skin texture, skin pores, and skin color.

5. The method of estimating a skin condition according to claim 4 , further comprising the step of estimating a comprehensive skin index from the skin condition.

6. The skin condition estimating method according to claim 1 , wherein the skin condition is a skin condition of any one of a part of a face, an entire face, and a plurality of parts of the face.

7. The method further includes estimating a shape of the user's facial anatomy based on the user's nose features; The skin condition estimating method according to claim 1 , wherein the estimation of the user's skin condition is based on a shape related to the user's facial bone structure.

8. The skin condition estimating method according to claim 7 , wherein the user's skin condition is caused by a shape related to the user's facial bone structure.

9. The method for estimating a skin condition according to claim 1 , wherein the nose feature is at least one of a root of the nose, a bridge of the nose, a tip of the nose, and an ala of the nose.

10. The skin condition estimating method according to claim 1 , wherein the user's skin condition is estimated using a trained model that outputs the skin condition when the nose features are input.

11. an identification unit for identifying a nose feature of a user; an estimation unit that estimates a skin condition of the user based on a feature of the user's nose; A skin condition estimation device comprising:

12. Computer an identification unit for identifying a nose feature of a user; an estimation unit that estimates a skin condition of the user based on a feature of the user's nose; A program to function as a

13. A system including a skin condition estimating device and a server, an identification unit for identifying a nose feature of a user; an estimation unit that estimates a skin condition of the user based on a feature of the user's nose; A system equipped with.

14. A step of acquiring teacher data in which input data is nose features and output data is skin condition; A step of performing machine learning using the training data to generate a trained model that outputs the skin condition when the nose features are input; A method for generating trained models, including:

15. A trained model generated by machine learning using training data in which input data are nose features and output data are skin condition, in which the skin condition is output when the nose features are input.