Information processing system

Abstract

This information processing system has a control unit. The control unit generates a learning model for estimating a skin classification of a user from at least one of skeleton information indicating the characteristics of a skeleton of the user or skin color information indicating the characteristics of the color of a skin of the user, which is acquired by analyzing a face image of the user, using skin classification information for classifying the skin of the user, which is generated by the analysis of user biometric information provided by a plurality of users, as training data. The control unit receives an input of the face image of the user, analyzes the received face image to acquire at least one of the skeleton information and the skin color information, and uses the learning model to output the skin classification information estimated from at least one of the acquired skeleton information or skin color information.

Description

Information Processing Systems

[0001] The present invention relates to an information processing system, an information processing method, and a program for estimating a user's skin classification.

[0002] There have been conventional techniques for classifying a user's skin by analyzing the user's biological information. For example, Patent Document 1 below describes a technique for classifying the user's skin into a group with high or low sebum content, or a group with high or low skin moisture content, by analyzing nucleic acids prepared from RNA extracted from lipids on the surface of the user's skin.

[0003] Japanese Patent Application Laid-Open No. 2020-74769

[0004] However, in the technology of Patent Document 1, in order to classify a user's skin, it is necessary to prepare nucleic acids through multiple steps, such as converting RNA collected from lipids on the surface of the user's skin into cDNA by reverse transcription, then subjecting the cDNA to multiplex PCR, and purifying the PCR reaction product, which is time-consuming and labor-intensive.

[0005] An object of the present invention is to provide an information processing system, an information processing method, and a program that can classify the properties of a user's skin more easily and quickly.

[0006] An information processing system according to one aspect of the present invention includes a control unit. The control unit generates a learning model for estimating a user's skin classification from at least one of skeletal information indicating skeletal characteristics of the user or skin color information indicating skin color characteristics of the user, which is acquired by analyzing a facial image of the user, using skin classification information that classifies the user's skin, generated by analyzing biometric information of the user provided by the user, as training data. The control unit then accepts an input of a user's facial image, analyzes the accepted facial image to acquire at least one of the skeletal information or the skin color information, and uses the learning model to output skin classification information estimated from at least one of the acquired skeletal information or the skin color information.

[0007] An information processing method according to another aspect of the present invention includes: generating a learning model that estimates a user's skin classification from at least one of skeletal information indicating skeletal features of the user or skin color information indicating skin color features of the user, obtained by analyzing a facial image of the user, using skin classification information that classifies the user's skin, generated by analyzing biometric information of the user provided by the user, as training data; accepting input of a facial image of the user; analyzing the accepted facial image to obtain at least one of the skeletal information or the skin color information; and outputting skin classification information estimated from at least one of the obtained skeletal information or skin color information using the learning model.

[0008] A program according to still another aspect of the present invention causes an information processing device to execute the following steps: generating a learning model that estimates a user's skin classification from at least one of skeletal information that indicates the user's skeletal characteristics or skin color information that indicates the user's skin color characteristics, obtained by analyzing a facial image of the user, using skin classification information that classifies the user's skin, generated by analyzing biometric information of the user provided by the user, as training data; accepting input of a user's facial image; analyzing the accepted facial image to obtain at least one of the skeletal information or the skin color information; and outputting skin classification information estimated from at least one of the obtained skeletal information or skin color information, using the learning model.

[0009] According to an information processing system according to an embodiment of the present invention, it is possible to classify the user's skin characteristics more easily and quickly.

[0010] 1 is a diagram showing the configuration of a cosmetics information providing system according to one embodiment of the present invention. 2 is a diagram showing the hardware configuration of a cosmetics information providing server according to one embodiment of the present invention. 3 is a diagram showing the configuration of a database held by a cosmetics information providing server according to one embodiment of the present invention. 4 is a flowchart showing the flow of processing for generating a learning model for classifying a user's skin by a cosmetics information providing server according to one embodiment of the present invention. 5 is a diagram showing bone structure information used in generating the learning model by a cosmetics information providing server according to one embodiment of the present invention. 6 is a diagram showing skin color information used in generating the learning model by a cosmetics information providing server according to one embodiment of the present invention. 7 is a flowchart showing the flow of processing for providing user's skin classification information and cosmetics information by a cosmetics information providing server according to one embodiment of the present invention. 8 is a diagram showing an example of display of cosmetics information provided by a cosmetics information providing server according to one embodiment of the present invention. 9 is a diagram showing skin color information and eye color information used in generating the learning model by a cosmetics information providing server according to one embodiment of the present invention.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [System Configuration] As shown in FIG. 1, this system includes a cosmetics information providing server 100 on the Internet 50 and a plurality of user terminals 200 .

[0013] Cosmetics information providing server 100 is a web server (information processing device) operated by the operator of a portal site (cosmetics information providing site) where information on cosmetics (including topical preparations such as quasi-drugs) can be searched. Cosmetics information providing server 100 is connected to multiple user terminals 200 via the Internet 50.

[0014] Cosmetics information providing server 100 provides a cosmetics information providing service on the portal site to users of user terminals 200. Specifically, cosmetics information providing server 100 searches for cosmetics information that matches the search criteria based on a search request from user terminal 200, generates a web page listing the search results, and transmits it to user terminal 200. The cosmetics information provided may include information such as the name, characteristics, and usage of each cosmetic product, as well as evaluation information (evaluation value, word of mouth, reviews) of each cosmetic product by multiple users.

[0015] In addition to the above-mentioned cosmetic information, the cosmetics information providing server 100 also provides, in response to a user's request, skin classification information that classifies the skin type of each user based on a facial image of the user. To provide this skin classification information, the cosmetics information providing server 100 generates and stores a learning model that estimates the skin classification from feature information acquired by analyzing the user's facial image. Specifically, the learning model uses each user's known skin classification information, generated by analyzing the user's biometric information (RNA) provided by the multiple users, as training data, and estimates the user's skin classification from skeletal information indicating the user's skeletal features and skin color information indicating the user's skin color features, acquired from the user's facial image. The process of generating this learning model will be described later.

[0016] User terminal 200 (200A, 200B, 200C, etc.) is a terminal used by a user, such as a smartphone, mobile phone, tablet PC (Personal Computer), notebook PC, or desktop PC. User terminal 200 accesses cosmetics information providing server 100, receives a web page containing the cosmetics information, etc., and displays it on a screen using a browser or the like. An application compatible with the cosmetics information providing service may also be installed on user terminal 200, and user terminal 200 may access cosmetics information providing server 100 using the application to display the cosmetics information.

[0017] [Hardware configuration of the cosmetics information providing server] As shown in FIG. 2, the cosmetics information providing server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface 15, and a bus 14 connecting these to each other.

[0018] The CPU 11 accesses the RAM 13 and other memory as needed, and performs various arithmetic processing while providing overall control over each block of the cosmetics information providing server 100. Multiple CPUs 11 may be provided depending on the processing. The ROM 12 is a non-volatile memory that permanently stores firmware such as the OS, programs, and various parameters to be executed by the CPU 11. The RAM 13 is used as a working area for the CPU 11, and temporarily stores the OS, various applications currently being executed, and various data currently being processed.

[0019] The input / output interface 15 is connected to a display unit 16, an operation reception unit 17, a storage unit 18, a communication unit 19, and the like.

[0020] The display unit 16 is a display device that uses, for example, an LCD (Liquid Crystal Display), an OELD (Organic ElectroLuminescence Display), a CRT (Cathode Ray Tube), or the like.

[0021] The operation reception unit 17 is, for example, a pointing device such as a mouse, a keyboard, a touch panel, or other input device. When the operation reception unit 17 is a touch panel, the touch panel can be integrated with the display unit 16.

[0022] The storage unit 18 is a non-volatile memory such as a hard disk drive (HDD), a flash memory (SSD; solid state drive), or other solid-state memory. The storage unit 18 stores the OS, various applications, and various data.

[0023] As will be described later, particularly in this embodiment, the storage unit 18 has a user information database, a cosmetics information database, and a skin classification information database in addition to programs such as applications required for the process of generating a learning model for skin classification processing (described later) and the skin classification processing using the learning model. The above programs can also be realized as a program product.

[0024] The communication unit 19 is, for example, a NIC (Network Interface Card) for Ethernet or various modules for wireless communication such as a wireless LAN, and is responsible for communication processing with the user terminal 200 .

[0025] Although not shown, the basic hardware configuration of the user terminal 200 is also substantially the same as the hardware configuration of the cosmetics information providing server 100 described above.

[0026] [Database configuration of the cosmetics information server]

[0027] 3, the cosmetics information providing server 100 has a user information database 31, a cosmetics information database 32, and a skin classification information database 33 in the storage unit 18. Note that each of these databases may be stored in a storage device or server externally connected to the cosmetics information providing server 100, rather than in the storage unit 18.

[0028] The user information database 31 stores attribute information for each user of the cosmetics information provision service provided by the cosmetics information provision server 100. User attribute information includes general information such as the user's name, a user ID for identifying the user, age (generation), occupation, address (area of ​​residence), gender, and email address, as well as information about the user's skin, such as the user's perceived or understood skin type, cosmetics preferences, and skin problems the user may have. The user ID functions as account information for using this cosmetics information provision service. It is preferable that the user's attribute information be selectable from multiple options. For example, skin type may be selected from multiple perspectives, such as classification of dry skin or oily skin, sensitivity level, skin color, etc.

[0029] The cosmetics information database 32 stores information about cosmetics, such as the manufacturer name, product name (brand name), price, cosmetic category (e.g., serum, lotion, cleanser, facial cleanser, makeup base, foundation, pack, cream, etc.), and expected effects (e.g., brightening, pore care, acne care, exfoliating care, skin firmness, sensitive skin care, moisture, anti-aging, shine, natural makeup, waterproof, color retention, color development, etc.).

[0030] The cosmetics information database 32 also stores user evaluation information for each cosmetic product (multi-level evaluation value information and text information describing word-of-mouth reviews). The evaluation information also includes data on the time (date or season) and location (region or area from which the user terminal 200 accessed) when the user provided the evaluation information. The evaluation information is stored in association with the user ID of the user who input the evaluation information and the skin classification information of the user.

[0031] The skin classification information database 33 stores each user's skin classification information generated using the learning model in association with the user ID. The skin classification information database 33 also stores each user's known skin classification information generated by analyzing each user's biometric information, which was acquired as training data for generating the learning model, as well as facial image data of each user corresponding to the known skin classification information used to generate the learning model, and information such as skeletal information, skin color information, and other parameters acquired by analyzing the facial image data. The facial image data may include VISIA images captured by a VISIA (registered trademark) skin image analysis device or smartphone images captured by the user using a user terminal 200 (smartphone). The image is captured, for example, from the front of the face, although images captured from other angles may also be used.

[0032] These databases are mutually referenced as necessary in the learning model generation process by the cosmetics information providing server 100 and the skin classification information providing process using the learning model, which will be described later.

[0033] [Skin Classification Information Derived from Biometric Information] Here, an example of a method for generating skin classification information by analyzing a user's biometric information will be described.

[0034] The business that generates the skin classification information sends a biological sampling kit to each user, and the user returns the biological sample collected using the kit to the administrator. The biological information is, for example, RNA information, and the biological sampling kit is a tool for collecting each user's biological sample, such as sebum, stratum corneum, saliva, urine, or blood. Sebum is preferred as the biological sample from the viewpoints of convenience in collecting RNA information and ability to identify skin surface regions.

[0035] Next, the business operator measures RNA information from the biological samples collected from users. Specifically, the RNA information is data obtained by extracting and preparing RNA from each biological sample, generating cDNA by reverse transcription, and then measuring the expression level of each RNA species by sequencing, PCR, or the like. In this case, the measurement and analysis of the RNA information from the biological samples can be performed using known devices, means, and algorithms.

[0036] Next, the business operator performs cluster analysis on the RNA expression level data. Specifically, two or more skin property items are selected from a plurality of skin property items (e.g., stratum corneum moisture content, transepidermal water loss (TEWL), sebum content, melanin content, erythema content, overall redness, skin elasticity, etc.) for evaluating the physical properties of the user's skin.

[0037] Then, for the genes related to the selected skin physical property items of each sample, similarity is determined based on their expression levels (expression patterns). A clustering method may be used to determine similarity, and either hierarchical clustering or non-hierarchical clustering may be used as the clustering method. Alternatively, other evaluation models such as machine learning models may be used.

[0038] Next, a predetermined number of skin classifications are generated according to the number of clusters obtained by the clustering. In this embodiment, for example, two skin classifications (e.g., keratinized type and immune type) are generated. Alternatively, three types of skin classifications may be generated, including an intermediate type between the two types.

[0039] Furthermore, as a result of the analysis, the following skin characteristics were found for the keratinized type and immune type. (Keratinized type) - High melanin - Firm - Elastic skin, sagging at the corners of the mouth is not noticeable (Immune type) - Low melanin, bright - No firmness - No elastic skin, sagging at the corners of the mouth is noticeable

[0040] In addition, those with the immune type were also found to have lifestyle characteristics such as irregular meal times, instability before and after menstruation, and a tendency to accumulate mental fatigue (stress).

[0041] Furthermore, multiple types of skin classifications may be generated by generating multiple different combinations of multiple skin property items. For example, RNA (genes) may be selected based on skin property items tailored to various skin concerns, such as age spots and wrinkles. Furthermore, clustering may be performed by combining proteome information and information on superficial skin bacteria in addition to RNA information.

[0042] In addition to skin classification based on specific skin properties, the clustering can also be based on RNA information with a gene count of 1,000 to 20,000. For example, by performing clustering based on the expression levels of genes related to a certain function (function X) and genes related to another function (function Y), four skin classifications can be generated: Class I (function X: medium expression level; function Y: medium expression level), Class II (function X: medium expression level; function Y: low expression level), Class III (function X: high expression level; function Y: high expression level), and Class IV (function X: low expression level; function Y: medium expression level). In other words, if users' RNA profiles are similar, it can be estimated that they have similar skin concerns or that the effects of cosmetics are similar. Therefore, it is also possible to generate skin classifications based solely on the similarity of genetic information (RNA expression level) without associating them with specific skin properties.

[0043] [Operation of Cosmetics Information Providing Server] Next, a description will be given of the operation of cosmetics information providing server 100 configured as described above. This operation is performed by the cooperation of hardware such as CPU 11 and communication unit 19 of cosmetics information providing server 100 and software stored in storage unit 18. For convenience, in the following description, CPU 11 is the subject of operation.

[0044] (Learning Model Generation Process) First, a process of generating a learning model for generating skin classification information of a user by the cosmetics information providing server 100 will be described.

[0045] As shown in FIG. 4, the CPU 11 of the cosmetics information providing server 100 acquires a facial image of the user from the skin classification information database 33 (step 41).

[0046] When VISIA images are used as face images for training, the number of images is increased by rotating, flipping, etc. the original VISIA images. At this time, the number of images is adjusted so that the number of training images for each skin classification is approximately equal.

[0047] Furthermore, when smartphone images are used as face images for learning, since the captured images may contain parts other than the face, preprocessing may be performed to extract the face part from each face image, and further preprocessing may be performed to cut out only images of specific feature areas from the face image. After these preprocessing steps, the image padding process is performed in the same way as for the VISIA images.

[0048] Next, the CPU 11 analyzes each facial image and acquires skeletal information and skin color information as feature quantities (step 42). The skeletal information is information that indicates the skeletal (shape) characteristics of the user's face, and the skin color information is information that indicates the color characteristics of the user's skin.

[0049] The reason why skeletal information is used here is that, as mentioned above, among the skin classifications, the keratinized type is characterized by firmness and elasticity and less noticeable sagging at the corners of the mouth, while the immune type is characterized by a lack of firmness and elasticity and more noticeable sagging at the corners of the mouth, and therefore it is thought that it may be possible to classify the keratinized type and the immune type by analyzing the skeleton (facial shape) from a facial image.

[0050] The reason why skin color information is used is that, as mentioned above, the keratinized type of skin classification has characteristics related to skin color, such as having a lot of melanin, while the immune type has characteristics related to skin color, such as having less melanin and being lighter, and it is assumed that analyzing skin color from facial images may make it possible to classify the keratinized type and the immune type.

[0051] Specifically, as the skeleton information, for example, as shown in FIG. 5, length (ratio) information of each part of the face for a total of 21 items is acquired from the face image.

[0052] That is, in the face image, the angles of the left and right eyes (2 and 1 in the same figure) and the angles of the left and right eyebrows (4 and 3 in the same figure), the ratio of the vertical width to the horizontal width of the face (5 in the same figure), the ratio of the horizontal width of the left and right eyes to the horizontal width of the face (9 and 7 in the same figure), the ratio of the distance between the left and right eyes (8 in the same figure), the right face line width (the horizontal distance between the right temple and the corner of the right eye) and the left face line width (the horizontal distance between the left temple and the corner of the left eye) to the horizontal width of the face. The acquired information includes the ratios of the vertical width of the forehead to the vertical width of the face (11 in the same figure), the ratios of the vertical widths of the left and right eyes to the vertical width of the face (13 in the same figure) and the ratios of the vertical widths of the left and right eyelids to the vertical width of the face (12 in the same figure), the ratio of the vertical width of the nose to the vertical width of the face (14 in the same figure), and the ratios of the vertical width of the upper lip, the vertical width of the lower lip, the vertical width of the white lip, and the vertical width of the chin to the vertical width of the face (16, 17, 15, 18 in the same figure).

[0053] The acquired skeletal information is not limited to that shown in the figure, and some, but not all, of the above 21 items may be acquired.

[0054] Furthermore, as skin color information, for example, as shown in Figure 6, the average values ​​(mean) of L*, a*, and b* in the Lab color space (skin color) and the SE (standard error) values ​​(color unevenness) of L*, a*, and b* are calculated for each of the forehead area, left and right cheek areas, and left and right corners of the mouth area in the facial image.

[0055] In addition to or instead of these, the median and standard deviation (SD) of L*, a*, and b* for each region may be calculated. Color values ​​in a color system other than the Lab color space may also be used. The target regions are not limited to the above five regions and may be other regions. Furthermore, the above values ​​may be calculated for some regions rather than all five regions.

[0056] Next, the CPU 11 obtains the user's known skin classification information (e.g., keratinization type, immune type) associated with the facial image from which the skeletal information and skin color information were obtained from the skin classification information database 33 (step 43).

[0057] Then, the CPU 11 uses the known skin classification information as training data to generate a learning model that estimates the skin classification from the acquired bone structure information and skin color information (step 44).

[0058] As a learning model, for example, a convolutional neural network (CNN), more specifically, VGG16 (a CNN consisting of 16 layers), is used, but this is not limited to this, and conventional machine learning models such as random forest, support vector machine, and logistic regression may also be used.

[0059] The CPU 11 repeatedly executes the learning process, including adjustment of the skeletal information, skin color information and other parameters, until the estimation accuracy of the skin classification by the learning model reaches a predetermined value or higher.

[0060] (Process for Providing Skin Classification Information and Cosmetic Information) Next, a description will be given of the process for providing skin classification information and cosmetic information to the user terminal 200. Fig. 7 is a flowchart showing the flow of the process for providing skin classification information and cosmetic information.

[0061] As shown in the figure, first, the CPU 11 determines whether or not a skin classification request has been received together with a facial image from the user terminal 200 (step 71). The skin classification request can be transmitted, for example, via a skin classification request transmission page provided on the cosmetics information providing website.

[0062] If it is determined that the skin classification request has been received (Yes in step 71), the CPU 11 analyzes the received face image and obtains the bone structure information and skin color information (step 72).

[0063] Next, the CPU 11 inputs the bone structure information and skin color information obtained from the face image into the learning model (step 73).

[0064] Next, the CPU 11 determines whether or not skin classification information has been output from the learning model (step 74). If it determines that skin classification information has not been output (No in step 74), the CPU 11 repeats the check until skin classification information has been output.

[0065] If it is determined that skin classification information has been output from the learning model (Yes in step 74), the CPU 11 stores the skin classification information in the skin classification information database 33 in association with the user ID of the requesting user, and transmits the skin classification information to the requesting user terminal 200 (step 75).

[0066] Next, the CPU 11 determines whether or not a search request for cosmetic information has been received from the user terminal 200 to which the skin classification information was sent (step 76).

[0067] If it is determined that a search request for cosmetics information has been received (Yes in step 76), the CPU 11 sends and displays a cosmetics information provision page to the user terminal that lists the search results for cosmetics information corresponding to the skin classification associated with the user ID of the user who made the search request (step 77).

[0068] FIG. 8 is a diagram showing an example of the display of the cosmetics information provision page.

[0069] As shown in the figure, the cosmetics information page displays, for example, search results for a beauty serum, along with information introducing the beauty serum (product name, photo, description, price, etc.), as well as product evaluation information (e.g., a 7-point scale) from users with the same skin classification as the user who made the search request (e.g., skin type B), and product evaluation information from users with other skin classifications (e.g., skin types A and C). This allows users to easily find highly rated products that suit their own skin classification.

[0070] In addition, evaluation information for skin classifications other than the skin classification of the user who made the search request (types A and C in the same figure) may not be posted, and only evaluation information for the skin classification of the user who made the search request (type B in the same figure) may be extracted and posted.

[0071] As described above, according to this embodiment, the cosmetics information providing server 100 can more easily and quickly classify the user's skin characteristics and provide corresponding cosmetics information simply by acquiring a facial image of the user, without having to take the trouble of collecting and analyzing biometric information (RNA) from the user to generate skin classification information.

[0072] [Modifications] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the gist of the present invention.

[0073] In the above-described embodiment, the cosmetics information providing server 100 generates a learning model based on skeletal information and skin color information acquired from a facial image, but a learning model that estimates skin classification using only one of skeletal information and skin color information may also be generated. Furthermore, a learning model that estimates skin classification may also be generated by taking into account skin surface information (information indicating the state of desquamation, scales, wrinkles, texture, pores, etc.) as a parameter in addition to or instead of skeletal information and skin color information.

[0074] In the above-described embodiment, the cosmetics information providing server 100 may train the learning model to estimate skin classification using eye color information in addition to skeletal information and / or skin color information acquired from a facial image. In this case, some of the evaluation items shown in FIGS. 5 and 6 may not be calculated as skeletal information and / or skin color information.

[0075] The cosmetics information providing server 100 then analyzes the received facial image to obtain at least one of skeletal information and / or skin color information, and eye color information, and outputs skin classification information estimated from at least one of the obtained skeletal information or skin color information and eye color information using a learning model.

[0076] For example, as shown in FIG. 9( a), the cosmetics information providing server 100 calculates, as skin color information, the average values ​​(mean) of R, G, and B in RGB, the average values ​​(mean) of L*, a*, and b* in Lab color space, and the SE (standard error) values ​​(color unevenness) of R, G, B, and L*, a*, and b* for the forehead region, left and right cheek regions, and left and right corners of the mouth region in the face image. Furthermore, as information indicating eye color (eye color information), it calculates the average values ​​and SE values ​​of R, G, B, and L*, a*, and b* for the iris (black part of the eye) regions of the left and right eyes and the white region surrounding them.

[0077] In this case, specifically, as shown in the same figure (a), the width of the nose, the width of the left and right cheeks (the distances from the left and right ends of the nose to the left and right ends of the face), the angle and width of the left and right eyes may be selectively calculated as skeletal information.

[0078] As shown in Figure 1(b), skin color information may be selectively calculated as the average values ​​of R, G, B, and L* in the left corner of the mouth area, G, B, L*, and b* in the forehead area, and R, L*, and b* in the right cheek area, as well as the SE values ​​of b* in the left corner of the mouth area and B and a* in the left cheek area.

[0079] Furthermore, as shown in FIG. 1C, the eye color information may be selectively calculated from the average values ​​of b* in the iris region of the left eye, R, B, and b* in the white region of the left eye, G in the iris region of the right eye, and b* in the white region of the right eye, as well as the SE values ​​of B and b* in the iris region of the left eye, B in the white region of the left eye, and G, B, and b* in the white region of the right eye.

[0080] Of course, the evaluation items to be calculated are not limited to these, and other areas may be calculated as evaluation items. For example, as eye color information, only the color of one of the eyes may be used, or only the color of the iris area or only the color of the white area of ​​both eyes or one eye may be used. Furthermore, even in this modified example using eye color information, in addition to or instead of the above, the median or standard deviation (SD) of R, G, B or L*, a*, b* of each area may be calculated, or color values ​​in a color system other than the RGB and Lab color spaces may be used.

[0081] In this way, by supplementarily using eye color information (color, unevenness of color) in addition to the bone structure information and / or skin color information, it is possible to improve the accuracy of estimating skin classification.

[0082] As a result of analyzing the user's skin images used to obtain skin color information and skeletal information in the above-mentioned embodiment, the inventors found that the higher the prediction confidence (a statistical measure of how certain the prediction or output is) when classifying skin using the above-mentioned learning model, the higher the classification accuracy (correct rate).

[0083] In particular, when the average values ​​of image features of images with high and low confidence levels were compared, it was found that images with high confidence levels (e.g., 70% or higher) had higher RGB B values ​​in the forehead, mouth, and eye areas compared to images with low confidence levels (e.g., 50-53%), and that images with low confidence levels had higher yellowness (Lab b* values) in the forehead, cheeks, and eye areas compared to images with high confidence levels.

[0084] Therefore, the cosmetics information providing server 100 may receive multiple facial images from a user, and from among the multiple facial images, extract facial images with a high average value of the B value of the RGB values ​​of each pixel of the facial images for analysis using a learning model, or may exclude facial images with a high average value of the b value of the Lab values ​​of each pixel from the above analysis.

[0085] Specifically, the cosmetics information providing server 100 may extract the face image with the highest average value of the B value of the RGB values ​​from multiple face images, or may accept face images from the user again until a face image with an average value of the B value of the RGB values ​​equal to or greater than a predetermined threshold is acquired. In this case, if the average value of the B value of the RGB values ​​of the acquired face image is less than the threshold, the server 100 may send and output a message to the user terminal 200 urging the user to take another face image.

[0086] Furthermore, if the average value of the b value of the Lab values ​​of the acquired facial image is equal to or greater than a predetermined threshold, the cosmetics information providing server 100 may accept another facial image from the user in the same manner as described above until a facial image in which the b value is less than the predetermined threshold can be acquired.

[0087] The cosmetics information providing server 100 may calculate the average value of the B value of the RGB values ​​by limiting the calculation to at least one of the forehead, mouth, and eye regions, or may calculate the average value for the entire face image regardless of these regions. Similarly, the calculation of the average value of the b value of the Lab values ​​may be limited to the forehead, cheeks, and eye regions, or may be calculated for the entire face image regardless of these regions.

[0088] Furthermore, as a result of analyzing the skin images, the inventors have found that images with a high degree of certainty show the unevenness of the skin more clearly and have more fine shading such as blemishes.

[0089] Therefore, the cosmetics information providing server 100 may receive multiple facial images from a user and extract facial images from the multiple facial images in which at least one of blemishes, pores, wrinkles, peeling, scales, or skin irregularities can be recognized for analysis using a learning model.

[0090] Specifically, when the cosmetics information providing server 100 receives a facial image from a user, it performs image recognition processing to determine whether the above-mentioned blemishes, pores, wrinkles, peeling, scales, or skin irregularities can be recognized, and if they cannot be recognized, it may send and output message information to the user terminal 200 such as "Please take another facial image as the resolution is low" or "Please take another facial image in a well-lit area" to encourage the user to take another photograph until a recognizable facial image is obtained.

[0091] In addition, instead of detecting blemishes, pores, etc., the cosmetics information providing server 100 may extract facial images for analysis in which edges have been detected using a predetermined edge detection filter that has a sufficiently high resolution and a high degree of certainty and is generated based on the facial image.

[0092] In the above-described embodiment, an example was shown in which a user who transmitted a facial image to the cosmetics information providing server 100 and received notification of their own skin classification information accessed the cosmetics search page to search for cosmetics information corresponding to that skin classification. However, it is conceivable that there may be users who access the search page without knowing the information about their skin classification. Therefore, when a user who has not been notified of skin classification information (a non-logged-in user or a logged-in user whose user ID has not been assigned a skin classification) accesses the search page, the cosmetics information providing server 100 may display, for example, suggestion information on the search page, suggesting the provision of skin classification information based on a facial image (submission of a facial image for skin classification), using a pop-up or the like.

[0093] In the above-described embodiment, RNA information was used as biological information, but other information may also be analyzed, such as proteomes (proteins) collected from the user's stratum corneum, sebum, etc., or epidermal bacteria collected from the skin surface.

[0094] In the above-described embodiment, cosmetics (including external preparations such as quasi-drugs) are used as an example of a product. However, the product is not limited to cosmetics, and information on any product that the user uses (or ingests) on their body (skin, hair, etc.) can be provided according to the skin classification information generated based on the user's facial image, such as hair care products such as shampoo, conditioner, and treatment, medicines (skin preparations, etc.), sweat wipes, cosmetic sheets, toothpaste, beverages, food, facial massagers, beauty tools, etc.

[0095] In the above embodiment, only one cosmetics information providing server 100 is shown, but the processes executed by the cosmetics information providing server 100 may be distributed and executed by multiple servers. For example, the process of generating the learning model shown in Fig. 4 and the process of providing skin classification information and cosmetics information using the learning model shown in Fig. 7 may be executed by separate servers.

[0096] In addition, in the above-described embodiment, the web page created by the cosmetics information providing server 100 is displayed on the browser of the user terminal 200, but if an application program corresponding to the cosmetics information providing service is installed on the user terminal 200, the web page may be displayed by the application program.

[0097] In the above-described embodiment, the present invention has been described as a cosmetics information providing system that outputs cosmetics information corresponding to skin classification information estimated by analyzing a facial image using a learning model to a user terminal 200. However, the present invention may also be realized as an information processing system that simply outputs estimated skin classification information without providing cosmetics information corresponding to the skin classification information.

[0098] In the above-described embodiment, the cosmetics information providing server 100 may acquire the skin color information using an estimation model that has been trained to output the skin condition of a human face appearing in a facial image from the facial image.

[0099] In other words, in order to conform to the input rules of the estimation model, the cosmetics information providing server 100 may convert the acquired user's facial image so that multiple points on the contour of the facial area of ​​the human face reflected in the acquired user's facial image are positioned on the outer edge of the image, obtain a converted facial image, input the converted facial image into the estimation model, and obtain skin condition information of the user's face from the estimation model as the skin color information.

[0100] Here, the cosmetics information providing server 100 may remove or reduce, in the converted face image, a predetermined local part area within the face region of the human face captured in the acquired face image. That is, in the conversion process, the predetermined local part area within the face region may be removed or reduced by aligning the positions of perimeter points that face each other across the predetermined local part area among a plurality of perimeter points that surround the predetermined local part area within the face region. The predetermined local part area is, for example, an eye area, an eyebrow area, or a lip area.

[0101] Furthermore, the cosmetics information providing server 100 may further execute a position specifying process for specifying the position coordinates of a plurality of predetermined locations in the facial region of the human face reflected in the acquired facial image. In this case, the conversion process may convert some or all of the specified position coordinates of the plurality of predetermined locations to convert the acquired facial image so that it conforms to the input rule of the estimation model.

[0102] In this case, the facial image may have a predetermined rectangular shape, and the position coordinates of the predetermined plurality of locations may further include position coordinates of a plurality of points on the contour of the facial region, the tip of the nose in the facial region, and a plurality of other points in the facial region. The cosmetics information providing server 100 may perform the coordinate transformation in the transformation process so that four first contour points included in the plurality of points on the contour of the facial region are positioned at the vertices of the predetermined rectangular shape, the tip of the nose is positioned at the center of the predetermined rectangular shape, the other points on the contour of the facial region are positioned on the sides of the predetermined rectangular shape, and the other points in the facial region move in response to movements of the plurality of points on the contour of the facial region and the tip of the nose.

[0103] Here, the other multiple points in the face region may include four third contour points that are located closer to the contour of the face region than the predetermined local site region and four second contour points that are located even closer to the contour of the face region than the four third contour points. The cosmetics information providing server 100 may perform the coordinate transformation in the transformation process so that the four second contour points and the four third contour points are each located on a diagonal line connecting the four first contour points and the nose tip, and so that the difference between the distance of the first contour point from the nose tip and the distance of the second contour point from the nose tip is smaller than the difference between the distance of the second contour point from the nose tip and the distance of the third contour point from the nose tip.

[0104] Among the inventions described in the claims of this application, the invention described as an "information processing method" is one in which each step is automatically performed by at least one device such as a computer through information processing by software, and is not performed by a human using a device such as a computer. In other words, the "information processing method" is an information processing method using computer software, and is not a method in which a human operates a calculation tool called a computer.

Claims

1. An information processing system comprising a control unit that generates a learning model for estimating a user's skin classification from at least one of skeletal information indicating the user's skeletal characteristics or skin color information indicating the user's skin color characteristics, obtained by analyzing a facial image of the user, using skin classification information that classifies the user's skin, generated by analyzing biometric information of the user provided by the user, as teacher data; accepts input of a user's facial image; analyzes the accepted facial image to obtain at least one of the skeletal information or the skin color information; and outputs skin classification information estimated from at least one of the acquired skeletal information or skin color information using the learning model.

2. The information processing system according to claim 1, wherein the control unit analyzes the facial image to obtain the skeletal information, and outputs the skin classification information estimated from the skeletal information using the learning model.

3. The information processing system according to claim 1, wherein the control unit generates the learning model using the skin classification information generated by analysis of the user's RNA as the training data.

4. The information processing system of claim 1, wherein the control unit generates the learning model from the skeletal information and the skin color information, analyzes the received facial image to obtain the skeletal information and the skin color information, and uses the learning model to output skin classification information estimated from the obtained skeletal information and skin color information.

5. The information processing system according to claim 1 or 2, wherein the control unit uses the ratio of the vertical width to the horizontal width of the face in the face image as the skeletal information.

6. An information processing system according to claim 1 or 2, wherein the control unit uses, as the skeletal information, the ratio of the width of the left eye and the right eye to the width of the face in the facial image, or the ratio of the distance between the left eye and the right eye.

7. An information processing system according to claim 1 or 2, wherein the control unit uses, as the skeletal information, the ratios of the vertical widths of the left and right eyes or the vertical widths of the left and right eyelids to the vertical width of the face in the facial image.

8. An information processing system according to claim 1 or 2, wherein the control unit uses the angles of the left and right eyes and the angles of the left and right eyebrows in the face image as the skeletal information.

9. An information processing system according to claim 1 or 2, wherein the control unit uses, as the skeletal information, the ratios of the vertical width of the upper lip, the vertical width of the lower lip, and the vertical width of the white lip to the vertical width of the face in the face image.

10. An information processing system according to any one of claims 1 to 7, wherein the control unit uses the ratio of the vertical width of the nose or the vertical width of the forehead to the vertical width of the face in the face image as the skeletal information.

11. An information processing system as described in claim 1 or 2, wherein the control unit uses, as the skeletal information, the ratios of the horizontal distance between the right temple and the right eye corner and the horizontal distance between the left temple and the left eye corner to the width of the face in the facial image.

12. The information processing system according to claim 1, wherein the control unit uses at least one value selected from the average, median, standard deviation, and standard error of color values ​​in at least one area in the facial image as the skin color information.

13. The information processing system according to claim 10, wherein the control unit uses at least one value in at least two of the forehead region, the left and right cheek regions, and the left and right corners of the mouth regions in the facial image as the skin color information.

14. The information processing system according to claim 1 or 2, wherein the control unit receives input of a facial image captured by the user terminal of the user from the user terminal, and outputs information regarding cosmetics corresponding to the skin classification information estimated by analysis of the facial image to the user terminal.

15. The information processing system of claim 1, wherein the control unit: trains the learning model to estimate the skin classification based on at least one of the skeletal information or the skin color information obtained by analysis of a user's facial image and eye color information indicating the user's eye color characteristics; analyzes the received facial image to obtain at least one of the skeletal information or the skin color information and the eye color information; and uses the learning model to output skin classification information estimated from the acquired at least one of the skeletal information or the skin color information and the eye color information.

16. The information processing system according to claim 1 or 2, wherein the control unit receives a plurality of facial images from the user, and extracts from the plurality of facial images those facial images having a high average B value in the RGB values ​​of each pixel of the facial images for the analysis, or excludes from the analysis those facial images having a high average b value in the Lab values ​​of each pixel.

17. The information processing system according to claim 1 or 2, wherein the control unit receives a plurality of facial images from the user, and extracts, from the plurality of facial images, facial images in which blemishes, pores, wrinkles, desquamation, scales or skin irregularities are recognized for the analysis.

18. The information processing system of claim 1, wherein the control unit converts the received facial image so that multiple points on the contour of the facial area of ​​the human face reflected in the facial image are positioned on the outer edge of the image to obtain a converted facial image, inputs the converted facial image to an estimation model trained to output the skin condition of the human face reflected in the facial image from the facial image, and obtains skin condition information of the user's face from the estimation model as the skin color information.

19. An information processing method comprising the steps of: generating a learning model that estimates a user's skin classification from at least one of skeletal information indicating the user's skeletal features or skin color information indicating the user's skin color features obtained by analyzing a facial image of the user, using skin classification information that classifies the user's skin, generated by analyzing biometric information of the user provided by the user, as training data; accepting input of a user's facial image; analyzing the accepted facial image to obtain at least one of the skeletal information or the skin color information; and outputting skin classification information estimated from at least one of the acquired skeletal information or skin color information using the learning model.

20. A program that causes an information processing device to execute the following steps: generating a learning model that estimates the skin classification of a user from at least one of skeletal information indicating the skeletal features of the user or skin color information indicating the skin color features of the user, obtained by analyzing a facial image of the user, using skin classification information that classifies the skin of the user, generated by analyzing biometric information of the user provided by a plurality of users, as teacher data; accepting input of a facial image of the user; analyzing the accepted facial image to obtain at least one of the skeletal information or the skin color information; and outputting skin classification information estimated from at least one of the acquired skeletal information or skin color information, using the learning model.

21. A storage medium storing a program to cause an information processing device to execute the following steps: generating a learning model that estimates the skin classification of a user from at least one of skeletal information indicating the skeletal features of the user or skin color information indicating the skin color features of the user, obtained by analyzing a facial image of the user, using skin classification information that classifies the skin of the user, generated by analyzing biometric information of the user provided by a plurality of users, as teacher data; accepting input of a facial image of the user; analyzing the accepted facial image to obtain at least one of the skeletal information or the skin color information; and outputting skin classification information estimated from at least one of the acquired skeletal information or skin color information, using the learning model.

Images