Information processing system

Abstract

To easily and quickly categorize a user skin type.SOLUTION: An information processing system has a control unit. By using skin category information generated by the analysis of biological information of a plurality of users provided from the plurality of users as teacher data to classify a skin of the user, the control unit generates a learning model for inferring a skin category of the user from at least one of skeleton information indicating a feature of a skeleton of the user and skin color information indicating a feature of a skin color of the user acquired by the analysis of a face image of the user. Then, the control unit receives an input of the face image of the user, acquires at least one of the skeleton information and the skin color information by analyzing the received face image, and outputs skin category information inferred from at least one of the acquired skeleton information and the acquired skin color information with the learning model.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program for estimating a user's skin classification. [Background technology]

[0002] Conventionally, there are technologies for classifying a user's skin by analyzing the user's biological information. For example, the following Patent Document 1 describes a method for classifying a user's skin into a group with a high or low amount of sebum, or a group with a high or low amount of skin moisture, by analyzing nucleic acids prepared from RNA extracted from lipids on the surface of the user's skin. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-74769 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology of Patent Document 1, in order to classify a user's skin, it is necessary to prepare nucleic acids through multiple steps, such as converting RNA collected from the lipids on the surface of the user's skin into cDNA by reverse transcription, then subjecting the cDNA to multiplex PCR, and purifying the PCR reaction product, which is time-consuming and labor-intensive.

[0005] An object of the present invention is to provide an information processing system, an information processing method, and a program that can classify the characteristics of a user's skin more easily and quickly. [Means for solving the problem]

[0006] An information processing system according to an embodiment of the present invention includes a control unit. The control unit generates a learning model for estimating a skin classification of a user from at least one of skeletal information indicating skeletal characteristics of the user or skin color information indicating skin color characteristics of the user, which are acquired by analyzing a face image of the user, using skin classification information for classifying the skin of the user, which is generated by analyzing biometric information of the user provided by the user, as teacher data. The control unit then receives an input of a face image of the user, analyzes the received face image to acquire at least one of the skeletal information or the skin color information, and uses the learning model to output skin classification information estimated from at least one of the acquired skeletal information or skin color information.

[0007] An information processing method according to another aspect of the present invention includes: A learning model is generated to estimate a skin classification of a user from at least one of skeletal information indicating skeletal characteristics of the user and skin color information indicating skin color characteristics of the user, the information being obtained by analyzing a face image of the user, using skin classification information that classifies the skin of the user, which is generated by analyzing biometric information of the user provided by the user, as training data; Accepting input of a user's face image; Analyzing the received face image to obtain at least one of the bone structure information and the skin color information; and outputting skin classification information estimated from at least one of the acquired skeletal information and skin color information using the learning model.

[0008] According to yet another aspect of the present invention, there is provided a program for an information processing device, A step of generating a learning model for estimating a skin classification of a user from at least one of skeletal information indicating skeletal characteristics of the user and skin color information indicating skin color characteristics of the user, which are obtained by analyzing a facial image of the user, using skin classification information that classifies the skin of the user, generated by analyzing biometric information of the user provided by the user, as training data; receiving an input of a face image of a user; analyzing the received face image to obtain at least one of the bone structure information and the skin color information; and outputting skin classification information estimated from at least one of the acquired skeletal information and skin color information using the learning model. Effect of the Invention

[0009] According to an information processing system according to an embodiment of the present invention, it is possible to classify the skin characteristics of a user more easily and quickly. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration of a cosmetics information providing system according to an embodiment of the present invention. [Diagram 2] 1 is a diagram showing a hardware configuration of a cosmetics information providing server according to an embodiment of the present invention. [Diagram 3] 2 is a diagram showing a configuration of a database included in a cosmetics information providing server according to an embodiment of the present invention. FIG. [Figure 4] 11 is a flowchart showing the flow of a process for generating a learning model for classifying a user's skin, performed by a cosmetics information providing server according to one embodiment of the present invention. [Diagram 5] A figure showing skeletal information used to generate the learning model by the cosmetics information providing server according to one embodiment of the present invention. [Figure 6] 13 is a diagram showing skin color information used to generate the learning model by the cosmetics information providing server according to one embodiment of the present invention. FIG. [Figure 7] 11 is a flowchart showing a flow of a process for providing skin classification information and cosmetics information of a user by a cosmetics information providing server according to one embodiment of the present invention. [Figure 8] 1 is a diagram showing an example of display of cosmetics information provided by a cosmetics information providing server according to an embodiment of the present invention. FIG. [Figure 9]A figure showing skin color information and eye color information used in generating the above-mentioned learning model by the cosmetics information providing server of one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [System configuration] As shown in FIG. 1, this system includes a cosmetics information providing server 100 on the Internet 50 and a plurality of user terminals 200.

[0013] Cosmetics information providing server 100 is a web server (information processing device) operated by the operator of a portal site (cosmetics information providing site) where information on cosmetics (including topical preparations such as quasi-drugs) can be searched. Cosmetics information providing server 100 is connected to multiple user terminals 200 via the Internet 50.

[0014] Cosmetics information providing server 100 provides a cosmetics information providing service for users of user terminals 200 on the portal site. Specifically, cosmetics information providing server 100 searches for cosmetics information that matches search conditions based on a search request from user terminal 200, generates a web page listing the search results, and transmits it to user terminal 200. The cosmetics information provided may include information such as the name, characteristics, and usage of each cosmetic product, as well as evaluation information (evaluation value, word of mouth, and reviews) of each cosmetic product by multiple users.

[0015] In addition to the above-mentioned cosmetic information, the cosmetic information providing server 100 provides skin classification information for classifying the skin type based on the facial image of each user in response to a user request. In order to provide the skin classification information, the cosmetic information providing server 100 generates and stores a learning model for estimating the skin classification from feature information acquired by analyzing the facial image of the user. Specifically, the learning model uses known skin classification information of each user generated by analyzing the biometric information (RNA) of the user provided by multiple users as teacher data, and estimates the skin classification of the user from skeletal information indicating the skeletal features of the user and skin color information indicating the skin color features of the user acquired from the facial image of the user. The generation process of the learning model will be described later.

[0016] User terminal 200 (200A, 200B, 200C...) is a terminal used by a user, and may be, for example, a smartphone, a mobile phone, a tablet PC (Personal Computer), a notebook PC, a desktop PC, etc. User terminal 200 accesses cosmetics information providing server 100, receives a web page containing the above-mentioned cosmetics information, etc., and displays it on a screen using a browser or the like. An application corresponding to the above-mentioned cosmetics information providing service may also be installed in user terminal 200, and user terminal 200 may access cosmetics information providing server 100 using the application to display the cosmetics information.

[0017] [Hardware configuration of cosmetics information server] As shown in FIG. 2, cosmetics information providing server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface 15, and a bus 14 connecting these to each other.

[0018] CPU 11 accesses RAM 13 etc. as necessary and performs various arithmetic processing while controlling all the blocks of cosmetics information providing server 100 in an integrated manner. Multiple CPUs 11 may be provided depending on the processing. ROM 12 is a non-volatile memory in which the OS, programs, various parameters, and other firmware to be executed by CPU 11 are fixedly stored. RAM 13 is used as a working area for CPU 11 and temporarily stores the OS, various applications being executed, and various data being processed.

[0019] The input / output interface 15 is connected to a display unit 16, an operation reception unit 17, a storage unit 18, a communication unit 19, and the like.

[0020] The display unit 16 is a display device using, for example, a Liquid Crystal Display (LCD), an Organic ElectroLuminescence Display (OLED), a Cathode Ray Tube (CRT), or the like.

[0021] The operation reception unit 17 is, for example, a pointing device such as a mouse, a keyboard, a touch panel, or other input device. When the operation reception unit 17 is a touch panel, the touch panel can be integrated with the display unit 16.

[0022] The storage unit 18 is, for example, a non-volatile memory such as a hard disk drive (HDD), a flash memory (SSD; Solid State Drive), or other solid-state memory. The storage unit 18 stores the OS, various applications, and various data.

[0023] As will be described later, particularly in this embodiment, the storage unit 18 has a user information database, a cosmetics information database, and a skin classification information database, in addition to programs such as applications required for generating a learning model for skin classification processing described later and for skin classification processing using the learning model. The above programs can also be realized as program products.

[0024] The communication unit 19 is, for example, a NIC (Network Interface Card) for Ethernet or various modules for wireless communication such as wireless LAN, and is responsible for communication processing with the user terminal 200.

[0025] Although not shown, the basic hardware configuration of user terminal 200 is also substantially the same as the hardware configuration of cosmetics information providing server 100 described above.

[0026] [Database configuration of cosmetics information server]

[0027] 3, cosmetics information providing server 100 has, in storage unit 18, a user information database 31, a cosmetics information database 32, and a skin classification information database 33. Note that each of these databases may be stored in a storage device or a server externally connected to cosmetics information providing server 100, rather than in storage unit 18.

[0028] The user information database 31 stores attribute information of each user of the cosmetics information provision service provided by the cosmetics information provision server 100. The user attribute information includes general information such as the name, a user ID for identifying the user, age (generation), occupation, address (area of ​​residence), sex, and email address, as well as information about the user's skin, such as the skin type that the user feels or understands, cosmetics preferences, and skin problems the user has. The user ID functions as account information for using this cosmetics information provision service. It is preferable that the user's attribute information can be selected from multiple options. For example, in the case of skin type, in addition to classification of dry skin and oily skin, it may be selected from multiple perspectives such as the level of sensitive skin and skin color.

[0029] The cosmetics information database 32 stores information about cosmetics, such as the manufacturer name, product name (brand name), price, cosmetic category (e.g., serum, lotion, cleansing agent, facial cleanser, makeup base, foundation, pack, cream, etc.), and expected effects (e.g., brightening, pore care, acne care, exfoliating care, skin firmness, sensitive skin care, moisturizing, anti-aging, shine, natural makeup, waterproof, color retention, color development, etc.).

[0030] The cosmetics information database 32 also stores user evaluation information for each cosmetic product (multi-level evaluation value information and text information describing word-of-mouth reviews). The evaluation information also includes data on the time (date or season) and place (region or area from which the user terminal 200 is accessed) when the user provided the evaluation information. The evaluation information is stored in association with the user ID of the user who input the evaluation information and the skin classification information of the user.

[0031] The skin classification information database 33 stores the skin classification information of each user generated using the learning model in association with the user ID. The skin classification information database 33 also stores the known skin classification information of each user generated by analyzing the biological information of each user acquired as teacher data for generating the learning model, the facial image data of each user corresponding to the known skin classification information used for generating the learning model, and information such as skeletal information, skin color information, and other parameters acquired by analyzing the facial image data. As the facial image data, a VISIA image taken by a skin image analyzer VISIA (registered trademark) or a smartphone image taken by a user using a user terminal 200 (smartphone) is used. The photographing direction is, for example, the front of the face, but in addition to or instead of that, an image taken in another photographing direction may be used.

[0032] These databases are mutually referenced and used as necessary in the learning model generation process by the cosmetics information providing server 100 and the skin classification information providing process using the learning model, which will be described later.

[0033] [Skin classification information derived from biological information] Here, an example of a method for generating skin classification information by analyzing a user's biological information will be described.

[0034] The business generating the skin classification information sends a biological sampling kit to each user, and the user returns the biological sample collected using the kit to the administrator. The biological information is, for example, RNA information, and the biological sampling kit is a tool for collecting each user's biological sample, such as sebum, stratum corneum, saliva, urine, and blood. Sebum is preferred as the biological sample from the viewpoint of convenience in collecting RNA information and the ability to identify the skin surface area.

[0035] Next, the business entity measures RNA information from the biological samples collected from the users. Specifically, the RNA information is data obtained by extracting and preparing RNA from each biological sample, generating cDNA by reverse transcription, and then measuring the expression level of each RNA species by sequencing, PCR, or the like. In this case, the measurement and analysis of the RNA information from the biological samples can be performed using known devices, means, and algorithms.

[0036] Next, the business entity performs cluster analysis on the RNA expression data. Specifically, the business entity selects two or more skin property items from a plurality of skin property items (e.g., stratum corneum moisture content, TEWL (transepidermal water loss), sebum amount, melanin amount, erythema amount, overall redness, skin elasticity, etc.) for evaluating the physical properties of the user's skin.

[0037] Then, the similarity is determined for the genes related to the selected skin physical property items of each sample based on their expression levels (expression patterns). A clustering method may be used to determine the similarity, and either hierarchical clustering or non-hierarchical clustering may be used as the clustering method. In addition, other evaluation models such as machine learning models may be used.

[0038] Next, a predetermined number of skin classifications are generated according to the number of clusters classified by the clustering. In this embodiment, for example, two skin classifications (e.g., keratinized type and immune type) are generated. Alternatively, three types of skin classifications may be generated by adding an intermediate type between the two types.

[0039] As a result of the analysis, the following skin characteristics were found for the keratinized type and immune type. (keratinized type) High in melanin -Firm - Elastic skin and less noticeable sagging at the corners of the mouth (Immune type) Less melanin, brighter No firmness -Skin lacks elasticity and sagging of the corners of the mouth is noticeable

[0040] In addition, those with the immune type were also found to have lifestyle characteristics such as irregular meal times, instability before and after menstruation, and a tendency to accumulate mental fatigue (stress).

[0041] In addition, multiple types of skin classifications may be generated by generating multiple different combinations of multiple skin physical property items. For example, RNA (genes) may be selected based on skin physical property items that correspond to various skin concerns such as age spots and wrinkles. Furthermore, in addition to RNA information, proteome information and information on skin surface bacteria may be combined to perform clustering.

[0042] In addition to skin classification based on specific skin properties, the above clustering can also be based on RNA information with a gene number of 1,000 to 20,000. For example, by performing clustering based on the expression levels of a gene related to a certain function (function X) and a gene related to another function (function Y), four skin classifications can be generated: Classification I (function X: medium expression level; function Y: medium expression level), Classification II (function X: medium expression level; function Y: low expression level), Classification III (function X: high expression level; function Y: high expression level), and Classification IV (function X: low expression level; function Y: medium expression level). In other words, if users' RNA profiles are similar, it can be estimated that they have similar skin concerns or that the effects of cosmetics are similar, so it is also possible to generate skin classifications based only on the similarity of gene information (RNA expression level) without associating them with specific skin properties.

[0043] [Operation of the cosmetics information server] Next, a description will be given of the operation of cosmetics information providing server 100 configured as described above. The operation is executed by the cooperation of hardware such as CPU 11 and communication unit 19 of cosmetics information providing server 100 and software stored in storage unit 18. For convenience, in the following description, CPU 11 is the subject of operation.

[0044] (Learning model generation process) First, a process of generating a learning model for generating skin classification information of a user by the cosmetics information providing server 100 will be described.

[0045] As shown in FIG. 4, the CPU 11 of the cosmetic information providing server 100 acquires a face image of the user from the skin classification information database 33 (step 41).

[0046] When VISIA images are used as face images for learning, the number of images is inflated by rotating, flipping, etc. the original VISIA images. At that time, the inflated number is adjusted so that the number of learning images for each skin classification is approximately equal.

[0047] In addition, when smartphone images are used as face images for learning, since there are cases where parts other than the face are reflected in the captured image, pre-processing is performed to extract the face part from each face image, and further pre-processing may be performed to cut out only the image of a specific feature area from the face image. After these pre-processing processes, image padding processing is performed in the same way as the VISIA images.

[0048] Next, the CPU 11 analyzes each face image and acquires bone structure information and skin color information as feature quantities (step 42). The bone structure information is information that indicates the bone structure (shape) characteristics of the user's face, and the skin color information is information that indicates the color characteristics of the user's skin.

[0049] The reason why skeletal information is used here is that, as mentioned above, among the skin classifications, the keratinized type is characterized by firmness and elasticity and less noticeable sagging at the corners of the mouth, while the immune type is characterized by a lack of firmness and elasticity and noticeable sagging at the corners of the mouth, and therefore it is speculated that it may be possible to classify the keratinized type and the immune type by analyzing the skeleton (facial shape) from facial images.

[0050] The reason that skin color information is used is that, as described above, the keratinized type of skin classification has characteristics related to skin color, such as having a lot of melanin, while the immune type has characteristics related to skin color, such as having less melanin and being lighter, and it is assumed that it may be possible to classify the keratinized type and the immune type by analyzing skin color from facial images.

[0051] Specifically, as the skeleton information, for example as shown in FIG. 5, information on the length (ratio) of each part of the face for a total of 21 items is acquired from the face image.

[0052] That is, in the face image, the angles of the left and right eyes (2 and 1 in the same figure) and the angles of the left and right eyebrows (4 and 3 in the same figure), the ratio of the vertical width to the horizontal width of the face (5 in the same figure), the ratio of the horizontal width of the left and right eyes to the horizontal width of the face (9 and 7 in the same figure), the ratio of the space between the left and right eyes (8 in the same figure), the right face line width (the horizontal distance between the right temple and the corner of the right eye) and the left face line width (the horizontal distance between the left temple and the corner of the left eye) to the horizontal width of the face. The information obtained includes the ratios of the vertical width of the face to the vertical width of the forehead (11 in the same figure), the ratios of the vertical widths of the left and right eyes to the vertical width of the face (13 in the same figure) and the ratios of the vertical widths of the left and right eyelids to the vertical width of the face (12 in the same figure), the ratio of the vertical width of the nose to the vertical width of the face (14 in the same figure), and the ratios of the vertical width of the upper lip, vertical width of the lower lip, vertical width of the white lip, and vertical width of the chin to the vertical width of the face (16, 17, 15, and 18 in the same figure).

[0053] The acquired skeletal information is not limited to that shown in the figure, and not all of the above 21 items but some of them may be acquired.

[0054] In addition, as skin color information, for example, as shown in Figure 6, the average value (mean) of L*, a*, and b* in the Lab color space (skin color) and the SE (standard error) values ​​of L*, a*, and b* (color unevenness) are calculated for each of the forehead area, left and right cheek areas, and left and right corners of the mouth areas in the face image.

[0055] In addition to or instead of these, the median and standard deviation (SD) of L*, a*, and b* of each region may be calculated. Color values ​​in a color system other than the Lab color space may also be used. The target regions are not limited to the above five regions and may be other regions, and the above values ​​may be calculated for some regions rather than all five regions.

[0056] Next, the CPU 11 acquires from the skin classification information database 33 the user's known skin classification information (for example, keratinization type, immune type) associated with the face image from which the skeletal information and skin color information were acquired (step 43).

[0057] Then, the CPU 11 generates a learning model that estimates the skin classification from the acquired bone structure information and skin color information, using the known skin classification information as training data (step 44).

[0058] As a learning model, for example, a convolutional neural network (CNN), more specifically, VGG16 (CNN consisting of 16 layers) is used, but this is not limited thereto, and conventional machine learning models such as random forest, support vector machine, logistic regression, etc. may also be used.

[0059] The CPU 11 repeatedly executes the learning process including adjustment of the skeletal information, skin color information and other parameters until the estimation accuracy of the skin classification by the learning model reaches a predetermined value or more.

[0060] (Provision and processing of skin classification information and cosmetic information) Next, a description will be given of the process of providing skin classification information and cosmetic information to the user terminal 200. Fig. 7 is a flowchart showing the flow of the process of providing skin classification information and cosmetic information.

[0061] As shown in the figure, first, the CPU 11 determines whether or not a skin classification request has been received together with a facial image from the user terminal 200 (step 71). The skin classification request can be transmitted, for example, via a skin classification request transmission page provided on the cosmetics information providing site.

[0062] If it is determined that the skin classification request has been received (Yes in step 71), the CPU 11 analyzes the received face image and obtains the bone structure information and skin color information (step 72).

[0063] Next, the CPU 11 inputs the bone structure information and skin color information obtained from the face image into the learning model (step 73).

[0064] Next, the CPU 11 judges whether or not skin classification information has been output from the learning model (step 74). If it is judged that skin classification information has not been output (No in step 74), the CPU 11 repeats the check until skin classification information is output.

[0065] If it is determined that skin classification information has been output from the learning model (Yes in step 74), the CPU 11 stores the skin classification information in the skin classification information database 33 in association with the user ID of the requesting user, and transmits the skin classification information to the requesting user terminal 200 (step 75).

[0066] Next, the CPU 11 determines whether or not a search request for cosmetics information has been received from the user terminal 200 to which the skin classification information was sent (step 76).

[0067] If it is determined that a search request for cosmetics information has been received (Yes in step 76), CPU 11 sends a cosmetics information provision page containing the search results for cosmetics information corresponding to the skin classification associated with the user ID of the user who made the search request to the user terminal and displays it (step 77).

[0068] FIG. 8 is a diagram showing a display example of the cosmetics information page.

[0069] As shown in the figure, the cosmetics information page displays, for example, information showing the search results for a beauty serum, along with information introducing the beauty serum (product name, photo, description, price, etc.), as well as evaluation information for the product (e.g., a 7-point rating scale) by users with the same skin classification as the user who made the search request (e.g., skin type B) and evaluation information for the product by users with other skin classifications (e.g., skin types A and C). This allows users to easily find products that are suitable for their own skin classification and are highly rated.

[0070] In addition, evaluation information for skin classifications that are not the skin classification of the user who made the search request (types A and C in the same figure) may not be posted, and only evaluation information for the skin classification of the user who made the search request (type B in the same figure) may be extracted and posted.

[0071] As described above, according to this embodiment, the cosmetics information providing server 100 can more easily and quickly classify the user's skin characteristics and provide corresponding cosmetics information simply by acquiring a facial image of the user, without having to take the trouble of collecting and analyzing biometric information (RNA) from the user to generate skin classification information.

[0072] [Variations] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

[0073] In the above embodiment, the cosmetics information providing server 100 generates a learning model based on skeletal information and skin color information acquired from a face image, but a learning model that estimates a skin classification using only one of skeletal information and skin color information may be generated. Also, in addition to or instead of skeletal information and skin color information, a learning model that estimates a skin classification may be generated by taking into account skin surface information (information indicating the state of desquamation, scales, wrinkles, texture, pores, etc.) as a parameter.

[0074] In the above-described embodiment, the cosmetics information providing server 100 may train a learning model to estimate a skin classification using eye color information in addition to bone structure information and / or skin color information acquired from a face image. In this case, some of the evaluation items shown in Figs. 5 and 6 may not be calculated as bone structure information and / or skin color information.

[0075] The cosmetics information providing server 100 then analyzes the received facial image to obtain at least one of skeletal information and / or skin color information and eye color information, and uses a learning model to output skin classification information estimated from at least one of the obtained skeletal information or skin color information and the eye color information.

[0076] For example, as shown in Figure 9(a), the cosmetics information providing server 100 calculates, as skin color information, the average values ​​(mean) of R, G, B in RGB, the average values ​​(mean) of L*, a*, b* in Lab color space, and SE (standard error) values ​​of R, G, B, and L*, a*, b* for the forehead area, the left and right cheek areas, and the left and right corners of the mouth in the face image, and calculates, as information indicating eye color (eye color information), the average values ​​and SE values ​​of R, G, B, and L*, a*, b* for the iris (pupil) areas of the left and right eyes and the white area surrounding them.

[0077] In this case, specifically, as shown in FIG. 1(a), the width of the nose, the width of the left and right cheeks (the distances from the left and right ends of the nose to the left and right ends of the face), the angle and width of the left and right eyes may be selectively calculated as skeletal information.

[0078] As shown in FIG. 1(b), skin color information may be selectively calculated as the average values ​​of R, G, B, and L* in the left corner of the mouth area, G, B, L*, and b* in the forehead area, and R, L*, and b* in the right cheek area, as well as the SE values ​​of b* in the left corner of the mouth area and B and a* in the left cheek area.

[0079] Furthermore, as shown in FIG. 1(c), the eye color information may be selectively calculated as the average values ​​of b* in the iris region of the left eye, R, B, and b* in the white region of the left eye, G in the iris region of the right eye, and b* in the white region of the right eye, as well as the SE values ​​of B and b* in the iris region of the left eye, B in the white region of the left eye, and G, B, and b* in the white region of the right eye.

[0080] Of course, the evaluation items to be calculated are not limited to these, and other areas may be calculated as evaluation items. For example, only the color of one of the left and right eyes may be used as eye color information, or only the color of the iris area or only the color of the white area of ​​both or one of the eyes may be used. Also, in this modified example using eye color information, in addition to or instead of the above, the median or standard deviation (SD) of R, G, B or L*, a*, b* of each area may be calculated, or color values ​​in a color system other than the RGB and Lab color spaces may be used.

[0081] In this way, by supplementarily using eye color information (color, uneven color) in addition to the bone structure information and / or skin color information, it is possible to improve the accuracy of estimating skin classification.

[0082] As a result of the inventors' analysis of the user's skin images used to obtain skin color information and skeletal information in the above-mentioned embodiment, it was found that the higher the prediction confidence (a statistical measure of how certain the prediction or output is) when classifying skin using the above-mentioned learning model, the higher the classification accuracy (correct answer rate).

[0083] In particular, when comparing the average values ​​of image features between images with high confidence and images with low confidence, it was found that images with high confidence (e.g., 70% or higher) had higher RGB B values ​​in the forehead, mouth, and eyes areas than images with low confidence (e.g., 50-53%), and that images with low confidence had higher yellowness (Lab b* value) in the forehead, cheeks, and eyes areas than images with high confidence.

[0084] Therefore, the cosmetics information providing server 100 may receive multiple face images from a user, and extract from the multiple face images face images having a high average B value of the RGB values ​​of each pixel of the face images for analysis using a learning model, or may exclude from the above analysis face images having a high average b value of the Lab values ​​of each pixel.

[0085] Specifically, the cosmetics information providing server 100 may extract a face image with the highest average value of the B value of the RGB values ​​from multiple face images, or may accept face images from the user again until a face image with an average value of the B value of the RGB values ​​equal to or greater than a predetermined threshold value is acquired. In this case, if the average value of the B value of the RGB values ​​of the acquired face image is less than the threshold value, a message such as "Please take another face image" may be sent to the user terminal 200 and output to prompt the user to take the image again.

[0086] In addition, if the average value of the b value of the Lab values ​​of the acquired facial image is equal to or greater than a predetermined threshold, the cosmetics information providing server 100 may re-accept a facial image from the user in the same manner as described above until a facial image in which the average value is less than the predetermined threshold is acquired.

[0087] The cosmetics information providing server 100 may calculate the average value of the B value of the RGB values ​​by limiting the calculation to at least one of the forehead, mouth, and eyes, or may calculate the average value of the B value of the RGB values ​​by limiting the calculation to the forehead, cheeks, and eyes, or may calculate the average value of the b value of the Lab ...

[0088] Furthermore, the inventors have found, as a result of analyzing the skin images, that images with a high certainty factor show the unevenness of the skin more clearly and have more fine shadows such as blemishes.

[0089] Therefore, the cosmetics information providing server 100 may receive multiple facial images from a user, and extract from the multiple facial images those in which at least one of blemishes, pores, wrinkles, desquamation, scales, or skin irregularities is recognized for analysis using a learning model.

[0090] Specifically, when the cosmetics information providing server 100 receives a facial image from a user, it executes an image recognition process and determines whether or not the above-mentioned blemishes, pores, wrinkles, peeling, scales or skin irregularities can be recognized. If they cannot be recognized, it may send and output message information to the user terminal 200 such as "Please take another facial image as the resolution is low" or "Please take another facial image in a bright place" to encourage the user to take the image again until a recognizable facial image is obtained.

[0091] In addition, instead of detecting blemishes, pores, etc., the cosmetics information providing server 100 may extract face images for analysis in which edges have been detected using a predetermined edge detection filter that has a sufficiently high resolution and a high degree of certainty and is generated based on the face image.

[0092] In the above embodiment, an example was shown in which a user who has transmitted a facial image to the cosmetics information providing server 100 and been notified of their own skin classification information accesses the cosmetics search page and searches for cosmetics information corresponding to the skin classification. However, it is considered that there may be users who access the search page without knowing the information on the skin classification. Therefore, when a user to whom skin classification information has not been notified (a non-logged-in user or a logged-in user to whom a skin classification has not been assigned to the user ID) accesses the search page, the cosmetics information providing server 100 may display suggested information, for example, by a pop-up on the search page, that suggests providing skin classification information based on a facial image (transmitting a facial image for skin classification).

[0093] In the above-described embodiment, RNA information was used as the biological information, but other than that, the proteome (protein) collected from the user's stratum corneum, sebum, etc. may be analyzed, or epidermal bacteria collected from the skin surface may be analyzed.

[0094] In the above embodiment, cosmetics (including external preparations such as quasi-drugs) are given as an example of products. However, products are not limited to cosmetics, and information on any product that the user uses (including ingests) on the body (skin, hair, etc.) of the user, such as hair care products such as shampoo, conditioner, and treatment, medicines (skin preparations, etc.), sweat wipes, cosmetic sheets, toothpaste, beverages, foods, facial massagers, beauty tools, etc., may be provided according to skin classification information generated based on the user's face image.

[0095] In the above embodiment, only one cosmetics information providing server 100 is shown, but the processes executed by the cosmetics information providing server 100 may be distributed and executed by a plurality of servers. For example, the process of generating the learning model shown in Fig. 4 and the process of providing skin classification information and cosmetics information using the learning model shown in Fig. 7 may be executed by separate servers.

[0096] In addition, in the above-described embodiment, the web page created by the cosmetics information providing server 100 is displayed on the browser of the user terminal 200, but if an application program corresponding to the cosmetics information providing service is installed on the user terminal 200, the web page may be displayed by the application program.

[0097] In the above-described embodiment, the present invention has been described as a cosmetics information providing system that outputs cosmetics information corresponding to skin classification information estimated by analyzing a face image using a learning model to a user terminal 200. However, the present invention may also be realized as an information processing system that simply outputs estimated skin classification information without providing cosmetics information corresponding to the skin classification information.

[0098] In the above-described embodiment, the cosmetics information providing server 100 may acquire the skin color information using an estimation model that is trained to output, from a face image, the skin condition of a human face that appears in the face image.

[0099] In other words, in order to conform to the input rules of the above-mentioned estimation model, the cosmetics information providing server 100 may convert the acquired user's facial image so that multiple points on the contour of the facial area of ​​the human face reflected in the acquired user's facial image are positioned on the outer edge of the image, acquire a converted facial image, input the converted facial image to the above-mentioned estimation model, and acquire skin condition information of the user's face from the estimation model as the above-mentioned skin color information.

[0100] Here, the cosmetics information providing server 100 may remove or reduce a predetermined local part area in the face area of ​​the human face captured in the acquired face image in the converted face image. That is, in the conversion process, the predetermined local part area in the face area may be removed or reduced by matching the positions of outer perimeter points that face each other across the predetermined local part area among a plurality of outer perimeter points that surround the predetermined local part area in the face area. The predetermined local part area is, for example, an eye area, an eyebrow area, and a lip area.

[0101] Furthermore, the cosmetics information providing server 100 may further execute a position specifying process for specifying the position coordinates of a plurality of predetermined locations in the face region of the human face captured in the acquired face image. In this case, the conversion process may convert a part or all of the specified position coordinates of the plurality of predetermined locations to convert the acquired face image so as to conform to the input rule of the estimation model.

[0102] In this case, the face image may have a predetermined rectangular shape, and the position coordinates of the predetermined plurality of locations may further include position coordinates of a plurality of points on the contour of the face region, the nose tip of the face region, and a plurality of other points in the face region. The cosmetics information providing server 100 may perform the coordinate conversion in the conversion process such that each of four first contour points included in the plurality of points on the contour of the face region is located at the vertices of the predetermined rectangular shape, the nose tip is located at the center of the predetermined rectangular shape, the other points on the contour of the face region are located on the sides of the predetermined rectangular shape, and the other points in the face region move in response to the movement of the plurality of points on the contour of the face region and the nose tip.

[0103] Here, the other multiple points in the face region may include four third contour points located closer to the contour of the face region than the predetermined local site region and four second contour points located even closer to the contour of the face region than the four third contour points. In the conversion process, the cosmetics information providing server 100 may perform the coordinate conversion such that the four second contour points and the four third contour points are respectively located on diagonals connecting the four first contour points and the nose tip, and the difference between the distance of the first contour point from the nose tip and the distance of the second contour point from the nose tip is smaller than the difference between the distance of the second contour point from the nose tip and the distance of the third contour point from the nose tip.

[0104] Among the inventions described in the claims of this application, the invention described as an "information processing method" is one in which each step is automatically performed by at least one device such as a computer through information processing by software, and is not performed by a human using a device such as a computer. In other words, the "information processing method" is an information processing method by computer software, and is not a method in which a human operates a calculation tool called a computer.

Claims

1. A learning model is generated to estimate a skin classification of a user from at least one of skeletal information indicating skeletal characteristics of the user and skin color information indicating skin color characteristics of the user, the information being obtained by analyzing a face image of the user, using skin classification information that classifies the skin of the user, which is generated by analyzing biometric information of the user provided by the user, as training data; Accepting input of a user's face image; Analyzing the received face image to obtain at least one of the bone structure information and the skin color information; outputting skin classification information estimated from at least one of the acquired skeletal information and skin color information using the learning model; Control Unit An information processing system comprising:

2. The control unit analyzes the face image to obtain the skeletal information, and outputs the skin classification information estimated from the skeletal information using the learning model. The information processing system according to claim 1 .

3. The control unit generates the learning model using the skin classification information generated by the analysis of the RNA of the user as the training data. The information processing system according to claim 1 .

4. The control unit is generating the learning model from the skeletal information and the skin color information; Analyzing the received face image to obtain the bone structure information and the skin color information; Using the learning model, skin classification information estimated from the acquired skeletal information and skin color information is output. The information processing system according to claim 1 .

5. The control unit uses a ratio of a vertical width to a horizontal width of a face in the face image as the skeleton information.

3. The information processing system according to claim 1 or 2.

6. The control unit uses, as the skeleton information, a ratio of the width of each of the left eye and the right eye to the width of the face in the face image or a ratio of the distance between the left eye and the right eye.

3. The information processing system according to claim 1 or 2.

7. The control unit uses, as the skeletal information, the ratio of the vertical width of each of the left eye and the right eye or the vertical width of each of the left eyelid and the right eyelid to the vertical width of the face in the face image.

3. The information processing system according to claim 1 or 2.

8. The control unit uses the angles of the left and right eyes and the angles of the left and right eyebrows in the face image as the skeleton information.

3. The information processing system according to claim 1 or 2.

9. The control unit uses, as the skeletal information, the ratios of the vertical width of the upper lip, the vertical width of the lower lip, and the vertical width of the white lip to the vertical width of the face in the face image.

3. The information processing system according to claim 1 or 2.

10. The control unit uses, as the skeletal information, a ratio of the vertical width of the nose or the vertical width of the forehead to the vertical width of the face in the face image.

8. An information processing system according to claim 1.

11. The control unit uses, as the skeleton information, the ratios of the horizontal distance between the right temple and the right eye corner and the horizontal distance between the left temple and the left eye corner to the horizontal width of the face in the face image.

3. The information processing system according to claim 1 or 2.

12. The control unit uses at least one of an average value, a median value, a standard deviation, and a standard error of color values ​​in at least one region in the face image as the skin color information. The information processing system according to claim 1 .

13. The control unit uses at least one value in at least two of a forehead region, left and right cheek regions, and left and right corners of a mouth in the face image as the skin color information. The information processing system according to claim 10.

14. The control unit receives an input of a face image captured by a user terminal of the user from the user terminal, and outputs information about cosmetics corresponding to the skin classification information estimated by analyzing the face image to the user terminal.

3. The information processing system according to claim 1 or 2.

15. The control unit is The learning model is trained so as to estimate the skin classification based on at least one of the skeletal information and the skin color information acquired by analyzing a face image of the user and eye color information indicating eye color characteristics of the user; Analyzing the received face image to obtain at least one of the bone structure information and the skin color information, and the eye color information; outputting skin classification information estimated from at least one of the acquired skeletal information and skin color information and the eye color information using the learning model; The information processing system according to claim 1 .

16. The control unit receives a plurality of face images from the user, and extracts, from among the plurality of face images, face images having a high average value of the B value of the RGB values ​​of each pixel of the face images for the analysis, or excludes face images having a high average value of the b value of the Lab values ​​of each pixel from the analysis.

3. The information processing system according to claim 1 or 2.

17. The control unit receives a plurality of face images from the user, and extracts, from the plurality of face images, face images in which blemishes, pores, wrinkles, desquamation, scales, or skin irregularities are recognized, for the analysis.

3. The information processing system according to claim 1 or 2.

18. The control unit converts the received face image so that a plurality of points on a contour of a face region of a human face reflected in the face image are positioned on an outer edge of the image to obtain a converted face image, inputs the converted face image to an estimation model that has been trained to output a skin condition of the human face reflected in the face image from the face image, and obtains skin condition information of the user's face from the estimation model as the skin color information. The information processing system according to claim 1 .

19. A learning model is generated to estimate a skin classification of a user from at least one of skeletal information indicating skeletal characteristics of the user and skin color information indicating skin color characteristics of the user, the information being obtained by analyzing a face image of the user, using skin classification information that classifies the skin of the user, which is generated by analyzing biometric information of the user provided by the user, as training data; Accepting input of a user's face image; Analyzing the received face image to obtain at least one of the bone structure information and the skin color information; Using the learning model, skin classification information estimated from at least one of the acquired skeletal information and skin color information is output. Information processing methods.

20. In the information processing device, A step of generating a learning model for estimating a skin classification of a user from at least one of skeletal information indicating skeletal characteristics of the user and skin color information indicating skin color characteristics of the user, which are obtained by analyzing a facial image of the user, using skin classification information that classifies the skin of the user, generated by analyzing biometric information of the user provided by the user, as training data; receiving an input of a face image of a user; analyzing the received face image to obtain at least one of the bone structure information and the skin color information; outputting skin classification information estimated from at least one of the acquired skeletal information and skin color information using the learning model; A program that executes the following.

21. In the information processing device, A step of generating a learning model for estimating a skin classification of a user from at least one of skeletal information indicating skeletal characteristics of the user and skin color information indicating skin color characteristics of the user, which are obtained by analyzing a facial image of the user, using skin classification information that classifies the skin of the user, generated by analyzing biometric information of the user provided by the user, as training data; receiving an input of a face image of a user; analyzing the received face image to obtain at least one of the bone structure information and the skin color information; outputting skin classification information estimated from at least one of the acquired skeletal information and skin color information using the learning model; A storage medium that stores a program for executing the above.

Images