Information processing system, information processing method, and program

Abstract

This information processing system comprises a storage unit and a control unit. The storage unit stores, in association with each other, information that indicates a plurality of skin types, and information that indicates, among a plurality of feature items indicating a feature of skin, a dominant feature item the degree which tends to be statistically high for the skin of each of the skin types. The control unit acquires information related to the skin type of a user of a user terminal and analyzes skin-related information related to the condition of the skin of the user, thereby determining the degree of the dominant feature item corresponding to the relevant skin type. In addition, the control unit transmits, to the user terminal, potential item information which is information related to a dominant feature item the degree of which is less than a predetermined threshold or is the lowest among the determined dominant feature items.

Description

Information processing system, information processing method and program

[0001] The present invention relates to an information processing system, an information processing method, and a program for estimating a user's skin type.

[0002] Conventionally, there have been technologies that examine the condition of a user's skin and, based on the results of the examination, present the user with information on recommended cosmetics and other skin care advice. For example, Patent Document 1 below discloses a counseling method that acquires genetic information representing the results of a genetic test of the user and stratum corneum information representing the results of a stratum corneum test of the user, determines the user's skin constitution based on the genetic information, determines the user's skin performance based on the stratum corneum information, identifies skin care advice and / or cosmetics to be suggested to the user based on the skin constitution and skin performance, and presents the user with counseling information including information on the skin care advice and / or cosmetics based on the identified skin constitution and skin performance.

[0003] International Publication No. 2020 / 130103

[0004] However, with the technology of Patent Document 1, although a user can receive advice according to their own skin constitution and skin condition, they cannot grasp which of the skin characteristics analyzed as being poor have the potential for improvement.

[0005] The present invention aims to provide an information processing system, an information processing method, and a program that can allow a user to understand, among the characteristic items analyzed as being poor in the user's skin, those that have the potential for improvement.

[0006] An information processing system according to one aspect of the present invention includes a storage unit and a control unit. The storage unit stores information indicating multiple skin types and, among multiple feature items indicating skin characteristics, information indicating dominant feature items in which the degree of the feature tends to be statistically high for skin of each of the skin types, in association with each other. The control unit acquires information about the skin type of a user of a user terminal and analyzes skin-related information related to the user's skin condition to determine the degree of the dominant feature item corresponding to the skin type. The control unit also transmits, to the user terminal, information about dominant feature items whose degree is less than a predetermined threshold or whose degree is the lowest, as potential item information.

[0007] An information processing method according to another aspect of the present invention includes: storing information indicating a plurality of skin types in association with information indicating dominant feature items among a plurality of feature items indicating skin characteristics, the degree of which tends to be statistically higher in skin of each of the skin types; acquiring information on the skin type of a user of a user terminal and analyzing skin-related information related to the user's skin condition to determine the degree of the dominant feature item corresponding to the skin type; and transmitting information on the dominant feature items whose degree is less than a predetermined threshold or whose degree is the lowest, among the determined dominant feature items, to the user terminal as potential item information.

[0008] A program according to yet another aspect of the present invention causes an information processing device to execute the steps of: storing information indicating a plurality of skin types in association with information indicating dominant feature items, among a plurality of feature items indicating skin characteristics, for which the degree of the feature tends to be statistically higher in skin of each of the skin types; acquiring information about the skin type of a user of a user terminal and analyzing skin-related information related to the user's skin condition to determine the degree of the dominant feature item corresponding to the skin type; and transmitting, as potential item information, information about dominant feature items, among the determined dominant feature items, whose degree is less than a predetermined threshold or whose degree is the lowest, to the user terminal.

[0009] According to an information processing system according to an embodiment of the present invention, it is possible to allow a user to understand, among the feature items analyzed as being unsatisfactory for the user's skin, feature items that have the potential for improvement, but this effect is not intended to limit the present invention.

[0010] 1 is a diagram showing the configuration of a skin analysis information providing system according to one embodiment of the present invention. 2 is a diagram showing the hardware configuration of a skin analysis information providing server according to one embodiment of the present invention. 3 is a diagram showing the configuration of a database held by a skin analysis information providing server according to one embodiment of the present invention. 4 is a flowchart showing the flow of a process for providing skin analysis information to a user by a skin analysis information providing server according to one embodiment of the present invention. 5 is a diagram showing an example of a display of skin analysis information generated and provided by a skin analysis information providing server according to one embodiment of the present invention. 6 is a diagram showing an example of a display of cosmetic recommendation information generated and provided by a skin analysis information providing server according to one embodiment of the present invention.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [System Configuration] As shown in FIG. 1, this system includes a skin analysis information providing server 100 on the Internet 50 and a plurality of user terminals 200.

[0013] The skin analysis information providing server 100 is a server (information processing device) that executes a service that provides a user's skin analysis information and corresponding skin care information. The skin analysis information providing server 100 is connected to a plurality of user terminals 200 via the Internet 50.

[0014] Based on the user's facial image received from the user terminal 200 along with the skin analysis request, the skin analysis information providing server 100 transmits to the user terminal 200 skin analysis information including skin type information indicating the user's skin type as the analysis result and skin score information corresponding to the skin type.

[0015] To acquire the skin type information, the skin analysis information providing server 100 uses a first trained model 10 that estimates a user's skin type from feature information acquired by analyzing a user's facial image. The first trained model 10 is trained to estimate a user's skin type from skeletal information indicating the user's skeletal features and skin color information indicating the user's skin color features, acquired from the user's facial image, using known skin type information of each user generated by analyzing the user's biometric information (e.g., RNA) provided by the users as training data. The first trained model 10 may be generated by a learning process via the skin analysis information providing server 100 or may be generated by another information processing device.

[0016] To acquire skin score information, the skin analysis information providing server 100 uses a second trained model 20 that estimates index values ​​(skin scores) for multiple skin feature items from feature information acquired by analyzing a user's facial image. The second trained model 20 is trained to output index values ​​(skin scores) indicating the degree of each feature item (skin attribute / skin condition) of the user's skin from multiple patch images acquired from the user's facial image, using patch images, which are partial section images of the user's facial image provided by multiple users, as training data. The second trained model 20 may be generated by a learning process via the skin analysis information providing server 100, or may be generated by another information processing device.

[0017] User terminal 200 (200A, 200B, 200C...) is a terminal used by a user, and may be, for example, a smartphone, a mobile phone, a tablet PC (Personal Computer), a notebook PC, or a desktop PC. User terminal 200 accesses skin analysis information providing server 100, receives a web page containing the skin analysis information, etc., and displays it on a screen using a browser or the like. An application corresponding to the skin analysis information providing service may also be installed on user terminal 200, and user terminal 200 may access skin analysis information providing server 100 using the application to display the skin analysis information.

[0018] In this embodiment, when providing the skin analysis information, the skin analysis information providing server 100 generates information on dominant feature items whose skin scores are less than a predetermined threshold, among dominant feature items whose skin scores tend to be statistically higher for the user's skin type, as potential item information, and provides this information to the user terminal 200. Details of the skin analysis information providing process, including the generation of the potential item information, will be described later.

[0019] [Hardware Configuration of Skin Analysis Information Providing Server] As shown in FIG. 2 , the skin analysis information providing server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface 15, and a bus 14 connecting these components to one another.

[0020] The CPU 11 accesses the RAM 13 and other memory as needed, and performs various arithmetic processing while providing overall control over each block of the skin analysis information providing server 100. Multiple CPUs 11 may be provided depending on the processing. The ROM 12 is a non-volatile memory in which firmware such as the OS, programs, and various parameters to be executed by the CPU 11 are permanently stored. The RAM 13 is used as a working area for the CPU 11, and temporarily stores the OS, various applications currently being executed, and various data currently being processed.

[0021] The input / output interface 15 is connected to a display unit 16, an operation reception unit 17, a storage unit 18, a communication unit 19, and the like.

[0022] The display unit 16 is a display device that uses, for example, an LCD (Liquid Crystal Display), an OELD (Organic ElectroLuminescence Display), a CRT (Cathode Ray Tube), or the like.

[0023] The operation reception unit 17 is, for example, a pointing device such as a mouse, a keyboard, a touch panel, or other input device. When the operation reception unit 17 is a touch panel, the touch panel can be integrated with the display unit 16.

[0024] The storage unit 18 is a non-volatile memory such as a hard disk drive (HDD), a flash memory (SSD; solid state drive), or other solid-state memory. The storage unit 18 stores the OS, various applications, and various data.

[0025] As will be described later, particularly in this embodiment, the storage unit 18 has a user information database, a skin type information database, a skin score information database, and a cosmetics information database, in addition to programs such as applications necessary for the skin analysis information provision processing described later.

[0026] The communication unit 19 is, for example, a NIC (Network Interface Card) for Ethernet or various modules for wireless communication such as a wireless LAN, and is responsible for communication processing with the user terminal 200 .

[0027] Although not shown, the basic hardware configuration of the user terminal 200 is also substantially the same as the hardware configuration of the skin analysis information providing server 100 described above.

[0028] [Database configuration of skin analysis information server]

[0029] 3, the skin analysis information providing server 100 has, in the storage unit 18, a user information database 31, a skin type information database 32, a skin score information database 33, and a cosmetics information database 34. Note that each of these databases may be stored in a storage device or server externally connected to the skin analysis information providing server 100, rather than in the storage unit 18.

[0030] The user information database 31 stores attribute information for each user of the cosmetics information provision service provided by the skin analysis information providing server 100. User attribute information includes general information such as name, user ID for identifying the user, age (generation), occupation, address (area of ​​residence), gender, and email address, as well as information about the user's skin, such as the user's perceived or understood skin type, cosmetic preferences, and skin problems the user has. The user ID functions as account information for using this skin analysis information provision service.

[0031] The skin type information database 32 stores the skin type information of each user classified using the first trained model 10 in association with the user ID. Examples of skin types include, but are not limited to, a "smooth predominant" type and a "bright predominant" type, which will be described later.

[0032] The skin type information database 32 also stores information indicating the dominant characteristic items among the multiple skin characteristic items stored in the skin score information database 33 that tend to statistically increase the skin score for each skin type.

[0033] In addition, when the first trained model 10 is generated by the skin analysis information providing server 100, the skin type information database 32 also stores information such as the subject's known skin type information and the corresponding facial image data of each subject used to generate the first trained model 10, as well as skeletal information, skin color information, and other parameters obtained by analyzing the facial image data.

[0034] The skin score information database 33 stores, in association with the user ID, skin score information of each user acquired using the second trained model 20. The skin score information is, for example, an index value (score) indicating the degree of each of a plurality of skin characteristic items (attributes / conditions) such as wrinkles, age spots, firmness, nasolabial folds, texture, moisture (moisture), dullness, pores, brightness, transparency, luster, smoothness, natural skin feel (natural skin-likeness), makeup feel (made-up skin-likeness), male / female skin-likeness, skin age, degree of makeup smearing, powdery feel, etc., as well as a score that comprehensively evaluates these, and is expressed as a value from 0 to 100, for example, but is not limited to this.

[0035] The skin score information database 33 also stores text information that explains the skin score of each characteristic item to the user, for example, for each skin score within a predetermined range. For example, if the characteristic item is wrinkles and the score is 90 to 100, explanatory text such as "Your skin has few wrinkles and a clean impression" is stored.

[0036] In addition, when the second trained model 20 is generated by the skin analysis information providing server 100, the skin score information database 33 also stores information such as facial images (patch images) of the subject for each feature item training data and other parameters used in generating the second trained model 20.

[0037] The cosmetics information database 34 stores information about cosmetics recommended to users, such as the manufacturer name, product name (brand name), price, cosmetic category (e.g., serum, lotion, cleanser, facial cleanser, makeup base, foundation, pack, cream, etc.), and expected efficacy (e.g., brightening, firming, wrinkle care, blemish care, nasolabial fold care, skin texture improvement, moisturizing, dullness care, enhancing transparency, smoothing, pore care, acne care, exfoliating care, sensitive skin care, anti-aging, radiance, natural makeup, waterproof, color retention, color development, etc.) (efficacy tag information). The efficacy tag information may include information about not only skin care effects but also makeup effects. The efficacy tag information is stored in association with at least some of the characteristic items stored in the skin score information database 33.

[0038] The cosmetics information database 34 also stores user evaluation information for each cosmetic product (multi-level evaluation value information and text information describing word-of-mouth reviews). The evaluation information may also include data on the time (date or season) and location (region or area from which the user terminal 200 accessed) when the user provided the evaluation information. The evaluation information may also include not only an evaluation of the overall impression of the product, but also evaluation information on the efficacy of the product that corresponds to the efficacy tag information (multi-level evaluation information and word-of-mouth reviews). The evaluation information is stored in association with the user ID of the user who input the evaluation information and the skin type information of the user.

[0039] These databases are used by mutual reference as needed in the skin analysis information providing process by the skin analysis information providing server 100, which will be described later.

[0040] [First Trained Model for Skin Type Classification] Next, the first trained model 10 for skin type classification will be described. As described above, the first trained model 10 is generated by learning using known skin type information of each user generated by analyzing the user's biometric information (e.g., RNA) as training data. Here, an example of a method for generating skin type information by analyzing the user's biometric information will be described.

[0041] The user's biological information is obtained by mailing a sample collected using a collection kit sent to the user to a testing institution, or by collecting the sample directly from the user who visits the business's testing institution. The biological information is, for example, RNA information, and is collected from the user's sebum, but may also be collected from the stratum corneum, saliva, urine, blood, etc.

[0042] The testing institution measures RNA information from biological samples collected from users. Specifically, the RNA information is data obtained by extracting and preparing RNA from each biological sample, generating cDNA by reverse transcription, and then measuring the expression level of each RNA species by sequencing, PCR, or the like.

[0043] Next, a cluster analysis of the RNA expression level data is performed. Specifically, two or more skin property items are selected from a plurality of skin property items for evaluating the physical properties of the user's skin (e.g., stratum corneum moisture content, TEWL (transepidermal water loss), sebum content, melanin content, erythema content, overall redness, skin elasticity, etc.).

[0044] Next, the similarity is determined for the genes related to the selected skin physical property items of each sample based on their expression levels (expression patterns). A clustering method may be used to determine the similarity, and either hierarchical clustering or non-hierarchical clustering may be used as the clustering method. Alternatively, other evaluation models such as machine learning models may be used.

[0045] Then, a predetermined number of skin types are generated according to the number of clusters classified by the clustering. In this embodiment, for example, two skin types (e.g., a smooth-predominant type and a light-predominant type) are generated as described above. Alternatively, three types of skin types may be generated, including an intermediate type between the two types.

[0046] Furthermore, as a result of the statistics and analysis, the following dominant skin characteristics were found for the smooth-dominant type and the brightness-dominant type. In other words, it was found that for the following characteristic items, the skin scores for each skin type tend to be statistically higher (dominant characteristic items). (Smooth-dominant type) - Strong firmness - Less noticeable nasolabial folds, giving a youthful impression - Less noticeable wrinkles, giving a dignified impression - Less noticeable pores, giving a smooth impression (Brightness-dominant type) - Bright skin tone - Less dullness, giving a clear impression - High transparency - Strong luster, giving a dignified impression

[0047] Furthermore, multiple types of skin types may be generated by generating multiple different combinations of multiple skin physical property items. For example, RNA (genes) may be selected based on skin physical property items tailored to various skin concerns, such as age spots and wrinkles. Furthermore, clustering may be performed by combining proteome information and information on superficial skin bacteria in addition to RNA information.

[0048] In addition to skin types based on specific skin properties, the clustering can also be based on RNA information with a gene count of 1,000 to 20,000. For example, by performing clustering based on the expression levels of genes related to a certain function (function X) and genes related to another function (function Y), four skin types can be generated: Class I (function X: medium expression level; function Y: medium expression level), Class II (function X: medium expression level; function Y: low expression level), Class III (function X: high expression level; function Y: high expression level), and Class IV (function X: low expression level; function Y: medium expression level). In other words, if users' RNA profiles are similar, it can be estimated that they have similar skin concerns or that their cosmetics have similar effects. Therefore, it is possible to generate skin types based only on the similarity of genetic information (RNA expression level) without associating them with specific skin properties.

[0049] In this embodiment, by using the first trained model 10, skin type classification, which has conventionally been performed by analyzing the user's biometric information as described above, can be performed based on the user's facial image. The generation process of the first trained model 10 will be described below.

[0050] When generating the first trained model 10, a VISIA image taken by a VISIA (registered trademark) skin image analysis device or a smartphone image taken by a user using a user terminal 200 (smartphone) is used as face image data for training. The photographing direction is, for example, the front of the face, but images taken from other photographing directions may also be used in addition to or instead of this.

[0051] When VISIA images are used as face images for learning, the number of images is increased by rotating, flipping, etc. the original VISIA images. At that time, the number of images is adjusted so that the number of learning images for each skin type is approximately equal.

[0052] Furthermore, when smartphone images are used as face images for learning, since the captured images may contain parts other than the face, preprocessing may be performed to extract the face part from each face image, and further preprocessing may be performed to cut out only images of specific feature areas from the face image. After these preprocessing steps, the image padding process is performed in the same way as for the VISIA images.

[0053] Next, each face image is analyzed to obtain bone structure information and skin color information as feature quantities. The bone structure information indicates the bone structure (shape) characteristics of the user's face, and the skin color information indicates the skin color characteristics of the user.

[0054] The reason why skeletal information is used here is that, as mentioned above, smooth-skinned skin types are characterized by firmness and elasticity (less noticeable sagging at the corners of the mouth), while bright-skinned skin types do not exhibit such characteristics, and therefore it is speculated that analyzing the skeleton (facial shape) from facial images may enable classification into smooth-skinned skin types and bright-skinned skin types.

[0055] The reason why skin color information is used is that, as mentioned above, the bright-dominant type has a characteristic related to skin color, that is, light (low melanin), while the smooth-dominant type does not have such a characteristic. Therefore, it is presumed that analyzing skin color from facial images may enable classification into the smooth-dominant type and the bright-dominant type.

[0056] Specifically, as the skeletal information, for example, 21 items of length (ratio) information of each part of the face are acquired from the face image. That is, for example, at least some of the information acquired from the face image includes the angles of the left and right eyes and the angles of the left and right eyebrows, the ratio of the vertical width to the horizontal width of the face, the ratio of the horizontal width of the left and right eyes to the horizontal width of the face, the ratio of the distance between the left and right eyes, the ratios of the right face line width (the horizontal distance between the right temple and the outer corner of the right eye) and the left face line width (the horizontal distance between the left temple and the outer corner of the left eye) to the horizontal width of the face, the ratio of the vertical width of the forehead to the vertical width of the face, the ratios of the vertical widths of the left and right eyes and the vertical widths of the left eyelid and the right eyelid to the vertical width of the face, the ratio of the vertical width of the nose to the vertical width of the face, and the ratios of the vertical width of the upper lip, the vertical width of the lower lip, the vertical width of the white lip, and the vertical width of the chin to the vertical width of the face.

[0057] Furthermore, as skin color information, the mean values ​​(skin color) of L*, a*, and b* in the Lab color space and the SE (standard error) values ​​(color unevenness) of L*, a*, and b* are calculated for each of the forehead region, left and right cheek regions, and left and right corners of the mouth in the facial image. In addition to or instead of these, the median and standard deviation (SD) of L*, a*, and b* for each region may be calculated. Color values ​​in a color system other than the Lab color space may also be used. Furthermore, the target regions are not limited to the above five regions and may be other regions, and the above values ​​may be calculated for only some of the five regions rather than all of them.

[0058] In addition to the above, information on the user's eye color (eye color information) may be used as skin color information. Specifically, the eye color information may selectively include the average values ​​of b* in the iris region of the left eye, R, B, and b* in the white region of the left eye, G in the iris region of the right eye, and b* in the white region of the right eye, as well as the SE values ​​of B and b* in the iris region of the left eye, B in the white region of the left eye, and G, B, and b* in the white region of the right eye.

[0059] Next, a learning model is generated that estimates the skin type from the acquired skeletal information and skin color information, using the user's known skin type information (e.g., smooth dominant type, light dominant type) associated with the facial image from which the skeletal information and skin color information were obtained as training data.

[0060] As a learning model, for example, a convolutional neural network (CNN), more specifically, VGG16 (a CNN consisting of 16 layers), is used, but this is not limited to this, and conventional machine learning models such as random forest, support vector machine, and logistic regression may also be used.

[0061] The learning process, including adjustment of the skeletal information, skin color information and other parameters, is repeatedly performed until the accuracy of skin type estimation by the learning model reaches a predetermined value or higher, thereby generating a first trained model 10.

[0062] Furthermore, if the user's skin type is already known, the skin type information may simply be received. The skin type may be classified by analyzing the user's biometric information as in the conventional method, or may be determined from a facial image as in the first trained model 10.

[0063] [Second Trained Model for Skin Score Determination] Next, the second trained model 20 for skin score determination will be described. As described above, the second trained model 20 is a learning model that has been trained to output, for example, information capable of determining the skin attributes or skin condition of a skin patch image in response to the input of the skin patch image (an index value (skin score) indicating the degree of characteristic of a characteristic item related to the skin attributes or skin condition).

[0064] The teacher skin image is an image used as teacher data and shows human skin. The skin shown in the teacher skin image may include skin that is the target of skin attribute or skin condition discrimination, such as the skin of a human face, the skin of a human neck, the skin of a human arm, the skin of the back or palm of a human hand, the skin of a human leg, etc. The teacher skin image may also show other body parts other than the skin that is the target of discrimination, such as the eyes, nostrils, mouth, etc.

[0065] The attributes of skin include, for example, attributes of the person who has the skin, such as age and gender, and the body part to which the skin belongs (cheeks, forehead, face, neck, legs, etc.). The skin condition includes, for example, a condition indicating whether the skin is bare or made-up, a condition indicating whether the makeup has come off, a condition indicating whether the skin is moisturized, a condition indicating whether the skin is dry, etc. The skin condition targeted in this embodiment may indicate the presence of local features such as blemishes or wrinkles, or may indicate the overall appearance impression.

[0066] The teacher skin patch image group is an image collection including multiple skin patch images extracted from one teacher skin image. A skin patch image is a small section image that contains a skin region (an image region showing skin) at a predetermined rate or more. The predetermined rate is at least 90%. The skin patch image has a predetermined shape, and is preferably rectangular for ease of processing, but the shape is not limited to circular, elliptical, or other shapes. The image size of the skin patch image is predetermined, and is set to, for example, 224 pixels x 224 pixels. However, the size of the skin patch image is not limited to this example and may be determined appropriately depending on the configuration of the discriminant model, etc.

[0067] The second trained model 20 is generated for each feature item that determines the skin score (index value). The feature items may be, for example, items related to visual (sensory) evaluation of skin. Examples of such feature items include index values ​​related to wrinkles, age spots, firmness, nasolabial folds, texture, moisture (moisture), dullness, pores, brightness, transparency, gloss, and smoothness, as well as their overall evaluation values. In addition to these, other feature items include bare skin feel (bare skin-likeness), makeup feel (makeup-likeness), male / female skin-likeness, skin age, makeup smudge level, powdery feel, etc.

[0068] Therefore, for each of these feature items, a learning process is performed to determine the skin score based on a group of teacher skin patch images with different evaluations and their feature information (for example, feature information obtained by actual measurement or visual inspection of each feature item converted into feature information on image data), and this process is repeated until the determination accuracy reaches a predetermined value or higher, thereby generating a second trained model 20.

[0069] [Operation of Skin Analysis Information Providing Server] Next, the operation of the skin analysis information providing server 100 configured as described above, i.e., the process of providing skin analysis information using the first trained model 10 and the second trained model 20, will be described. This operation is performed by hardware such as the CPU 11 and communication unit 19 of the skin analysis information providing server 100 in cooperation with software stored in the memory unit 18. For convenience, in the following description, the CPU 11 will be the subject of operation. Figure 4 is a flowchart showing the flow of the process of providing skin analysis information.

[0070] As shown in the figure, first, the CPU 11 determines whether or not a skin analysis request has been received together with a facial image from the user terminal 200 (step 41). The skin analysis request can be sent, for example, via a menu (such as a skin analysis request button) provided in an application installed on the user terminal 200.

[0071] If it is determined that the skin analysis request has been received (Yes in step 41), the CPU 11 analyzes the received face image and acquires the bone structure information and skin color information (step 42).

[0072] Next, the CPU 11 inputs the skeleton information and skin color information acquired from the face image into the first trained model 10 (step 43).

[0073] Next, the CPU 11 receives the skin type information output from the first trained model 10 (step 44) and stores the skin type information in the skin type information database 32 in association with the user ID of the user who has requested the skin analysis (step 45). If the user's skin type is known to the user, the CPU 11 may receive the user's skin type information along with the skin analysis request from the user terminal 200 without using the first trained model 10. If the user information database 31 contains the user's skin type information, the CPU 11 only needs to obtain, from the user information database 31, the skin type information corresponding to the user ID of the user who has requested the skin analysis.

[0074] Next, the CPU 11 acquires a skin patch image from the facial image (step 46) and inputs the skin patch image for each feature item into the second trained model 20 (step 47). Note that the facial image input into the second trained model 20 may be different from the facial image input into the first trained model 10 as long as the user is the same.

[0075] Next, the CPU 11 receives the skin score information output from the second trained model 20 for each of the above feature items (step 48).

[0076] Next, when the CPU 11 receives the skin score information for all target feature items, it tallies the skin scores (step 49). The skin score information is stored in the skin score information database 33.

[0077] Next, the CPU 11 acquires the dominant feature items of the skin type received from the first trained model 10 from the skin type information database 32, and determines whether or not there is any item among the dominant feature items received from the second trained model 20 whose skin score is less than a predetermined threshold (step 50). Here, the predetermined threshold is, for example, when the skin score is expressed as a value from 0 to 100, a value that is a predetermined value (e.g., 10 points) lower than the average value of the skin scores of many users for the dominant feature item, but is not limited to this.

[0078] If it is determined that there is a superior feature item whose skin score is less than the threshold value (Yes in step 50), the CPU 11 generates potential item information for the superior feature item (step 51). The potential item information is information that notifies the user that the feature item has the potential to increase its skin score in the future. If there are multiple superior feature items whose skin score is less than the threshold value, multiple pieces of potential item information are also generated.

[0079] Next, CPU 11 generates skin analysis information displaying the skin analysis results and transmits it to user terminal 200 (step 52). That is, based on the above-mentioned aggregation results, CPU 11 selects the top predetermined number (e.g., three) of characteristic items from the skin scores of each characteristic item of the user who has requested the skin analysis, generates top item information together with explanatory information corresponding to the scores of those characteristic items stored in skin score information database 33, and, if there is information about the potential items, also includes this as skin analysis information and transmits it to user terminal 200.

[0080] 5 is a diagram showing an example of the skin analysis information sent to and displayed on the user terminal 200. As shown in the figure, the skin analysis information includes the user's name and skin type information, as well as the above-mentioned higher-level item information 61, potential item information 62, and advice information 63.

[0081] The top item information 61 includes the top three feature items with the highest skin scores, their scores, and explanatory information. If the top item information 61 includes a dominant feature item of the user's skin type, the explanatory information also includes an explanation to that effect. In the example shown in the same figure, three items, "wrinkles," "blemishes," and "firmness," are extracted as the top item information 61. Of these, "wrinkles" and "firmness" are dominant feature items for the user's skin type "smooth dominant."

[0082] The potential item information 62 includes the dominant feature items and their scores that are less than the threshold, as well as explanatory information. The explanatory information includes information indicating that the feature items have the potential to increase the skin score in the future for the user's skin type. In the example shown in the same figure, "laugh lines" is extracted as the potential item information 62.

[0083] The advice information 63 is information summarizing the user's skin analysis results, including the above-mentioned upper item information 61 and potential item information 62, and can be automatically generated, for example, by inputting the above-mentioned upper item information 61 and potential item information 62 into a specified template.

[0084] The skin analysis information may further include a cosmetics recommendation information display button 64 that is set with a link to information on recommended cosmetics corresponding to the potential item information 62. In the example shown in the figure, the cosmetics recommendation information display button 64 is displayed with a link to information on cosmetics recommended for the "laugh lines." Pressing the cosmetics recommendation information display button 64 on the user terminal 200 displays the cosmetics recommendation information.

[0085] 6 is a diagram showing an example of cosmetic recommendation information generated by the skin analysis information providing server 100 and displayed on the user terminal 200. As shown in the figure, the cosmetic recommendation information includes cosmetic information 65 and a purchase button 66.

[0086] The cosmetic information includes information such as the name, manufacturer, price, and photo of the cosmetic product, as well as information indicating the user's skin type and the evaluation value of the cosmetic product for that skin type based on reviews and other information from other users. As described above, the cosmetic product information database 34 stores each characteristic item in association with the efficacy tag information of each cosmetic product. The CPU 11 extracts, from the cosmetic product information database 34, cosmetic product information that corresponds to the user's skin type, has an evaluation value equal to or greater than a predetermined threshold (e.g., 80 / 100 points or more), and corresponds to the potential item information 62, thereby generating cosmetic recommendation information. The evaluation information used to extract the cosmetic products is not limited to the evaluation value (numerical value). For example, if multiple users' evaluations of the cosmetic product are classified into high evaluations (positive evaluations) and low evaluations (negative evaluations), cosmetics that have a predetermined percentage (e.g., 80%) or more of the former may be extracted.

[0087] In the example shown in the figure, information on cosmetics that are highly rated among users of the user's "smooth" skin type and have the effect of improving "laugh lines," a characteristic that is a dominant feature of the skin type but has a low skin score, is displayed as cosmetic recommendation information. When the purchase button 66 is pressed on the user terminal 200, a purchase page for the cosmetics is displayed, for example, on an external EC site. This allows the user to easily identify and purchase products that are highly rated for their skin type and that are intended to improve characteristic features of their skin type that have room for improvement.

[0088] When there are multiple pieces of potential item information 62, multiple pieces of cosmetic recommendation information may be generated correspondingly, or cosmetic recommendation information may be generated only for the potential item with the lowest skin score or for only the potential items with the lowest predetermined number of skin scores (e.g., two) of potential items among the multiple pieces of potential item information 62. Furthermore, even when multiple pieces of cosmetic recommendation information are generated according to multiple pieces of potential item information 62, recommendation priorities may be set according to the skin scores, such as displaying the cosmetic information corresponding to the lowest skin scores.

[0089] As described above, according to this embodiment, the skin analysis information providing server 100 can allow the user to understand which feature items have the potential for improvement among the feature items analyzed as being poor in the user's skin.

[0090] [Modifications] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the gist of the present invention.

[0091] In the above-described embodiment, RNA information was used as the biological information, but other information such as DNA information, blood information, proteomes (proteins) collected from the user's stratum corneum or sebum, etc. may also be analyzed, or epidermal bacteria collected from the skin surface may also be analyzed.

[0092] In the above-described embodiment, the skin analysis information providing server 100 includes, as potential item information, information on dominant feature items whose skin scores are less than a predetermined threshold (e.g., a value that is a predetermined value (e.g., 10 points) lower than the average skin score) in the skin analysis information and transmits this information to the user terminal 200. The skin analysis information providing server 100 may also include, as potential item information, information on the dominant feature item with the lowest skin score or a predetermined number of subordinate dominant feature items (e.g., two) regardless of the average value.

[0093] In the above-described embodiment, the skin analysis information providing server 100 includes the top three skin scores in the skin analysis information and transmits it to the user terminal 200. However, it is sufficient that the skin analysis information includes at least the potential item information 62, and the skin analysis information providing server 100 does not need to include the top item information 61 in the skin analysis information.

[0094] In the above-described embodiment, a link to cosmetic recommendation information that recommends cosmetics that have the effect of improving potential item information 62 was provided in the skin analysis information, and the cosmetic recommendation information was displayed based on this link. However, the generation of this cosmetic recommendation information is not required, and the processing may be completed by simply presenting the skin analysis information.

[0095] In the above-described embodiment, the user's skin type information and skin score information were obtained as skin-related information related to the user's skin condition by analyzing a facial image using the first trained model 10 and the second trained model 20. However, the skin-related information is not limited to facial images, and the skin type information and skin score information may be obtained by analyzing skin-related information other than facial images. For example, the skin type may be determined as in the past from biological information obtained from the above-described biological tissues (sebum, stratum corneum, saliva, urine, blood, etc.), or the skin type information or skin score information may be obtained using a trained model trained to estimate the skin type or skin score based on various measurements of the user's skin, physical information (height, weight, age, sex, BMI, blood type, etc.), and questionnaire information regarding the user's lifestyle, eating habits, etc.

[0096] It is preferable that the skin type information be classified by analyzing the relationship between information that does not directly correspond to skin condition, such as biometric information, and information on skin physical properties. The information on skin physical properties may be determined from a facial image or various measured values.

[0097] In the above-described embodiment, the first trained model 10 was generated based on skeletal information and skin color information acquired from a face image, but it may also be generated by learning to estimate a skin type using only one of skeletal information and skin color information. Furthermore, the first trained model 10 may be a model that estimates a skin type by taking into account skin surface information (information indicating the state of desquamation, scales, wrinkles, texture, pores, etc.) as a parameter in addition to or instead of skeletal information and skin color information.

[0098] In the above-described embodiment, the second trained model 20 was trained to determine the skin score from a face patch image extracted from a face image, but it may also be trained to determine the face score by learning the features of the entire face image.

[0099] In the above-described embodiment, cosmetics (including external preparations such as quasi-drugs) are used as examples of products. However, products are not limited to cosmetics, and information on other products that have some effect of improving the user's skin characteristics, such as hair care products such as shampoo, conditioner, and treatment, medicines (skin preparations, etc.), sweat wipes, cosmetic sheets, toothpaste, beverages, foods, facial massagers, beauty tools, etc., may be provided in accordance with the skin type information generated based on the user's facial image.

[0100] In the above embodiment, only one skin analysis information providing server 100 is shown, but the processing executed by the skin analysis information providing server 100 may be distributed and executed by multiple servers. For example, there may be a server that acquires skin type information from the first trained model 10 and a server that acquires skin score information from the second trained model 20, and there may be a server that executes the skin analysis result information providing processing using the skin type information and skin score information shown in FIG. 4 separately from these.

[0101] Among the inventions described in the claims of this application, the invention described as an "information processing method" is one in which each step is automatically performed by at least one device such as a computer through information processing by software, and is not performed by a human using a device such as a computer. In other words, the "information processing method" is an information processing method using computer software, and is not a method in which a human operates a calculation tool called a computer.

[0102] The functions performed by the components (such as memory and controller) described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in memory.

[0103] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0104] If the hardware is a processor that is considered to be a type of circuitry, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or processor. Also, configurations described as units or means in this specification can be implemented as circuitry.

Claims

1. An information processing system comprising: a memory unit that stores information indicating a plurality of skin types and, among a plurality of feature items indicating skin characteristics, information indicating dominant feature items whose degree of the feature tends to be statistically higher in skin of each of the skin types, in association with each other; and a control unit that acquires information about the skin type of a user of a user terminal and analyzes skin-related information related to the user's skin condition to determine the degree of the dominant feature item corresponding to the skin type, and transmits information about the dominant feature item whose degree is less than a predetermined threshold or has the lowest degree to the user terminal as potential item information.

2. The information processing system of claim 1, wherein the control unit classifies the user into the skin type and / or determines the degree of the dominant feature items by analyzing a facial image of the user's face received from the user terminal as the user's skin-related information.

3. The information processing system of claim 2, wherein the control unit acquires information regarding the skin type by classifying the user's skin type using a first trained model trained to estimate the skin type from the facial image.

4. The information processing system according to claim 2, wherein the control unit determines the degree of the dominant feature item using a second trained model trained to estimate the degree of characteristic of each feature item from the facial image.

5. The information processing system of claim 1, wherein the control unit transmits to the user terminal analysis result information that displays at least the feature item with the highest degree among the feature items of the user's skin and information indicating that degree, together with the potential item information and information indicating that degree.

6. The information processing system of claim 1, wherein the memory unit stores product information about products that have the effect of improving the characteristic items in association with the characteristic items, and stores evaluation information indicating user evaluations of the products for each skin type in association with the skin type, and the control unit transmits to the user terminal product information that corresponds to the classified skin types, has the predetermined evaluation information, and corresponds to the potential item information.

7. An information processing method that associates and stores information indicating multiple skin types with information indicating dominant feature items among multiple feature items indicating skin characteristics, the degree of which tends to be statistically higher for skin of each of the skin types; acquires information about the skin type of a user of a user terminal and analyzes skin-related information related to the user's skin condition to determine the degree of the dominant feature item corresponding to the skin type; and transmits information about the dominant feature item whose degree is less than a predetermined threshold or has the lowest degree to the user terminal as potential item information.

8. A program that causes an information processing device to execute the following steps: storing information indicating multiple skin types and, among multiple feature items indicating skin characteristics, information indicating dominant feature items whose degrees tend to be statistically higher in skin of each of the skin types, in association with each other; acquiring information about the skin type of the user of the user terminal and analyzing skin-related information related to the user's skin condition to determine the degrees of the dominant feature items corresponding to the skin type; and transmitting information about dominant feature items whose degrees are less than a predetermined threshold or have the lowest degrees, among the determined dominant feature items, to the user terminal as potential item information.

Images