Information processing device, program, and method of operating information processing device

Abstract

To support sale of cosmetics by precisely reproducing the evaluation of impressions by a skilled evaluator.SOLUTION: The information processor includes: a storage unit for storing a model that has mechanically learned the correspondence between a feature part of a facial image of a person that contributes to an impression evaluation and the impression evaluation, by using the facial image and the impression evaluation of the facial image as training data; and a control unit for determining and outputting the feature part of the input facial image and the impression evaluation about the feature part by using the model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a program, and an operation method of an information processing device. [Background technology]

[0002] In the sale of cosmetics, there is a demand for customization of beauty treatments according to diversifying customer needs. For example, for a customer who wishes to improve their skin condition, a salesperson will provide counseling by asking about the customer's age, skin type, skin concerns, etc. using a questionnaire, and measuring the physical characteristics of the skin using equipment, and then recommend products suitable for the customer's skin type. As a technology to support this process, for example, Patent Document 1 proposes an information processing device that estimates a customer's skin age from a facial image captured of the customer's face. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-512797 Summary of the Invention [Problem to be solved by the invention]

[0004] To further enhance customer satisfaction, it is desirable to objectively evaluate the overall impression of a customer's appearance on others and provide cosmetic procedures to improve that impression. While skin type can be quantitatively measured using conventional methods, such as measuring physical skin characteristics or estimating skin age through image recognition of those characteristics, it is only one of several factors that influence impression. Many of the factors that influence impression, such as youthfulness and vitality, are difficult to measure quantitatively and are instead grasped intuitively. Objective evaluation of these non-quantitative, sensory factors relies on the personal skills of an experienced beauty technician. Therefore, to objectively evaluate impression assessments, it is necessary to ensure highly accurate reproducibility of impression assessments by experienced evaluators.

[0005] In view of the above, the following describes an information processing device and the like that can reproduce impression evaluations by skilled evaluators with high accuracy and support the sale of cosmetics. [Means for solving the problem]

[0006] The information processing device of the present disclosure includes a memory unit that stores a model that uses a person's facial image and an impression evaluation of the facial image as training data to machine-learn the correspondence between the characteristic parts of the facial image that contribute to the impression evaluation and the impression evaluation, and a control unit that uses the model to find and output the characteristic parts of an input facial image and the impression evaluation corresponding to the characteristic parts.

[0007] In addition, the operating method of the information processing device in the present disclosure includes a first step of using a person's facial image and an impression evaluation for the facial image as training data to machine-learn the correspondence between the characteristic parts of the facial image that contribute to the impression evaluation and the impression evaluation to create a model, and a second step of using the model to determine and output the characteristic parts of an input facial image and the impression evaluation corresponding to the characteristic parts. [Effects of the Invention]

[0008] According to the information processing device and the like of the present disclosure, it is possible to support the sale of cosmetics by reproducing with high accuracy impression evaluations by skilled evaluators. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 1 illustrates an example of the configuration of an information processing device. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a terminal device. [Figure 4] FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. [Figure 5] FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. [Figure 6]FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. [Figure 7A] FIG. 10 is a diagram illustrating an example of information to which the information processing system refers. [Figure 7B] FIG. 10 is a diagram illustrating an example of information to which the information processing system refers. [Figure 8A] FIG. 10 is a diagram illustrating processing of a face image. [Figure 8B] FIG. 10 is a diagram illustrating processing of a face image. [Figure 8C] FIG. 10 is a diagram illustrating processing of a face image. [Figure 8D] FIG. 10 is a diagram illustrating processing of a face image. [Figure 9] FIG. 10 is a diagram illustrating information on recommended cosmetics. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described.

[0011] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. The information processing system 1 supports, for example, a salesperson at a cosmetics store providing counseling on beauty treatments to customers. The information processing system 1 includes a terminal device 11 and an information processing device 12. The terminal device 11 and the information processing device 12 are connected to each other peer-to-peer via a network 10 or, without the network 10, via wired or short-range wireless communication, and transmit and receive information. The network 10 includes the Internet, at least one wide area network (WAN), at least one metropolitan area network (MAN), or any combination thereof. The network 10 may also include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may be, for example, an ad hoc network, a cellular network, a wireless local area network (LAN), a satellite communication network, or a terrestrial microwave network. The terminal device 11 is an information processing terminal device equipped with an imaging function, such as a tablet terminal or a smartphone, and used by a cosmetics salesperson. The information processing device 12 is, for example, a computer such as a server that belongs to a cloud computing system or other computing system.

[0012] The information processing device 12 stores a model (hereinafter referred to as the impression evaluation model) that has been machine-learned in advance to associate the characteristic parts of a facial image that contribute to the impression evaluation with the impression evaluation, using as training data facial images obtained by capturing a person's face and impression evaluations of the facial images by experienced evaluators. The impression evaluation is, for example, an evaluation of the degree of satisfaction of elements such as friendliness, youthfulness, cuteness, cheerfulness, sociability, and liveliness. The impression evaluation model obtained by machine learning of such training data reflects the impression evaluations by the experienced evaluators. In a sales store, a salesperson assigned to the store captures an image of a customer's face using a terminal device 11. The information processing device 12 receives an input of the customer's facial image from the terminal device 11, calculates the characteristic parts of the input facial image and the impression evaluations corresponding to the characteristic parts using the impression evaluation model, and outputs the calculated impression evaluations to the terminal device 11. The salesperson outputs the impression evaluations and the characteristic parts on the terminal device 11 by displaying them or the like, and presents them to the customer. By using the impression evaluation model in the impression evaluation of a customer's facial image, the information processing device 12 can reproduce with high accuracy the impression evaluation by an experienced evaluator. In addition, by presenting the characteristic parts that contribute to the impression evaluation, it is possible to improve the customer's sense of satisfaction with the impression evaluation.

[0013] The information processing device 12 also stores recommendation information indicating cosmetics to be recommended for combinations of characteristic parts associated in the impression evaluation model and the impression evaluation. The recommendation information includes, for example, wrinkle-reducing skin care cosmetics, eyebrow cosmetics to tone the eyes, skin care cosmetics to reduce sagging, effects required for impression improvement, and product group information. The information processing device 12 then selects cosmetics to be recommended for the combination of characteristic parts and impression evaluation determined using the impression evaluation model based on the recommendation information, and further outputs information indicating the selected cosmetics. The salesperson receives the information output by the information processing device 12 via the terminal device 11 and presents it to the customer. This makes it possible to recommend cosmetics to customers that reflect the impression evaluations of experienced evaluators.

[0014] 2 shows an example of the configuration of the information processing device 12. The information processing device 12 is, for example, a server computer that functions as a server that implements various functions. The information processing device 12 may be one or more server computers that are connected to each other so that they can communicate information with each other and operate in cooperation with each other. The information processing device 12 has an input / output unit 20, a communication unit 21, a storage unit 22, and a control unit 23.

[0015] The input / output unit 20 has an input interface that detects user input and sends the input information to the control unit 23. The input interface is any input interface including, for example, physical keys, capacitance keys, a touch screen integrated with a panel display, various pointing devices, etc. The input / output unit 20 also has an output interface that outputs information generated by the control unit 23 or read from the storage unit 22 to the user. Such an output interface is any output interface including, for example, a display that outputs information as an image or video, a speaker that outputs information as sound, or an interface for connecting to an external output device.

[0016] The communication unit 21 has a communication module compatible with one or more wired or wireless LAN standards for connecting to the network 10. Alternatively, the communication unit 21 may have a communication module compatible with one or more wired or wireless standards for connecting peer-to-peer with the terminal device 11. Such a communication module may include a communication module compatible with short-range communication such as Bluetooth, AirDrop, IrDA, ZigBee, FeliCa, RFID, etc.

[0017] The storage unit 22 includes one or more memories. Each memory included in the storage unit 22 is, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these. Each memory functions, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 22 stores any information and control / processing programs used in the operation of the information processing device 12. The storage unit 22 may also store application programs that provide various functions and are downloaded via the network 10 or other networks.

[0018] The control unit 23 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a specific process. The control unit 23 comprehensively controls the operation of the information processing device 12 in accordance with a control / processing program stored in the storage unit 22.

[0019] In the information processing device 12, the storage unit 22 stores an impression evaluation model 24 created by the control unit 23 using a procedure described below. The storage unit 22 also stores recommendation information 25 including information on cosmetics to be recommended for combinations of characteristic parts and impression evaluations associated in the impression evaluation model 24, and standard information 26 regarding criteria for the impression evaluation.

[0020] 3 shows an example of the configuration of the terminal device 11. The terminal device 11 is an information processing terminal device such as a tablet terminal or a smartphone. The terminal device 11 includes an input / output unit 30, a communication unit 31, a storage unit 32, and a control unit 33.

[0021] The input / output unit 30 has an input interface that detects user input and sends the input information to the control unit 33. Such an input interface may be any input interface including, for example, physical keys, capacitive keys, a touch screen integrated with a panel display, various pointing devices, a microphone that accepts audio input, a camera that captures captured images or image codes, etc. The input / output unit 30 also has an output interface that outputs information generated by the control unit 33 or read from the storage unit 32 to the user. Such an output interface may be, for example, a display that outputs information as an image or video, a speaker that outputs information as audio, or any output interface including a connection interface with an external output device.

[0022] The communication unit 31 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as 4G (4th Generation) and 5G (5th Generation), and a communication module compatible with short-range communication such as Bluetooth, AirDrop, IrDA, ZigBee, FeliCa, RFID, etc. The terminal device 11 communicates information with the information processing device 12 via the network 10 or peer-to-peer using the communication unit 31.

[0023] The storage unit 32 includes one or more memories. Each memory included in the storage unit 32 may be, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these. Each memory functions as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 32 stores any information used in the operation of the terminal device 11. For example, the storage unit 32 stores control and processing programs, embedded software, application programs that provide various functions and are downloaded via the network 10 or other networks, and the like.

[0024] The control unit 33 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a specific process. The control unit 33 operates in accordance with a control / processing program stored in the storage unit 32, thereby providing overall control of the operation of the terminal device 11.

[0025] In the terminal device 11, the control unit 33 responds to an operation input by the salesperson to the input / output unit 30 by causing the camera of the input / output unit 30 to capture an image of the customer's face, and sends the captured facial image to the information processing device 12 via the communication unit 31. The control unit 33 also receives information sent from the information processing device 12 via the communication unit 31 and displays it on the display of the input / output unit 30.

[0026] 4 is a flowchart showing an example of the operation procedure of the information processing device 12 when the information processing device 12 creates the impression evaluation model 24 by machine learning. This flowchart corresponds to the information processing procedure by the control unit 23.

[0027] In step S400, the control unit 23 reads the training data and stores it in the storage unit 22. The training data includes facial images obtained by capturing images of people's faces with a digital camera and impression evaluations assigned to the facial images by experienced evaluators. Hereinafter, the facial images included in the training data are referred to as training facial images. The training facial images are obtained, for example, by capturing images of the faces of several hundred Japanese women, approximately uniformly distributed between their 20s and 70s, from the front. Each person is photographed wearing a hair net to maintain a uniform hairstyle, without makeup, and with a straight face. The impression evaluation is a discrete value (hereinafter referred to as an evaluation value) assigned to each training facial image for each given element. The evaluation value is assigned by the experienced evaluator's visual evaluation of the displayed facial image, eliminating the influence of the facial expression, personality, and preferences of the evaluator of the person in the training facial image. The elements include, for example, friendliness, youthfulness, cuteness, cheerfulness, sociability, and liveliness. The higher the degree to which each element is satisfied, the better the evaluation. The better the evaluation, the lower the evaluation value, which is assigned as a discrete value on a scale of, for example, 7 levels from 1 to 7. The dynamic range of the discrete values ​​may be set arbitrarily according to the range within which the evaluator can classify. Here, learning of evaluation values ​​using a classification model is used as an example, but a regression model may also be used.

[0028] In step S402, the control unit 23 performs image processing on each training face image as a preliminary step for machine learning. This image processing includes, for example, normalization of the training face image, extraction of facial features from the training face image, random cropping, random horizontal and vertical flipping, random rotation, random saturation adjustment, vertical dropout, and the like. In the facial feature extraction, facial features can be extracted using any open source library to eliminate the influence of features such as hair and neck in the training face image, or facial features can be extracted by specifying the vertices of a bounding box. In addition, in the random cropping, for example, a training face image from 512 × 512 pixels is resized to 256 × 256 pixels, and then randomly cropped to 224 × 224 pixels.

[0029] In step S404, the control unit 23 creates the impression evaluation model 24 through machine learning using training data. The control unit 23 performs machine learning using, for example, a convolutional neural network. For example, the control unit 23 extracts feature parts from the training face images using a convolutional layer and a pooling layer, and compares the extracted feature parts with impression evaluations using a multi-layer neural network, thereby learning to associate the feature parts that contribute to the impression evaluations assigned to the training face images with the impression evaluations, and creates the impression evaluation model 24.

[0030] 5 is a flowchart showing an example of the operation procedure of the information processing device 12 when the information processing device 12 performs impression evaluation of a customer's facial image using the impression evaluation model 24. This flowchart corresponds to the information processing procedure by the control unit 23.

[0031] In step S500, the control unit 23 accepts input of a facial image and customer information. For example, a salesperson at a store takes an image of the face of a customer who visits the store using the terminal device 11, and inputs the customer's age, gender, and other customer information into the terminal device 11. The terminal device 11 sends the captured image and customer information to the information processing device 12. In the information processing device 12, the control unit 23 accepts the captured image and customer information via the communication unit 21. The facial image and customer information are stored in the memory unit 22. Hereinafter, the facial image that the control unit 23 accepts as input and for which an impression evaluation is to be obtained is referred to as an evaluation facial image.

[0032] In step S502, the control unit 23 performs image processing on the evaluation face image as a preliminary step for obtaining an impression evaluation. The image processing includes normalization of the evaluation face image, extraction of facial features from the evaluation face image, random cropping, random flipping in horizontal and vertical directions, random rotation, random saturation adjustment, vertical dropout, etc.

[0033] In step S504, the control unit 23 obtains an impression evaluation of the evaluation face image using the impression evaluation model 24. The control unit 23 obtains, as the impression evaluation, the class of the evaluation value corresponding to the maximum value in the final layer output of the impression evaluation model 24. The evaluation value is assigned as a discrete value from 1 to 7.

[0034] In step S506, the control unit 23 uses the impression evaluation model 24 to find feature parts that contribute to the impression evaluation of the evaluation face image. The feature parts that contribute to the impression evaluation are represented, for example, as a heat map. The control unit 23 creates a heat map indicating the feature parts via a convolutional neural network. The control unit 23 converts the maximum output value in the final output layer into a probabilistic expression using a numerical value between 0 and 1 using an activation function such as a sigmoid function or a softmax function. The control unit 23 calculates the gradient with respect to the maximum output value of the final output layer, i.e., the partial differential value, for each pixel of the feature map, which consists of 512 channels in the final convolutional layer. The control unit 23 then calculates the average value of the gradient for each pixel and creates an activation map for 512 channels by multiplying each pixel of the feature map by this average value. The number of channels in the feature map shown here is an example, and the number of channels may be other than 512. The control unit 23 then integrates the created activation map across all channels and scales the integrated activation factors to values ​​between 0 and 1 to generate a two-dimensional activation map, i.e., a heat map. The control unit 23 then resizes the heat map to the size of the face image for evaluation to generate a heat map image corresponding to the face image for evaluation. The control unit 23 may also add the heat map image to the face image for evaluation and scale it with a gradation value between 0 and 255 to generate a display image in which the heat map is superimposed on the face image for evaluation. By superimposing the heat map on the face image for evaluation, characteristic portions in the face image for display are displayed with a predetermined color distribution that is arbitrarily set. Note that when scaling the size of the heat map, the control unit 23 may eliminate abnormal values ​​using an arbitrarily set threshold value to remove noise from the heat map.

[0035] In step S508, the control unit 23 selects recommended cosmetics based on the impression evaluation and the characteristic features. Since a smaller evaluation value corresponds to a better impression evaluation, characteristic features that contribute to a relatively low evaluation value are considered to indicate parts that contribute to a good impression, and characteristic features that contribute to a relatively high evaluation value are considered to indicate parts that contribute to a bad impression. Therefore, the control unit 23 identifies parts of the face that contribute to a bad impression based on the impression evaluation and the characteristic features, and selects recommended cosmetics to improve the impression evaluation according to the characteristics of those parts. The control unit 23 then outputs information on the recommended cosmetics together with the impression evaluation and the characteristic features.

[0036] A specific example of step S508 will now be described with reference to Figures 6 to 9. Figure 6 is a flowchart showing the subroutine of step S508. Here, using the degree of liveliness as an example of an impression element, the procedure is shown for the case where the impression evaluation for the liveliness is expressed as an evaluation value on a seven-level scale from 1 to 7.

[0037] In step S600, the control unit 23 determines whether the liveliness evaluation value is equal to or greater than a reference value of 3. If the evaluation value is equal to or greater than 3 (Yes), the control unit 23 proceeds to step S602. On the other hand, if the evaluation value is less than 3 (No), the control unit 23 proceeds to step S604 and determines that the liveliness evaluation value belongs to the "sufficient liveliness group." Note that the reference value can be adjusted as appropriate and may be other than 3.

[0038] In step S602, the control unit 23 determines whether the evaluation value of the liveliness is higher than the average for the customer's age. The control unit 23 compares the customer's age included in the customer information received in step S500 with the judgment criteria included in the reference information 26 stored in the storage unit 22. For example, the reference information 26 stores the average evaluation values ​​for each age group of people who were subjects of the learning face images as judgment criteria for impression evaluation, and this is used for comparison with the evaluation value.

[0039] FIG. 7A schematically illustrates reference information 26. It shows the average liveliness evaluation scores for each age group from teens to 70s. In addition to liveliness, reference information 26 also includes age-group average evaluation scores for each impression element. The criteria included in reference information 26 may be other than age-group averages, such as medians. As shown in the distribution of evaluation scores in the age group of FIG. 7B, the same evaluation value tends to span multiple age groups. In other words, compared to features that have a strong correlation with a person's age, such as skin age, elements that have a low correlation with age tend to contribute significantly to impression evaluation.

[0040] Returning to Figure 6, if the evaluation value is higher than the age average (Yes in step S602), that is, if the impression evaluation is not better than the age average, the control unit 23 proceeds to step S606 and determines that the liveliness evaluation value belongs to the "liveliness 'needs improvement' group." If the evaluation value is lower than the age average (No in step S602), that is, if the impression evaluation is at or better than the age average, the control unit 23 proceeds to step S608 and determines that the liveliness evaluation value belongs to the "liveliness 'slightly lacking' group."

[0041] When determining the attribution of the evaluation value in step S604, S606, or S608, the control unit 23 calculates the feature parts that contributed to the evaluation value and their contribution levels in step S610. For example, the control unit 23 divides the evaluation image into any number of regions. For example, FIG. 8A shows an example of four regions in an evaluation face image 80: a forehead region 80A, an eye region 80B, a cheek region 80C, and a mouth region 80D. For each region, the control unit 23 calculates the sum of the activity elements in the heat map for that region (e.g., the number of pixels) as the feature activity level of that region. Here, a feature activity value ranging from 0 to 1.0 is calculated as the contribution level. The control unit 23 then generates a display image that displays the feature parts as color-coded regions in the heat map and the contribution levels as the area of ​​the color-coded regions, and outputs the display image. The display image is sent to the terminal device 11 by the communication unit 21 and displayed on the terminal device 11.

[0042] 8B to 8D are schematic diagrams illustrating examples of display images generated by the control unit 23 of the information processing device 12 and displayed on the terminal device 11. FIGS. 8B, 8C, and 8D illustrate examples for liveliness evaluation scores of "2," "3," and "6," respectively. Here, the average evaluation score for each age group is assumed to be between "3" and "6." The example in FIG. 8B corresponds to the "sufficient liveliness group," the example in FIG. 8C corresponds to the "slightly insufficient liveliness group," and the example in FIG. 8D corresponds to the "needs improvement liveliness group." In FIG. 8B, a wide range of characteristic features 82 are shown in the cheek region 80C of the evaluation face image 80. This indicates that the features in the cheek region 80C contribute significantly to a good liveliness score (evaluation score of "2"). Furthermore, in FIG. 8C, characteristic features 83 are shown in the forehead region 80A and the mouth region 80D of the evaluation face image 80. That is, it is shown that the characteristic portions 83 in the forehead region 80A and the mouth region 80D make a high contribution to a medium level of liveliness (evaluation value "3"). Furthermore, in FIG. 8D, characteristic portions 84 are shown in the forehead region 80A, the cheek region 80C, and the mouth region 80D in the evaluation face image 80. That is, it is shown that the characteristic portions 84 in the forehead region 80A, the cheek region 80C, and the mouth region 80D make a high contribution to a poor level of liveliness (evaluation value "6").

[0043] Returning to FIG. 6, in steps S612, S614, and S616, the control unit 23 identifies areas of the face that contribute to a poor impression based on the evaluation, the characteristic parts, and the contribution of the characteristic parts, and selects recommended cosmetics to improve the impression evaluation according to the characteristics of those parts. If the evaluation value of the liveliness belongs to the "sufficient liveliness group" (step S604), the control unit 23 determines in step S612 the region with the highest contribution of the characteristic part as the "good region." For example, in the example shown in FIG. 8B, the cheek region 80C is determined to be the good region. On the other hand, if the evaluation value of the liveliness belongs to the "slightly insufficient liveliness group" (step S608), the control unit 23 determines in step S614 the region with a high contribution of the characteristic part as the "good region," and the region with a low contribution of the characteristic part as the "slightly insufficient region." For example, in the example shown in FIG. 8C, the mouth region 80D is determined to be the "good region," and the forehead regions 80B and 80C are determined to be the "slightly insufficient regions." For example, the control unit 23 determines whether the area is a "good area" or a "slightly deficient area" depending on the feature activity of the feature part. For example, it is possible to determine that a feature activity of 0 to 0.3 is a "slightly deficient area", and that a feature activity of 0.7 to 1.0 is an "good area". There may be three or more types of determinations depending on the feature activity, and the numerical range serving as the basis for the determination can be set arbitrarily. Then, if the liveliness evaluation value belongs to the "liveliness 'needs improvement' group" (step S606), the control unit 23 determines in step S616 that the area with the highest contribution of the feature part is a "area needing improvement". For example, in the example shown in FIG. 8D, the forehead area 80A is determined to be an area needing improvement.

[0044] In step S618, the control unit 23 selects recommended cosmetics for the improvement-needing areas and the slightly deficient areas by referring to the recommendation information 25, and displays information about the selected recommended cosmetics.

[0045] FIG. 9 schematically illustrates an example of recommendation information 25 including information on recommended cosmetics for improving liveliness. The recommendation information 25 includes information for each impression element, as shown in FIG. 9. The control unit 23 references the recommendation information 25 to select recommended cosmetics for a "slightly deficient area" (evaluation value of 3.0 or more and less than the age average) or a "area requiring improvement" (evaluation value of the age average or more and 7.0 or less). For example, if the forehead area 80A is determined to be a "slightly deficient area," skin care cosmetics for improving wrinkles between the eyebrows are selected for skin care for the forehead area 80A. If the forehead area 80A is determined to be an "area requiring improvement," cosmetics for highlighting the temples are selected for makeup for the forehead area 80A. If the eye area 80B is determined to be a "slightly deficient area," eye makeup cosmetics for emphasizing the eyes are selected for makeup for the eye area 80B. Furthermore, if the eye area 80B is determined to be an "area requiring improvement," eyebrow cosmetics to shape the eyes are selected for makeup application to the eye area 80B. Furthermore, if the cheek area 80C is determined to be an "area requiring slight improvement," cheek color cosmetics are selected for makeup application to the cheek area 80C. Furthermore, if the cheek area 80C is determined to be an "area requiring improvement," cheek skin care cosmetics to improve sagging are selected for skin care of the cheek area 80C. Furthermore, if the mouth area 80D is determined to be an "area requiring slight improvement," skin care cosmetics to improve sagging around the lips are selected for skin care of the mouth area 80D. Furthermore, if the mouth area 80D is determined to be an "area requiring improvement," skin care cosmetics to improve sagging around the chin are selected for skin care of the mouth area 80D. Information on the cosmetics selected in this manner is sent to the terminal device 11 by the communication unit 21 and displayed on the terminal device 11.

[0046] On the terminal device 11, for example, a lively feeling assessment result such as sufficient, slightly insufficient, or requiring improvement is displayed together with a display image showing the characteristic parts that contributed to the assessment result, and further, information on recommended cosmetics for skin care or makeup is displayed. The information on the recommended cosmetics may be the type of cosmetics or the specific name of the cosmetics.

[0047] In Figures 6 to 9, the impression evaluation of liveliness has been explained as an example, but the same procedure as above is also performed for other impression elements, such as friendliness, youthfulness, cuteness, cheerfulness, and sociability.

[0048] According to this embodiment, it is possible to reproduce with high accuracy the objective impression evaluation of an experienced evaluator even for non-quantitative impression elements. Furthermore, it is possible to visualize the characteristic parts that contributed to the impression evaluation, thereby improving the customer's sense of satisfaction with the impression evaluation. Furthermore, the information processing device 12 that reproduces the impression evaluation of an experienced evaluator can be used to train unskilled evaluators in impression evaluation.

[0049] In the above description, the information processing device 12 acquires facial images and displays various output information via the terminal device 11. However, the information processing procedures shown as the operation of the information processing device 12 may be shared between the information processing device 12 and the terminal device 11 as appropriate. In this case, the information processing device 12 and the terminal device 11 constitute an "information processing device." For example, the information processing device 12 may execute the machine learning procedure shown in FIG. 4 to create the impression evaluation model 24, and the terminal device 11 may execute the procedure shown in FIG. 5 using the impression evaluation model 24 of the information processing device 12. Alternatively, for example, the information processing device 12 may operate standalone and acquire facial images and display various output information using the input / output unit 20. Furthermore, the terminal device 11 does not have to be installed in a store; for example, it may be a smartphone used by a customer. In this case, the customer captures an image of their face and sends the facial image to the information processing device 12. The smartphone then receives information from the information processing device 12 and displays it to the customer. This allows, for example, the customer to obtain information about recommended cosmetics without visiting a store.

[0050] Furthermore, in the above explanation, the information processing device 12 generates the impression evaluation model 24 once and then performs an impression evaluation of the customer. However, with the customer's consent, the evaluation captured image can be incorporated into the training data, and an experienced evaluator can provide an impression evaluation using the information processing device 12 or a terminal device capable of communicating with the information processing device 12, thereby updating the training data at any time.

[0051] Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present invention. For example, the functions included in each means, step, etc. can be rearranged so as not to be logically inconsistent, and multiple means, steps, etc. can be combined or divided into one. [Explanation of symbols]

[0052] 1. Information Processing Systems 10 Network 11 Terminal equipment 12 Information processing equipment 20, 30 input / output section 21, 31 Communications Department 22, 32 Storage section 23, 33 Control section

Claims

1. a storage unit for storing a model obtained by machine learning a correspondence between a facial image of a person and an impression evaluation of the facial image, the characteristic portion of the facial image contributing to the impression evaluation, and the impression evaluation, using the facial image and the impression evaluation as training data; a control unit that uses the model to determine characteristic parts of an input face image and impression evaluations corresponding to the characteristic parts in accordance with the proportions of the characteristic parts for each area included in the face image, and outputs information on the impression evaluations and an image showing the characteristic parts contributing to the impression evaluation in a manner corresponding to the impression evaluations together with the input captured image; An information processing device having the above.

2. In claim 1, the storage unit further stores recommendation information indicating a cosmetic product to be recommended for a combination of the characteristic portion associated in the model and the impression evaluation; the control unit selects, based on the recommendation information, a cosmetic product to be recommended for the combination of the characteristic portion and the impression evaluation obtained using the model, and further outputs information indicating the selected cosmetic product. Information processing device.

3. In claim 1 or 2, In the training data, the same impression evaluation is associated with face images of a plurality of people of different ages. Information processing device.

4. When executed by a computer, the computer is caused to operate as the information processing device according to any one of claims 1 to 3. program.

5. A method for operating an information processing device, comprising: a first step of using a person's face image and an impression evaluation of the face image as training data to machine-learn the correspondence between the characteristic parts of the face image that contribute to the impression evaluation and the impression evaluation, and creating a model; a second step of using the model to determine characteristic parts of an input face image and impression evaluations corresponding to the characteristic parts in accordance with the proportions of the characteristic parts in each area included in the face image, and outputting information on the impression evaluations and an image showing the characteristic parts contributing to the impression evaluation in a manner corresponding to the impression evaluations together with the input captured image; A method for operating an information processing device having the above-mentioned components.

6. In claim 5, a third step of referring to recommendation information indicating cosmetics to be recommended for the combination of the feature portion and the impression evaluation associated in the model, and selecting cosmetics to be recommended for the combination of the feature portion and the impression evaluation obtained using the model; a fourth step of further outputting information indicating the selected cosmetic product; A method for operating an information processing device further comprising:

7. In claim 5 or 6, In the training data, the same impression evaluation is associated with face images of a plurality of people of different ages. A method for operating an information processing device.

Images