Information processing device, program, and method of operating information processing device

Abstract

To provide an information processing device, a program and a method for operating the information processing device, capable of identifying a region that largely contributing to an age estimation prediction of a person, and indicating an influence degree of color information, for performing counseling for cosmetics based thereon.SOLUTION: In an information processing system in which a terminal device and an information processing device are connected with each other by peer-to-peer through wired or short-range radio communication or the like via a network or not via a network to transmit / receive information, the information processing device 12 includes: a storage unit 22 that stores a machine-learning model obtained by machine-learning an association between a feature portion of a lower half part of a face in a face image contributing to an age prediction value, as teacher data of the face image of a person and the age prediction value for the face image; and a control unit 23 that determines the feature portion of the input face image and color information on the feature portion by using the model and outputs the result.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a program, and an operation method of an information processing device. [Background technology]

[0002] There has been a great deal of interest in the predicted age that others perceive based on one's own face, and various evaluation methods have been investigated. For example, a visual evaluation by a dedicated technician and a method of estimating skin age from facial images using AI (Patent Document 1) have been proposed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-512797 Summary of the Invention [Problem to be solved by the invention]

[0004] It is known that the lower half of the face, particularly the area around the mouth, is likely to have an impact on how others predict one's age. Factors thought to influence this include the appearance of wrinkles such as nasolabial folds and marionette lines, noticeable sagging cheeks, and dull red lips. While quantitative assessment techniques have been established using visual evaluation by dedicated technicians and image processing technology using digital images, age-related changes around the mouth are not only diverse but also vary greatly between individuals. Therefore, it remains difficult to objectively define which characteristics are most influential among those common to age-related changes across generations and those that differ between individuals.

[0005] Furthermore, the current COVID-19 pandemic has led to the habit of wearing masks, which has reduced opportunities for people to see the lower half of their faces, especially their mouths. However, the amount of activity around the mouth has significantly decreased, which is feared to accelerate the aging process around the mouth. Therefore, there is a need for technology to identify the areas of the face that affect age prediction values ​​and the color information that contributes to lowering the age, as well as technology to provide cosmetic consultation based on that information. [Means for solving the problem]

[0006] Therefore, the present inventors have conducted extensive research and have discovered the following technology. The information processing device according to the present disclosure includes a storage unit that stores a machine learning model 1 that has performed machine learning to learn associations between characteristic parts of the lower half of a face of a face image that contribute to a predicted age value and the predicted age value using a face image of a person and a predicted age value for the face image as training data, and a machine learning model 2 that has performed machine learning to learn influences of the predicted age value based on color information of the characteristic parts; The apparatus has a control unit that uses the model to determine and output characteristic parts of an input face image and color information relating to the characteristic parts.

[0007] Further, the method of operating an information processing device according to the present disclosure includes a first step of creating a model by machine learning a correspondence between a feature part of a face image that contributes to a predicted age value and the predicted age value, using a face image of a person and a predicted age value for the face image as training data; a second step of creating a machine learning model that learns the influence level by machine learning using color information of the characteristic part; and a third step of determining and outputting characteristic parts of the input face image and color information relating to the characteristic parts using the model. [Effects of the Invention]

[0008] The information processing device and the like of the present disclosure can identify areas that contribute significantly to age estimation prediction, and can also represent the influence of color information in the identified areas, enabling calculation of an estimated age with objectivity and accuracy. Furthermore, this information can also be used for cosmetic counseling. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 1 illustrates an example of the configuration of an information processing device. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a terminal device. [Figure 4] FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. [Figure 5] FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. [Figure 6A] This is an example of a mouth image divided into 16 parts. [Figure 6B] This is an example of a mouth image in which the image is divided into 16 parts and some of them are masked. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described.

[0011] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. The information processing system 1 supports, for example, a salesperson at a cosmetics store providing counseling on beauty treatments to customers. The information processing system 1 includes a terminal device 11 and an information processing device 12. The terminal device 11 and the information processing device 12 are connected to each other peer-to-peer via a network 10 or, without the network 10, via wired or short-range wireless communication, and transmit and receive information. The network 10 includes the Internet, at least one wide area network (WAN), at least one metropolitan area network (MAN), or any combination thereof. The network 10 may also include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may be, for example, an ad hoc network, a cellular network, a wireless local area network (LAN), a satellite communication network, or a terrestrial microwave network. The terminal device 11 is an information processing terminal device equipped with an imaging function, such as a tablet terminal or a smartphone, and used by a cosmetics salesperson. The information processing device 12 is, for example, a computer such as a server that belongs to a cloud computing system or other computing system.

[0012] The information processing device 12 uses the lower half of a facial image (hereinafter referred to as a mouth image) obtained by dividing a facial image obtained by capturing a person's face into upper and lower parts based on the cheekbones as training data, and the person's actual age corresponding to the mouth image, and stores in advance a machine learning model (hereinafter referred to as an age prediction value model) that associates characteristic parts of the facial image that contribute to age prediction with the age prediction. In a sales store, a salesperson stationed at the store captures an image of the face of a visiting customer using a terminal device 11. The information processing device 12 receives an input of the customer's facial image from the terminal device 11, acquires a mouth image from the input facial image, calculates characteristic parts of the mouth image and a predicted age value using an age prediction model, and outputs the calculated predicted age value to the terminal device 11. The salesperson uses the terminal device 11 to display or otherwise output to the customer the predicted age value, the image region that contributes to the predicted age value, a heat map showing the contribution of the region to an increase or decrease in the predicted age value, or changes in color information that contribute to lowering the predicted age value in the region. Presenting the information to the customer can improve the customer's sense of satisfaction with the predicted age value.

[0013] The information processing device 12 also stores recommendation information indicating the cosmetics to be recommended to the customer. For example, the recommended information can include skin care cosmetics to reduce sagging, makeup cosmetics to even out blemishes with the skin tone, and product group information and effects required to enhance a youthful appearance. The information processing device 12 then selects the cosmetics to be recommended to the user based on the recommendation information and further outputs information indicating the selected cosmetics. The salesperson receives the information output by the information processing device 12 via the terminal device 11 and presents it to the customer. This makes it possible to understand the customer's aging phenomenon and recommend cosmetics that enhance a youthful appearance.

[0014] 2 shows an example of the configuration of the information processing device 12. The information processing device 12 is, for example, a server computer that functions as a server that implements various functions. The information processing device 12 may be one or more server computers that are connected to each other so that they can communicate information with each other and operate in cooperation with each other. The information processing device 12 has an input / output unit 20, a communication unit 21, a storage unit 22, and a control unit 23.

[0015] The input / output unit 20 has an input interface that detects user input and sends the input information to the control unit 23. The input interface is any input interface including, for example, physical keys, capacitance keys, a touch screen integrated with a panel display, various pointing devices, etc. The input / output unit 20 also has an output interface that outputs information generated by the control unit 23 or read from the storage unit 22 to the user. Such an output interface is any output interface including, for example, a display that outputs information as an image or video, a speaker that outputs information as sound, or an interface for connecting to an external output device.

[0016] The communication unit 21 has a communication module compatible with one or more wired or wireless LAN standards for connecting to the network 10. Alternatively, the communication unit 21 may have a communication module compatible with one or more wired or wireless standards for peer-to-peer connection with the terminal device 11. Such a communication module may include a communication module compatible with short-range communication such as Bluetooth (registered trademark), AirDrop, IrDA, ZigBee, FeliCa, RFID, etc.

[0017] The storage unit 22 includes one or more memories. Each memory included in the storage unit 22 is, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these. Each memory functions, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 22 stores any information and control / processing programs used in the operation of the information processing device 12. The storage unit 22 may also store application programs that provide various functions and are downloaded via the network 10 or other networks.

[0018] The control unit 23 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a specific process. The control unit 23 comprehensively controls the operation of the information processing device 12 in accordance with a control / processing program stored in the storage unit 22.

[0019] In the information processing device 12, the storage unit 22 stores an age predicted value model 24 created by the control unit 23 using a procedure described below. The storage unit 22 also stores recommendation information 25 including information on cosmetics to be recommended for combinations of characteristic parts and predicted age values ​​associated in the age predicted value model 24, and standard information 26 regarding criteria for determining the predicted age values.

[0020] 3 shows an example of the configuration of the terminal device 11. The terminal device 11 is an information processing terminal device such as a tablet terminal or a smartphone. The terminal device 11 includes an input / output unit 30, a communication unit 31, a storage unit 32, and a control unit 33.

[0021] The input / output unit 30 has an input interface that detects user input and sends the input information to the control unit 33. Such an input interface may be any input interface including, for example, physical keys, capacitive keys, a touch screen integrated with a panel display, various pointing devices, a microphone that accepts audio input, a camera that captures captured images or image codes, etc. The input / output unit 30 also has an output interface that outputs information generated by the control unit 33 or read from the storage unit 32 to the user. Such an output interface may be, for example, a display that outputs information as an image or video, a speaker that outputs information as audio, or any output interface including a connection interface with an external output device.

[0022] The communication unit 31 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as 4G (4th Generation) and 5G (5th Generation), and a communication module compatible with short-range communication such as Bluetooth (registered trademark), AirDrop, IrDA, ZigBee, FeliCa, RFID, etc. The terminal device 11 communicates information with the information processing device 12 via the network 10 or peer-to-peer using the communication unit 31.

[0023] The storage unit 32 includes one or more memories. Each memory included in the storage unit 32 may be, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these. Each memory functions as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 32 stores any information used in the operation of the terminal device 11. For example, the storage unit 32 stores control and processing programs, embedded software, application programs that provide various functions and are downloaded via the network 10 or other networks, and the like.

[0024] The control unit 33 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a specific process. The control unit 33 operates in accordance with a control / processing program stored in the storage unit 32, thereby providing overall control of the operation of the terminal device 11.

[0025] In the terminal device 11, the control unit 33 responds to an operation input by the salesperson to the input / output unit 30 by causing the camera of the input / output unit 30 to capture an image of the customer's face, and sends the captured facial image to the information processing device 12 via the communication unit 31. The control unit 33 also receives information sent from the information processing device 12 via the communication unit 31 and displays it on the display of the input / output unit 30.

[0026] 4 is a flowchart illustrating an example of the operation procedure of the information processing device 12 when the information processing device 12 creates the age predicted value model 24 by machine learning. This flowchart corresponds to the information processing procedure by the control unit 23.

[0027] In step S400, the control unit 23 reads the training data and stores it in the storage unit 22. The training data includes facial images obtained by capturing a person's face with a digital camera and the actual age of the person in the image. Hereinafter, the facial images included in the training data are images of the lower half of the face, obtained by dividing the image into upper and lower halves using the cheekbones of the individual as a guide (hereinafter referred to as "mouth images"), and are referred to as training facial images. First, the training facial images are obtained by capturing images of the faces of several hundred Japanese women, approximately uniformly distributed between their 20s and 70s, from the front. Each person's hairstyle is standardized using a hair net, and they are not wearing makeup, and a mouth image is created from the image. For example, facial features may be extracted by using an open-source library such as OpenCV, or by specifying the vertices of a bounding box on the device, but this is not limiting.

[0028] In step S402, the control unit 23 performs image processing on each training face image as a preliminary step for machine learning. This image processing includes, for example, normalization of the training face image, extraction of facial features from the training face image, random cropping, random horizontal and vertical flipping, random rotation, random saturation adjustment, vertical dropout, and the like. In the facial feature extraction, facial features can be extracted using any open source library to eliminate the influence of features such as hair and neck in the training face image, or facial features can be extracted by specifying the vertices of a bounding box. In addition, in the random cropping, for example, a training face image from 512 × 512 pixels is resized to 256 × 256 pixels, and then randomly cropped to 224 × 224 pixels.

[0029] In step S404, the control unit 23 creates the age-predicted value model 24 through machine learning using training data. The control unit 23 performs machine learning using, for example, a convolutional neural network. For example, the control unit 23 extracts feature parts from the training face images using a convolutional layer and a pooling layer, and compares the extracted feature parts with the age-predicted value using a multi-layer neural network, thereby learning the feature parts that contribute to the age-predicted value assigned to the training face images and the age-predicted value in association with each other, and creates the age-predicted value model 24. Next, the mouth image, partially masked as shown in Figure 6B, is input to the supervised learning model 1, and the output predicted age value and the rate of change from the predicted age value before masking are calculated. This predicted age may be an age range with a certain range. Areas with a high rate of change are then recognized as areas that contribute to age prediction.

[0030] 5 is a flowchart illustrating an example of the operation procedure of the information processing device 12 when the information processing device 12 predicts the age of a customer's face image using the age prediction value model 24. This flowchart corresponds to the information processing procedure by the control unit 23.

[0031] In step S500, the control unit 23 accepts input of a facial image and customer information. For example, a salesperson at a store takes an image of the face of a customer who visits the store using the terminal device 11, and inputs the customer's age, gender, and other customer information into the terminal device 11. The terminal device 11 sends the captured image and customer information to the information processing device 12. In the information processing device 12, the control unit 23 accepts the captured image and customer information via the communication unit 21. The facial image and customer information are stored in the memory unit 22. Hereinafter, the facial image accepted by the control unit 23 and used to calculate an age prediction value is referred to as an evaluation facial image.

[0032] In step S502, the control unit 23 performs image processing on the evaluation face image as a preliminary step for obtaining a predicted age value. The image processing includes normalization of the evaluation face image, extraction of facial features from the evaluation face image, random cropping, random flipping in horizontal and vertical directions, random rotation, random saturation adjustment, vertical dropout, etc., and creates a mouth image using OpenCV or the like.

[0033] In step S504, the control unit 23 obtains a predicted age value of the evaluation face image using the age predicted value model 24. The control unit 23 obtains, as the predicted age value, the class of the evaluation value corresponding to the maximum value in the final layer output of the age predicted value model 24. The evaluation value is assigned as a discrete value from 1 to 9 for each age group.

[0034] In step S506, the control unit 23 determines the characteristic features that contribute to the predicted age value. The mouth image is divided into multiple analysis regions, and one region is selected from each analysis region and initialized with a brightness value of 1 (hereinafter referred to as masking). This creates an occlusion region in the mouth image. By processing the mouth image to a pure white state with a brightness value of 1, all information contained in each region is deleted, allowing the influence of each region on age prediction to be examined. For example, in Figure 6A, the number of occlusion regions is set to 16. If the number is too small, the output change of the machine learning model becomes minimal, while if it is too large, multiple pieces of information are included in the analysis region, making interpretation difficult. A partially masked mouth image is input, and the output predicted age value and the rate of change compared to the predicted age value before masking are calculated. This predicted age may be a range of ages. Considering the bilateral symmetry of the face, the number of regions is preferably 2n × 2n (n is an arbitrary integer), such as 4, 8, or 16. Regions with a high rate of change are then recognized as characteristic features contributing to age prediction. The control unit 23 also obtains color information that contributes to lowering the predicted age value. It then performs occlusion processing of the a* value, b* value, and grayscale of the characteristic portion. The a value corresponds to a, which is one of the complementary color dimensions in the L*a*b color space. By converting the RGB color information for the feature areas to L*a*b color information, setting the a value to 0, and then converting back to RGB color information, only the a* color information within the area is eliminated, allowing us to examine the effect of occlusion processing, which sets the a* value to 0 for each area, on age prediction. The b* value corresponds to b, one of the complementary color dimensions in the L*a*b color space. By converting the RGB color information for the feature areas to L*a*b color information, setting the b* value to 0, and then converting back to RGB color information, only the b* color information within the area is eliminated, allowing us to examine the effect of occlusion processing, which sets the b* value to 0 for each area, on age prediction. Furthermore, grayscale is a method of representing color in computers and photographs, and represents the absence of information other than luminosity. Therefore, by processing the image to grayscale, i.e., black and white, the color information within each area is eliminated, leaving only shape information, allowing us to examine the effect of color information for each area on age prediction.

[0035] In step S506, the control unit 23 may use the age prediction value model 24 to visualize the feature portions that contribute to the predicted age value of the evaluation face image. For example, the feature portions may be represented as a heat map. The control unit 23 creates a heat map showing the feature portions via a convolutional neural network. The control unit 23 converts the maximum output value of the final output layer into a probabilistic expression using a numerical value between 0 and 1 using an activation function such as a sigmoid function or a softmax function. The control unit 23 calculates the gradient with respect to the maximum output value of the final output layer, i.e., the partial differential value, for each pixel of the feature map, which consists of 512 channels of the final convolutional layer. The control unit 23 then calculates the average value of the gradient for each pixel and creates an activation map for 512 channels by multiplying each pixel of the feature map by the average value. The number of channels of the feature map shown here is an example, and the number of channels may be other than 512. The control unit 23 then integrates the created activation map across all channels and scales the integrated activation factors to values ​​between 0 and 1 to generate a two-dimensional activation map, i.e., a heat map. The control unit 23 then resizes the heat map to the size of the face image for evaluation to generate a heat map image corresponding to the face image for evaluation. The control unit 23 may also add the heat map image to the face image for evaluation and scale it with a gradation value between 0 and 255 to generate a display image in which the heat map is superimposed on the face image for evaluation. By superimposing the heat map on the face image for evaluation, characteristic portions in the face image for display are displayed with a predetermined color distribution that is arbitrarily set. Note that when scaling the size of the heat map, the control unit 23 may eliminate abnormal values ​​using an arbitrarily set threshold value to remove noise from the heat map.

[0036] In step S508, the control unit 23 selects recommended cosmetics based on the predicted age value, the characteristic part, and the color information. For example, if the characteristic part is the cheek area, a cosmetic for improving cheek sagging is selected as the cosmetic corresponding to the characteristic part. For the mouth area, a cosmetic for improving lip sagging is selected. For the chin area, a cosmetic for improving chin sagging is selected. Furthermore, if the color information significantly contributes to increasing the predicted age value, a skin care cosmetic containing an anti-inflammatory agent that improves inflammation by suppressing localized redness is selected because localized inflammation is expected when the a* value has a large influence. Alternatively, a whitening cosmetic for improving dullness is selected because localized dullness is expected when the b* value has a large influence.

[0037] According to this embodiment, it is possible to reproduce an objective predicted age value with high accuracy. In addition, it is possible to visualize the characteristic parts that contributed to the predicted age value by displaying them, which makes it possible to improve the customer's satisfaction with the predicted age value.

[0038] In the above description, the information processing device 12 acquires facial images and displays various output information via the terminal device 11. However, the information processing procedures shown as the operations of the information processing device 12 may be shared between the information processing device 12 and the terminal device 11 as appropriate. In this case, the information processing device 12 and the terminal device 11 constitute an "information processing device." For example, the information processing device 12 may execute the machine learning procedure shown in FIG. 4 to create the age prediction value model 24, and the terminal device 11 may execute the procedure shown in FIG. 5 using the age prediction value model 24 of the information processing device 12. Alternatively, for example, the information processing device 12 may operate standalone and acquire facial images and display various output information using the input / output unit 20. Furthermore, the terminal device 11 does not have to be installed in a store; for example, it may be a smartphone used by a customer. In this case, the customer captures an image of their face and sends the facial image to the information processing device 12. The smartphone then receives information from the information processing device 12 and displays it to the customer. This allows, for example, the customer to obtain information about recommended cosmetics without visiting a store.

[0039] In addition, in the above explanation, the information processing device 12 generates the age prediction value model 24 once and then executes the customer's age prediction value, but with the customer's consent, the evaluation captured image can be imported into the training data, and an experienced evaluator can assign an age prediction value using the information processing device 12 or a terminal device that can communicate with the information processing device 12, thereby updating the training data at any time. [Explanation of symbols]

[0040] 1. Information Processing Systems 10 Network 11 Terminal equipment 12 Information processing equipment 20, 30 input / output section 21, 31 Communications Department 22, 32 Storage section 23, 33 Control section

Claims

1. a storage unit for storing a model that has been machine-learned to associate a person's lip image with a predicted age value, using the person's lip image and the person's actual age as training data; a control unit that uses the model to determine and output an input first lip image and a first predicted age value corresponding to the input first lip image, the control unit outputs information for superimposing on the analysis region obtained by dividing the first mouth image information indicating a rate of change in the first age predicted value before and after a process for changing a predetermined value in a color space in the analysis region, and displaying the information; the storage unit further stores recommendation information indicating a cosmetic product to be recommended for a combination of the analysis region and the age predicted value associated in the model; The control unit selects cosmetics to be recommended for a combination of the analysis area having a predetermined change rate and the age prediction value based on the recommendation information, and further outputs information indicating the selected cosmetics.

2. In claim 1, In the process of changing the predetermined value, one or more of a decrease in the a* value, a decrease in the b* value, and setting the grayscale value to the predetermined value is executed.

3. In claim 1 or 2, In the information processing device, the information indicating the rate of change is displayed in a manner superimposed on the analysis region as a heat map.

4. In any one of claims 1 to 3, The information processing device, wherein the subdivided locations are 2n x 2n (n is any integer) locations.

5. A program that, when executed by a computer, causes the computer to operate as the information processing device according to any one of claims 1 to 4.

6. a first step of creating a model by machine learning the correspondence between a person's lip image and a predicted age value using a person's lip image and the person's actual age as training data; a second step of calculating and outputting the input first lip image and a first predicted age value corresponding to the first lip image using the model; a third step of outputting information for superimposing on the analysis region the information indicating a rate of change in the first predicted age value before and after a process for changing a predetermined value in a color space in the analysis region obtained by dividing the first mouth image; and a fourth step of referring to recommendation information indicating cosmetics to be recommended for a combination of the analysis area and the predicted age value associated in the model, and selecting cosmetics to be recommended for the combination of the analysis area and the predicted age value having a predetermined rate of change; a fifth step of further outputting information indicating the selected cosmetic product; A method for operating an information processing device having the above-mentioned components.

Images