Method for evaluating the degree of fit of ornaments or daily commodities worn on or around the face, or makeup or hairstyle, to the user's face, system for evaluating the degree of fit, recommendation system, and eyeglass design system

Abstract

To provide a technique and the like which can objectively explain why ornaments or daily necessaries worn on the face or the periphery of the face, or the makeup or hairstyle are good for the face of a user, and can select ornaments or daily necessaries worn on the periphery of the face, or the makeup, hairstyle or the like matching the face of the user without a need of creating an enormous database.SOLUTION: A method calculates a first evaluation value as an evaluation value of a first face image and a second evaluation value as an evaluation value of a second face image by evaluating with a face image evaluator the second face image obtained by combining the first face image of a user with at least one image of ornaments, daily necessaries, makeup and hairstyle to be evaluated, and calculates a matching degree to the face of the user from the first evaluation value and second evaluation value.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a method and system for evaluating the degree of fit of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, a system for recommending ornaments or daily necessities, makeup or hairstyle, etc., and a system for designing eyeglasses, etc. [Background technology]

[0002] Users have a desire to purchase items that suit them when shopping, so when serving customers in a store, staff can recommend items such as clothes, shoes, glasses, hats, cosmetics, and accessories based on whether they suit the user, thereby increasing the user's satisfaction with their purchase. In particular, when selecting items to wear on or around a person's face, it is particularly important that they look good on the user, as the user will see their face in the mirror in their daily life and the impression they make on others. Therefore, the development of systems that take photographs of a person's face and select whether the items look good on them has been proposed. An example of such a selection system is cited in Reference 1. Reference 1 proposes a system that assists in the selection of subjects by presenting an image of the subject's face with virtual glasses superimposed on it. Furthermore, cited reference 2 is an example of a selection system that uses a facial photograph. Cited reference 2 proposes a hairstyle selection system and method that can accurately select a hairstyle that suits a subject based on various information analyzed using the subject's facial photograph data, and display and output a composite image of the selected hairstyle and the subject's facial photograph data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-30296 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-143158 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the system in Cited Document 1 superimposes eyeglasses onto a photograph of the user's face and allows the user to select a pair, but does not provide any explanation as to why a particular pair of eyeglasses is suitable for the user's face, so the selection of eyeglasses is left to the user's subjective opinion. Furthermore, since the cited document 2 is a system that references a pre-created database, it can only calculate evaluation values ​​for products that exist in the database, and is therefore unable to evaluate new products that did not exist at the time the database was created, or even if it could, the accuracy would be low. Furthermore, in order to create such a database, it is generally necessary to create a database that links "faces," "products," and "evaluation values," but creating such a database requires a huge amount of effort and cost. In view of these problems, the present invention provides a technology that can more objectively explain why an ornament or daily necessities to be worn on or around the face, or makeup or hairstyle, is suitable for the user's face, and that can select an ornament or daily necessities to be worn around the face, or makeup or hairstyle, that is suitable for the user's face, without the need to create an enormous database. [Means for solving the problem]

[0005] In order to solve the above problem, the first means is a method for evaluating the degree of suitability of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, which comprises: evaluating a first facial image of the user and a second facial image obtained by combining an image of at least one of the ornament, the daily necessities, the makeup, and the hairstyle to be evaluated with the first facial image using a facial image evaluator to calculate a first evaluation value as an evaluation value of the first facial image and a second evaluation value as an evaluation value of the second facial image; and calculating the degree of suitability to the user's face from the first evaluation value and the second evaluation value, thereby evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to the user's face. This makes it possible to objectively evaluate and judge whether or not an ornament, daily necessities, or makeup or hairstyle suits the wearer based on the degree of suitability calculated each time.

[0006] Examples of "accessories or everyday items worn on or around the face of a user" include eyeglasses, eyeglass displays, earrings, piercings, ear cuffs, face paint, hair bands, hats, headbands, headphones, earphones, earphones with microphones, eye masks, colored contact lenses, masks, mask charms, face masks, face guards, wearable biometric devices, etc. They are not limited to these, as long as they are worn on or around the face of a user. In the context of "makeup or hairstyle," makeup includes makeup using cosmetics such as lipstick, blush, eye shadow, and eyebrow pencil, as well as makeup using components that can be repeatedly attached and removed, such as false eyelashes and wigs. It also includes, for example, ultra-thin transparent stretch tape with a thickness of 20 μm or less that is attached to the user's face to reduce wrinkles and dullness. It also includes not only facial makeup but also changing hair color through hair dye. Makeup and hairstyles also include the use of components such as glitter. The "decorative item or daily necessities, or makeup or hairstyle" that is worn or attached may be just one, or multiple of these elements may be worn selectively. The "first face image" is an image of the head including the face taken from the front, and may include only the face or the face and its associated parts, such as hair, head, ears, neck, etc. The front may be directly in front, or may be taken from a diagonal angle below or to the side, as long as it shows the main parts of the face. The "first face image" should preferably be "a face image without the accessory or daily necessities, without the makeup, or before the makeup or hairstyle is evaluated," because this allows the user's usual appearance and appearance at the time of suitability evaluation to be appropriately reflected in the evaluation (suitability). The first facial image is acquired using an imaging device such as a camera. The acquired facial image may be digital facial image data or a printed image such as a printed photograph. The user may also take a photograph of their own face using a device such as a smartphone or tablet, or a store that sells accessories, daily necessities, or makeup or hairstyles may take a photograph using a camera, video, or device equipped with a camera or video. If the user acquires the first facial image at home, the image may be transmitted via an online service such as the Internet. The data may also be stored in memory and brought to a location where it will be evaluated. The "second facial image synthesized with an image of the ornaments or other items to be evaluated" is a first facial image synthesized with an image of the ornaments or other items, and therefore the second facial image itself is not captured using an imaging device. When the second facial image itself is captured, the environment may be different from that of the first facial image, and the facial expression or position or orientation may be different. Even if the image is captured exactly as the first facial image, it is not possible to obtain an image that is exactly the same as the first facial image, which may affect the evaluation value. In other words, the difference between the evaluation value for the first facial image and the evaluation value for the second facial image includes not only the change due to the influence of the synthesized image of the ornaments or other items to be evaluated, but also the influence of the differences between the first and second facial images, and therefore does not reflect the influence of only the ornaments or other items to be evaluated. However, if a second facial image is created by synthesizing an image (photo) of the ornaments or other items to be evaluated with the first facial image and the evaluation value of the first facial image and the evaluation value of the second facial image are calculated, the difference in the evaluation values ​​reflects the influence of only the ornaments or other items to be evaluated. It is also preferable that images of accessories, etc., be prepared in advance. For example, the images may be captured as photographs or videos using an imaging device such as a camera, or may be created using computer graphics rather than actual images of the items, or may be created by obtaining copyright-free images from web pages on the Internet. Furthermore, images of accessories, etc., the suitability of which for the user is to be evaluated, may be obtained by photographing them separately from the user's facial image. It is preferable that the images of accessories, etc., be created, for example, as a layer screen, overlaid on the first facial image. This composition work may be performed, for example, by an operator visually using an operating means, such as a mouse, to grip and arbitrarily move and place the ornaments, etc. on the first face image displayed on a monitor by the control means of the computer device, or the control means may control a program in the computer device to automatically recognize appropriate positions according to the ornaments, etc., and automatically place the ornaments, etc. on the first face image. Also, the face identification function of the face image evaluator may be used to have the program automatically place the images of the ornaments, etc., in appropriate positions.

[0007] A "face image evaluator" is a device that has the function of identifying an input face image and outputting an evaluation value, and returns one or more evaluation values ​​for the input face image. The image evaluator may be implemented as a program (software) in the storage device of a computer device, or may be implemented as a hardware device with a circuit design connected to a computer device, etc. Furthermore, it may be an integrated device with (or part of) an imaging device such as a camera or video. The data input to the facial image evaluator can be image data such as photographs or videos, and either still image input data such as photographs or video input data such as video can be used. Preprocessing image data such as photographs or videos, such as images, can be performed to improve the accuracy of identifying and recognizing the user's face. It is also effective to convert the data into 3D data using 3D modeling, for example, to obtain depth information. The facial image evaluator preferably extracts at least some of the facial image features from the input facial image (still image, video, etc.) (feature extraction), calculates the similarity and estimated values ​​for the evaluation items set by the creator based on the correlations learned when the evaluator was created, and outputs the evaluation value. The facial image evaluator is preferably created by statistically learning the correlation between features extracted from at least a portion of a facial image and evaluation items set by the creator of the facial image evaluator, such as emotion and age. For example, a facial image evaluator can be created and used by using deep learning or the like from facial images linked to numerous emotions, ages, and other features. The present invention may also use commercially available software (APIs, etc.) or embedded hardware as the facial image evaluator. A "feature" refers to information constituting a facial image that is statistically correlated with the evaluation items evaluated by the facial image evaluator. The creator of the facial image evaluator may set feature values, such as facial wrinkles and sagging, the position and angle of a portion of the contour line of the corners of the mouth and the outer corners of the eyes, the angle of the eyebrows, and their positional relationship with the eyes, and extract the feature values ​​from the facial image using an image filter or calculation. Alternatively, a computer may set feature values ​​from a large number of facial image data and evaluation item data using a machine learning algorithm such as deep learning. Alternatively, feature extraction may be performed by extracting characteristic points from a facial image, such as the corners of the mouth or the corners of the eyes, as "feature points" and calculating the feature values ​​from those feature points. By extracting feature points and calculating feature values ​​from those feature points, it is possible to extract features that are easy for evaluators to understand and reduce the computational cost required for calculating feature values. Such feature extraction can be performed, for example, by extracting feature points related to facial features, such as the corners of the mouth, the corners of the eyes, or part of a contour line, and then using one or more of those feature points to quantify the distance between two feature points, the shape of a figure connecting multiple feature points, or the RGB values, hue, brightness, or saturation of the pixels in the area surrounded by the feature points. The facial image evaluator may be configured to include classifiers such as an emotion classifier, an age classifier, and a gender classifier, depending on the evaluation items for which the facial image to be evaluated is to be identified, classified, and evaluated. By evaluating the facial image for more specific evaluation items such as emotion, age, and gender, a more subtle evaluation can be made of the facial image to be evaluated. The facial image evaluator may be configured with multiple classifiers, or may consist of just one classifier. The emotion classifier, age classifier, and gender classifier will be described below.

[0008] (1) Emotion Classifier An "emotion classifier" is a facial image evaluator that estimates and classifies the emotion of an input facial image. It extracts features such as the contours of a person's face and facial features, as well as their arrangement and orientation (feature extraction), and uses these features to identify and quantify emotions. It is preferable that the emotion classifier be trained using an algorithm such as machine learning to determine the correlation between emotions obtained through subjective evaluation using a separate questionnaire or the like and the features of the facial image. The emotion classifier statistically calculates an evaluation value from a pre-trained "correlation between features and evaluation value." This correlation can be estimated, for example, by using the maximum likelihood method based on the sample correlation coefficient obtained using a large amount of data on features and evaluation values ​​for features extracted from an input facial image. For example, it is best to evaluate "emotions" as "emotional valence" and output a number between 0 and 1 (0 to 100%) probabilistically. For example, in Figure 1(d), the evaluations are neutral 0.51, joy 0.18, anger 0.12, sadness 0.11, and surprise 0.08. This means that the input facial image has a high probability of being "neutral," followed by "joy." Therefore, the input facial image is interpreted as having an emotional valence between neutral and joy, or as having the highest probability of being neutral. The "emotions" evaluated by the emotion classifier may be one or multiple. It is also possible to evaluate multiple emotions and use only the emotion you want to evaluate. An example of an emotion model for quantifying emotions will be described below. The well-known Plutchik's Wheel of Emotions model shown in Figure 3 represents human emotions using eight basic emotions: Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger, and Anticipation. Joy and sadness, trust and disgust, fear and anger, and anticipation and surprise are each treated as pairs of emotions. In addition, this model also lists emotions that progress from basic emotions by intensity, such as Serenity, Ecstasy, Acceptance, Adoration, Apprehension, Terror, Distraction, Amazement, Pensiveness, Grief, and Boredom. The basic emotions include Boredom, Loathing, Annoyance, Rage, Interest, and Vigilance, and the mixed emotions include Love, Submission, Awe, Disapproval, Remorse, Contempt, Aggressiveness, Optimism, Pride, Guilt, Hope, Curiosity, Despair, Unbelief, Envy, Cynicism, Delight, Sentimentality, Shame, Outrage, Pessimism, Morbidity, Dominance, and Anxiety. These can be used as evaluation items for emotion classifiers. Various types of such emotion models are known, such as two-axis, four-axis, and circular, and can be used when creating an emotion classifier.

[0009] (2) About the age classifier An "age classifier" is a facial image evaluator that estimates and classifies the age of an input facial image and outputs it. It is preferable that the age classifier is one that has been trained using an algorithm such as machine learning to learn the correlation between features extracted from a large number of facial images and age. For example, an age classifier can be created by training at least one of the visual facial features that change as people age, such as wrinkles, sagging skin, and drooping of the corners of the eyes, as a feature. (3) Gender Classifier A "gender classifier" is a facial image evaluator that estimates and classifies the gender of an input facial image and outputs it. It is trained on the correlation between features extracted from numerous facial images and gender. A gender classifier outputs the statistical probability of the gender of an input facial image. It does not simply determine whether an input facial image is male or female; for example, it outputs a statistical probability for a given facial image, such as male = 0.9 (female = 0.1). A value of 0.5 indicates neutral. Using such a gender classifier, it is possible to objectively determine whether an input facial image appears masculine or feminine. Since users may desire to appear more feminine or more neutral, the numerical value output by a gender classifier can be used as a basis for determining such preferences. Furthermore, apparent age and facial expressions generally vary significantly depending on gender. Therefore, for example, an age classifier is often combined with a gender classifier, and an emotion classifier is often combined with a gender classifier.

[0010] Next, an outline of the principle of calculating the degree of suitability of a product or the like to be evaluated for a face using a facial image evaluator will be explained with reference to Figures 1 and 2. The following calculation is performed by a computer device equipped with a display device and an input device. A program (software) that calculates the degree of suitability based on the evaluation value obtained by the facial image evaluator is stored in the storage device of the computer device. The control means of the computer device executes the program based on input by the operator to calculate the degree of suitability. The following explanation is an example in which glasses are used as a product, but other products other than glasses may also be used, and this is an example of a virtual person with evaluation values ​​for facial feature points (feature amounts). This section describes whether a pair of eyeglasses suits (looks good on) the person whose face is shown in Figure 1(a). First, a first facial image is captured using a camera or other device and input into a facial image evaluator. This facial image evaluator is composed of an emotion classifier and an age classifier. To improve classification accuracy, it may also include other classifiers, such as a gender classifier. The facial image evaluator extracts feature points from the input first facial image data, calculates feature values ​​from the feature points, and statistically calculates an evaluation value for the first facial image based on a pre-trained correlation between feature values ​​and evaluation values. The feature value calculation process is an example; some facial image evaluators may calculate feature values ​​directly from the facial image without calculating feature points. In this example, as shown in Figures 1(b) and 1(c), the eyebrows, eyes, mouth, and contours from the chin to the cheeks are extracted and used as feature points for calculating feature values. Figure 1(d) shows the evaluation value. Here, for each evaluation item set in the face image evaluator, the age classifier calculated an age of 29, and the emotion classifier calculated neutral 0.51, joy 0.18, anger 0.12, sadness 0.11, and surprise 0.08. Next, as shown in Figure 2(a), a second facial image is created by combining the eyeglasses, which are accessories to be evaluated, with the first facial image. When combining the eyeglasses to be evaluated with the first facial image as shown in Figure 2(a), the position and orientation of the eyeglasses may be adjusted manually by the evaluator, or may be determined automatically by a program using the feature points of the first facial image (Figure 1(c)). The second facial image data created in this way is input into the facial image evaluator. As shown in Figure 2(b) and (c), the facial image evaluator extracts features from the facial image 2 data in the same way as for the first facial image data, and statistically calculates an evaluation value for the second facial image from the "correlation between features and evaluation values" that it has trained in advance.For the second facial image, as shown in Figure 2(d), the age classifier calculated an age of 30, and the emotion classifier calculated evaluation values ​​as neutral 0.59, joy 0.04, anger 0.01, sadness 0.35, and surprise 0.01.

[0011] The degree of suitability of a product or the like to be evaluated to the user's face can be calculated from the first evaluation value and the second evaluation value. Various calculation methods are possible for calculating the degree of suitability using the first evaluation value and the second evaluation value. For example, the difference between the first evaluation value and the second evaluation value, or a comparison value between the first evaluation value and the second evaluation value, can be used. The difference value may be evaluated as an absolute value or a relative value. When the face image evaluator calculates multiple evaluation values, multiple first evaluation values ​​and multiple second evaluation values ​​are calculated. However, by multiplying each of the multiple evaluation values ​​by a weighting coefficient, it is possible to prioritize the evaluation of the evaluation values ​​of the evaluation items required when calculating the degree of suitability, thereby increasing the accuracy of the evaluation values. The degree of conformance is calculated based on at least one of the first evaluation value for each evaluation item of the first face image obtained as described above and the second evaluation value for each evaluation item of the second face image. In the following example of calculating the degree of conformance, it is important to calculate the degree of conformance from the first evaluation value and the second evaluation value, but the calculation method is not limited to this. Now, let P1 to Px be the evaluation items such as "neutral" and "joy." Then, the first evaluation values ​​A1 to Ax calculated for the first face image have the following relationship for each evaluation item. First evaluation value of evaluation item P1 = A1, second evaluation value of evaluation item P1 = A2, First evaluation value of evaluation item Px = Ax Similarly, for the second face image, the second evaluation values ​​B1 to Bx calculated when wearing a certain "accessory or daily necessities, or makeup or hairstyle" have the following relationship for each evaluation item. The second evaluation value of evaluation item P1=B1, the second evaluation value of evaluation item P2=B2, ...The second evaluation value of evaluation item Px = Bx The degree of suitability of a product, etc. to be evaluated to the user's face is calculated from the first evaluation values ​​A1 to Ax and the second evaluation values ​​B1 to Bx. When calculating the degree of suitability, a weighting coefficient may be set for each evaluation item and multiplied by the first evaluation value and the second evaluation value. The weighting coefficient is set according to the user's wishes. Here, if the weighting coefficients for the evaluation items P1 to Px are w1 to wx, the evaluation value obtained by reflecting the weighting coefficients in the first evaluation value is w1×A1 to wx×Ax, and the evaluation value obtained by reflecting the weighting coefficients in the second evaluation value is w1×B1 to wx×Bx. The degree of suitability of a product, etc. to be evaluated to the user's face is calculated from the evaluation value obtained by reflecting the weighting coefficients in the first evaluation value and the evaluation value obtained by reflecting the weighting coefficients in the second evaluation value. The weighting coefficient may be calculated by multiplying the difference between the first evaluation value and the second evaluation value, for example, (A1-B1) to (Ax-Bx), or by multiplying the comparison value between the first evaluation value and the second evaluation value, for example, (A1 / B1) to (Ax / Bx), since these are equivalent mathematical formulas.

[0012] The weighting coefficient does not have to be positive, but can also be 0 or negative. For evaluation items that are inversely correlated with the evaluation item to be evaluated when calculating the degree of compatibility, the weighting coefficient should be set to negative, and for evaluation items that have no correlation at all, the weighting coefficient should be set to 0. By adjusting the weighting coefficient in this way, various degrees of compatibility can be calculated using the same algorithm.

[0013] In the present invention, the key point is that the second facial image is created by superimposing an image of a product or the like (glasses in this example) to be evaluated on the first facial image itself. By inputting the second facial image, in which glasses are superimposed on the first facial image, into the facial image evaluator, the "feature amount of the second facial image" extracted by the facial image evaluator will change compared to the "feature amount of the first facial image." This is because the image of the product to be evaluated overlaps with part of the feature amount (feature point), which prevents the facial image evaluator from recognizing it as a feature amount. For example, as shown in FIG. 2(b), part of the product overlaps with a feature point (eyebrows in FIG. 2(b)), resulting in the facial image evaluator determining that part of the product is the feature point, resulting in a "change in the feature amount" used by the facial image evaluator for evaluation. As described above, in the face image evaluator, feature amounts corresponding to (correlated with) the evaluation items for which the evaluation is to be output are set in advance (for example, by machine learning, etc.). The feature amounts may be set by an operator (the person who sets the feature amounts) targeting, for example, the shape and direction of the eyes and lips, the shape of facial parts such as the open / closed state, etc. Alternatively, there is a method in which the feature amounts are obtained by a computer using deep learning, etc., from a large number of image data and the output results linked to each of those images. For example, age classifiers and emotion classifiers learn features correlated with age (looking older or younger) and emotion (in this example, neutral, joy, anger, sadness, and surprise) as evaluation items. That is, if the features of the input facial image and the features related to the evaluation value to be calculated statistically match well, the facial image evaluator calculates the evaluation value as a statistically high evaluation value. If they do not match well, the facial image evaluator calculates a low evaluation value. Therefore, if the "features of the second facial image" change from the "features of the first facial image" due to the synthesis of a product whose suitability (suitability) to the first facial image is to be evaluated, the change reflects the suitability (suitability) of the product to be evaluated to the facial image. That is, if the evaluation value for the input second facial image changes from the evaluation value for the input first facial image, the change is the result of an increase or decrease in the features used to calculate the evaluation value due to the product being visually overlaid, and the suitability of the product to be evaluated to the facial image is being evaluated. So far, we have used the example of glasses as the product to be evaluated, but any product that can be overlaid on a facial image can be evaluated in the same way because the features will change. This does not have to be glasses. For example, this applies to accessories worn on the face, such as eyeglass displays, earrings, hair bands, hats, headbands, headphones, earphones, piercings, ear cuffs, face paint, earphones with microphones, colored contacts, eye masks, masks, mask charms, face masks, face guards, wearable biometric devices, and other accessories, as well as makeup and hairstyles.

[0014] Next, the significance of calculating the degree of suitability of a product to be evaluated for a facial image by using the means of the present invention will be explained. Typically, when creating a classifier that can determine whether a product (e.g., eyeglasses) suits a person, images (photos or videos) are created of various people wearing various products to be evaluated. Then, a set of data is generated in which the images are evaluated by subjective evaluation, such as through questionnaires, to determine whether the product suits the person. Training data and evaluation data for validation are then generated from the set of data, and the classifier is then created using the set of data. For example, when using algorithms such as machine learning, approximately 1,000 to 10,000 images are typically required, which not only requires time and effort to create the data, but also makes it extremely difficult to obtain the corresponding subjective evaluations. Another problem with such classifiers is that they cannot evaluate products outside the product variations evaluated at the time of classifier creation (training) or products in a different category. In other words, even if a classifier is painstakingly created to determine whether a new product or a different product suits a person, it will be completely useless. While methods such as transfer learning have been proposed to reduce this effort, some kind of classifier tuning is still required.

[0015] The age classifier and emotion classifier used in the present invention are set by learning features for evaluation items (for example, age and emotion (neutral, happy, anger, etc.)) for a large number of faces that are "not wearing the product to be evaluated." Changing the product to be evaluated changes the second facial image, and the feature values ​​of the facial image increase or decrease accordingly, which changes the evaluation value. In this case, for example, if it is an age classifier, the evaluation value will be calculated as age, and this age will be communicated to the user as information on the suitability of the product to be evaluated. A younger age is not necessarily better. The ideal age at which a user would like to be perceived varies depending on their actual age and position, and users' ideals vary, such as "I want to appear a little more mature" or "I want to appear five years younger than I actually am, but it's better not to appear too young." Furthermore, if it is an emotion classifier, for example, if wearing this product increases the evaluation value for happiness, it will "make the user appear more likely to smile," or if the evaluation values ​​for neutral and anger increase, it will "make the user appear more likely to look serious (serious face)," and this information will be communicated as information on the suitability of the product to be evaluated. Other possible user preferences include, for example, "looking sincere," "looking kind," "looking humble," "looking mature," "looking youthful," "looking friendly," "looking unapproachable," "looking troubled," "looking weak," "looking strong," "looking reliable," "looking gentle," "looking short-tempered," "looking fun," "looking happy," "looking unhappy," "looking fragile," "looking energetic," "looking intelligent," "focus on the eyes," and "focus on the mouth." Here again, each user's ideal of how they want to be perceived differs. Therefore, when conducting an evaluation, it is advisable to prompt the user in advance to input their wishes and requests (ideals) regarding "how they want to be perceived," and then calculate the degree of conformity with the input user's ideal.

[0016] For example, if a user wishes to look five years younger, the evaluation value is calculated by giving a high rating (high compatibility) to a condition where the age determined by the age classifier of the second face image is five years younger than the first face image. Alternatively, if the user wishes to "appear more serious (serious face)," the evaluation value can be calculated by giving a high rating (high compatibility) to a condition where the emotion classifier of the second face image determines more neutral and angry emotions than the first face image. Alternatively, if the user wishes to "stay as unchanged as possible," the absolute value of the difference between the output values ​​of the emotion classifier and age classifier for the first and second face images is calculated, and the second face image with the smaller absolute value is given a high rating (high compatibility) to calculate the evaluation value.

[0017] As a second means, the face image evaluator is an emotion classifier that estimates emotions based on face images. By using an emotion classifier, it is possible to estimate and evaluate emotions that are unconsciously and consciously expressed as facial expressions in facial images. When people judge whether something looks good, they often use what emotion it appears to be (what impression it gives to others) as a basis for their judgment, so using an emotion classifier is advantageous because it allows them to evaluate the suitability of products, etc. that they are trying to evaluate. In addition, emotion classifiers have many evaluation items, so by selecting multiple evaluation items that they want to evaluate and calculating composite evaluation values, it is possible to evaluate subtle facial expressions, and it is possible to more objectively and accurately evaluate the suitability of products, etc. that they are trying to evaluate. As a third measure, at least one of the emotions estimated by the emotion classifier is joy. Among emotions, "joy" is evoked when a person perceives a situation or event in a positive light, and it reduces the sense of caution or tension felt by those around them who express joy on their face, and gives the impression that they are approachable. Therefore, "joy" is good because it is easily correlated with the suitability of the product or other item being evaluated. As a fourth means, the degree of suitability for the face is evaluated as being high as the evaluation value of the joy increases. Many users expect that wearing a product will ease the sense of caution or tension felt by those around them, and will make them appear more approachable to others. Therefore, by assessing that the degree of suitability is high when the joy evaluation value is high, it becomes possible to provide an evaluation that meets such user expectations. As a fifth means, at least one of the emotions estimated by the emotion classifier is disgust. Among emotions, "disgust" is an evaluation item that, like "happiness," changes facial expression significantly. "Disgust" is the most universally expressed human facial expression, and when a person dislikes something, their facial expression changes significantly. Therefore, it is good to use "disgust," whose changes are easy to understand, as an evaluation item for calculating an evaluation value. Furthermore, "disgust" is an emotion that is evoked when a person perceives a surrounding situation or event negatively, and can increase tension around a person who expresses disgust on their face, or give the impression that they are difficult to approach. Therefore, "disgust" is a good evaluation item because it is easily correlated (related) with the suitability of the product, etc., to be evaluated. As a sixth means, the degree of suitability for the face is evaluated as being low as the evaluation value of dislike increases. Some users feel that wearing a product increases tension in the surroundings or gives the impression that they are unapproachable, and so by assessing the suitability as the dislike rating increases, the system can provide an evaluation that reflects such users' thoughts. As a seventh means, at least one of the emotions estimated by the emotion classifier is confusion. "Confusion" is an emotion that occurs when a person is struggling to accept the situation or events around them, and it can increase the concern of those around them who show confusion on their face, or prompt them to re-examine the situation. Therefore, "confusion" is a good evaluation item because it is easily correlated with the degree of suitability of the product, etc., that you are evaluating. As an eighth means, at least one of the emotions estimated by the emotion classifier is neutral. "Neutral" refers to a neutral emotion that is not based on any particular emotion. Because "neutral" means that there is no major change in emotion, using it as an evaluation item to base emotions estimated by emotion classifiers is important in comparison with cases where there are major changes. Furthermore, neutral is an emotion that indicates that a person is not influenced by the surrounding situation or events, and it can lead to others maintaining the same impression of a person with a neutral face, or to an increased sense of security and trust. Therefore, "neutral" can be a good evaluation item because it is likely to correlate (relate) to the suitability of the product, etc., being evaluated.

[0018] As a ninth means, the face image evaluator is an age classifier that estimates age based on a face image. When evaluating the suitability of a product, etc., age is an important evaluation item, and how users are perceived is also an evaluation item of great interest. Using an age classifier as the face image evaluator makes it possible to objectively evaluate how wearing the product changes the age that others estimate. Changing the impression of age by wearing the product can meet the user's wishes, for example, by making the user appear older, they can infer that they have accumulated that amount of experience and gain trust, or by making the user appear younger, they can infer that they have plenty of physical strength and be entrusted with more work. As a tenth means, the face image evaluator is a gender classifier that estimates gender based on a face image. If a gender classifier is used as the facial image evaluator, it becomes possible to objectively evaluate the gender that is assumed by others when wearing the product. The gender classifier not only statistically distinguishes between male and female, but also determines how masculine or feminine the person is. For example, it becomes possible to evaluate according to the user's wishes, such as wanting to appear more masculine, androgynous, or more feminine. In addition, as an eleventh means, the apparatus has a means for acquiring the user's requirements for how to be viewed, calculates a target evaluation value that suits the user from the requirements, and calculates the degree of suitability for the user's face by comparing the evaluation value calculated from the first evaluation value and the second evaluation value with the target evaluation value. This makes it possible to objectively provide the user with the most suitable accessories, daily necessities, makeup or hairstyle that meets their requirements for how they want to be seen by others. "User's desire for how they want to be seen" is a desire that can be evaluated by a face image evaluator, such as "I want to look five years younger," "I want to look intelligent," "I want to look sincere," or "I want to look feminine." Also, "means for acquiring desires" is, for example, when a user visually looks at a monitor screen attached to a computer device and inputs a desire using an input means such as operating a mouse pointer or using a touch panel to select from buttons or lists showing the desired options, or when the user conveys a desire in writing or verbally to a store clerk or the like who inputs it. It is advisable to set in advance the method for calculating the target value from the input request. For example, in response to a request such as "I want to look five years younger," the difference between the first evaluation value and the second evaluation value evaluated by the age classifier can be used as the evaluation value, and compared with the target evaluation value of -5 years old, and the smaller the difference between the evaluation value and the target evaluation value, the higher the degree of adaptability can be evaluated. Similarly, it is advisable to set in advance rules for the evaluation values ​​calculated by the emotion classifier and gender classifier on how to set the target evaluation value in accordance with the user's request. As a twelfth means, the first face image is a photograph of the face without the ornaments, the daily necessities, and the makeup. This is advantageous because it makes it possible to evaluate the degree of suitability of the product to be evaluated for the user's natural or ordinary face and expression. As a thirteenth means, the first facial image is based on photographed image data photographed by the user and sent by communication means. If the first facial image can be taken by the user and sent via a communication means, it is not necessary for the user's photo to be captured at, for example, a store that sells accessories, daily necessities, makeup, or hairstyles, thereby saving time and effort. Furthermore, the first facial image can be acquired without the user actually visiting a store that sells accessories, daily necessities, makeup, or hairstyles, eliminating the user's time and travel constraints. The first facial image does not have to be acquired at the time of evaluation, but may be one taken previously. For example, by using a facial image with a user's favorite expression, such as a natural expression taken while traveling, as the first facial image, the calculated compatibility level will be more in line with the user's wishes. As a fourteenth means, the degree of suitability of the user is evaluated based on two or more of the ornaments or daily necessities worn on or around the face of the user, or the makeup or hairstyle. Because multiple accessories, daily necessities, makeup, or hairstyles are often combined, it is meaningful to be able to evaluate the degree of compatibility when multiple combinations are used in this way, as this would enable evaluations that are closer to the user's everyday life. Such evaluations are possible by using a facial image evaluator. This is because the facial image evaluator is not trained on just one accessory or daily necessities worn on or around the face, or makeup or hairstyle, and therefore can evaluate a combination of two or more items without the need to create or adjust a new classifier. As a fifteenth means, the ornament or the daily necessities is a pair of glasses. Eyes are important for the impression of the face (how it is perceived), and eyeglasses are an accessory that sits close to the eyes, and are therefore an everyday item, so evaluating eyeglasses is particularly important for users. Also, there are a wide variety of eyeglass designs, and comparing many pairs at once to see which suits the user can lead to confusion during the evaluation process and is time-consuming, so there is great value in calculating the degree of suitability of eyeglasses and calculating an evaluation score.

[0019] Further, as a sixteenth means, there is provided a system using a computer device for evaluating the degree of suitability of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, comprising a first means for acquiring a first facial image, a second means for synthesizing an image of at least one of the ornament, the daily necessities, the makeup, and the hairstyle with the first facial image to create a second facial image, a third means for calculating a first evaluation value of the first facial image obtained by the first means, a fourth means for calculating a second evaluation value of the second facial image obtained by the second means, and a fifth means for calculating the degree of suitability to the user's face from the first evaluation value obtained by the third means and the second evaluation value obtained by the fourth means. The sixteenth means is the first means claimed as a product invention. This makes it possible to provide a system that objectively evaluates whether an ornament, daily necessities, or makeup or hairstyle suits a person based on the degree of suitability calculated each time. The "first means" may be, for example, an imaging device such as a camera. The image acquired by the imaging device may be digital facial image data, may be printed, such as a printed photograph, or may be displayed visually on a monitor screen. The facial image data may be transmitted via a communication means such as the Internet. The "second means" is a means for synthesizing an image of an accessory or the like with the first facial image that has already been captured, and is a control means of a computer device based on an image synthesis program. The "third means" is a device capable of calculating an evaluation value, such as a face image evaluator. The "fourth means" is a device capable of calculating an evaluation value, such as a face image evaluator. The third means and the fourth means may be the same. The "fifth means" is a control means of a computer device based on a program for calculating the degree of compatibility.

[0020] As a seventeenth means, a display means is provided for displaying information regarding the degree of suitability for the user's face. By providing such a display means, it is possible to present the first face image, the second face image, the first evaluation value, the second evaluation value, the degree of fit to the user's face, products with a high degree of fit to the user or store clerk, etc., and the process of calculating the degree of fit, thereby increasing the user's sense of satisfaction and understanding. Furthermore, as an 18th means, a display means is provided for displaying to the user two or more of the first evaluation value, the second evaluation value, and an evaluation value calculated from the first evaluation value and the second evaluation value. By presenting information about the evaluation value to the user, the changes caused by wearing accessories or daily necessities or changing makeup or hairstyle can be objectively and immediately recognized, providing an auxiliary means of consideration that increases the user's sense of satisfaction. As a nineteenth means, an input means for inputting the user's request is provided. By doing so, it is possible to provide a system for calculating the degree of compatibility in accordance with the user's requirements. in The input may include not only the user's request for how they would like to appear, but also, for example, the user's request for adjusting the position of a product or the like to be superimposed on the second image, or a request to select a product or the like that the user would like to evaluate. Furthermore, as a 20th means, a recommendation system is provided which is characterized by presenting to the user information on at least one of the accessories, the daily necessities, the makeup, and the hairstyle that suits the user based on the degree of suitability obtained by any of the systems of the 16th to 19th means. The system above presents suitable accessories or daily necessities, or makeup or hairstyles to the user, allowing the user to consider the suitable accessories or daily necessities, or makeup or hairstyles.

[0021] Also, as 21st means, there is provided a design system using a computer device for eyeglasses that fit a user's face, comprising first means for acquiring a first facial image, second means for synthesizing a plurality of images of eyeglasses to be evaluated with the first facial image to create a plurality of second facial images, third means for calculating a first evaluation value of the first facial image obtained by the first means, and fourth means for calculating a plurality of second evaluation values ​​of the second facial image obtained by the second means, and fifth means for calculating the degree of fit of a plurality of eyeglasses to be evaluated to the user's face from the first evaluation value obtained by the third means and the plurality of second evaluation values ​​obtained by the fourth means, and eyeglasses with the highest degree of fit based on the plurality of degrees of fit obtained by the fifth means are designed as eyeglasses that fit the user's face. This makes it possible to design actual eyeglasses based on the eyeglasses image that best suits the wearer based on the degree of fit calculated each time. Designing eyeglasses means designing and determining the shapes of the eyeglass frames, such as the rims, temples, bridges, end pieces, hinges, and nose pads. Also, tinting eyeglass lenses can make wrinkles less noticeable, and if the eyeglass lenses have prescription lenses, the size of the eyes and wrinkles seen through the eyeglass lenses will change. In other words, since facial feature points change depending on the eyeglass lenses, the eyeglasses to be evaluated may be those with prescription lenses or tinted eyeglass lenses when the second face image is created. "First Measure" to "Fifth Measure" are the same as the fifteenth measure. As a 22nd means, the images of the plurality of spectacles to be evaluated are virtually created by computer simulation. By creating the images virtually through computer simulation, it becomes possible to synthesize images of many types of eyeglasses to be evaluated against the first facial image, and it becomes possible to evaluate frame designs and lens colors that the user or eyeglass designer would never have thought of, and to provide the user with eyeglasses that fit best.

[0022] Also, as a 23rd means, there is provided a design system for eyeglasses using a computer device that fits to a user's face, comprising: a first means for acquiring a first facial image; a second means for synthesizing a plurality of images of eyeglass frames to be evaluated with the first facial image to create a plurality of second facial images; a third means for calculating a first evaluation value of the first facial image; and a fourth means for calculating a plurality of second evaluation values ​​of the second facial image. The system further comprises: a fifth means for calculating a degree of fit of a plurality of eyeglasses to be evaluated to the user's face from the first facial image evaluation value obtained by the third means and the plurality of second facial image evaluation values ​​obtained by the fourth means; a fifth means for calculating a group of coordinates where the eyeglass frames and the user's face overlap in the plurality of second facial images; and a fifth means for calculating a group of coordinates that improves or decreases the degree of fit to the user's face from a combination of the group of coordinates and the degree of fit. The system is configured to design eyeglasses that fit the user's face using the group of coordinates obtained by the fifth means. As in the fifth means, by calculating a coordinate group that improves or decreases the degree of fit to the user's face from a combination of the coordinate group and the degree of fit, it is possible to calculate a feature point (coordinate group) that is particularly important for the evaluation item to be evaluated, among the features of the face image evaluator related to the degree of fit.The coordinate group calculated by this fifth means is associated with improving or decreasing the degree of fit to the user's face, so it is possible to predict the degree of fit of the eyeglass lens frame to the facial image when the eyeglass lens frame passes over the coordinate group or passes around the coordinate group.This makes it possible to design eyeglasses that fit the user's face using the coordinate group.In addition, it is possible to present multiple recommended eyeglass designs to the user, such as showing options for eyeglasses that pass through the coordinate group. As a 24th means, the face image evaluator is an emotion classifier that identifies positive emotions and is designed to evaluate the degree of compatibility as high when the positive emotion matches the positive emotion desired by the user. In this way, it will be possible to design glasses that match the user's positive emotions, such as joy and happiness. As a 25th means, the face image evaluator is an emotion classifier, and the emotion classifier is designed to classify negative emotions and to have a high degree of compatibility when the negative emotion matches the negative emotion desired by the user. This makes it possible to design glasses that match the user's negative emotions. Negative emotions include confusion, disgust, and sadness. Normally, product design would be done with positive emotions as the target value for evaluation, but negative emotions are sometimes easier to measure than positive emotions. Therefore, designing with negative emotions as the target is advantageous because it can be expected that positive emotions will be closer to the design target. As a 26th means, the face image evaluator is an emotion classifier, and is designed to evaluate the degree of suitability as low when the emotion is opposite to the emotion desired by the user. It may be difficult to directly evaluate the emotion desired by the user in response to a request. In such cases, the accuracy of the evaluation of the degree of compatibility can be improved by evaluating the emotion opposite to the emotion desired by the user and evaluating the degree of compatibility. Also, as a 27th means, there is provided a method for evaluating the degree of suitability of an accessory or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, in which the second facial image is presented to the user or an evaluator other than the user, a facial image of the user or the evaluator other than the user when viewing the second facial image is obtained as a third facial image, the third facial image is evaluated by a facial image evaluator to calculate a third evaluation value as an evaluation value of the third facial image, and the degree of suitability to the user's face is calculated from the third evaluation value. The "third facial image" is obtained by capturing a facial image with a camera or the like when a second facial image (i.e., a facial image of the user wearing an accessory or daily necessities to be worn around the face, or a user wearing makeup, hairstyle, etc.) is presented to the user or an evaluator other than the user. The third facial image may be recorded as a still image or a video. By evaluating this third facial image with a facial image evaluator to obtain a third evaluation value, it becomes possible to evaluate how the subject (the user or an evaluator other than the user) felt when viewing the second facial image through the third facial image. In this case, if the subject is the user, the evaluation value obtained will be how the user felt when seeing themselves wearing the accessory or daily necessities, etc. that they are considering purchasing. Furthermore, if the subject is not the user, the evaluation value obtained will be how a third party felt when seeing the user wearing the accessory or daily necessities, etc. The "evaluator other than the user" is preferably, for example, a friend or family member of the user, someone who visits a store with the user, or a store clerk. The third image may be a photograph of the user looking at the same image in a store, or may be a photograph of the user looking at the second image from a remote location via communication means such as the Internet or videophone. The camera may be attached to a PC or tablet, or may function as a standalone camera. The camera position for acquiring the third facial image may be from the front, diagonal, or side of the subject (user or evaluator other than the user). A frontal view facilitates evaluation by the facial image evaluator, while a diagonal or side view allows for the capture of a natural facial expression without the subject being aware of the camera's presence. It is particularly preferable for the facial image evaluator to use an expression classifier. The degree of fit to the user's face is preferably calculated from the first evaluation value, the second evaluation value, and the third evaluation value. The first evaluation value and the second evaluation value are calculated using a facial image evaluator, and therefore reflect how many other people (an unspecified majority) perceived them. The third evaluation value based on the third face image reflects how a specific person or the user (the user's acquaintances, family, etc., or the user himself / herself) felt, rather than how many other people felt.Some users place more importance on how they are perceived by specific individuals than by an unspecified number of people. Therefore, using a third evaluation value is advantageous because it allows the evaluation value to reflect how specific individuals feel. Some users are more concerned with how they or specific others feel, while others place more importance on how an unspecified number of others feel. Because such values ​​differ from user to user, it is desirable to have a means for inputting the user's judgment criteria into the system in advance. For example, it is desirable to input the ratio of specific to unspecified. It is desirable to obtain multiple third facial images by presenting multiple second facial images in sequence, obtain multiple third evaluation values ​​as evaluation values ​​for the multiple third facial images, and compare the multiple third evaluation values. In this way, it is possible to obtain how the subject felt about multiple potential purchases and present this to the user. Furthermore, as a 28th means, a system for evaluating the degree of fit of an accessory or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face is provided, which includes a display means for presenting the second facial image to the user or an evaluator other than the user, a sixth means for acquiring a facial image of the user or the evaluator other than the user when viewing the second facial image as a third facial image, and a seventh means for evaluating the third facial image with a facial image evaluator and calculating a third evaluation value as an evaluation value of the third facial image. The 28th means is a system configuration that realizes the 27th means. In addition to the system of the 21st means, by providing a "display means" that presents the second face image to the user or an evaluator other than the user, a "sixth means" that is an imaging means (camera, video camera, etc.) for acquiring the third face image, and a "seventh means" that calculates the third evaluation value, it becomes possible to objectively measure how the user or an evaluator other than the user felt when viewing the second face image, that is, for example, whether the user looks good when wearing an accessory, daily necessities, etc. that the user wants to evaluate. Also, as a 29th means, a glasses design system is provided, which includes a display means for presenting the plurality of second facial images to the user or an evaluator other than the user, a sixth means for acquiring facial images of the user or the evaluator other than the user when viewing the plurality of second facial images as a plurality of third facial images, and a seventh means for evaluating the plurality of third facial images with a facial image evaluator and calculating a plurality of third evaluation values ​​as evaluation values ​​of the plurality of third facial images. This makes it possible to provide an eyeglass design system that reflects not only the evaluations of an unspecified number of people but also the evaluations of specific people (for example, the user, the user's acquaintances, family, store clerks, etc.). Here, since users have values ​​that emphasize how specific people or themselves feel, and values ​​that emphasize how an unspecified number of people feel, the system should preferably be equipped with an input means for inputting the user's values ​​in advance. By using the third evaluation value, eyeglasses that the user prefers can be selectively recommended from among eyeglasses designed by the means 21 to 23. In addition, if eyeglasses designed by the means 21 to 23 include designs that are too unusual compared to other eyeglasses, or designs that the user wants to avoid because they know someone wearing the same design, the system can efficiently eliminate them from the options. The inventions of the first to twenty-ninth means described above can be combined in any way. In particular, it is preferable to have the configuration of the first means, the sixteenth means, the twenty-first means, or the twenty-third means, and combine it with at least one configuration of each of the other inventions of the above means. Any component of each of the inventions of the first to twenty-ninth means may be extracted and combined with another component. [Effects of the Invention]

[0023] The above invention makes it possible to objectively determine whether an accessory or daily necessities to be worn on or around the face, or makeup or hairstyle, etc., suits the user's face based on the degree of suitability. Furthermore, it is possible to select an accessory or daily necessities to be worn around the face, or makeup or hairstyle, etc., that suits the user's face, without the need to create a huge database. [Brief explanation of the drawings]

[0024] [Figure 1] 10(a) to 10(d) are schematic explanatory diagrams outlining the principle of calculating the degree of suitability of a product to be evaluated for a face using the facial image evaluator of the present invention, illustrating the flow of calculating an evaluation value from a first facial image. [Figure 2] 10(a) to 10(d) are schematic explanatory diagrams outlining the principle of calculating the degree of suitability of a product to be evaluated for a face using the facial image evaluator of the present invention, illustrating the flow of calculating an evaluation value from a second facial image. [Figure 3] An illustration of the well-known Plutchik's wheel of emotions model. [Figure 4] FIG. 2 is a block diagram illustrating an electrical configuration according to the embodiment. [Figure 5] 10 is a flowchart illustrating a process for calculating the suitability of a plurality of eyeglass images in consideration of the customer's needs, and then displaying eyeglass images on a monitor as recommended products for the customer. [Figure 6] FIG. 3 is a schematic explanatory view illustrating a state in which a first facial image of a user and choice buttons are displayed on a monitor screen in the first embodiment. [Figure 7] FIG. 3 is a diagram showing an eyeglasses image displayed on a monitor screen in the first embodiment. [Figure 8] 8 is a diagram of a second face image obtained by combining the three types of eyeglass images in FIG. 7 with the first face image. [Figure 9] FIG. 9 is an explanatory diagram illustrating the results of evaluating the second face image in FIG. 8 by a face image evaluator. [Figure 10] FIG. 10 is a schematic explanatory view illustrating a state in which a first facial image of a user and choice buttons are displayed on a monitor screen in the second embodiment. [Figure 11] FIG. 11 is a schematic explanatory diagram outlining the process from obtaining a second face image obtained by combining a number of eyeglasses images to adopting an eyeglasses design with a high degree of suitability in the third embodiment. [Figure 12] FIG. 13 is an explanatory diagram illustrating the relationship between a second face image obtained by combining a number of eyeglass images and the degree of compatibility in the fourth embodiment. [Figure 13] (A) is a diagram of a face image in which a group of coordinates common to products with a predetermined or higher degree of compatibility is combined with a first face image in embodiment 4, (B) is an example of a frame design that includes the common group of coordinates and has a high degree of compatibility, and (C) is another example of a frame design that includes the common group of coordinates and has a high degree of compatibility. [Figure 14] FIG. 13 is a schematic explanatory diagram outlining the process of obtaining a second face image synthesized with an eyebrow image, calculating first and second evaluation values, and calculating the degree of compatibility based on the difference between the evaluation values ​​in the fifth embodiment. [Figure 15] FIG. 20 is a schematic explanatory diagram outlining the process of obtaining a second face image by combining images of lips (lipstick) and glasses, calculating first and second evaluation values, and calculating the degree of compatibility based on the difference between the evaluation values ​​in the sixth embodiment. [Figure 16] A flowchart explaining the process of taking into consideration the customer's requests for multiple eyeglasses images, as well as estimating and considering the evaluations of specific people to calculate the degree of suitability, and then displaying eyeglasses images on a monitor as recommended products to the customer. DETAILED DESCRIPTION OF THE INVENTION

[0025] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First, an outline of the electrical configuration common to the following embodiments will be described. As shown in Figure 4, a monitor 2, an input device 3, a camera 4, and a facial image evaluator 5 are connected to a calculation computer device 1. The monitor 2 displays the user's facial image on the screen, displays a GUI input screen for inputting information for calculation, and displays the calculation results. The input device 3 is composed of a keyboard, a mouse, etc., and by operating these, a program for evaluating the degree of compatibility can be started, and calculations based on the program can be carried out by predetermined input, and the calculation results can be obtained. The camera 4 captures the user's facial image. The facial image evaluator 5 is a device having a hardware circuit, and calculates an evaluation value based on predetermined evaluation items. The facial image evaluator 5 may also be software built into the storage device 8. Although the basic configuration has been described above, it is also possible to apply user face image data stored in another device to the calculation computer device 1. In this case, the device does not need to be directly connected to the calculation computer device 1, but may be transferred from another device such as another computer or data storage device connected via wireless communication or LAN.

[0026] The calculation computer device 1 is electrically configured by a CPU (Central Processing Unit) 7 and peripheral devices such as a storage device 8 consisting of ROM, RAM, etc. The CPU 7, which is a control means, executes the following main control based on input operations from the input device 3 and in accordance with a compatibility calculation program stored in the storage device 8. (1) The image (first facial image) captured by the camera 4 is displayed on the screen of the monitor 2, and the acquired first facial image data of the user is temporarily stored in the storage device 8. (2) The facial image evaluator 5 calculates an evaluation value (first evaluation value) based on the facial image data stored in the storage device 8, and the result is temporarily stored in the storage device 8. (3) The first facial image data stored in the storage device 8 is called up and displayed on the screen of the monitor 2, and multiple eyeglass image data as accessories (or everyday items) stored in the storage device 8 are called up and combined with the first facial image data, and temporarily stored in the storage device 8. The combined multiple images (second facial images) are also called up and displayed on the screen of the monitor 2. (4) The facial image evaluator 5 calculates an evaluation value (second evaluation value) for each of the plurality of second facial image data, and the results are stored in the storage device 8. (5) A difference value is calculated between each of the first evaluation value and the plurality of second evaluation values, and a degree of compatibility is calculated based on the difference value. The degree of compatibility is calculated by changing the calculation method according to the input user request. (6) The results of the calculation of the degree of fit are displayed on the screen of monitor 2. Furthermore, the order of the degree of fit is determined from the calculation results of the multiple degrees of fit. For example, the eyeglasses image data with the best degree of fit may be displayed, or the calculation results may be displayed in descending order of degree of fit. For example, the first face image may be displayed on the left side of the screen, and the second face image may be displayed on the right side of the screen together with numerical values ​​and graphs of the degree of fit.

[0027] Specific embodiments will be described below with reference to the drawings. (Embodiment 1) The first embodiment will be described with reference to the flowchart of FIG. 5, taking as an example a case where a store selling eyeglasses recommends eyeglasses to a customer based on a calculation of the degree of suitability. In step S1, as shown in FIG. 6, the CPU 7 of the calculation computer device 1 displays the captured image of the user's face (first face image) on the screen 2a of the monitor 2. Three option buttons 11A to 11C, "Looks sincere," "Looks kind," and "As unchanged as possible," are displayed in a GUI adjacent to the user's first face image 10 to prompt the user for input. The option buttons 11A to 11C are themselves input areas associated with links to other screens. The user looks at the screen 2a, selects one of these, and operates the input device 3 to select one of their desires (desired appearance). The user desires are used to set a target value when calculating the fitness of the product to be evaluated. In step S2, if the CPU 7 determines that one of "appears sincere," "appears kind," and "remains as unchanged as possible" has been selected, it selects a weighting factor and a calculation method for calculating the degree of suitability according to the selection. First, suppose the user selects "Looks honest." In this case, the coefficients (weighting coefficients) used to calculate the degree of compatibility from the output of the face image evaluator 5 are set to 1.0 for "Neutral," -0.5 for "Happy," and 0.0 for the rest. The larger the calculated value, the higher the importance of the evaluation item in calculating the degree of compatibility. Also, suppose the user selects "Looks kind." In this case, the coefficients (weighting coefficients) used to calculate the degree of compatibility from the output of the face image evaluator 5 are set to 0.5 for "Neutral," 1.0 for "Happy," -1.0 for "Anger," and 0.0 for the rest. The larger the calculated value, the higher the importance of the evaluation item in calculating the degree of compatibility. Also, suppose the user selects "as unchanged as possible." The smaller the sum of the absolute values ​​of the differences between all outputs of the facial image evaluator 5 between the first facial image and the second facial image synthesized with the product to be evaluated, the higher the degree of conformance is evaluated. In this case, it is not necessary to evaluate all evaluation items, and it is possible to increase (or decrease) the importance of some evaluation items by setting weighting coefficients.

[0028] Next, the process proceeds to step S3, where CPU 7 simultaneously displays a plurality of types of eyeglass images 12A-12C shown in Fig. 7 on screen 2a of monitor 2. When the user inputs that they have been authenticated, the screen changes and second facial images 13A-13C, which are obtained by combining eyeglass images 12A-12C with first facial image 10, are displayed side by side, as shown in Fig. 8. When the user inputs that they have been authenticated, the suitability (degree of suitability) is calculated in step S4. The degree of conformance is calculated based on the evaluation value obtained by the face image evaluator 5. The CPU 7 calculates the difference between the first evaluation value of the first face image 10 serving as a reference and the second evaluation value of each of the second face images 13A to 13C, and calculates the degree of conformance for the difference. The evaluation results of the face image evaluator 5 are shown in Fig. 9. For example, the first face image 10 was evaluated as having an age of 29, with a neutral value of 0.51, a happy value of 0.18, an angry value of 0.12, a sad value of 0.11, and a surprised value of 0.0. When the "appears honest" case is selected, the suitability of the second face image 13A in the case of the eyeglasses image 12A is -0.62, the suitability of the second face image 13B in the case of the eyeglasses image 12B is -0.75, and the suitability of the second face image 13C in the case of the eyeglasses image 12C is 0.07, and the eyeglasses image 12C is evaluated as having the highest suitability. Therefore, the eyeglasses image 12C is determined to be the product recommended to the user. Furthermore, when "looks gentle" is selected, the suitability of second face image 13A for eyeglasses image 12A is 0.24, the suitability of second face image 13B for eyeglasses image 12B is 0.48, and the suitability of second face image 13C for eyeglasses image 12C is 0.01, so eyeglasses image 12B is evaluated as having the highest suitability. Therefore, eyeglasses image 12B is determined to be the product recommended to the user. Furthermore, when "keep as unchanged as possible" is selected, the suitability of the second face image 13A for the eyeglasses image 12A is 0.98, the suitability of the second face image 13B for the eyeglasses image 12B is 1.20, and the suitability of the second face image 13C for the eyeglasses image 12C is 0.74, and the eyeglasses image 12C, which has the smallest value, is evaluated as having the highest suitability. Therefore, the eyeglasses image 12C is determined to be the product recommended to the user. Finally, only the eyeglasses image 12C is presented on the display as the recommended product. For comparison, other eyeglasses images 12A and 12B may also be displayed. In step S5, the CPU 7 displays on the screen 2a of the monitor 2 an eyeglasses image of a recommended product based on the calculated suitability. In the case of "looks sincere," only eyeglasses image 12C is presented on the display as a recommended product. In the case of "looks kind," only eyeglasses image 12B is presented on the display as a recommended product. In the case of "as unchanged as possible," only eyeglasses image 12C is presented on the display as a recommended product. For comparison, other eyeglasses images 12A and 12B may also be displayed.

[0029] With this configuration, the first embodiment provides the following effects. (1) The cost of building the system can be reduced because the face image evaluator can evaluate whether the glasses suit the user's face and the product each time and calculate the degree of suitability, without creating a database that links "faces," "products," and "evaluation values." (2) The user's face only needs to be captured once as the first face image 10, and the burden on the user is significantly reduced because the user does not have to go through the trouble of having to wear glasses multiple times. (3) Because the user's requests can be reflected as options, the system is more user-friendly and can provide eyeglasses that are suited to the user. (4) It is possible to provide objective, recommended eyeglass shapes without subjectivity.

[0030] (Embodiment 2) The second embodiment is also an example of a case where a store that sells eyeglasses as decorative items etc. recommends eyeglasses to a customer by calculating the degree of suitability. In the second embodiment, options are provided for age as an evaluation item. The basic routine is the same as in the first embodiment, and only the content of step S2 is different, so only the content unique to the second embodiment will be explained. As shown in Fig. 10, the CPU 7 displays the captured image of the user's face (first face image) on the screen 2a of the monitor 2. Option buttons 15A to 15C for receiving the user's request regarding age are displayed in a GUI adjacent to the first face image 10 to prompt the user to input. In the second embodiment, the user is asked to be older and / or younger than the actual age, and here, as an example, a three-year age difference is set. When calculating the degree of compatibility, the CPU 7 evaluates the option buttons 15A to 15C so that the age desired by the user will have a high degree of compatibility (evaluation value). Assume that the user selects "-3 years old." When a second face image, which is a composite of first face image 10 and eyeglasses images 12A to 12C, is evaluated by a face image evaluator, the age of the first face image is evaluated as 29 years old, 27 years old when wearing eyeglasses image 12A, 28 years old when wearing eyeglasses image 12B, and 30 years old when wearing eyeglasses image 12C. The age difference from first face image 10 is -2 years old for eyeglasses image 12A, -1 year old for eyeglasses image 12B, and +1 year old for eyeglasses image 12C. If the compatibility is calculated with "-3 years old" as a compatibility of 1.0, the compatibility of eyeglasses image 12A with first face image 10 is 0.9, the compatibility of eyeglasses image 12B is 0.8, and the compatibility of eyeglasses image 12C is 0.6. Therefore, eyeglasses image 12A is determined to have the highest compatibility. Therefore, eyeglasses image 12A is determined to be recommended to the user, and this image is displayed on screen 2a of monitor 2. At this time, a sentence explaining the effect of wearing the glasses is attached, for example, "Glasses image 12A can make the person look two years younger." The matching score may be displayed as 0.9. As a simple example of how to calculate the matching score, an estimated age of 26 is set as 1.0, and 0.1 is subtracted for each year of age difference. By configuring in this way, in addition to the effect of embodiment 1, embodiment 2 can provide glasses that are ideal for making the wearer appear younger or more mature than their age.

[0031] (Embodiment 3) The third embodiment is an application of the compatibility evaluation system of the present invention to a product design system. The above first and second embodiments are based on the viewpoint of selecting eyeglasses recommended to the user, but this embodiment is based on the viewpoint of newly designing eyeglasses that fit well to a person. As shown in FIG. 11 , many new designs of eyeglasses A to X are created. Here, X represents the majority of the eyeglasses. Then, as in the first embodiment, a first facial image 21 is acquired, and each of the designed eyeglasses A to X is combined with the first facial image 21 to obtain multiple second facial images 22A to 22X. An evaluation value is then calculated using a facial image evaluator 5. A suitability score is calculated for each pattern based on the difference between the first evaluation score of the first facial image 21 and the second evaluation scores of the multiple second facial images 22A to 22X, and the eyeglass design with the highest suitability score is adopted. When calculating the suitability score, if this product design system is used at the user's purchasing stage, the user's request for the first facial image 21 is acquired via input means, and the suitability score for the request is calculated. Furthermore, if this product design system is used at the product development stage by a manufacturer, for example, a designer may predict the user's request and determine a product concept, such as "glasses that make the user look five years younger and smile beautifully," and the suitability score may be calculated using that product concept as the request. This allows us to select the eyeglasses that best fit the individual and offer them as a product. In this example, design B has the highest fit score, so we can determine that it fits the individual best.

[0032] (Fourth embodiment) The fourth embodiment is a variation of the third embodiment. First, a first face image is acquired in the same manner as in embodiment 3. Then, the following steps (1) to (3) are performed. (1) Create as many images as possible of the product you are designing. While the images can be designed by a designer, in this case we will have a computer automatically design virtual eyeglass frames. (2) Next, the created images are combined with the first face image to obtain a plurality of second face images 22a, 22b, . . . 22x as shown in Fig. 12. Then, the degree of fit is calculated for all glasses in the same way as in embodiment 3. At this time, the calculation method, which evaluation items to use from the output of the face image evaluator and how to add them up to obtain the degree of fit can be arbitrarily decided for each design. (3) Next, a level of conformance to be used in design is set. First, only images of products calculated to have conformance levels equal to or higher than the set level are retained. In this fourth embodiment, only designs with conformance levels of 0.6 or higher are adopted. Then, second facial images with conformance levels of 0.6 or higher are selected from the group of second facial images 22a to 22x, and coordinates common to many of the selected second facial images are determined. The coordinates are set two-dimensionally with the center of the face as the origin, but depth coordinates may also be acquired to make them three-dimensional.

[0033] To find the common coordinates, do the following: First, each image can be divided into two: coordinates occupied by the product and the background image. The coordinates occupied by the eyeglasses image data to be used as the product are binarized as 1, and the background image is binarized as 0, creating two-dimensional data for the second facial images 22a-22x. Next, data is created by adding together the coordinate values ​​of the binarized second facial images 22a-22x that have a compatibility score of 0.6 or higher. The data thus created has a larger value for the coordinates that are more common between designs with a high compatibility score. In other words, these coordinates represent feature points common to designs with a high compatibility score. Figure 13 shows a facial image in which a set of coordinates 23 common to products with a compatibility score of 0.6 or higher is superimposed on the first facial image. In Figures 13(A)-(C), dots are placed on the common coordinates 23, making it appear as if part of the frame is displayed. Then, the product is designed so that it contains many coordinates with large values ​​(feature points in Figure 13). For example, glasses with a high degree of fit are designed as shown in Figure 13(B) or 13(C), which contain the feature point (coordinate point 23) in Figure 13. In this way, it is possible to design a product that is likely to be highly rated for fit by an evaluation system. This type of design can be performed by the manufacturer when the product is manufactured, and the product is processed and delivered to the user, or it can be performed within an application operated on a PC, smartphone, tablet, etc. when the user places an order, and the data determined by the user after comparison and consideration is received by the manufacturer, which processes the product and delivers it.

[0034] (Embodiment 5) The fifth embodiment shows an example of product recommendation and design for other types of products, not just eyeglasses. Evaluation systems and recommendation and design systems can be configured for other types of products in the same manner as above. FIG. 14 is a schematic diagram of the process of obtaining a second facial image 27 by combining a first facial image 25 with an image 26 of eyebrows decorating the eyebrows, determining the degree of compatibility using the facial image evaluator 5 in the same manner as described above, and then calculating the degree of compatibility from the difference between the first and second evaluation values. The first evaluation value for the first facial image and the second evaluation value for the second facial image are as shown in FIG. 14. Here, when the user selects "appears sincere" as a preference, the weighting coefficients are set in advance so that the weighting coefficient for "neutral" is 1.0 and the weighting coefficient for "joyed" is -0.5. By multiplying the difference between the first and second facial images by the weighting coefficients, the result is 1.0 x (0.65 - 0.51) - 0.5 x (0.09 - 0.18) = 0.19, and the degree of compatibility is calculated as 0.19. Furthermore, suppose that when a second face image is created by combining another eyebrow image (not shown) and the compatibility is evaluated, the compatibility is calculated to be 0.1. In this case, it can be shown to the user that eyebrow image 26, which has a compatibility of 0.19, is more suitable for the user's needs. (Embodiment 6) The sixth embodiment is a variation of the fifth embodiment. The first facial image 25 in the fifth embodiment and the first facial image 28 in the sixth embodiment are the same facial image, and are examples of the same user comparing and considering different products. FIG. 15 is a schematic diagram illustrating the process of simultaneously combining eyeglasses and lip-decorating products (lipstick) with a first facial image 28 of the fifth embodiment to obtain a second facial image 29, calculating the degree of compatibility using the facial image evaluator 5 as described above, and then calculating the degree of compatibility from the difference between the first and second evaluation values. When a user selects "looking sincere" as a preference, the degree of compatibility of the second facial image 29 is calculated as 0.25. In other words, for the user's desired appearance in the first facial image, applying makeup to the "eyebrows" as in the fifth embodiment is 0.19, while applying "glasses" and "lipstick" as in the sixth embodiment is 0.25. This indicates that coordinating eyeglasses and lipstick results in a higher degree of compatibility (a way of looking that meets the user's preference) than applying makeup to the eyebrows alone, and this result can be presented to the user as objective data. It should be noted that combinations of "eyebrows" and "lipstick," "eyebrows" and "glasses," "eyebrows" and "glasses" and "lipstick," or other combinations of products are also possible. In particular, since both glasses and makeup are items that decorate the face and are commonly worn in everyday life, it is significant that they can be evaluated in combination as in the sixth embodiment.

[0035] (Embodiment 7) In the seventh embodiment, a process of designing specific eyeglasses based on recommended eyeglass images and providing them to users will be described. When a user decides to purchase the glasses recommended by the recommendation system in the above embodiments 1 to 5, a set of image data and fit data of the recommended glasses is sent to the eyeglass frame manufacturer as manufacturing information. This manufacturing information can be either online information such as the Internet or offline information such as by letter or facsimile. If the system is located in an eyeglass store, data about the eyeglass lenses will also be sent at the same time. Based on the provided data, the eyeglass frame manufacturer manufactures eyeglass frames that match the image data, for example by molding plastic. Next, the manufactured eyeglass frames or their data are sent to an eyeglass lens manufacturer, along with fitting data and eyeglass lens information. The eyeglass lens manufacturer manufactures eyeglass lenses based on the provided data. The eyeglass frame manufacturer and eyeglass lens manufacturer then send the eyeglass frames and eyeglass lenses to an eyeglass store, where the eyeglass lenses are assembled into the eyeglass frames and provided to the user of the recommendation system. The manufacturing method for eyeglass frames when they are designed using the design system of embodiment 3 will be described below. When it is decided to manufacture eyeglass frames designed by the design system, a set of product image data and fit data is sent to the eyeglass frame manufacturer. Based on the sent data, the eyeglass frame manufacturer manufactures eyeglass frames that match the image data by molding plastic or the like. The manufactured eyeglass frames are sent to the user of the design system or to a customer who has signed a purchase contract.

[0036] (Embodiment 8) As with the first and second embodiments, the eighth embodiment is an example of a case where a store selling eyeglasses as accessories or the like recommends eyeglasses to a customer by calculating the degree of suitability. The eighth embodiment is an example of a case where a user visits an eyeglass store accompanied by a friend, and the third image is acquired by the user or the friend to correct the evaluation value. In the eighth embodiment, as in the first embodiment, processing is performed by the CPU 7 of the calculation computer device 1. In the eighth embodiment, the main control of paragraphs (1) to (6) of the first embodiment is performed, and furthermore, the following additional control (calculation) is performed. (7) The second image is displayed on the screen of the monitor 2, and a facial image of the user or a third party other than the user (e.g., a friend, family member, etc.) when the user or the third party other than the user views the displayed second image is captured by the camera 4 to obtain a third facial image, and the data of the third facial image is temporarily stored in the storage device 8. (8) The facial image evaluator 5 calculates an evaluation value (third evaluation value) based on the facial image data of the third facial image stored in the storage device 8, and the result is temporarily stored in the storage device 8. (9) If there is another second image to be evaluated, change the second facial image and repeat the steps (7) and (8). A plurality of third evaluation values ​​are stored in the storage device 8. (10) A plurality of third evaluation values ​​are called up from the storage device 8, and the plurality of ornaments (or everyday items) called up in the control of (3) are associated with the plurality of third evaluation values, and the degree of compatibility is calculated from the third evaluation values. (11) The degree of compatibility calculated from the first evaluation value and the second evaluation value is corrected using the degree of compatibility calculated from the third evaluation value. Control (6) The corrected degree of compatibility is displayed on the screen of monitor 2. At this time, for example, in addition to the corrected degree of compatibility, the degree of compatibility calculated from the first evaluation value and the second evaluation value and the degree of compatibility calculated from the third evaluation value may also be displayed on the screen of monitor 2.

[0037] The following description will be given according to the flowchart in Fig. 16. Contents that are the same as those in the first and second embodiments will be omitted. In step S11, CPU 7 displays the captured image of the user's face as a first image on the monitor screen. Along with the first image, the monitor screen displays three pairs of eyeglasses (12A to 12C in FIG. 7) that the user is considering purchasing. Along with the desired appearance input (desired appearance) of the first embodiment, a user's desired appearance input screen is displayed, asking the question "How important is your opinion or that of people close to you about whether eyeglasses suit you?" Answers "1: Very important, 2: Important, 3: Don't care, 4: Don't care at all" are displayed on the screen so that the user can enter them via a touch panel. If "1: Very important" or "2: Important" is entered, the weighting coefficient of the third evaluation value is increased and reflected, as described below. In step S12, the CPU 7 sets a predetermined weighting factor based on the input and performs calculations. For example, if "2: Important" is selected as the input above, a further question is asked: "Which is more important in terms of whether something looks good on you? 1: Your opinion or 2: Other people's opinions?" Here, it is assumed that "1: Your opinion" is selected. Here, by selecting "2: Important" for the first question, the first evaluation value and the second evaluation value are weighted with a weight of 1 (weight coefficient set to 0.33) when calculating the degree of suitability, and the third evaluation value is weighted with a weight of 2 (weight coefficient set to 0.66). Also, by selecting "1: My opinion" for the second question, an instruction "Please sit in front of the screen" is displayed on the screen, and by the user following this instruction and sitting in front of the screen, the user's facial image can be acquired at any time by the built-in camera 4. This is the process when the third facial image is acquired from the user himself. On the other hand, if the user selects "2: Other people's opinions" in the second question, the user is prompted to "Please have the person you want to rate sit in front of the screen," and the friend accompanying the user is asked to sit in front of the screen, and the friend's facial image is acquired using the built-in camera 4. This is the process for acquiring a third facial image from another person. Note that in the eighth embodiment, for the sake of explanation, an example is shown in which the subject is instructed to position themselves in front of the camera 4. However, if the subject moves in front of the camera for the evaluation, it may be impossible to acquire a natural facial expression from the subject. For this reason, it is preferable to not specify the positions of the evaluator and the camera, but to use a wide-angle camera as the camera 4, for example, to simultaneously capture images of the user and an evaluator other than the user (a friend in this embodiment), and then select the user or the evaluator other than the user from the multiple facial images using a classifier to acquire the third facial image.

[0038] Next, in step S13, a process for acquiring a third image is executed based on the presentation of the second image. Specifically, the process is executed as follows. First, CPU 7 displays a screen of the user's face wearing glasses 12B as the second image. Then, the person seated in front of the monitor is prompted to look at this face image on the monitor screen. CPU 7 controls built-in camera 4 to capture an image of the person seated in front of the monitor between just before and just after the second face image is displayed, for example, from two seconds before to two seconds after the second face image is displayed. This makes it possible to compare the face image (expression) of the person just before the second face image is presented with the face image (expression) of the person just after the second face image is presented, and obtain an image of the user's face whose expression has changed due to the presentation of glasses 12B (the face of the user wearing them). After viewing the second image, the image is processed by cropping only the face area and adjusting the angle, etc., to create a "third face image." In addition, a facial image of the subject immediately before the second facial image is presented is similarly cut out and processed to produce a "fourth facial image." The fourth facial image and the third facial image are evaluated by a facial expression evaluator to calculate a fourth evaluation value and a third evaluation value, and the difference between the third evaluation value and the fourth evaluation value is calculated, thereby making it possible to evaluate a change in facial expression, i.e., how the subject felt when viewing the second facial image. Here, in order to obtain how the subject felt when viewing the second facial image, it is more accurate to observe the changes before and after the second facial image is presented, so it is more preferable to obtain how the "third facial image" changed relative to the "fourth facial image." However, information on the "third facial image" alone also contains information on how the subject felt when viewing the second facial image. The change in the user's facial expression upon being presented with the second image reflects how the user felt when looking at their own face wearing glasses 12B. While the above explanation was given in detail for the case where glasses 12B were worn, the screen is similarly displayed with the user's face wearing glasses 12A and 12C to obtain a "third face image" and its evaluation value. The compatibility is calculated based on the evaluation values ​​of the multiple (three) types of "third face images" obtained based on each pair of glasses. The acquisition of the "third face image" for the glasses 12A to 12C and the calculation of its evaluation value may be done once each, but since a comparison is usually desired, it is preferable to repeatedly display the "second face image" in random order, and it is better for the user or the user's friend, etc., to switch images themselves. It is preferable to have input means such as a touch panel, mouse, input button, etc. for changing the second face image so that the user or the user's friend, etc., can switch images themselves.

[0039] Next, in step S14, the CPU 7 calculates the degree of compatibility based on the instruction to "execute evaluation" from the user. In the compatibility calculation, a facial image evaluator outputs a first evaluation value from the first facial image and a second evaluation value from the second facial image. The compatibility calculated from the first and second evaluation values ​​is then corrected using a third evaluation value obtained by evaluating a third facial image with the facial image evaluator. The specific compatibility calculation in step S14 is as follows. The process up to calculating the compatibility from the first and second facial images is the same as in the first and second embodiments, so details will be omitted. At this point, the compatibility was 0.24 for glasses 12A, 0.48 for glasses 12B, and 0.01 for glasses 12C. From this point, the compatibility is further corrected using a third evaluation value. In this calculation, the third evaluation value is calculated by adding weighting coefficients (1.0 for "happy"), (-1.0 for "anger"), and (0.0) for all other expressions from the facial expression classifier. The outputs of the facial expression classifier for glasses 12B were 0.33 for neutral, 0.22 for happy, 0.13 for angry, 0.13 for sad, and 0.10 for surprised, so the third evaluation value for glasses 12B was 1.0 × 0.22 - 1.0 × 0.13 = 0.09. Similarly, for glasses 12A, when the third image was evaluated with the facial expression classifier, the outputs were 0.35 for neutral, 0.39 for happy, 0.14 for angry, 0.04 for sad, and 0.08 for surprised, so the third evaluation value for glasses 12A was 1.0 × 0.39 - 1.0 × 0.14 = 0.25. For glasses 12C, when the third image was evaluated using the facial expression classifier, the output was 0.41 for neutral, 0.18 for happy, 0.11 for angry, 0.13 for sad, and 0.14 for surprised, so the third evaluation value for glasses 12C was 1.0 × 0.18 - 1.0 × 0.11 = 0.07. When the fitness was corrected using these third evaluation values ​​and the weights set in step S12, the results were 0.33 × 0.24 + 0.66 × 0.25 = 0.24 for glasses 12A, 0.33 × 0.48 + 0.66 × 0.09 = 0.22 for glasses 12B, and 0.33 × 0.01 + 0.66 × 0.07 = 0.05 for glasses 12C.That is, the evaluation scores by AI assuming an unspecified number of evaluators are 12A:12B:12C=0.24:0.48:0.01, with 12B being the highest evaluation, but the evaluation score taking into account how the person being photographed (the user in this embodiment) felt is 12A:12B:12C=0.24:0.22:0.05, with 12A being the highest evaluation score. Furthermore, the evaluation scores of the person being photographed are 12A:12B:12C=0.25:0.09:0.07, and it can be inferred from the person being photographed's facial expression that 12A is their favorite. Next, in step S15, CPU 7 recommends glasses by displaying glasses 12A, 12B, and 12C on the screen of monitor 2 in the order of decreasing suitability calculated in step S14, or by displaying glasses 12A, 12B, and 12C side by side on the screen of monitor 2. At this time, as the evaluation results, the evaluation values ​​before correction with the third evaluation value as "recommended by AI" (glasses 12A 0.24, glasses 12B 0.48, glasses C 0.01), the third evaluation values ​​as "your favorite degree" (glasses 12A 0.25, glasses 12B 0.09, glasses C 0.07), and the evaluation values ​​after correction with the third evaluation value as "overall recommendation" (glasses 12A 0.24, glasses 12B 0.22, glasses C 0.05) are displayed below the image of each pair of glasses. Based on this information, users can obtain an evaluation value based on the evaluations of an unspecified number of evaluators, or the user's own evaluations or evaluations by specific evaluators such as friends or family members, to determine which product suits them best, and can then consider purchasing the product. By configuring it in this way, in addition to the evaluation of the glasses that the computer has determined to be optimal, glasses can be recommended to the user by reflecting information about how the user or an evaluator other than the user felt about the user when wearing the glasses.

[0040] The above-described embodiment has been described merely as a specific embodiment for illustrating the principles and concepts of the present invention. In other words, the present invention is not limited to the above-described embodiment. The present invention can also be embodied in modified forms, for example, as follows. The evaluation items and feature quantities used in the calculations in the above embodiments are merely examples, and calculations may be performed using other evaluation items and feature quantities, or evaluation items and feature quantities different from those described above. The above-mentioned embodiments 1 to 5, 7 and 8 are embodiments relating to eyeglasses as products, etc., but this concept may also be applied to other ornaments or daily necessities, or makeup or hairstyles, etc. The calculation methods used to calculate the evaluation value or the degree of compatibility in the above embodiments are merely examples. Other calculation methods may also be applied. The weighting values ​​above are just examples and can be changed freely. In the above embodiment, the calculations are performed within the computer device 1 by the CPU 7 of the computer device 1, but the calculations may be performed via an internet connection, for example, on a cloud. Furthermore, the calculation results may also be stored in a cloud, for example, via an internet connection, instead of in the storage device 8. Although glasses, eyebrows, and lipstick have been given as examples above, any other accessory, daily necessities, makeup, or hairstyle may be used as long as it can be superimposed on the first facial image and second facial image data different from the first facial image data can be obtained and an evaluation value can be obtained using a facial image evaluator. In the above-described eighth embodiment, the third facial image acquired by the camera 4 may be evaluated as a still image or as a moving image. The third evaluation value as the evaluation value of the third facial image may be acquired by having the facial image evaluator evaluate a still image, or alternatively, changes in facial expression may be recorded as the third facial image and evaluated by the facial image evaluator. The third facial image may be evaluated at a certain moment, or multiple images captured at slightly different times may be evaluated, and the highest, lowest, or average of the evaluations may be used as the third evaluation value. Arrangements are free. The third facial image is preferably calculated as a difference from the fourth facial image. The facial values ​​of the third and fourth facial images may be calculated separately and then the difference may be calculated. Alternatively, a difference image between the third and fourth facial images may be calculated and evaluated by a facial image evaluator. This is advantageous because it makes it easier to understand the correspondence between the evaluation value and changes in facial expression. The present invention is not limited to the configurations described in the above-described embodiments. The components of the above-described embodiments and variations may be arbitrarily selected and combined. Furthermore, any component of each embodiment or variation may be arbitrarily combined with any component described in the Summary of the Invention or any component embodying any component described in the Summary of the Invention. The present invention also intends to obtain rights to these by filing an amendment or divisional application of this application. Furthermore, the applicant intends to obtain rights to the overall design or partial design by filing a conversion application to a design application. The drawings depict the entire device in solid lines, but they also include partial designs claimed for parts of the device. For example, not only can a part of the device be a partial design, but the drawings also include a part of the device as a partial design regardless of the part. A part of the device may be a part of the device, or a part of that part. [Explanation of symbols]

[0041] 1...computer device, 4...camera which is the first means, 5...facial image evaluator, 7...CPU which constitutes the second means, third means, fourth means, and fifth means, 10, 21, 25, 28...first facial image, 12A-12C...images of eyeglasses which are ornaments or everyday items, 13A-13C, 22A-22X, 27, 29...second facial image.

Claims

1. A method for evaluating the suitability of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, comprising: a first face image of the user and a second face image obtained by combining an image of at least one of the accessory, the daily necessities, the makeup, and the hairstyle to be evaluated with the first face image; an age classifier as a face image evaluator estimates and evaluates an age based on a face image to calculate a first evaluation value as an evaluation value of the first face image and a second evaluation value as an evaluation value of the second face image; A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, characterized by calculating the degree of suitability to the user's face from the first evaluation value and the second evaluation value.

2. A method for evaluating the suitability of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, comprising: a first face image of the user and a second face image obtained by combining an image of at least one of the accessory, the daily necessities, the makeup, and the hairstyle to be evaluated with the first face image; a gender classifier serving as a facial image evaluator estimates and evaluates gender based on the facial image to calculate a first evaluation value as an evaluation value of the first facial image and a second evaluation value as an evaluation value of the second facial image; A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, characterized by calculating the degree of suitability to the user's face from the first evaluation value and the second evaluation value.

3. A method for evaluating the degree of fit of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, comprising: a first face image of the user and a second face image obtained by combining an image of at least one of the accessory, the daily necessities, the makeup, and the hairstyle to be evaluated with the first face image; a facial image evaluator is used to calculate a first evaluation value as an evaluation value of the first facial image and a second evaluation value as an evaluation value of the second facial image; A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, comprising: a means for acquiring the user's requirements for how they are perceived; calculating a target evaluation value that suits the user from the requirements; and calculating the degree of suitability to the user's face by comparing an evaluation value calculated from the first evaluation value and the second evaluation value with the target evaluation value.

4. A method for evaluating the degree of fit of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, comprising: a first face image of the user and a second face image obtained by combining the first face image with two or more images selected from the accessories, the daily necessities, the makeup, and the hairstyle to be evaluated; calculating a first evaluation value as an evaluation value of the first facial image and a second evaluation value as an evaluation value of the second facial image by evaluating the first facial image with a facial image evaluator; A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, characterized by calculating the degree of suitability to the user's face from the first evaluation value and the second evaluation value.

5. A method for evaluating the degree of fit of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, comprising: a first face image of the user and a second face image obtained by combining an image of at least one of the accessory, the daily necessities, the makeup, and the hairstyle to be evaluated with the first face image; an emotion classifier as a face image evaluator estimates and evaluates emotions based on the face image to calculate a first evaluation value as an evaluation value of the first face image and a second evaluation value as an evaluation value of the second face image; A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, characterized by calculating the degree of suitability to the user's face from the first evaluation value and the second evaluation value.

6. A method for evaluating the degree of suitability of an ornament or everyday item worn on or around the face, or makeup or hairstyle, to a user's face, as described in claim 5, characterized in that at least one of the emotions estimated by the emotion classifier is joy.

7. A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to the user's face, as described in claim 6, characterized in that the degree of suitability to the face is evaluated to be higher as the joy evaluation value increases.

8. A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in any one of claims 5 to 7, wherein at least one of the emotions estimated by the emotion classifier is disgust.

9. A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to the user's face, as described in claim 8, characterized in that the degree of suitability to the face is evaluated as low as the evaluation value of the dislike increases.

10. A method for evaluating the degree of suitability of an accessory or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in any one of claims 5 to 7, wherein at least one of the emotions estimated by the emotion classifier is confusion.

11. A method for evaluating the degree of suitability of an accessory or daily necessities worn on or around the face, or makeup or hairstyle, to the user's face, as described in any one of claims 5 to 7, wherein at least one of the emotions estimated by the emotion classifier is neutral.

12. A method for evaluating the degree of fit of an ornament or daily necessities worn on or around the face, or makeup or hairstyle to the user's face, as described in any one of claims 3 to 11, wherein the facial image evaluator is an age classifier that estimates age based on a facial image.

13. A method for evaluating the degree of fit of an ornament or daily necessities worn on or around the face, or makeup or hairstyle to a user's face, as described in any one of claims 3 to 11, wherein the facial image evaluator is a gender classifier that estimates gender based on a facial image.

14. A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in any one of claims 1, 2, and 5 to 13, characterized in that it has a means for acquiring the user's requirements for how they are perceived, calculates a target evaluation value that suits the user from the requirements, and calculates the degree of suitability to the user's face by comparing an evaluation value calculated from the first evaluation value and the second evaluation value with the target evaluation value.

15. A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in any one of claims 1 to 14, wherein the first facial image is a photograph of the user without the ornament, daily necessities, and makeup.

16. A method for evaluating the degree of suitability of an accessory or daily necessities worn on or around the face, or makeup or hairstyle, to the user's face, as described in any one of claims 1 to 15, characterized in that the first facial image is based on photographed image data taken by the user and sent by communication means.

17. A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in any of claims 1 to 16, characterized in that the degree of suitability for the user is evaluated based on two or more of the ornaments or daily necessities worn on or around the face of the user, or makeup or hairstyle.

18. A method for evaluating the degree of fit of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in any one of claims 1 to 17, wherein the ornament or daily necessities are glasses.

19. A method for evaluating the degree of suitability of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in any one of claims 1 to 18, characterized in that the second facial image is presented to the user or an evaluator other than the user, a facial image of the user or the evaluator other than the user when they view the second facial image is obtained as a third facial image, the third facial image is evaluated using a facial image evaluator to calculate a third evaluation value as an evaluation value of the third facial image, and the degree of suitability to the user's face is calculated from the third evaluation value.

20. A system using a computer device that evaluates the degree of suitability of an ornament or daily necessities worn on or around a user's face, or makeup or hairstyle, to the user's face, comprising: a first means for acquiring a first facial image; and a second means for creating a second facial image by combining an image of at least one of the ornament, the daily necessities, the makeup, and the hairstyle with the first facial image; a third means for calculating a first evaluation value of the first face image obtained by the first means, and a fourth means for calculating a second evaluation value of the second face image obtained by the second means, a fifth means for calculating a degree of suitability for the user's face from the first evaluation value obtained by the third means and the second evaluation value obtained by the fourth means, A system for evaluating the degree of suitability of decorative items or daily necessities worn on or around the face, or makeup or hairstyle, to the user's face, characterized by having a display means for displaying to the user two or more of the first evaluation value, the second evaluation value, and an evaluation value calculated from the first evaluation value and the second evaluation value.

21. A system for evaluating the degree of fit of an ornament or daily necessities worn on or around the face, or makeup or hairstyle, to a user's face, as described in Claim 20, characterized in that it comprises: a display means for presenting the second facial image to the user or an evaluator other than the user; a sixth means for acquiring a facial image of the user or the evaluator other than the user when viewing the second facial image as a third facial image; and a seventh means for evaluating the third facial image with a facial image evaluator and calculating a third evaluation value as an evaluation value of the third facial image.

22. A recommendation system characterized by presenting to the user information on at least one of the ornaments, daily necessities, makeup, and hairstyles that suit the user based on the degree of suitability obtained by the system of claim 20 or 21.

23. A computer-aided design system for eyeglasses that fit a user's face, comprising: a first means for acquiring a first face image; and a second means for synthesizing a plurality of images of eyeglasses to be evaluated with the first face image to create a plurality of second face images; a third means for calculating a first evaluation value of the first face image obtained by the first means, and a fourth means for calculating a plurality of second evaluation values ​​of the second face image obtained by the second means, A glasses design system comprising: a fifth means for calculating the degree of fit of a plurality of pairs of glasses to be evaluated to the user's face from the first evaluation value obtained by the third means and the plurality of second evaluation values ​​obtained by the fourth means; and a glasses design system for designing glasses that fit the user's face with the highest degree of fit based on the plurality of degrees of fit obtained by the fifth means.

24. The eyeglasses design system for the face of the user of the decorative item or daily necessities to be worn on or around the face, or the makeup or hairstyle, as described in claim 23, characterized in that the images of the eyeglasses to be evaluated are virtually created by computer simulation.

25. The eyeglass design system described in Claim 23 or 24, characterized in that it comprises a display means for presenting the plurality of second facial images to the user or an evaluator other than the user, a sixth means for acquiring facial images of the user or an evaluator other than the user when viewing the plurality of second facial images as a plurality of third facial images, and a seventh means for evaluating the plurality of third facial images with a facial image evaluator and calculating a plurality of third evaluation values ​​as evaluation values ​​of the plurality of third facial images.

26. A system for designing eyeglasses using a computer device that fits a user's face, comprising: a first means for acquiring a first face image; and a second means for synthesizing a plurality of images of eyeglass frames to be evaluated with the first face image to create a plurality of second face images; a third means for evaluating the first facial image with a facial image evaluator to calculate a first evaluation value of the first facial image; and a fourth means for evaluating a plurality of the second facial images with the facial image evaluator to calculate a plurality of second evaluation values, a fifth means for calculating the degree of fit of the plurality of eyeglasses to be evaluated to the user's face from the first evaluation value obtained by the third means and the plurality of second evaluation values ​​obtained by the fourth means; calculating a group of coordinates where the eyeglass frame and the user's face overlap in the plurality of second face images; and calculating a group of coordinates that improves or decreases the degree of fit to the user's face from a combination of the group of coordinates and the degree of fit; and a glasses design system characterized by having: a fifth means for calculating the degree of fit of the plurality of eyeglasses to be evaluated to the user's face from the first evaluation value obtained by the third means and the plurality of second evaluation values ​​obtained by the fourth means;

27. 27. The eyeglasses design system according to claim 25, wherein the facial image evaluator is an emotion classifier that classifies positive emotions, and evaluates the degree of suitability as high when the positive emotion matches a positive emotion desired by the user, and designs the eyeglasses accordingly.

28. 27. The eyeglasses design system according to claim 25 or 26, wherein the facial image evaluator is an emotion classifier that identifies negative emotions and is designed to have a high degree of compatibility when the negative emotion matches the negative emotion desired by the user.

29. The eyeglass design system described in Claim 25 or 26, characterized in that the facial image evaluator is an emotion classifier and is designed to have a low degree of compatibility when the emotion is opposite to the emotion desired by the user.

Images