Face type diagnostic device, face type diagnostic method, and program

Abstract

To provide a face-type diagnosis device capable of improving the diagnostic accuracy of face type, and a face-type diagnosis method and program.SOLUTION: The face-type diagnosis device includes: an image acquisition unit that acquires a first image capturing a first facial expression of a user and a second image capturing a second facial expression of the user; a feature extraction unit that extracts first features of the face part of the user in the first image and second features of the face parts of the user in the second image; and a face-type determination unit that determines the face type of the user based on the first features and the second features.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a face type diagnosis apparatus, a face type diagnosis method, and a program.

Background Art

[0002] Patent Document 1 discloses an apparatus for diagnosing a face type using face feature points. The apparatus disclosed in Patent Document 1 acquires position data of feature points such as eyes, face contour, forehead, and jaw from face image data of a subject, and classifies the face type of the subject into a preset face type based on the position data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology disclosed in Patent Document 1, since the face shape is diagnosed based on feature points obtained from a specific expression of the user, there is a deviation between the overall face impression and the diagnosis result. For example, even when the facial expression changes, the face impression can change, but the conventional technology cannot reflect this change in the diagnosis result, and there is a problem that the face type that conforms to the user's overall face impression cannot be accurately diagnosed.

[0005] The non-limiting embodiments of the present disclosure contribute to providing a face type diagnosis apparatus, a face type diagnosis method, and a program that can improve the diagnosis accuracy of the face type.

Means for Solving the Problems

[0006] A face type diagnosis apparatus according to an embodiment of the present disclosure includes an image acquisition unit that acquires a first image capturing a first expression of a user and a second image capturing a second expression of the user, a feature amount extraction unit that extracts a first feature amount of a face part of the user in the first image and a second feature amount of the face part of the user in the second image, and a face type determination unit that determines the face type of the user based on the first feature amount and the second feature amount.

[0007] A face type diagnosis method according to an embodiment of the present disclosure includes acquiring a first image capturing a first expression of a user and a second image capturing a second expression of the user, extracting a first feature amount of a face part of the user in the first image and a second feature amount of the face part of the user in the second image, and determining the face type of the user based on the first feature amount and the second feature amount, and is executed by a computer.

[0008] A program according to an embodiment of the present disclosure causes a computer to acquire a first image capturing a first expression of a user and a second image capturing a second expression of the user, extract a first feature amount of a face part of the user in the first image and a second feature amount of the face part of the user in the second image, and determine the face type of the user based on the first feature amount and the second feature amount.

Advantages of the Invention

[0009] According to an embodiment of the present disclosure, it is possible to provide a face type diagnosis apparatus, a face type diagnosis method, and a program that can improve the diagnosis accuracy of the face type.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 4E

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. FIG. 1 is a schematic diagram of a face type diagnostic device 100 according to an embodiment of the present disclosure.

[0012] As shown in FIG. 1, the face type diagnostic apparatus 100 receives a frontal face image and a smiling face image, diagnoses a face type considering the change in the expression of a person's face based on the received frontal face image and smiling face image, classifies the diagnosed face, and outputs it as a diagnosis result. The frontal face image and the smiling face image are images of the user's face taken by a camera. In the following embodiments, attention is paid to two expressions, a frontal face and a smiling face. However, the present disclosure is not necessarily limited to these, and may be applied to any different plurality of expressions according to face type diagnosis. Also, the number of expressions is not limited to two and may be three or more. However, generally, since it is rare to show expressions such as anger or sadness in daily life, it is considered that there are few situations where the results of face type diagnosis considering angry faces or sad faces are useful. Therefore, in the present embodiment, an example of performing face type analysis targeting the frontal face and the smiling face that frequently appear in daily life will be described.

[0013] Diagnosis of face type can be used, for example, in industries such as apparel and makeup to promote the sale of products. By performing diagnosis of face type, it is possible to propose products suitable for the face type classified based on the face image to the user.

[0014] The face type represents the type of face analyzed from the position, size, shape, etc. of the facial parts. Facial parts are parts of the face that have differences in shape, arrangement, color, etc. for each individual. The facial parts may be parts having specific organ names such as eyes, mouth, nose, or eyebrows, or may be parts without specific organ names such as the vertical width or horizontal width of the face, the angle of the tip of the jaw, or the contour of the face.

[0015] (Classification of face type) The face type will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of a face type. FIG. 2 shows a graph in a rectangular coordinate system having orthogonal vertical and horizontal axes as coordinate axes.

[0016] The vertical axis represents the tendency of whether the face type is a child's face or an adult's face. A child's face may have features such as roundness in the contour and facial parts like eyes and nose being lower than the contour. An adult's face may have features such as an oblong contour and facial parts like eyes and nose being higher than the contour. The details of the criteria for determining whether the face type is a child's face or an adult's face will be described later.

[0017] The horizontal axis represents the tendency of whether the face gives a curvilinear impression or a linear impression. A curvilinear face may have features such as not giving a sense of skeleton in the contour, roundness in the jaw, round eyes, thick lips, and thin nasolabial folds. A linear face may have features such as giving a sense of skeleton in the contour, narrow and slanting eyes, thin lips, and prominent nasolabial folds. The details of the criteria for determining whether the face gives a curvilinear impression or a linear impression will be described later.

[0018] With reference to FIG. 3A, the criteria for determining whether the face type is a child's face or an adult's face will be described. FIG. 3A is a diagram showing an example of the criteria for determining whether the face type is a child's face or an adult's face.

[0019] The items for determining whether the face type is a child's face or an adult's face may include, for example, face shape (contour), eye position, nose length, mouth size, distance between eyes and lips, distance between eyes, etc. As shown in FIG. 3A, predetermined criteria are associated with these determination items. The face type diagnosis device 100 can determine whether the face type is a child's face or an adult's face by referring to a table in which the determination items shown in FIG. 3A and the criteria are associated.

[0020] For example, when the vertical width of the face is less than or equal to the horizontal width, when the position of the eyes is lower on the face (located on the jaw side), when the length of the nose is shorter than a predetermined length, when the size of the mouth is smaller than a predetermined size, when the distance between the eyes and the lips is closer than a predetermined value, when the distance between the eyes is farther than a predetermined value, etc., the face type diagnosis device 100 can determine that it is a child's face. On the other hand, when the vertical width of the face exceeds the horizontal width, when the position of the eyes is higher on the face (located on the forehead side), when the length of the nose is longer than a predetermined length, when the size of the mouth is larger than a predetermined size, when the distance between the eyes and the lips is farther than a predetermined value, when the distance between the eyes is closer than a predetermined value, etc., the face type diagnosis device 100 can determine that it is an adult's face.

[0021] Referring to FIG. 3B, the criteria for determining whether the face gives a curvilinear impression or a linear impression will be described. FIG. 3B is a diagram showing an example of the criteria for determining whether the face gives a curvilinear impression or a linear impression.

[0022] The items for determining whether the face gives a curvilinear impression or a linear impression may include, for example, the face shape (contour), the shape of the eyes, the thickness of the lips, etc. As shown in FIG. 3B, predetermined determination criteria are associated with these determination items. The face type diagnosis device 100 can determine whether the face gives a curvilinear impression or a linear impression by referring to a table in which the determination items shown in FIG. 3B and the determination criteria are associated.

[0023] For example, when the face type diagnostic device 100 has roundness in the jaw, when the shape of the eyes is rounder compared to a predetermined shape, when the thickness of the lips is thicker than a predetermined width, etc., it can determine that the face gives a curvilinear impression. On the other hand, when the jaw of the face type diagnostic device 100 is sharp, when the shape of the eyes is thinner compared to a predetermined shape, when the thickness of the lips is thinner than a predetermined width, etc., it can determine that the face gives a linear impression. Note that the face type diagnostic device 100 may determine whether it is a child's face or an adult's face, or whether it is curvilinear or linear, by integrating the evaluation results of multiple face parts, or it may determine based on the evaluation result of only a single face part. Also, when determining by integrating the evaluation results of multiple face parts, for example, if some face parts indicate an adult face but most face parts are child faces, a more detailed determination may be made, such as determining that it is a "child face leaning towards an adult face".

[0024] In this way, the face type diagnostic device 100 can classify the face type into four types according to the determination criteria shown in FIGS. 3A and 3B.

[0025] Specifically, in the case of a curvilinear child's face, the face type diagnostic device 100 classifies the face type into the "Cute" area that gives an impression such as bright and cute (see FIG. 2).

[0026] In the case of a linear child's face, the face type diagnostic device 100 classifies the face type into the "Fresh" area that gives an impression such as energetic, lively, and boyish.

[0027] In the case of a linear adult face, the face type diagnostic device 100 classifies the face type into the "Cool" area that gives an impression such as having sharp insight and being calm and imposing.

[0028] In the case of a curvilinear adult face, the face type diagnostic device 100 classifies the face type into the "Elegant" area that gives a refined impression.

[0029] The face type diagnostic device 100 that performs these classifications displays, on the screen of the display, an image representing four regions such as "Cute", "Fresh", "Cool", and "Elegant" as shown in FIG. 2, and plots an icon indicating the classification result on the image to provide the user's face type. Details of the display example will be described later.

[0030] (Feature points of facial parts) Next, referring to FIGS. 4A to 4E, the feature points of the facial parts used for determining the face type will be described. FIGS. 4A to 4E are diagrams showing an example of the feature points used for determining the face type.

[0031] In FIGS. 4A and 4B, the feature points of facial parts such as the facial contour, eyes, mouth, nose, and eyebrows are shown. The face type diagnostic device 100 can determine whether the vertical width D1 of the face is less than or equal to the horizontal width D2, and whether the angle of the tip of the jaw is equal to or greater than a predetermined angle based on the feature points obtained from the face image (see FIG. 4A). The face type diagnostic device 100 can determine the position of the eyes by comparing the width D3 from the eyes to the lower end of the jaw and the width D4 from the eyes to the center of the forehead based on the feature points (see FIG. 4B). The face type diagnostic device 100 can determine whether the distance D5 from the eyes to the lips is closer than a predetermined distance based on the feature points (see FIG. 4B).

[0032] In FIG. 4C, the feature points such as the eyes and eyebrows are shown. When the ratio of the vertical width D7 of the eyes to the horizontal width D6 of the eyes is equal to or greater than a predetermined value based on the feature points, the face type diagnostic device 100 determines that the eyes are round, and when the ratio of the vertical width D7 of the eyes to the horizontal width D6 of the eyes is less than the predetermined value, the face type diagnostic device 100 can determine that the eyes are narrow. Also, the face type diagnostic device 100 can determine whether the distance D8 between the eyes is farther than a predetermined value based on the feature points.

[0033] In FIGS. 4D and 4E, the feature points such as the nose and mouth are shown. When the ratio of the width D10 of the lips to the distance D9 between the nose and the mouth is equal to or greater than a predetermined value based on the feature points, the face type diagnostic device 100 determines that the lips are thick, and when the ratio of the width D10 of the lips to the distance D9 between the nose and the mouth is less than the predetermined value, the face type diagnostic device 100 can determine that the lips are thin (see FIG. 4D).

[0034] When the ratio of the width D12 of the mouth to the width D11 of the nose is equal to or greater than a predetermined value based on the feature points, the face type diagnostic device 100 can determine that the mouth is large, and when the ratio of the width D12 of the mouth to the width D11 of the nose is less than the predetermined value, it can determine that the mouth is small (see FIG. 4E).

[0035] Next, with reference to FIG. 5, a hardware configuration example of the face type diagnostic device 100 will be described. FIG. 5 is a block diagram showing the hardware configuration of the face type diagnostic device 100 according to an embodiment of the present disclosure.

[0036] The face type diagnostic device 100 is realized by a computing device such as a smartphone, a tablet, or a personal computer, and may have a hardware configuration as shown in FIG. 5, for example. That is, the face type diagnostic device 100 includes a storage device 101, a processor 102, a user interface device 103, and a communication device 104 that are interconnected via a bus B.

[0037] Programs or instructions for realizing various functions and processes described later in the face type diagnostic device 100 may be downloaded from any external device via a network or the like, or may be provided from a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a flash memory.

[0038] The storage device 101 is realized by a random access memory, a flash memory, a hard disk drive, or the like, and stores files, data, etc. used for the execution of the installed programs or instructions together with the installed programs or instructions. The storage device 101 may include a non-transitory storage medium.

[0039] The processor 102 may be implemented by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuitry, etc., which may be composed of one or more processor cores, and executes various functions and processes of the face type diagnosis device 100 described below according to programs, instructions, data such as parameters necessary for executing the programs or instructions, etc. stored in the storage device 101.

[0040] The user interface device 103 may be composed of input devices such as keyboards, mice, cameras, microphones, output devices such as displays, speakers, headsets, printers, etc., and input / output devices such as touch panels, and realizes the interface between the user and the face type diagnosis device 100. For example, the user operates the face type diagnosis device 100 by operating a GUI (Graphical User Interface) displayed on the display or touch panel using a keyboard, mouse, etc.

[0041] The communication device 104 is realized by various communication circuits that execute communication processing with external devices, communication networks such as the Internet, LAN (Local Area Network), etc.

[0042] However, the above-described hardware configuration is merely an example, and the face type diagnosis device 100 according to the present disclosure may be realized by any other appropriate hardware configuration. For example, each function provided in the face type diagnosis device 100 may be distributed and realized by a plurality of computers. In this case, the exchange of information between functions provided in different computers is performed via a communication network (LAN or Internet) connecting the computers.

[0043] FIG. 6 is a block diagram showing the functional configuration of the face type diagnosis device 100 according to the embodiment of the present disclosure. The face type diagnosis device 100 includes an image acquisition unit 110, a feature amount extraction unit 120, and a face type determination unit 130.

[0044] The image acquisition unit 110 acquires a first image capturing the user's first facial expression and a second image capturing the user's second facial expression. The first facial expression may be, for example, the user's natural face, and the second facial expression may be, for example, the user's smiling face.

[0045] Specifically, the camera captures the user's face, and the image acquisition unit 110 acquires an image of the user's face captured by the camera. The face image acquired by the face type diagnosis device 100 may be a still image or a moving image. The camera captures the measurement target area and generates an RGB image of the measurement target area. For example, the camera may be a monocular camera and generates a monocular RGB image. The generated RGB image is passed to the image acquisition unit 110.

[0046] The feature extraction unit 120 extracts a first feature amount of the user's facial parts in the first image and a second feature amount of the user's facial parts in the second image. Specifically, the feature extraction unit 120 extracts feature points from each of the user's natural face image and smiling face image, and calculates the feature amounts of the facial parts corresponding to each determination item based on the extracted feature points. For example, the extraction of feature points from the face image may be performed by a trained machine learning model. Such a trained machine learning model may be any known one, and for example, it may be trained to extract the feature points of the subject's face image included in the learning face images using a plurality of learning face images as teacher data.

[0047] The first feature amount is, for example, a feature amount representing each facial part such as the face shape (contour), eye shape, lip thickness, jaw shape, etc. as described above when the user's facial expression is a natural face. The second feature amount is, for example, a feature amount representing each facial part such as the face shape (contour), eye shape, lip thickness, jaw shape, etc. as described above when the user's facial expression is a smiling face. Note that the feature amounts of each facial part may be appropriately scaled by normalization or the like so that the subsequent calculations are appropriate.

[0048] The face type determination unit 130 executes a predetermined operation on the first feature amount and the second feature amount based on the first weight set for each face part and the second weight set for each face part in each expression. Further, the face type determination unit 130 determines the user's face type based on the operation result and a predetermined determination criterion.

[0049] A specific example of the process in the face type determination unit 130 will be described below. When the user's expression is a neutral face, as shown in Equation (1), the face type determination unit 130 assigns a predetermined weight (first weight) set for each face part to the feature amount of the face part (first feature amount) when the expression is a neutral face, and further assigns a predetermined weight (second weight α) set for each face part when the expression is a neutral face to the first feature amount. The face type determination unit 130 sums up the feature amounts of each face part to which these weights are assigned.

[0050] Feature amount of each face part in the case of a neutral face × First weight for each face part × Second weight α ··· (1)

[0051] When the user's expression is a smiling face, as shown in Equation (2), the face type determination unit 130 assigns a predetermined weight (first weight) set for each face part to the feature amount of the face part (second feature amount) when the expression is a smiling face, and further assigns a predetermined weight (second weight β) set for each face part when the expression is a smiling face to the first feature amount. The face type determination unit 130 sums up the feature amounts of each face part to which these weights are assigned.

[0052] Feature amount of each face part in the case of a smiling face × First weight for each face part × Second weight β ··· (2)

[0053] As shown in Equation (3), the face type determination unit 130 executes an operation of summing up and averaging the feature amount to which the second weight α in the case of a neutral face is assigned and the feature amount to which the second weight β in the case of a smiling face is assigned.

[0054] (Feature amount for each facial part in the case of a straight face × First weight for each facial part × Second weight α + Feature amount for each facial part in the case of a smiling face × First weight for each facial part × Second weight β) / 2 ··· (3)

[0055] For example, in the case of curve / straight line determination, the face type determination unit 130 executes the operation shown in the following formula (4) on the feature amounts of the straight face and the smiling face of three facial parts: the face shape, the eye shape, and the lip thickness.

[0056] 1 / 2{(Feature amount f of the face shape in the straight face 11 × First weight w1 of the face shape × Second weight α1 of the face shape in the straight face) + (Feature amount f of the face shape in the smiling face 12 × First weight w1 of the face shape × Second weight β1 of the face shape in the smiling face)} + 1 / 2{(Feature amount f of the eye shape in the straight face 21 × First weight w2 of the eye shape × Second weight α2 of the eye shape in the straight face) + (Feature amount f of the eye shape in the smiling face 22 × First weight w2 of the eye shape × Second weight β2 of the eye shape in the smiling face)} + 1 / 2{(Feature amount f of the lip thickness in the straight face 31 × First weight w3 of the lip thickness × Second weight α3 of the lip thickness in the straight face) + (Feature amount f of the lip thickness in the smiling face 32 × First weight w3 of the lip thickness × Second weight β3 of the lip thickness in the smiling face)} ··· (4)

[0057] Note that the second weights α and β for each expression may be predetermined values given from outside the face type diagnosis device 100, or may be predetermined values set in the face type diagnosis device 100.

[0058] Here, the values of the second weight α in the case of a straight face and the second weight β in the case of a smiling face will be described. For example, the second weights α and β may be set according to the susceptibility to the influence of the change in expression (the degree of influence by the change in expression) for each facial part.

[0059] The face type determination unit 130 may set the second weight α in the case of a straight face to be larger than the second weight β in the case of a smiling face for facial parts such as the eyes and mouth that are susceptible to the influence of the change in expression. This is because the straight face is taken as the basis for face type diagnosis.

[0060] Further, the face type determination unit 130 may set the second weight α in the case of a straight face to be equal to the second weight β in the case of a smiling face for face parts such as contours and noses that are less affected by changes in expression.

[0061] For example, when the expression changes from a straight face to a smiling face, the degree of eye opening tends to decrease, and the closed mouth tends to open wide and the teeth become visible. Therefore, as described above, by setting different values for the weights of the face parts that are easily affected by changes in expression, and setting equal values for the weights of the face parts that are less affected by changes in expression, the diagnostic accuracy of the face type can be improved.

[0062] In this way, based on the above-described calculation result (4), the face type determination unit 130 can perform curve / straight line determination and map the calculation result to any coordinate on the straight line-curve axis. Similarly, the face type determination unit 130 can execute a calculation for child face / adult face determination and map the calculation result to any coordinate on the child face-adult face axis. As a result, the face type determination unit 130 can map the calculation result as a point on a plane as shown in FIG. 2, and can perform an appropriate face type diagnosis.

[0063] Note that the determination process in the face type determination unit 130 may include the following modes.

[0064] (First Modified Example) In the above-described embodiment, the second weight β in the case of a smiling face is a constant value, but the second weight β may be set according to the degree of the user's expression.

[0065] The second expression, the smiling face, can generally be classified into various smile levels from a faint smile to a big laugh. Therefore, the face type determination unit 130 may further adjust the above-described second weight β in the case of a smiling face according to the degree of the smile (smile level).

[0066] Specifically, the face type determination unit 130 may specify the smile level from the smile image by any known measurement technique. For example, the face type determination unit 130 may classify the smile image into three smile levels (for example, a smile with the mouth closed is level "1", a smile with the mouth slightly open is level "2", a smile with the mouth wide open is level "3", etc.). Note that the smile level according to the present disclosure is not necessarily limited to three levels and may be classified into any other number of levels.

[0067] Then, the face type determination unit 130 changes the feature amount for each face part by assigning smile scores a, b, c, which are coefficients corresponding to the smile level, to the second weight β for each face part. For example, the smile scores a, b, c may have a relationship of a < b < c.

[0068] For example, when the smile level is determined to be "1", the face type determination unit 130 may further multiply the smile score a by the second weight β as follows to adjust the feature amount of the face part in the case of a smile. (Feature amount for each face part in the case of a smile × First weight for each face part × Second weight β × Smile score a)

[0069] Also, when the smile level is determined to be "2", the face type determination unit 130 may further multiply the smile score b by the second weight β as follows to adjust the feature amount of the face part in the case of a smile. (Feature amount for each face part in the case of a smile × First weight for each face part × Second weight β × Smile score b)

[0070] Similarly, when the smile level is determined to be "3", the face type determination unit 130 may further multiply the smile score c by the second weight β as follows to adjust the feature amount of the face part in the case of a smile. (Feature amount for each face part in the case of a smile × First weight for each face part × Second weight β × Smile score c)

[0071] In this way, by calculating the feature amount according to the degree of the user's smile, it is possible to further improve the diagnostic accuracy of the face type while taking into account the degree of the smile.

[0072] (Second Modified Example) In the above-described embodiment, an example of diagnosing a face type based on a still image has been described. On the other hand, there are people with many smiles and people with few smiles, and by reflecting such smile frequencies, more appropriate face type diagnosis may be possible. Specifically, the second weight β in the case of a smile may be changed according to the smile frequency. Thereby, it is possible to perform face type diagnosis according to the life scene. With reference to FIG. 7, the determination process of the face type according to the smile frequency will be described.

[0073] FIG. 7 is a diagram for explaining an example of calculating a feature amount taking into account the frequency of smiles detected during a certain period. The image acquisition unit 110 acquires the video of the user during conversation from the camera for a predetermined time, and transmits the acquired video to the feature amount extraction unit 120. In the example of FIG. 7, a 3-minute video is acquired.

[0074] The feature amount extraction unit 120 calculates the frequency of smiles of the user in the video for a predetermined time. That is, the feature amount extraction unit 120 calculates the frequency for each expression of the user in the video for a predetermined time. For example, in the illustrated specific example, the smile frequency of a certain user is 12%, and the smile frequency of another user is 75%.

[0075] When the video of the user is unclear due to the camera vibrating or the like, the feature amount extraction unit 120 may thin out the video of the time period that cannot be accurately captured from the video as being undetermined, and calculate the frequency for each expression of the user based on the remaining video. Further, when the feature amount extraction unit 120 cannot determine whether it is a front face or a smile, such as when the face parts necessary for expression recognition are hidden or a special expression is made, the video of that time period may be thinned out as being undetermined, and the frequency for each expression of the user may be calculated.

[0076] The face type determination unit 130 adjusts the second weight β according to the frequency for each expression, and executes the above-described operation on the second feature amount based on the adjusted second weight. Specifically, as shown in Equation (5), the face type determination unit 130 adjusts the second weight β by assigning a predetermined coefficient corresponding to the frequency of smiling faces to the second weight β, and new second weights β', β'', etc. may be applied.

[0077] Feature amount for each facial part in the case of a smiling face × First weight for each facial part × Second weight β' Feature amount for each facial part in the case of a smiling face × First weight for each facial part × Second weight β'' ···(5)

[0078] The second weight β' may be a weight corresponding to the smiling face frequency of "12%", and the second weight β'' (β' < β'') may be a weight corresponding to the smiling face frequency of "75%".

[0079] Note that the first modification example may be combined with the second modification example, and the second weight β in the case of a smiling face may be adjusted in consideration of the smiling face frequency together with the smiling face level. For example, if the second weight is set to β' according to the measured smiling face frequency, and the smiling face level is "1", the face type determination unit 130 may execute the operation shown in the following Equation (6). Here, the smiling face level may be the average of the smiling face levels in the video.

[0080] Feature amount for each facial part in the case of a smiling face × First weight for each facial part × Second weight β' × Smiling face score a ···(6)

[0081] Next, with reference to FIG. 8, a face type diagnosis process according to an embodiment of the present disclosure will be described. The face type diagnosis process is executed by the above-described face type diagnosis apparatus 100, and more specifically, may be realized by one or more processors 102 of the face type diagnosis apparatus 100 executing one or more programs or instructions stored in one or more storage devices 101. For example, the face type diagnosis process may be started when a user of the face type diagnosis apparatus 100 activates an application or the like related to the process.

[0082] Figure 8 is a flowchart showing the face type diagnosis process according to an embodiment of the present disclosure. As shown in Figure 8, in step S101, the face type diagnosis device 100 acquires a true face image and a smiling face image. Specifically, the camera captures the face of the user for each expression, and the face type diagnosis device 100 acquires the image of the face of the user captured by the camera. The face image acquired by the face type diagnosis device 100 may be a still image or a moving image.

[0083] Next, in step S102, the face type diagnosis device 100 extracts the feature amounts of the face parts of the acquired true face image and the smiling face image. Specifically, the face type diagnosis device 100 extracts the feature points of the face of the user to be diagnosed in the true face image and the smiling face image, and calculates the feature amounts for each face part based on the extracted feature points.

[0084] Next, in step S103, the face type diagnosis device 100 performs the following operation of formula (7) on the feature amounts of the true face and the smiling face of three face parts, namely the face shape, the shape of the eyes, and the thickness of the lips, for example, for curve / straight line determination.

[0085] 1 / 2{(feature amount f of the face shape of the true face 11 × first weight w1 of the face shape × second weight α1 of the face shape of the true face)+(feature amount f of the face shape of the smiling face 12 × first weight w1 of the face shape × second weight β1 of the face shape of the smiling face)}+1 / 2{(feature amount f of the shape of the eyes of the true face 21 × first weight w2 of the shape of the eyes × second weight α2 of the shape of the eyes of the true face)+(feature amount f of the shape of the eyes of the smiling face 22 × first weight w2 of the shape of the eyes × second weight β2 of the shape of the eyes of the smiling face)}+1 / 2{(feature amount f of the thickness of the lips of the true face 31 × first weight w3 of the thickness of the lips × second weight α3 of the thickness of the lips of the true face)+(feature amount f of the thickness of the lips of the smiling face 32 × first weight w3 of the thickness of the lips × second weight β3 of the thickness of the lips of the smiling face)} ··· (7)

[0086] The face type diagnostic device 100 can identify the position of the straight line-curve axis on the plane shown in FIG. 2 based on the calculation result of equation (7).

[0087] Similarly, for example, for child / adult face determination, the face type diagnostic device 100 performs similar calculations on the feature amounts of the true face and smiling face of six face parts, namely face shape, eye position, nose length, mouth size, distance between eyes and lips, and interpupillary distance, and can identify the position of the child face-adult face axis on the plane shown in FIG. 2 based on the calculation result.

[0088] Thereby, the face type diagnostic device 100 can determine the position of the face type of the user to be measured on the plane shown in FIG. 2.

[0089] For example, the face type diagnostic device 100 may provide the user with the diagnostic result of the face type through a display screen as shown in FIG. 9. FIG. 9 is a diagram showing an example of a display screen of the diagnostic result of the face type by the face type diagnostic process according to an embodiment of the present disclosure.

[0090] The face type diagnostic device 100 may display, on a display installed in, for example, a cosmetics store, an image representing four regions such as "Cute", "Fresh", "Cool", and "Elegant" shown in FIG. 9, and further show the diagnostic result of the face type of the user to be measured (for example, a heart mark) on the image.

[0091] FIG. 9 shows an example of the display of the face type determined to be a face giving a curvilinear impression, and the image (heart mark) showing the diagnostic result of the face type is located in the "Elegant" region, closer to "Cute".

[0092] In addition, the face type diagnostic device 100 may display guidance for explaining the face type next to the screen representing the four regions such as "Cute", "Fresh", "Cool", and "Elegant".

[0093] Here, the face type determination unit 130 of the face type diagnosis device 100 may calculate a smile contribution degree indicating the ratio of smiles that contributed to the determination of the face type, and display it on the screen. In an embodiment where a straight face and a smile contribute to face type determination by fixed values, the smile contribution degree also becomes a common value among users, so there is not much meaning in calculating the smile contribution degree. For this reason, it is preferable that the smile contribution degree is calculated in an embodiment in which one or both of the smile level and the smile frequency are considered.

[0094] (Example of the calculation formula for the smile contribution degree) The smile contribution degree of the first modified example described above is calculated as shown in formula (8).

[0095] Smile contribution degree = (β × smile score) / α + (β × smile score) ··· (8)

[0096] The smile contribution degree of the second modified example is calculated as shown in formula (9).

[0097] Smile contribution degree = (β' × smile score) / α + (β' × smile score) β' = smile score a × weighting γ + smile score b × weighting γ' + smile score c × weighting γ'' ··· (9) Here, γ, γ', γ'' are set as weights corresponding to the lengths of time of smile levels 1, 2, and 3, respectively. In the illustrated display screen, smile level 1 is 30 seconds, smile level 2 is 1 minute and 10 seconds, and smile level 3 is 26 seconds. According to the above-described calculation formula, the smile contribution degree is calculated as 70%.

[0098] As described above, the face type diagnostic apparatus according to the embodiment of the present disclosure includes an image acquisition unit that acquires a first image capturing a first expression of a user and a second image capturing a second expression of the user, a feature amount extraction unit that extracts a first feature amount of the user's face parts in the first image and a second feature amount of the user's face parts in the second image, and a face type determination unit that executes a first operation on the first feature amount and the second feature amount based on a first weight set for each face part and a second weight set for each face part in each expression, and determines the user's face type based on the operation result and a predetermined determination criterion.

[0099] With this configuration, since the face type can be diagnosed based on the feature amounts of the respective images of the second expression that is different from the first expression and the first expression, the diagnostic accuracy of the face type can be improved compared to the case of using an image of a single expression.

[0100] Therefore, when the face type diagnostic apparatus according to the embodiment of the present disclosure is applied to a dedicated diagnostic machine installed in a cosmetics corner such as a department store, for example, based on the diagnostic result of the face type diagnosed in consideration of the change in the facial expression, cosmetics optimal for the face type can be proposed. Also, when the face type diagnostic apparatus according to the embodiment of the present disclosure is applied to a dedicated diagnostic machine installed in a clothing store, for example, based on the diagnostic result of the face type diagnosed in consideration of the change in the facial expression, clothes optimal for the face type can be proposed. Further, when a part of the functions of the face type diagnostic apparatus is realized by a smartphone or the like, based on the diagnostic result of the face type, it may be possible to guide the user to a communication sales site or made-to-order of optimal cosmetics.

[0101] (Other Modification Examples) In the above-described embodiments, the face type diagnosis was performed based on the result of assigning weights to the feature amounts for each face part. However, the face type diagnosis apparatus may perform the face type diagnosis using the result of calculation by assigning the same weight to all face parts, or by omitting the assignment of weights to the feature amounts of some or all face parts. Further, when the face type diagnosis apparatus assigns weights to the feature amounts for each face part, it may assign different weights for each face part. In this case, the face type diagnosis apparatus may be configured such that the greater the influence on the impression of the face type, the greater the weight assigned to the face part.

[0102] In the above-described embodiments, multiplication was performed when assigning weights in the process of face type diagnosis. However, the face type diagnosis apparatus may assign weights by other methods such as addition.

[0103] In the above-described embodiments, the smile contribution degree indicating the ratio of smiles contributing to the determination of the face type was calculated and displayed. However, the contribution degree to be calculated is not limited to that for smiles. For example, the face type diagnosis apparatus may calculate and display the true face contribution degree indicating the ratio of true faces. Further, if the face type diagnosis apparatus also uses expressions other than smiles and true faces in the determination of the face type, it may calculate and display the contribution degree of one or more of those expressions.

[0104] In the above-described embodiments, only the result considering both true faces and smiles was displayed as the result of face type diagnosis. However, the face type diagnosis apparatus may also display other results. For example, the face type diagnosis apparatus may also display the face type diagnosed using only true faces or the face type diagnosed using only smiles. Note that generally, it is known that smiles tend to be determined as having a childlike face and a curvilinear face type rather than true faces. Therefore, the possibility of obtaining a result deviating from this tendency is low. However, since the degree to which the face type changes between true faces and smiles varies from person to person, visualizing this degree can inform the user of the characteristics of their own expressions. Note that when the face type diagnosis apparatus also uses expressions other than true faces and smiles in the determination of the face type, it may also display the determination result of the face type based on each expression alone or on an arbitrary combination of expressions.

[0105] It should be noted that, for example, the following aspects are also understood to belong to the technical scope of the present disclosure.

[0106] (1) The face type diagnosis device according to the embodiment of the present disclosure includes an image acquisition unit that acquires a first image of a user's first expression and a second image of the user's second expression, a feature amount extraction unit that extracts a first feature amount of the user's face parts in the first image and a second feature amount of the user's face parts in the second image, and a face type determination unit that determines the user's face type based on the first feature amount and the second feature amount.

[0107] (2) The face type determination unit determines the face type by comparing the result of an operation using the first feature amount and the second feature amount with a predetermined standard.

[0108] (3) The face type determination unit performs an operation by assigning a weight set according to the influence degree of the expression change for each face part to the second feature amount.

[0109] (4) The image acquisition unit acquires a video of the user for a predetermined time, the feature amount extraction unit calculates the frequency at which the user's second expression is detected in the video, and the face type determination unit executes the operation by assigning a weight according to the frequency to the second feature amount.

[0110] (5) The second expression is classified into a plurality of expression levels, and the face type determination unit executes the operation by assigning a weight according to the plurality of expression levels to the second feature amount.

[0111] (6) The face type determination unit further calculates a contribution degree indicating the ratio that contributed to the determination of the face type for one or more expressions of the first expression and the second expression.

[0112] (7) The first expression is a straight face, and the second expression is a smile.

[0113] (8) The face type includes four types classified according to a first criterion based on face shape, eye position, nose length, mouth size, distance between eyes and lips, and distance between eyes, and a second criterion based on face shape, eye shape, and lip thickness.

[0114] (9) The face type diagnosis method according to an embodiment of the present disclosure includes obtaining a first image capturing a first expression of a user and a second image capturing a second expression of the user, extracting a first feature amount of the user's face parts in the first image and a second feature amount of the user's face parts in the second image, and determining the user's face type based on the first feature amount and the second feature amount.

[0115] (10) The program according to an embodiment of the present disclosure causes a computer to obtain a first image capturing a first expression of a user and a second image capturing a second expression of the user, extract a first feature amount of the user's face parts in the first image and a second feature amount of the user's face parts in the second image, and determine the user's face type based on the first feature amount and the second feature amount.

[0116] As described above, the embodiments of the present disclosure have been described in detail. However, the present disclosure is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of the present disclosure described in the claims.

Explanation of Signs

[0117] 100 Face type diagnosis device 101 Storage device 102 Processor 103 User interface device 104 Communication device 110 Image acquisition unit 120 Feature amount extraction unit 130 Face type determination unit

Claims

1. An image acquisition unit that acquires a first image capturing a first expression of a user and a second image capturing a second expression of the user; A feature quantity extraction unit that extracts a first feature quantity of the user's facial parts in the first image and a second feature quantity of the user's facial parts in the second image; A face type determination unit that determines the face type of the user based on the first feature quantity and the second feature quantity; A face type diagnosis device having the above.

2. The face type determination unit according to claim 1, wherein the face type is determined by comparing the result of an operation using the first feature quantity and the second feature quantity with a predetermined standard.

3. The face type diagnosis device according to claim 2, wherein the face type determination unit performs the operation by assigning a weight set according to the degree of influence of the change in expression for each facial part to the second feature quantity.

4. The image acquisition unit acquires a video of the user for a predetermined time, The feature quantity extraction unit calculates the frequency at which the second expression of the user is detected in the video, The face type diagnosis device according to claim 2 or 3, wherein the face type determination unit performs the operation by assigning a weight according to the frequency to the second feature quantity.

5. The second expression is classified into a plurality of expression levels, The face type diagnosis device according to any one of claims 2 to 4, wherein the face type determination unit performs the operation by assigning a weight according to the plurality of expression levels to the second feature quantity.

6. The face type determination unit further calculates a contribution degree indicating the ratio that contributed to the determination of the face type for one or more expressions of the first expression and the second expression, according to any one of claims 1 to 5. The face type diagnosis device described.

7. The face type diagnosis device according to any one of claims 1 to 6, wherein the first expression is a straight face and the second expression is a smile.

8. The face type includes four types classified according to a first determination criterion based on face shape, eye position, nose length, mouth size, distance between eyes and lips, and distance between eyes, and a second determination criterion based on face shape, eye shape, and lip thickness. The face type diagnosis device according to claim 7.

9. Acquiring a first image capturing a first expression of a user and a second image capturing a second expression of the user; Extracting a first feature amount of the user's face part in the first image and a second feature amount of the user's face part in the second image; Determining the user's face type based on the first feature amount and the second feature amount; A face type diagnosis method executed by a computer, including the above.

10. Obtaining a first image capturing the user's first expression and a second image capturing the user's second expression; Extracting a first feature amount of the user's face part in the first image and a second feature amount of the user's face part in the second image; Determining the user's face type based on the first feature amount and the second feature amount; A program for causing a computer to execute the above.

Images