Emotion estimation device and emotion estimation method

Abstract

To provide an emotion estimation device capable of appropriately estimating emotion of a driver of a vehicle.SOLUTION: An emotion estimation device includes a threshold setting part for setting a face feature amount threshold on the basis of a plurality of face feature amounts representing the magnitude of deviation from a state to be a reference of a prescribed portion of the face, detected from each of a plurality of face images representing the face of a driver of a vehicle generated at times included within a prescribed time range by a photographing part mounted on the vehicle, a face feature amount specification part for specifying a face feature amount larger than the face feature amount threshold among the plurality of face feature amounts detected from each of the plurality of the face images generated at times that are not included within the prescribed time range by the photographing part, and an emotion estimation part for estimating emotion of the driver by using a normalized feature amount obtained by normalizing the specified face feature amount downward.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an emotion estimation device and an emotion estimation method for estimating the emotion of a vehicle driver. [Background technology]

[0002] A technology is known that estimates emotions based on facial features that are detected from a facial image representing a human face and indicate deviations of each part of the face from a reference state. The technology can change the driving control of a vehicle in accordance with the driver's emotions estimated from a facial image representing the driver's face.

[0003] Patent document 1 describes an image processing device that calculates the amount of change in each feature point of a predetermined group of facial parts in an image from the feature points of a face with a predetermined expression, normalizes the amount of change according to changes in face size and face rotation, and determines the facial expression based on the normalized amount of change. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-056388 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the image processing device disclosed in Patent Document 1, feature points of a predetermined reference group of parts are detected in advance from an image containing a face with a predetermined expression, and the facial expression is judged by comparing these feature points detected from the image. It is cumbersome to obtain an image containing a face with a specific expression from the person to be judged in advance. Furthermore, it may be difficult to properly estimate the emotion of the person to be judged due to the influence of feature points detected from an image of the person with a neutral expression.

[0006] An object of the present disclosure is to provide an emotion estimation device that can appropriately estimate the emotion of a vehicle driver. [Means for solving the problem]

[0007] The emotion estimation device according to the present disclosure includes: a threshold setting unit that sets a facial feature amount threshold based on a plurality of facial feature amounts that indicate the magnitude of deviation from a reference state of a predetermined part of the face, the facial feature amounts being detected from each of a plurality of facial images representing the face of the vehicle driver, which have been generated by an imaging unit mounted on the vehicle at times included in a predetermined time range; a facial feature amount identification unit that identifies a facial feature amount that is larger than the facial feature amount threshold, from the plurality of facial feature amounts detected from each of the plurality of facial images generated by the imaging unit at times not included in the predetermined time range; and an emotion estimation unit that estimates the emotion of the driver using normalized feature amounts that have been normalized downwardly from the identified facial feature amounts.

[0008] In the feeling estimation device according to the present disclosure, the predetermined time range preferably includes a time immediately after the feeling estimation device is started up.

[0009] In the feeling estimation device according to the present disclosure, it is preferable that the threshold setting unit sets, as the facial feature amount threshold, a value of a facial feature amount that is located at a predetermined percentage from a minimum value when a plurality of facial feature amounts detected from each of a plurality of facial images generated at times included in a predetermined time range are arranged in ascending order.

[0010] In the feeling estimation device according to the present disclosure, it is preferable that the threshold setting unit sets a facial feature amount threshold for each of a plurality of facial action units included in each of the plurality of facial images, and the facial feature amount identifying unit identifies, for each of the plurality of facial action units, a facial feature amount that is greater than the facial feature amount threshold set for the facial action unit.

[0011] The emotion estimation method according to the present disclosure includes setting a facial feature threshold based on a plurality of facial features that indicate the magnitude of deviation from a reference state of a predetermined part of the face, the facial features being detected from each of a plurality of facial images representing the face of the vehicle driver that were generated by an imaging unit mounted on the vehicle at a time that is included in a predetermined time range; identifying facial features that are larger than the facial feature threshold from among the plurality of facial features detected from each of the plurality of facial images that were generated by the imaging unit at a time that is not included in the predetermined time range; and estimating the emotion of the driver using normalized features that are normalized downward by the identified facial features.

[0012] The emotion estimation device according to the present disclosure can appropriately estimate the emotion of a vehicle driver. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic configuration diagram of a vehicle in which the emotion estimation device is implemented. [Figure 2] FIG. 1 is a hardware schematic diagram of an emotion estimation device. [Figure 3] FIG. 2 is a functional block diagram of a processor included in the emotion estimation device. [Figure 4] FIG. 1A is a diagram illustrating a plurality of face images, and FIG. 1B is a diagram illustrating a plurality of face feature amounts detected from each of the plurality of face images. [Figure 5] (a) is a diagram showing the distribution of facial feature values in a specified time range, (b) is a diagram showing the distribution of facial feature values in a time range different from the time range in (a), and (c) is a diagram showing the distribution of facial feature values in which facial feature values larger than a facial feature value threshold are normalized downward. [Figure 6] 10 is a flowchart of an emotion estimation process. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, with reference to the drawings, a feeling estimation device capable of appropriately estimating the feeling of a vehicle driver will be described in detail. The feeling estimation device sets a facial feature amount threshold based on facial feature amounts that represent the magnitude of deviation from a reference state of a predetermined facial part, detected from a facial image of the vehicle driver's face generated by an imaging unit mounted on the vehicle at a time included in a predetermined time range. The feeling estimation device also identifies facial feature amounts that are larger than the facial feature amount threshold, among facial feature amounts detected from facial images generated by the imaging unit at a time not included in the predetermined time range. The feeling estimation device then estimates the feeling of the driver using normalized feature amounts that are obtained by normalizing the identified facial feature amounts downward.

[0015] FIG. 1 is a schematic configuration diagram of a vehicle in which the emotion estimation device is implemented.

[0016] The vehicle 1 has a driver monitor camera 2 and an emotion estimation device 3. The driver monitor camera 2 and the emotion estimation device 3 are communicably connected via an in-vehicle network that complies with a standard such as a controller area network.

[0017] The driver monitor camera 2 is an example of an imaging unit for generating a facial image representing the face of the vehicle driver. The driver monitor camera 2 has a two-dimensional detector configured with an array of photoelectric conversion elements, such as a CCD or C-MOS, that are sensitive to infrared light, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The driver monitor camera 2 also has a light source that emits infrared light. The driver monitor camera 2 is mounted, for example, in the upper front part of the vehicle interior, facing the face of the driver seated in the driver's seat. The driver monitor camera 2 irradiates the driver with infrared light at a predetermined photographing interval (for example, 1 / 30 to 1 / 10 seconds) and outputs an image showing the driver's face.

[0018] The emotion estimation device 3 is an ECU (Electronic Control Unit) having a communication interface, a memory, and a processor. The emotion estimation device 3 estimates the emotion of the driver of the vehicle 1 using an image generated by the driver monitor camera 2.

[0019] The vehicle 1 appropriately operates each part of the vehicle 1 according to the driver's emotion estimated by the emotion estimation device 3. For example, if anxiety is detected as the driver's emotion, the vehicle 1 reduces the driving speed during automatic driving by a driving control device (not shown). Furthermore, if boredom is detected as the driver's emotion, the vehicle 1 outputs music preferred by the driver from a speaker (not shown).

[0020] 2 is a hardware schematic diagram of the emotion estimation device 3. The emotion estimation device 3 includes a communication interface 31, a memory 32, and a processor 33.

[0021] The communication interface 31 is an example of a communication unit, and has a communication interface circuit for connecting the feeling estimation device 3 to an in-vehicle network. The communication interface 31 supplies received data to the processor 33. In addition, the communication interface 31 outputs data supplied from the processor 33 to the outside.

[0022] The memory 32 includes a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 32 stores various data used in processing by the processor 33, such as parameters of a neural network used as a classifier that detects facial features from a facial image. The memory 32 also stores various application programs, such as an emotion estimation program that executes emotion estimation processing.

[0023] The processor 33 is an example of a control unit and includes one or more processors and their peripheral circuits. The processor 33 may further include other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphics processing unit.

[0024] FIG. 3 is a functional block diagram of the processor 33 included in the emotion estimation device 3.

[0025] The processor 33 of the emotion estimation device 3 has, as functional blocks, a threshold setting unit 331, a facial feature identification unit 332, and an emotion estimation unit 333. Each of these units included in the processor 33 is a functional module implemented by a program executed on the processor 33. Alternatively, each of these units included in the processor 33 may be implemented in the emotion estimation device 3 as an independent integrated circuit, microprocessor, or firmware.

[0026] The threshold setting unit 331 sets a facial feature threshold based on facial feature values representing the degree of deviation of a predetermined facial part from a reference state, detected from facial images of the driver of the vehicle 1 captured by a vehicle-mounted camera at a time within a predetermined time range. In this embodiment, the threshold setting unit 331 acquires multiple facial images of the driver of the vehicle 1 captured by the driver monitor camera 2 at a time within a predetermined time range. The threshold setting unit 331 inputs each of the acquired facial images to a classifier trained to detect facial action units (FAUs), such as the corners of the eyes, the inner corners of the eyes, and the corners of the mouth, which move in response to changes in facial emotions, thereby detecting the positions of the facial action units. The threshold setting unit 331 then detects, as multiple facial feature values, the amount of change in the positions of the facial action units detected from each of the multiple facial images from the positions of the facial action units in a reference face model. The facial action units are an example of predetermined facial parts.

[0027] The classifier can be, for example, a convolutional neural network (CNN) having multiple convolution layers connected in series from the input side to the output side. By inputting facial images containing facial action units as training data into the CNN in advance and performing learning, the CNN operates as a classifier that identifies facial action units.

[0028] The threshold setting unit 331 sets the value of the facial feature amount that is located at a predetermined percentage (e.g., 95%) from the minimum value when the facial feature amounts detected from each of the facial images are arranged in ascending order as the facial feature amount threshold.

[0029] The threshold setting unit 331 may set the facial feature amount threshold based on facial feature amounts detected from multiple facial images generated at times included in a time range including the time immediately after the feeling estimation device 3 is started. The time range including the time immediately after the feeling estimation device 3 is started is, for example, a range from when the feeling estimation device 3 is started until five minutes have elapsed.

[0030] FIG. 4(a) is a diagram for explaining a plurality of face images, and FIG. 4(b) is a diagram for explaining a plurality of face feature amounts detected from each of the plurality of face images.

[0031] 4(a), facial images P1 to Pn are generated in the time range from time t1 to time tn. A plurality of facial feature amounts are detected from each of the plurality of facial images generated in this manner.

[0032] From each facial image, a plurality of facial feature amounts corresponding to a plurality of facial action units are detected. In this embodiment, the facial feature amount for a certain facial action unit is detected as a real number greater than or equal to 0 and less than 10. In FIG. 4(b), for example, facial feature amounts corresponding to FAU1, FAU2, ..., FAU9 are detected from a facial image generated at time t1. In this way, the facial feature amounts detected from each of a plurality of facial images generated within a certain time range can be expressed as a tensor.

[0033] FIG. 5(a) is a diagram showing the distribution of facial feature amounts in a predetermined time range.

[0034] 5(a), for multiple facial feature amounts for one facial action unit detected from multiple facial images generated within a predetermined time range, the number of facial feature amounts included in each range of facial feature amounts on the horizontal axis is indicated by the height of the bar on the vertical axis. That is, for example, the height of the bar indicated at the position of "0" on the horizontal axis indicates the number of facial feature amounts whose value is equal to or greater than 0 and less than 1 among multiple facial feature amounts detected from multiple facial images generated within a predetermined time range.

[0035] The threshold setting unit 331 sets the value of the facial feature amount at a predetermined percentage from the minimum value when the detected facial feature amounts are sorted in ascending order as the facial feature amount threshold. In the example of Fig. 5(a), 7, which is the value of the facial feature amount at a position that is 95% from 0, is set as the facial feature amount threshold.

[0036] Although FIG. 5(a) shows an example of facial feature amounts for one facial action unit, the threshold setting unit 331 similarly sets facial feature amount thresholds for facial feature amounts for other facial action units.

[0037] The facial feature identification unit 332 identifies facial features that are greater than a facial feature threshold value from among the facial features detected from each of the facial images generated by the driver monitor camera 2 at a time that is not included in a specified time range.

[0038] FIG. 5(b) is a diagram showing the distribution of facial feature amounts in a time range different from the time range in FIG. 5(a).

[0039] In the example of FIG. 5(b), the facial feature amount identifying unit 332 identifies a facial feature amount having a value greater than 7 from among a plurality of facial feature amounts for one facial action unit generated in the target time range.

[0040] Although FIG. 5(b) shows an example of facial feature amounts for one facial action unit, the facial feature amount specifying unit 332 similarly specifies facial feature amounts for other facial action units.

[0041] The emotion estimation unit 333 estimates the emotion of the driver of the vehicle 1 by inputting normalized features obtained by downwardly normalizing the identified facial features into an estimator trained to estimate the emotion of a person expressed based on facial features detected from an image showing the face of a person with an expression.

[0042] FIG. 5(c) is a diagram showing the distribution of facial feature amounts in which facial feature amounts greater than the facial feature amount threshold are normalized downward.

[0043] The emotion estimation unit 333 changes the value of each of the identified facial feature quantities so that the range from the facial feature quantity threshold to the maximum possible value of the facial feature quantity corresponds to the range from the minimum possible value to the maximum possible value of the facial feature quantity. In the example of Fig. 5(c), the emotion estimation unit 333 changes the value of each of the facial feature quantities identified by being included in the range of 7-9 in Fig. 5(b) so that the range of 7-9 corresponds to the range of 0-9.

[0044] The estimator can be, for example, a neural network with multiple layers connected in series from the input side to the output side. By inputting facial features detected in advance from images showing human faces with different expressions as training data into the neural network and performing learning, the CNN operates as an estimator that estimates emotions.

[0045] 6 is a flowchart of the emotion estimation process. The emotion estimation device 3 executes the emotion estimation process every time it is started up.

[0046] First, the threshold setting unit 331 of the feeling estimation device 3 sets a facial feature amount threshold based on a plurality of facial feature amounts detected from each of a plurality of facial images generated by the driver monitor camera 2 at a time included in a predetermined time range after the feeling estimation device 3 is started (Step S1).

[0047] Next, the facial feature identification unit 332 of the feeling estimation device 3 identifies facial feature values that are larger than a facial feature threshold value from among the multiple facial feature values detected from each of the facial images generated by the driver monitor camera 2 at a time not included in the predetermined time range (Step S2).

[0048] Then, the feeling estimation unit 333 of the feeling estimation device 3 generates normalized features by normalizing the identified facial features downward (step S3), and estimates the feeling of the driver of the vehicle 1 using the normalized features (step S4).

[0049] After executing the process of step S4, the process of the feeling estimation device 3 returns to step S2, and the processes from step S2 to step S4 are repeated.

[0050] By performing the emotion estimation process as described above, the emotion estimation device 3 becomes less susceptible to the influence of small values of facial feature amounts detected from a facial image of a driver with a neutral expression, and is therefore able to appropriately estimate the emotion of the vehicle driver.

[0051] It should be understood that those skilled in the art can make various changes, substitutions, and alterations thereto without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0052] 1 vehicle 3 Emotion estimation device 331 Threshold setting unit 332 Facial feature identification unit 333 Emotion estimation part

Claims

1. a threshold setting unit that sets a facial feature amount threshold based on a plurality of facial feature amounts that indicate the magnitude of deviation from a reference state of a predetermined part of the face detected from each of a plurality of facial images that represent the face of the driver of the vehicle, the facial feature amounts being generated at times included in a predetermined time range by an imaging unit mounted on the vehicle; a facial feature amount specifying unit that specifies a facial feature amount that is greater than the facial feature amount threshold value among a plurality of facial feature amounts detected from each of a plurality of the facial images generated by the photographing unit at a time that is not included in the predetermined time range; an emotion estimation unit that estimates the emotion of the driver using normalized features obtained by changing the values of the specified facial feature quantities so that a range from the facial feature quantity threshold to a maximum value that the facial feature quantity can take corresponds to a range from a minimum value that the facial feature quantity can take to a maximum value that the facial feature quantity can take; An emotion estimation device comprising:

2. The emotion deduction device according to claim 1 , wherein the predetermined time range includes a time immediately after the emotion deduction device is started.

3. 3. The feeling estimation device according to claim 1 or 2, wherein the threshold setting unit sets, as the facial feature amount threshold, a value of the facial feature amount that is located at a predetermined percentage from a minimum value when the facial feature amounts detected from each of the facial images generated at times included in the predetermined time range are arranged in ascending order.

4. the threshold setting unit sets the facial feature amount threshold for each of a plurality of facial action units included in each of the plurality of facial images; the facial feature amount specifying unit specifies, for each of the plurality of facial action units, a facial feature amount that is greater than the facial feature amount threshold set for that facial action unit; The emotion estimation device according to any one of claims 1 to 3.

5. a facial feature amount threshold is set based on a plurality of facial feature amounts that indicate the magnitude of deviation from a reference state of a predetermined part of the face detected from each of a plurality of facial images that represent the face of the driver of the vehicle, the facial feature amounts being generated at times included in a predetermined time range by an imaging unit mounted on the vehicle; Identifying a facial feature amount that is greater than the facial feature amount threshold value from a plurality of facial feature amounts detected from each of a plurality of facial images generated by the photographing unit at a time that is not included in the predetermined time range; estimating the emotion of the driver using normalized features obtained by changing the values of the identified facial feature quantities so that the range from the facial feature quantity threshold value to the maximum value that the facial feature quantities can take corresponds to the range from the minimum value that the facial feature quantities can take to the maximum value that the facial feature quantities can take; The emotion estimation method includes:

Images