Emotion inference device, emotion inference method, and non-transitory storage medium

Abstract

The present application relates to an emotion inference device, an emotion inference method, and a program for inferring an emotion with high accuracy even when there are individual differences in expressions. The emotion inference device has an image acquisition unit, an evaluation value setting unit, a correction value calculation unit, and an emotion inference unit. The image acquisition unit acquires a facial image of a user. The evaluation value setting unit sets an evaluation value for each of emotions classified into a plurality including a calm state, based on the facial image acquired by the image acquisition unit. The correction value calculation unit calculates a correction value using the evaluation value set based on a calm-time facial image acquired by the image acquisition unit at a timing of the calm state. The emotion inference unit infers the emotion of the user by reflecting the correction value calculated by the correction value calculation unit on the evaluation value set by the evaluation value setting unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an emotion inference device, an emotion inference method, and a non-transitory storage medium. BACKGROUND

[0002] For example, an emotion inference device that infers a user's emotion is disclosed in International Patent Application Publication No. 2019 / 193781. In the device disclosed in International Patent Application Publication No. 2019 / 193781, a set of evaluation values for each emotion attribute, i.e., a first evaluation value group, is acquired, and correction processing is performed in accordance with a relationship with a second evaluation value group acquired earlier than the first evaluation value group. Specifically, in a case where a difference in evaluation value in the first evaluation value group and the second evaluation value group does not exceed a reference, by reducing a correction value for an emotion attribute of the highest evaluation value and uniformly increasing correction values for other emotion attributes, each emotion's evaluation value is corrected to gradually equalize.

[0003] In the device disclosed in International Patent Application Publication No. 2019 / 193781, by correcting each emotion's evaluation value to be equal, a slight change in expression is reflected in the emotion. However, there is a possibility that an incorrect emotion is output due to detection errors of the image or the like, and there is room for improvement in terms of increasing accuracy. SUMMARY

[0004] The present disclosure obtains an emotion inference device, an emotion inference method, and a program that can infer an emotion with high accuracy even in a case where there are individual differences in expression.

[0005] The emotion inference device of the first aspect has: an image acquisition unit that acquires a facial image of a user; an evaluation value setting unit that sets an evaluation value for each of emotions classified into a plurality of emotions including a calm state, based on the facial image acquired by the image acquisition unit; a correction value calculation unit that calculates a correction value using the evaluation value set based on a calm-time facial image acquired by the image acquisition unit at a time of the calm state; and an emotion inference unit that infers the user's emotion by reflecting the correction value calculated by the correction value calculation unit on the evaluation value set by the evaluation value setting unit.

[0006] In the emotion inference device of the first aspect, the facial image of the user is acquired by the image acquisition unit. In addition, the evaluation value setting unit sets an evaluation value for each of the emotions classified into a plurality of emotions based on the facial image acquired by the image acquisition unit.

[0007] Here, the emotion inference device has a correction value calculation unit that calculates a correction value using the evaluation value set based on a calm-time facial image acquired by the image acquisition unit at a time of the calm state.

[0008] Further, the emotion inference unit infers the user's emotion by reflecting the correction value to the evaluation value of each emotion set by the evaluation value setting unit. In this way, by taking the calm state as a reference for the user's expression and using the evaluation value set based on the facial image in the calm state, it is possible to suppress variation in the evaluation value due to individual differences.

[0009] The emotion inference device of the second aspect is completed on the basis of the first aspect, wherein in the case where the emotion corresponding to the highest evaluation value among the evaluation values set by the evaluation value setting unit based on the facial image in the calm state is an emotion other than the calm state, the correction value calculation unit sets, as the correction value, the difference between the evaluation value of the other emotion and the evaluation value of the calm state.

[0010] In the emotion inference device of the second aspect, if the emotion corresponding to the highest evaluation value among the evaluation values set based on the facial image in the calm state is an emotion other than the calm state, the correction value calculation unit sets, as the correction value, the difference between the evaluation value of the other emotion and the evaluation value of the calm state. Thus, it is possible to correct the error between the calm state of the user and the emotion of the general model inferred by the emotion inference unit. Here, the "other emotion" refers to, for example, emotions such as happiness, surprise, anger, sadness, tension, and fatigue.

[0011] The emotion inference device of the third aspect is completed on the basis of the second aspect, wherein in the case where the emotion to which the highest evaluation value is set by the evaluation value setting unit is the other emotion, the emotion inference unit reflects the correction value, and in the case where the emotion to which the highest evaluation value is set by the evaluation value setting unit is an emotion other than the other emotion, the emotion inference unit does not reflect the correction value.

[0012] In the emotion inference device of the third aspect, the correction value is reflected in the case where the emotion to which the highest evaluation value is set by the evaluation value setting unit is the other emotion. Thus, it is possible to appropriately absorb the difference between the user and the evaluation value of the general model. On the other hand, since the correction value is not reflected in the case where the emotion is an emotion other than the other emotion, the emotion is inferred by the emotion inference unit based on the evaluation value of the general model in the case where the emotion is an emotion other than the other emotion.

[0013] The emotion inference device of the fourth aspect is completed on the basis of the third aspect, wherein in the case where the emotion to which the highest evaluation value is set is the other emotion, when the difference between the evaluation value of the second emotion corresponding to the second highest evaluation value and the evaluation value of the other emotion is the same as or smaller than the correction value, the emotion inference unit infers the second emotion as the user's emotion.

[0014] In the emotion inference device of the fourth aspect, in the case where the user is in a state close to an expression of another emotion in a calm state, if the correction value is not reflected, an erroneous inference as the other emotion is made. In contrast, by making the second emotion the emotion of the user in the case where the difference between the evaluation value of the second emotion and the evaluation value of the other emotion is the same as or smaller than the correction value, even the user whose evaluation value in the calm state is different from the general model can be more correctly inferred.

[0015] The emotion inference device of the fifth aspect is achieved on the basis of any one of the first to fourth aspects, wherein the image acquisition unit acquires a facial image of the driver as the facial image in the calm state in the case where the vehicle is in a prescribed driving state and the driver is in a prescribed driving state.

[0016] In the emotion inference device of the fifth aspect, the facial image acquired on the basis of the driving state of the vehicle and the driving state of the driver is used as the facial image in the calm state. Thus, compared to the case where the facial image in the calm state is acquired on the basis of only one of the driving state and the driving state, the facial image in the calm state can be acquired with high accuracy.

[0017] The emotion inference device of the sixth aspect is achieved on the basis of the fifth aspect, wherein the prescribed driving state is a state where the vehicle is straight ahead.

[0018] In the emotion inference device of the sixth aspect, since the driver's line of sight is directed to the front of the vehicle in the state where the vehicle is straight ahead, the facial image in the calm state can be acquired with high accuracy compared to the timing when the vehicle is turning left or right.

[0019] The emotion inference device of the seventh aspect is achieved on the basis of the fifth aspect, wherein the prescribed driving state is a state where the driver is not speaking.

[0020] In the emotion inference device of the seventh aspect, the possibility of being in a calm state is high in the state where the driver is not speaking compared to the state where the driver is speaking. Thus, the facial image in the calm state can be acquired with high accuracy.

[0021] In the emotion inference method of the eighth aspect, a facial image of a user is acquired by a processor, an evaluation value is set for each of emotions classified into a plurality of emotions including a calm state on the basis of the acquired facial image, a correction value is calculated using an evaluation value set on the basis of a facial image in the calm state acquired at a timing in the calm state, and an emotion of the user is inferred by reflecting the calculated correction value on the set evaluation value.

[0022] The non-transitory storage medium of Mode 9 stores a program for causing a processor to execute the following emotion inference process: acquiring a facial image of a user, setting an evaluation value for each of emotions classified into a plurality including a calm state based on the acquired facial image, calculating a correction value using the evaluation value set based on a calm-time facial image acquired at a time of the calm state, and inferring an emotion of the user by reflecting the calculated correction value on the set evaluation value.

[0023] As explained above, according to the emotion inference apparatus, the emotion inference method, and the program according to the present disclosure, it is possible to infer an emotion with high precision even in a case where there is individual difference in expression. BRIEF DESCRIPTION OF DRAWINGS

[0024] Embodiments of the present application are described in detail based on the following drawings, in which:

[0025] Figure 1 is a block diagram showing a hardware structure of an emotion inference apparatus according to an embodiment.

[0026] Figure 2 is a block diagram showing a functional structure of an emotion inference apparatus according to an embodiment.

[0027] Figure 3 is a flowchart showing one example of a flow of a correction value acquisition process of an emotion inference apparatus according to an embodiment.

[0028] Figure 4 is a flowchart showing one example of a flow of an emotion inference process of an emotion inference apparatus according to an embodiment.

[0029] Figure 5A is a diagram showing an emotion inference result inferred by an emotion inference apparatus according to an embodiment.

[0030] Figure 5B is a diagram showing an emotion inference result in a comparative example.

[0031] Figure 6 is a part of a flowchart showing another example of a flow of a correction value acquisition process of an emotion inference apparatus according to an embodiment.

[0032] Figure 7 is a part of a flowchart showing a subsequent flow of Figure 6 . DETAILED DESCRIPTION

[0033] An emotion inference apparatus 10 according to an embodiment is described with reference to the drawings.

[0034] As Figure 1As shown, the emotion inference device 10 of the present embodiment is provided with a control section, that is, an ECU (Electronic Control Unit) 12, mounted on a vehicle. That is, as one example, the emotion inference device 10 of the present embodiment is mounted on a vehicle. Therefore, the user of the emotion inference device 10 becomes an occupant of the vehicle.

[0035] The ECU 12 is configured to include a CPU (Central Processing Unit) 14, a ROM (Read Only Memory) 16, a RAM (Random Access Memory) 18, a storage 20, a communication interface 22, and an input / output interface 24. Each structure is connected via a bus 26 so as to be communicable with each other.

[0036] The CPU 14 as a processor is a central arithmetic processing unit that executes various programs and controls each section. That is, the CPU 14 reads out a program from the ROM 16 as a memory or the storage 20 as a memory, and executes the program using the RAM 18 as a work area. The CPU 14 performs the control of each structure and various arithmetic processing according to the program stored in the ROM 16 or the storage 20.

[0037] The ROM 16 stores various programs and various data. The RAM 18 temporarily stores a program or data as a work area. The storage 20 is configured by a HDD (Hard Disk Drive) or a SSD (Solid State Drive), and is a non-transitory storage medium that stores various programs including an operating system and various data. In the present embodiment, a correction value acquisition program for acquiring a correction value, an emotion inference program, and the like are stored in the ROM 16 or the storage 20.

[0038] The communication interface 22 is an interface for the ECU 12 to communicate through a computer network, and for example, a standard such as 5G, LTE, Wi-Fi (registered trademark), Ethernet (registered trademark), or the like can be used.

[0039] The in-vehicle camera 28, the vehicle periphery camera 30, the microphone 32, and the display panel 34 are electrically connected to the input / output interface 24. The in-vehicle camera 28 is an optical camera for capturing an occupant in a vehicle cabin, and in the present embodiment, as one example, is disposed at a dashboard of a front portion of the vehicle and faces a driver. Further, an image including a face of the driver is captured by the in-vehicle camera 28, and the captured image is transmitted to the ECU 12.

[0040] The vehicle periphery camera 30 is installed in the vehicle cabin or the vehicle frame or the like, and is configured to be able to capture the periphery of the vehicle. In the present embodiment, as one example, the vehicle periphery camera 30 is configured from two optical cameras, an optical camera that captures the front of the vehicle, and an optical camera that captures the rear of the vehicle. Furthermore, the image of the front of the vehicle and the image of the rear of the vehicle captured by the vehicle periphery camera 30 are transmitted to the ECU 12.

[0041] The microphone 32 is disposed in the vicinity of the driver's seat in the vehicle cabin, and in the present embodiment, as one example, is a directional microphone that is disposed toward the driver's seat. Therefore, the microphone 32 collects the voice of the driver speaking, and transmits the collected voice to the ECU 12. In addition, since the microphone 32 has directivity, it is configured so that the voice of the occupant seated in the passenger seat or the like speaking is not collected at all or almost not collected.

[0042] The display panel 34 is disposed in the instrument panel or the like, and displays information to the occupant including the driver. For example, the display panel 34 displays information related to the navigation system, information related to entertainment, and information related to the attention alert, and the like.

[0043] (Functional configuration of the emotion inference device 10)

[0044] The emotion inference device 10 implements various functions using the hardware resources described above. The functional configuration implemented by the emotion inference device 10 will be described with reference to Figure 2

[0045] As shown in Figure 2 , the emotion inference device 10 is configured to include an image acquisition section 40, a periphery information detection section 42, a rating value setting section 44, a calm state detection section 46, a correction value calculation section 48, and an emotion inference section 50 as the functional configuration. Each functional configuration is implemented by the CPU 14 reading out the program stored in the ROM 16 or the storage 20 and executing.

[0046] The image acquisition section 40 acquires the image of the face of the driver captured by the in-cabin camera 28. In addition, the periphery information detection section 42 detects the periphery information of the vehicle based on the image captured by the vehicle periphery camera 30. Specifically, the periphery information detection section 42 detects other vehicles in front of the vehicle and behind the vehicle.

[0047] ​The evaluation value setting section 44 sets an evaluation value for each of the emotions classified into a plurality including the neutral state, based on the face image acquired by the image acquisition section 40. In the present embodiment, as one example, an evaluation value is set for each of the emotions classified into five of Neutral, Happy, Irritated, Nervous, and Tired. Also, in the present embodiment, the probability of each emotion is taken as the evaluation value, and normalized so that the sum of the five evaluation values is 100%. For example, the data of the evaluation value set by the evaluation value setting section 44 is expressed by Neutral: 5%, Happy: 40%, Irritated: 50%, Nervous: 5%, Tired: 0%, and the like, and transmitted as one data to the ECU 12. The interval at which the evaluation value is set by the evaluation value setting section 44 can be several seconds, or several minutes. Also, the evaluation value can be set at a prescribed timing, rather than at a constant interval. The method of setting the evaluation value can employ, for example, a method using a learned model that has learned using teaching data in which a face image of a person is grouped with an emotion.

[0048] The neutral state detection section 46 detects that the driver is in the neutral state, in a case where the vehicle is in a prescribed driving state and the driver is in a prescribed driving state. In the present embodiment, as one example, it is detected that the driver is in the neutral state when the vehicle is in a straight-ahead state and the driver is in a non-speaking state. The state of the vehicle can be determined to be in a straight-ahead state, for example, from an image of the front of the vehicle acquired from the vehicle periphery camera 30.

[0049] The state of the driver can be determined to be a non-speaking state, for example, from a signal acquired from the microphone 32. Also, in a case where the same directional microphone is provided in the other seat as well, and no speech is recognized from any of the microphones, it can be determined to be non-speaking.

[0050] The correction value calculation section 48 calculates the correction value Δ using the evaluation value set based on the face image at the time of the neutral state, i.e., the neutral-time face image, acquired by the image acquisition section 40 at the time of the neutral state. Hereinafter, the specific calculation method of the correction value involved in the correction value calculation section 48 will be described with reference to Figure 5A The specific calculation method of the correction value involved in the correction value calculation section 48 will be described.

[0051] Figure 5A The upper face image in the upper and lower two face images illustrated in FIG. 6 is a neutral-time face image acquired by the image acquisition section 40 at the time of the neutral state detected by the neutral state detection section 46. Also, on the right of the neutral-time face image, the data of the evaluation value set by the evaluation value setting section 44 based on the neutral-time face image is shown.

[0052] In this evaluation data, the percentages are: Calm: 40%, Happy: 0%, Annoyed: 50%, Tensive: 5%, and Fatigued: 5%. That is, although the state is calm, the evaluation value for the emotion of annoyance is higher than the evaluation value for the emotion of calmness. Thus, when the emotion corresponding to the highest evaluation value set by the evaluation value setting unit 44 based on the facial image during calmness is an emotion other than calmness, the correction value calculation unit 48 calculates the difference between the evaluation value of the other emotion and the evaluation value of the calm state as a correction value Δ.

[0053] That is, in Figure 5A In the example, the emotion corresponding to the highest evaluation value is annoyance, which belongs to other emotions. Therefore, the correction value calculation unit 48 calculates 10% of the difference between the evaluation value of annoyance (50%) and the evaluation value of calm (40%) as a correction value Δ. Furthermore, the correction value Δ calculated by the correction value calculation unit 48 is stored in the storage unit, i.e., the memory 20. Additionally, the case where the evaluation value of other emotions is higher than calm in the calm state, i.e., annoyance, is stored in the memory 20. That is, the case where other emotions are annoyance and the correction value Δ is 10% are stored.

[0054] like Figure 2 As shown, the emotion inference unit 50 infers the user's emotion by reflecting the correction value Δ calculated by the correction value calculation unit 48 on the evaluation values ​​of each emotion set by the evaluation value setting unit 44. Hereinafter, refer to... Figure 5A The specific calculation method for emotion inference involved in the emotion inference section 50 is explained.

[0055] Figure 5A The lower facial image in the two illustrated facial images is a facial image acquired by the image acquisition unit 40 at a predetermined time. Furthermore, the evaluation value data set by the evaluation value setting unit 44 is shown on the right side of this facial image.

[0056] In this evaluation data, the percentages are: Calm: 5%, Happy: 40%, Annoyed: 50%, Tensive: 5%, Fatigue: 0%. That is, since annoyance is the highest evaluation value, it is inferred to be the driver's emotion before reflecting the state before the correction value Δ.

[0057] Here, when the emotion for which the highest evaluation value is set by the evaluation value setting unit 44 is set to another emotion, the emotion inference unit 50 of this embodiment reflects a correction value Δ. Figure 5AIn the example of the face image of the lower side, since the emotion stored as the other emotion is anger, which coincides with the emotion set as the highest evaluation value, the correction value Δ is reflected. On the other hand, in a case where the emotion set as the highest evaluation value by the evaluation value setting section 44 is an emotion other than the other emotion, the emotion estimation section 50 does not reflect the correction value Δ. For example, in a case where the emotion stored as the other emotion is anger and the emotion set as the highest evaluation value is happiness, the correction value Δ is not reflected.

[0058] In a case where the difference between the evaluation value of the second emotion corresponding to the second highest evaluation value and the evaluation value of the other emotion is the same as or smaller than the correction value Δ, the emotion estimation section 50 estimates the second emotion as the emotion of the user. In a case where the difference is larger than the correction value Δ, the emotion estimation section 50 estimates the other emotion as the emotion of the user. Figure 5A In the example of the face image of the lower side, the second emotion corresponding to the second highest evaluation value is happiness and the evaluation value is 40%. Therefore, the difference from the evaluation value of the other emotion, that is, anger, becomes 50% - 40% = 10%.

[0059] Here, since the correction value Δ calculated by the correction value calculation section 48 is 10%, it becomes the same value as the difference in the evaluation value. Therefore, the emotion estimation section 50 reflects the correction value Δ and estimates the second emotion, that is, happiness, as the emotion of the driver. On the other hand, for example, in a case where the difference is 15%, since the difference is larger than the correction value Δ, the emotion estimation section 50 estimates the other emotion with the highest evaluation value as the emotion of the driver.

[0060] (Action)

[0061] Next, the action of the present embodiment will be described.

[0062] (One example of correction value acquisition processing)

[0063] Figure 3 is a flowchart showing one example of the flow of the correction value acquisition processing involved in the emotion estimation apparatus 10. The correction value acquisition processing is executed by the CPU 14 reading out a program from the ROM 16 or the storage 20 and expanding and executing it on the RAM 18. In the present embodiment, as one example, the correction value acquisition processing is performed at a prescribed interval after the ignition switch is turned on or after the power is turned on. In addition, in a case where the correction value Δ is registered, the correction value acquisition processing is not performed until the driving of the vehicle ends. That is, the correction value Δ is registered once per driving of the vehicle.

[0064] As Figure 3As shown, the CPU 14 determines whether the shift range of the vehicle is D range in step S102. Specifically, the CPU 14 determines the current shift range by receiving a signal from the shift range. Also, in the case where the shift range is D range, the CPU 14 moves to the processing of step S104, and in the case where the shift range is not D range, the CPU 14 ends the correction value acquisition processing.

[0065] In step S104, the CPU 14 causes the evaluation value setting section 44 to perform the setting processing of the evaluation value concerned. That is, in the present embodiment, the setting of the evaluation value is started after the vehicle becomes in a state where the vehicle can travel. Then, the CPU 14 moves to the processing of step S106.

[0066] In step S106, the CPU 14 determines whether the vehicle is in straight traveling. Specifically, in the case where the road ahead of the vehicle is a road without a turn based on a signal from the vehicle periphery camera 30, the CPU 14 determines that the vehicle is in straight traveling by the function of the calm state detection section 46. Then, in the case where it is determined that the vehicle is in straight traveling, the CPU 14 moves to the processing of step S108. On the other hand, in the case where it is determined that the vehicle is not in straight traveling, that is, in the case where it is determined that the vehicle is in left or right turn traveling or the like, the CPU 14 ends the correction value acquisition processing.

[0067] In step S108, the CPU 14 determines whether the occupant is not speaking. Specifically, in the case where no sound of the driver is collected from the microphone 32, the CPU 14 determines that the driver is not speaking by the function of the calm state detection section 46. Also, in the case where it is determined that the driver is not speaking, the CPU 14 moves to the processing of step S110. On the other hand, in the case where it is determined that the driver is speaking based on the collection of the sound of the driver from the microphone 32 or the like, the CPU 14 ends the correction value acquisition processing.

[0068] In step S110, the CPU 14 acquires the evaluation value. Specifically, the CPU 14 acquires the evaluation value set by the function of the evaluation value setting section 44. Here, the evaluation value setting section 44 sets the evaluation value based on the calm facial image in the calm state where the vehicle is in straight traveling and the driver is not speaking.

[0069] Next, in step S112, the CPU 14 determines whether there is an emotion whose evaluation value is higher than calm. Specifically, it is determined whether the emotion with the highest evaluation value is other than calm. Also, if there is an evaluation value higher than the evaluation value of calm among the evaluation values of the emotions acquired in step S110, the CPU 14 moves to the processing of step S114. For example, in the case where the evaluation value of the emotion of "happy" is higher than the evaluation value of calm, the CPU 14 moves to the processing of step S114. Figure 5AIn the upper facial image, since the rating value of anger is higher than the rating value of calmness, the process moves to step S114. On the other hand, if there is no rating value higher than the rating value of calmness, that is, if the rating value of calmness is the highest rating value, CPU14 does not obtain a correction value and ends the correction value acquisition process.

[0070] like Figure 3 As shown, in step S114, CPU 14 calculates the correction value Δ and registers it in memory 20. Specifically, the correction value Δ is calculated using the function of the correction value calculation unit 48 (correction value calculation step). Figure 5A In the upper facial image, 50% - 40% = 10% becomes the correction value Δ. Furthermore, for the emotion of anger, the correction value Δ is registered as 10%. Then, CPU14 finishes the correction value acquisition process.

[0071] (An example of emotion inference processing)

[0072] Figure 4 This is a flowchart illustrating an example of the emotion inference processing involved in the emotion inference device 10. The emotion inference processing is performed by the CPU 14 reading the program from the ROM 16 or memory 20, expanding it in the RAM 18, and executing it. Furthermore, the emotion inference processing is performed at intervals of several seconds to several minutes after the vehicle is started.

[0073] In step S202, CPU 14 acquires an image. Specifically, CPU 14 acquires a facial image of the driver through the function of image acquisition unit 40 (image acquisition step).

[0074] In step S204, CPU14 sets an evaluation value. Specifically, CPU14 sets an evaluation value based on the driver's facial image using the function of the evaluation value setting unit 44 (evaluation value setting step). In the following description, the evaluation value of the highest emotion set in step S204 is set as Va1, and the evaluation value of the second highest emotion, i.e., the second emotion, is set as Va2.

[0075] In step S206, CPU 14 determines whether the correction value Δ has been registered. If it is determined that the correction value Δ calculated by the correction value calculation unit 48 has been registered, CPU 14 proceeds to step S208. If it is determined that the correction value Δ has not been registered, CPU 14 proceeds to step S214. The process of step S214 will be described later.

[0076] In step S208, the CPU 14 determines whether the evaluation value of the emotion for which the correction value is registered is the highest. That is, when it is determined that the emotion of Va1 set in step S204 is the emotion for which the correction value is registered, the CPU 14 proceeds to the processing of step S210. On the other hand, when it is determined that the emotion of Va1 is different from the emotion for which the correction value is registered, the CPU 14 proceeds to the processing of step S214. The processing of step S214 will be described later.

[0077] In step S210, the CPU 14 determines whether the difference between Va1 and Va2 is equal to or smaller than the correction value Δ. Also, when the difference between Va1 and Va2 is equal to or smaller than the correction value Δ, the CPU 14 proceeds to the processing of step S212. On the other hand, when the difference between Va1 and Va2 is larger than the correction value Δ, the CPU 14 proceeds to the processing of step S214.

[0078] In step S212, the CPU 14 infers the emotion corresponding to Va2 as the emotion of the driver (emotion inference step). For example, in the case of the face image on the lower side of Figure 5A Va1 becomes the evaluation value of anger, that is, 50, and Va2 becomes the evaluation value of happiness, that is, 40. Also, since the difference of 10 is the same as the correction value Δ, happiness corresponding to Va2 is inferred as the emotion of the driver.

[0079] On the other hand, when it is determined in step S206 that the correction value is not registered, when it is determined in step S208 that the emotion of Va1 is different from the emotion for which the correction value Δ is registered, and when it is determined in step S210 that the difference between Va1 and Va2 is larger than the correction value Δ, the CPU 14 proceeds to the processing of step S214. Also, in step S214, the CPU 14 infers the emotion corresponding to Va1 as the emotion of the driver (emotion inference step). Then, the CPU 14 ends the emotion inference processing.

[0080] As described above, in the emotion inference device 10 according to the present embodiment, the emotion inference section 50 infers the emotion of the user by reflecting the correction value Δ on the evaluation value of each emotion set by the evaluation value setting section 44. In this way, by taking the calm state as a reference of the expression of the user and using the evaluation value set based on the face image of the calm state, it is possible to suppress the variation of the evaluation value due to individual differences. As a result, even in the case where there are individual differences in the expression, it is possible to infer the emotion with high accuracy. Also, compared to the case where the emotion is inferred using only the learned model, it is possible to suppress the error in the inference of the expression due to individual differences.

[0081] Further, in the present embodiment, if the emotion corresponding to the highest evaluation value set based on the facial image in the calm state is an emotion other than the calm state, the correction value calculation section 48 sets the difference between the evaluation value of the other emotion and the evaluation value of the calm state as the correction value Δ. Thus, it is possible to correct the error of the emotion of the general model inferred by the emotion inference section 50 from the calm state of the user.

[0082] Further, in the present embodiment, the correction value is reflected in a case where the emotion for which the highest evaluation value is set by the evaluation value setting section 44 is an emotion other than the calm state. Thus, it is possible to appropriately absorb the difference in the evaluation value between the user and the general model. On the other hand, in a case where the emotion for which the highest evaluation value is set by the evaluation value setting section 44 is an emotion other than the emotion other than the calm state, the correction value is not reflected. Thus, it is possible to infer the emotion based on the evaluation value of the general model by the emotion inference section 50.

[0083] Further, in a case of a user whose expression in the calm state is close to the expression of the emotion other than the calm state, if the correction value Δ is not reflected, the emotion other than the calm state is erroneously inferred. In contrast, in the present embodiment, in a case where the difference between the evaluation value of the second emotion and the evaluation value of the emotion other than the calm state is the same as or smaller than the correction value Δ, the second emotion is regarded as the emotion of the user. Thus, even in a case of a user whose evaluation value in the calm state is different from the general model, it is possible to more accurately infer the emotion. Referring to Figure 5B The effect will be described.

[0084] Figure 5B is the emotion inference method in the comparative example, which is a method of setting the evaluation value by the evaluation value setting section 44 and inferring the emotion corresponding to the highest evaluation value as the emotion of the driver. Here, the driver is in the calm state, but the facial image is judged to be more angry than the general model. Thus, in the emotion inference method of the comparative example, the emotion of the driver is judged to be angry.

[0085] On the other hand, in the present embodiment, in a case where the emotion other than the calm state and the second emotion are angry and happy, respectively, and the difference between the evaluation values thereof is smaller than the correction value Δ, it is possible to infer the second emotion, i.e., happy, as the emotion of the driver.

[0086] In addition, in the present embodiment, the face image obtained based on the running state of the vehicle and the driving state of the driver is used as the face image at the time of calm, so that the face image at the time of calm can be obtained with high accuracy compared to a case where the face image at the time of calm is obtained based on only one of the running state and the driving state. In particular, by using the face image obtained while the vehicle is in a straight running state and the driver is in a state of not speaking as the face image at the time of calm, the face image at the time of calm can be obtained with high accuracy compared to a timing such as a time of a left or right turn. In addition, since the state of not speaking is more likely to be a calm state than a state of speaking, the face image at the time of calm can be obtained with high accuracy.

[0087] Further, in the above embodiment, as the condition for obtaining the face image at the time of calm, the vehicle is in a straight running state and the driver is in a state of not speaking, but the present application is not limited thereto, and a further condition can be added to improve the detection accuracy of the calm state. In addition, in the above embodiment, the correction value Δ is calculated and registered once per vehicle driving, but the present application is not limited thereto. For example, the correction value obtaining process shown in Figure 6 and Figure 7 may be employed.

[0088] (Other Example of Correction Value Obtaining Process)

[0089] Figure 6 and Figure 7 is a flowchart showing another example of the flow of the correction value obtaining process involved in the emotion inference device 10. This correction value obtaining process is executed by the CPU 14 reading out a program from the ROM 16 or the storage 20 and expanding and executing it on the RAM 18. In the following description, the same processes as those described in the above embodiment are appropriately omitted.

[0090] As shown in Figure 6 , the CPU 14 executes the processes of steps S102 to S108 in the same manner as the flowchart of Figure 3 . Further, when it is determined in step S108 that the driver is not speaking, the CPU 14 moves to the process of step S109.

[0091] In step S109, the CPU 14 determines whether the inter-vehicle distance is equal to or greater than a predetermined distance. Specifically, the CPU 14 measures the inter-vehicle distance to another vehicle running in front of the vehicle by the function of the surrounding information detection section 42. In addition, the CPU 14 measures the inter-vehicle distance to another vehicle running behind the vehicle by the function of the surrounding information detection section 42. Further, if the inter-vehicle distance in front of and behind the vehicle is equal to or greater than a predetermined distance, the CPU 14 determines that the inter-vehicle distance is equal to or greater than a predetermined distance and moves to the process of step S110.

[0092] In this way, by adding the distance between vehicles to the vehicle and the vehicle in front as a condition for a calm state, a calm state can be detected with high precision. That is, in situations where emotions other than calmness, such as anger, are present, the distance between the vehicle and the vehicle in front is often too narrow. Similarly, when being urged on by a vehicle behind, emotions other than calmness are more likely to be present. Therefore, by increasing the distance between vehicles to the vehicle and the vehicle in front as a condition for a calm state, the likelihood of being in a calm state can be increased.

[0093] like Figure 7 As shown, in step S112, CPU14 determines whether there is an emotion with a higher evaluation value than calm. Furthermore, if there is an emotion with a higher evaluation value than calm among the evaluation values ​​obtained in step S110, CPU14 moves to step S113 for processing.

[0094] In step S113, CPU 14 calculates the correction value Δ. Next, CPU 14 proceeds to step S116 to determine whether the correction value Δ is less than a predetermined value. This predetermined value is set to be a value that deviates significantly from the normal range. Therefore, if the correction value Δ is above the predetermined value, it can be determined that a false detection of a calm state has occurred.

[0095] If the correction value Δ is less than the specified value in step S116, the CPU14 moves to step S118 and registers the correction value Δ in the same way as in the above embodiment.

[0096] On the other hand, if the correction value Δ is above a predetermined value in step S116, CPU 14 proceeds to step S120. Furthermore, CPU 14 terminates the correction value acquisition process in step S120 without registering the correction value Δ. This prevents erroneous correction values ​​Δ from being registered.

[0097] The emotion inference device 10 according to the embodiments described above, but it can of course be implemented in various ways without departing from the spirit of this disclosure. For example, in the above embodiments, five emotions are classified: neutral, happy, irritated, nervous, and tired, but it is not limited to these. More emotions can be classified, such as surprise, anger, and sadness. Alternatively, it can be classified into four or fewer emotions.

[0098] Further, in the above embodiment, it is determined that the vehicle is in a straight-ahead state based on the image of the front of the vehicle acquired from the vehicle periphery camera 30, but the present application is not limited to this. The vehicle can be determined to be in a straight-ahead state by another method. For example, the steering angle can be detected to determine that the vehicle is in a straight-ahead state. Further, in the case where a travel route is set using a navigation system, it can be determined to be in a straight-ahead state based on information of the travel route.

[0099] Further, in the above embodiment, the condition that the driver does not speak is used as the condition of the calm state, but the present application is not limited to this. For example, a living body sensor can be mounted on the driver's seat, and information such as the heart rate and the breathing state acquired from the living body sensor can be used to determine whether the driver is in a calm state.

[0100] Further, in the above embodiment, the emotion estimation unit 50 estimates the second emotion as the user's emotion in the case where the difference between Va1 and Va2 is the same as or smaller than the correction value Δ, but the present application is not limited to this. For example, the second emotion can be estimated as the user's emotion only in the case where the difference between Va1 and Va2 is smaller than the correction value Δ. Further, in the case where the difference between Va1 and Va2 is the same as the correction value Δ, it can be estimated that the user's emotion is both the emotion corresponding to Va1 and the second emotion corresponding to Va2.

[0101] Further, in the above embodiment, the case where the emotion estimation apparatus 10 is applied to a vehicle is described, but the present application is not limited to this. The present application can be widely used when the user's emotion is estimated in the home, a hospital, and the like.

[0102] Further, in the above embodiment and the modified example, various processors other than the CPU can execute the correction value acquisition process and the emotion estimation process in which the software (program) is read into and executed by the CPU 14. As the processor in this case, a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) that can change the circuit structure after manufacture, an ASIC (Application Specific Integrated Circuit) that has a circuit structure designed to be dedicated for executing a specific process, and the like can be exemplified. Further, the correction value acquisition process and the emotion estimation process can be executed by one of these various processors, or can be executed by a combination of two or more processors of the same kind or different kinds (for example, a combination of a plurality of FPGAs and a CPU and an FPGA, and the like). Further, the hardware configuration of these various processors is more specifically a circuit in which circuit elements such as semiconductor elements are combined.

[0103] Also, in the above-described embodiments, the storage 20 is used as the storage section, but is not limited thereto. For example, various programs can be distributed by being stored in a non-transitory storage medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), and a USB (Universal Serial Bus) memory. Also, the programs can be downloaded from an external device via a network.

Claims

1. An emotion inferring apparatus, wherein, having: an image acquisition unit that acquires a facial image of a user; an evaluation value setting unit that sets an evaluation value for each of emotions classified into a plurality of emotions including a calm state, based on the facial image acquired by the image acquisition unit; a correction value calculation unit that calculates a correction value using the evaluation value set based on a calm-time facial image acquired by the image acquisition unit at a timing of the calm state; and an emotion inference unit that infers an emotion of the user by reflecting the correction value calculated by the correction value calculation unit on the evaluation value set by the evaluation value setting unit, in a case where an emotion corresponding to a highest evaluation value among the evaluation values set by the evaluation value setting unit based on the calm-time facial image is an emotion other than the calm state, the correction value calculation unit sets, as the correction value, a difference between the evaluation value of the other emotion and the evaluation value of the calm state, in a case where the emotion for which the highest evaluation value is set by the evaluation value setting unit is the other emotion, the emotion inference unit reflects the correction value, and in a case where the emotion for which the highest evaluation value is set by the evaluation value setting unit is an emotion other than the other emotion, the emotion inference unit does not reflect the correction value, in a case where the emotion for which the highest evaluation value is set is the other emotion, when a difference between the evaluation value of a second emotion corresponding to a second highest evaluation value and the evaluation value of the other emotion is the same as or smaller than the correction value, the emotion inference unit infers the second emotion as the emotion of the user.

2. The emotion inference apparatus according to claim 1, wherein in a case where the vehicle is in a prescribed driving state and the driver is in a prescribed driving state, the image acquisition unit acquires a facial image of the driver as the calm-time facial image.

3. The emotion inference apparatus according to claim 2, wherein the prescribed driving state is a state in which the vehicle is straight ahead.

4. The emotion inference apparatus according to claim 2, wherein the prescribed driving state is a state in which the driver does not speak.

5. An emotion inference method, wherein a facial image of a user is acquired by a processor, an evaluation value is set for each of emotions classified into a plurality of emotions including a calm state, based on the acquired facial image, a correction value is calculated using an evaluation value set based on a calm-time facial image acquired at a timing of the calm state, an emotion of the user is inferred by reflecting the calculated correction value on the set evaluation value, in a case where an emotion corresponding to a highest evaluation value among the evaluation values set based on the calm-time facial image is an emotion other than the calm state, a difference between the evaluation value of the other emotion and the evaluation value of the calm state is set as the correction value by the processor, in a case where the emotion for which the highest evaluation value is set is the other emotion, the correction value is reflected by the processor, and in a case where the emotion for which the highest evaluation value is set is an emotion other than the other emotion, the correction value is not reflected by the processor, ​ In a case where the emotion corresponding to the highest evaluation value among the evaluation values set based on the calm-time face image is the other emotion than the calm state, the second emotion corresponding to the second highest evaluation value is inferred as the user's emotion when the difference between the evaluation value of the second emotion and the evaluation value of the other emotion is the same as or smaller than the correction value.

6. The emotion inference method according to claim 5, wherein In a case where the vehicle is in a prescribed driving state and the driver is in a prescribed driving state, the processor acquires a face image of the driver as the calm-time face image.

7. A non-transitory storage medium in which a program for causing a processor to execute the following emotion inference processing is stored: a face image of a user is acquired, evaluation values are set for emotions classified into a plurality of emotions including a calm state, respectively, based on the acquired face image, a correction value is calculated using an evaluation value set based on a calm-time face image acquired at a time of the calm state, an emotion of the user is inferred by reflecting the calculated correction value on the set evaluation values, in a case where an emotion corresponding to the highest evaluation value among the evaluation values set based on the calm-time face image is an other emotion than the calm state, the difference between the evaluation value of the other emotion and the evaluation value of the calm state is set as the correction value, a program for causing a processor to execute the following emotion inference processing is stored: in a case where the emotion set with the highest evaluation value is the other emotion, the correction value is reflected, and in a case where the emotion set with the highest evaluation value is an emotion other than the other emotion, the correction value is not reflected, a program for causing a processor to execute the following emotion inference processing is stored: in a case where the emotion set with the highest evaluation value is the other emotion, the second emotion corresponding to the second highest evaluation value is inferred as the user's emotion when the difference between the evaluation value of the second emotion and the evaluation value of the other emotion is the same as or smaller than the correction value.

8. The non-transitory storage medium according to claim 7, wherein a program for causing a processor to execute the following emotion inference processing is stored: in a case where the vehicle is in a prescribed driving state and the driver is in a prescribed driving state, a face image of the driver is acquired as the calm-time face image.

Images