Emotion determination device, emotion determination method, and program product

Abstract

The emotion determination device has: a first inference unit that infers an emotion of a user based on a change in an expression of the user detected from a facial image of the user; a second inference unit that infers an emotion of the user based on a change in a temperature of the user detected from the user in a non-contact manner; and a determination unit that determines an emotion of the user based on an inference result of the first inference unit and an inference result of the second inference unit.

Description

Technical Field

[0001] This invention relates to a technology for determining a user's normal, angry, sad, and joyful emotions. Background Technology

[0002] To enable appropriate control, technologies are proposed to determine the user's emotions. If emotions can be accurately determined, it is possible to control the actions (speech, etc.) of the chatbot to engage in appropriate communication with the user.

[0003] Prior techniques for determining emotions have been disclosed in patent documents 1-3. Patent documents 1-3 disclose techniques for determining emotions based on various biological information such as brain waves and heart rate measured by wearable devices.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2014-178970

[0007] Patent Document 2: Japanese Patent Application Publication No. 2017-144222

[0008] Patent Document 3: Japanese Patent Application Publication No. 2012-059107 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] However, in the prior art disclosed in Patent Documents 1-3, users must wear wearable devices, resulting in poor usability. Furthermore, due to the use of a large amount of biological information, the processing load for emotion determination is high, and emotion determination takes time.

[0011] This invention was made in view of the above-mentioned actual situation, and its purpose is to provide a technology that can accurately determine emotions with a simple structure that is highly usable.

[0012] Methods for solving problems

[0013] To achieve the above objectives, the present invention employs the following structure.

[0014] A first aspect of the present invention provides an emotion determination device, characterized by comprising: a first inference unit that infers the user's emotion based on changes in the user's facial expression detected from a user's facial image; a second inference unit that infers the user's emotion based on changes in the user's temperature detected from the user in a non-contact manner; and a determination unit that determines the user's emotion based on the inference results of the first inference unit and the inference results of the second inference unit. Here, the facial image is an image obtained by an image sensor (camera) capturing a face, or a portion of the face detected in an image captured by an image sensor, etc. The temperature is detected non-contactly by a thermal sensor, etc.

[0015] Based on the above structure, emotions can be accurately determined using a simple structure that combines emotion inference based on facial images and emotion inference based on temperature, without relying on extensive biological information. Furthermore, both facial images and temperature (detection values) are obtained non-contactly, eliminating the inconvenience of users wearing wearable devices and achieving high usability. Moreover, due to individual differences in the degree of change in facial expressions and the correspondence between emotions and expressions, it is difficult to accurately determine emotions solely based on facial images. A user's temperature also varies greatly depending on the environment (season, location (indoor / outdoor)) and clothing, making it difficult to accurately determine emotions solely based on temperature. Based on the above structure, by combining the results of emotion determination (emotion inference) using facial images and the results of emotion determination using temperature, emotions can be inferred with high accuracy compared to emotion determination based on only one method.

[0016] The first inference unit and the second inference unit can also calculate the probability (i.e., a score) that the user possesses for each of the multiple emotions. The determination unit determines the user's emotion based on the scores of each emotion calculated by the first inference unit and the scores of each emotion calculated by the second inference unit. Here, in emotion inference using facial images and emotion inference using temperature, the scores are normalized using the same benchmark, and the score range is shared. For example, in emotion inference using facial images and emotion inference using temperature, the maximum score is a shared, predetermined value such as 100. The multiple emotions are normal, angry, sad, joyful, etc.

[0017] For example, if the emotion for which the maximum score is calculated by the first inference unit is consistent with the emotion for which the maximum score is calculated by the second inference unit, the determination unit may determine that emotion as the user's emotion. If the emotion for which the maximum score is calculated by the first inference unit is inconsistent with the emotion for which the maximum score is calculated by the second inference unit, the determination unit may determine the user's emotion based on the scores of each emotion calculated by the first inference unit and the scores of each emotion calculated by the second inference unit.

[0018] Specifically, if the emotion with the highest score calculated by the first inference unit is inconsistent with the emotion with the highest score calculated by the second inference unit, the determination unit may determine the emotion with the higher calculated score as the user's emotion. In this way, the emotion with higher trust level can be obtained as the determination result.

[0019] Alternatively, if the emotion with the highest score calculated by the first inference unit is inconsistent with the emotion with the highest score calculated by the second inference unit, the determination unit may determine the emotion with the larger sum of the scores calculated by the first and second inference units as the user's emotion. This allows for the determination of an emotion with higher reliability.

[0020] Alternatively, if the emotion for which the maximum score is calculated by the first inference unit is inconsistent with the emotion for which the maximum score is calculated by the second inference unit, the determination unit may identify the emotion with the larger change in score from the previous result as the user's emotion. This allows for earlier detection of emotion changes.

[0021] Alternatively, if the emotion with the highest score calculated by the first inference unit is inconsistent with the emotion with the highest score calculated by the second inference unit, the determination unit may determine the emotion with the larger variance value among the scores of these emotions as the user's emotion. This allows for the acquisition of an emotion with higher reliability as the determination result. Here, the variance value can be considered as the degree to which the score stands out.

[0022] The system may also include a modification unit that detects when the variance of the user's inferred emotion (i.e., the score) is less than a predetermined threshold during a specified period. This modification unit then modifies the inference results of the first and second inference units, tracing back from the first time point when the variance is less than the predetermined threshold, replacing the inference results at the immediately following time point when the variance is less than the predetermined threshold with the inference results from the first time point. The determination unit then determines the user's emotion based on the modified inference results. This allows for smooth capture of emotion shifts and high-precision capture of emotion trends. Consequently, it enables appropriate control over predicting emotion shifts. For example, if a user's emotion is trending from anger to sadness, the chatbot's actions (speech, etc.) can be controlled to communicate with the user who is currently experiencing anger, assuming a shift towards sadness.

[0023] The first inference unit and the second inference unit infer the user's emotion in a first cycle, and the determination unit updates and uses the inference results of the first inference unit and the second inference unit in a second cycle that is longer than the first cycle. In this way, the processing load of emotion determination can be reduced.

[0024] A second aspect of the present invention provides an emotion determination method, characterized in that it includes: a first inference step, inferring the user's emotion based on changes in the user's facial expression detected from a user's facial image; a second inference step, inferring the user's emotion based on changes in the user's temperature detected from the user in a non-contact manner; and a determination step, determining the user's emotion based on the inference results in the first inference step and the inference results in the second inference step.

[0025] Furthermore, the present invention can be used as an emotion determination system having at least a portion of the aforementioned structure or function. Additionally, the present invention can also be implemented as an emotion determination method or control method for an emotion determination system including at least a portion of the aforementioned processes, a program for causing a computer to execute these methods, or a computer-readable recording medium that non-temporarily records such a program. The aforementioned structures and processes can be combined with each other to constitute the present invention, provided that no technical contradiction arises.

[0026] According to the present invention, emotions can be determined with high accuracy using a simple structure that is highly usable. Attached Figure Description

[0027] Figure 1 This is a block diagram illustrating an example of the configuration of the emotion determination device according to the present invention.

[0028] Figure 2 (A) is a schematic diagram showing a general configuration example of the communication system (emotion determination system) of the first embodiment. Figure 2 (B) is a block diagram showing an example of the configuration of the PC (emotion determination device) in the first embodiment.

[0029] Figure 3 This is a flowchart illustrating an example of the processing flow of the first embodiment.

[0030] Figure 4 (A) to (C) are diagrams showing examples of operation in the first embodiment.

[0031] Figure 5 This is a flowchart illustrating an example of the processing flow in the second embodiment.

[0032] Figure 6 (A) to (C) are diagrams showing examples of operation in the second embodiment.

[0033] Figure 7 This is a flowchart illustrating an example of the processing flow in the third embodiment.

[0034] Figure 8 (A) to (C) are diagrams showing examples of operation in the third embodiment.

[0035] Figure 9 This is a flowchart illustrating an example of the processing flow in the fourth embodiment.

[0036] Figure 10 (A) to 10 (C) are diagrams showing examples of operation in the fourth embodiment.

[0037] Figure 11 This is a block diagram illustrating a configuration example of the PC (emotion determination device) in the fifth embodiment.

[0038] Figure 12 (A) to 12 (C) are diagrams showing examples of operation in the fifth embodiment.

[0039] Figure 13 (A) and (B) are diagrams illustrating an example of the operation of the sixth implementation method. Detailed Implementation

[0040] <Application Examples>

[0041] An application example of the present invention will be described. In conventional techniques for determining a user's emotions such as normal, angry, sad, and joy, the user must wear a wearable device, resulting in poor usability. Furthermore, because it uses a great deal of biometric information, the processing load for emotion determination is high, and emotion determination takes time.

[0042] Figure 1 This is a block diagram illustrating an example configuration of the emotion determination device 100 according to the present invention. The emotion determination device 100 includes a first emotion inference unit 101, a second emotion inference unit 102, and an emotion determination unit 103. The first emotion inference unit 101 infers the user's emotion based on changes in the user's facial expressions detected from a user's facial image. The second emotion inference unit 102 infers the user's emotion based on changes in the user's temperature detected non-contactly from the user. The emotion determination unit 103 determines the user's emotion based on the inference results of the first emotion inference unit 101 and the second emotion inference unit 102. The first emotion inference unit 101 is an example of the first inference unit of the present invention, the second emotion inference unit 102 is an example of the second inference unit of the present invention, and the emotion determination unit 103 is an example of the determination unit of the present invention. Here, the facial image is an image obtained by an image sensor (camera) capturing a face, or a portion of the face detected in an image obtained by an image sensor. The temperature is detected non-contactly by a thermal sensor or the like.

[0043] According to the structure of the emotion determination device 100 described above, without using much biological information, emotions can be determined with high accuracy using a simple structure based on the results of emotion inference using facial images and the results of emotion inference using temperature. If emotions can be correctly determined, the actions (speech, etc.) of the communication robot can be controlled, such as engaging in appropriate communication with the user. Furthermore, since both facial images and temperature (detection values) are obtained in a non-contact manner, the inconvenience of users wearing wearable devices is avoided, achieving high usability. In addition, since there are individual differences in the amount of change in facial expressions and the correspondence between emotions and facial expressions, it is difficult to determine emotions with high accuracy using only facial images. The user's temperature varies greatly depending on the environment (season, location (indoor / outdoor)) and clothing, making it difficult to determine emotions with high accuracy using only temperature. According to the above structure, by combining the results of emotion determination (emotion inference) using facial images and the results of emotion determination using temperature, emotions can be inferred with high accuracy compared to emotion determination using only one method.

[0044] <First Embodiment>

[0045] The first embodiment of the present invention will be described.

[0046] (structure)

[0047] Figure 2 (A) is a schematic diagram showing a general configuration example of the communication system (emotion determination system) according to the first embodiment. The communication system of the first embodiment includes: a camera 10, a temperature measuring device 20, a PC 200 (personal computer, emotion determination device), and a communication robot 30. The camera 10 and the PC 200 are interconnected via wired or wireless means. Similarly, the temperature measuring device 20 and the PC 200 are interconnected via wired or wireless means, and the PC 200 and the communication robot 30 are interconnected via wired or wireless means.

[0048] Camera 10 uses an image sensor to capture images and outputs the captured images to PC 200. In the first embodiment, camera 10 outputs a facial image obtained by capturing the user's face to PC 200. Temperature measuring device 20 uses a thermal sensor to measure temperature and outputs the measurement result (temperature information) to PC 200. In the first embodiment, temperature measuring device 20 detects the user's temperature (facial temperature, etc.) and outputs temperature information (detected temperature value) representing the detected temperature to PC 200. PC 200 determines the user's emotion based on the user's facial image and temperature and outputs the emotion determination result to communication robot 30. Communication robot 30 communicates with the user based on the determined emotion result.

[0049] Furthermore, camera 10 can also capture a range wider than the face. In this case, for example, processing to obtain a facial image from the image captured by camera 10 (processing to detect the face from the captured image) can be performed by camera 10 or PC 200. Additionally, temperature measuring device 20 can also detect a temperature distribution over a range wider than the user's (face). In this case, for example, processing to obtain the user's temperature from the detected temperature distribution can be performed by temperature measuring device 20 or PC 200. Furthermore, the application of the user's emotion determination result is not particularly limited and may not be related to the control of the communication robot 30.

[0050] In addition, in the first embodiment, the camera 10, temperature measuring device 20, PC 200, and communication robot 30 are independent devices, but they may not be. For example, at least two of the camera 10, temperature measuring device 20, PC 200, and communication robot 30 may be included in one device. Specifically, the camera 10, temperature measuring device 20, and PC 200 may also be built into the communication robot 30. Furthermore, there is no particular limitation on the location of the PC 200. For example, the PC 200 may be located in the same room as the camera 10, temperature measuring device 20, and communication robot 30, or it may not be located in the same room. The PC 200 may or may not be a computer in the cloud.

[0051] Figure 2 (B) is a block diagram showing an example of the configuration of PC200. PC200 includes: an image acquisition unit 210, a temperature acquisition unit 220, a control unit 230, a storage unit 240, and an output unit 250.

[0052] The image acquisition unit 210 acquires facial images from the camera 10 and outputs them to the control unit 230 (first emotion inference unit 231). In the first embodiment, the camera 10 captures moving images, and the image acquisition unit 210 sequentially acquires one frame of the moving image from the camera 10 and outputs it to the control unit 230. The period for acquiring facial images by the image acquisition unit 210 is not particularly limited. In the first embodiment, the image acquisition unit 210 acquires facial images (one frame of the moving image) synchronously with the shooting of the camera 10. That is, the image acquisition unit 210 performs the process of acquiring facial images from the camera 10 and outputting them to the control unit 230 at the frame rate of the camera 10's shooting (e.g., 30fps). Furthermore, the camera 10 can sequentially capture still images; in this case, the image acquisition unit 210 sequentially performs the process of acquiring still images from the camera 10 and outputting them to the control unit 230.

[0053] The temperature acquisition unit 220 acquires the user's temperature information from the temperature measuring device 20 and outputs it to the control unit 230 (the second emotion inference unit 232). The temperature measurement cycle of the temperature measuring device 20 and the temperature information acquisition cycle of the temperature acquisition unit 220 are not particularly limited. In the first embodiment, the temperature measuring device 20 measures the temperature synchronously with the camera 10's shooting, and the temperature acquisition unit 220 acquires temperature information synchronously with the camera 10's shooting (the image acquisition unit 210 acquires facial images). In other words, the temperature acquisition unit 220 processes the acquisition of temperature information from the temperature measuring device 20 and outputs it to the control unit 230 at the same frame rate as the camera 10's shooting.

[0054] The control unit 230 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and performs control of each component and various information processing. Details will be described later, but in the first embodiment, the control unit 230 determines the user's emotion based on the facial image acquired by the image acquisition unit 210 and the temperature information acquired by the temperature acquisition unit 220, and outputs the emotion determination result to the output unit 250.

[0055] Storage unit 240 stores programs executed by control unit 230, various data used by control unit 230, etc. For example, storage unit 240 is an auxiliary storage device such as hard disk drive or solid-state drive.

[0056] The output unit 250 outputs the judgment result (emotion judgment result) output by the control unit 230 to the communication robot 30.

[0057] The storage unit 240 will be described in more detail. The storage unit 240 has a first emotion inference dictionary 241 and a second emotion inference dictionary 242. The first emotion inference dictionary 241 is a dictionary that pre-registers the correspondence between facial expression changes and emotions. The second emotion inference dictionary 242 is a dictionary that pre-registers the correspondence between temperature changes and emotions.

[0058] The control unit 230 will be described in more detail. The control unit 230 includes: a first emotion inference unit 231, a second emotion inference unit 232, and an emotion determination unit 233.

[0059] The first emotion inference unit 231 sequentially performs processing (expression detection) to detect the user's facial expressions from the facial image output by the image acquisition unit 210. Referring to the first emotion inference dictionary 241, the first emotion inference unit 231 infers the user's emotion based on the detected changes in facial expressions. Furthermore, the first emotion inference unit 231 outputs the inferred emotion result to the emotion determination unit 233. The first emotion inference unit 231 is an example of the first inference unit of the present invention.

[0060] Furthermore, the expression detection and emotion inference performed by the first emotion inference unit 231 (both based on facial image processing) can use any algorithm. For example, existing expression detection methods can be used to detect the user's expression; specifically, a detector (recognizer) combining image features such as HoG and Haar-like can be used to detect expressions. Alternatively, a learned model generated through existing machine learning can be used to detect expressions; specifically, a learned model generated through deep learning (e.g., R-CNN, Fast R-CNN, YOLO, SSD, etc.) can be used to detect expressions. Similarly, the user's emotion can be inferred using existing emotion inference methods, or a learned model generated through existing machine learning can be used to infer the user's emotion. Specifically, a learned model generated through deep learning (e.g., R-CNN, Fast R-CNN, YOLO, SSD, etc.) can also be used to infer the user's emotion.

[0061] The second emotion inference unit 232 sequentially acquires temperature information output by the temperature acquisition unit 220, and infers the user's emotion based on changes in the acquired temperature information (changes in the user's temperature) referring to the second emotion inference dictionary 242. Furthermore, the second emotion inference unit 232 outputs the inferred emotion result to the emotion determination unit 233. The second emotion inference unit 232 is an example of the second inference unit of the present invention.

[0062] Furthermore, the emotion inference (temperature-based emotion inference) performed by the second emotion inference unit 232 can use any algorithm. For example, the user's emotion can be inferred using existing emotion inference methods, or it can be inferred using a fully learned model generated through existing machine learning. Specifically, the user's emotion can also be inferred using a fully learned model generated through deep learning (e.g., R-CNN, Fast R-CNN, YOLO, SSD, etc.).

[0063] The emotion determination unit 233 determines the user's emotion based on the inference results of the first emotion inference unit 231 and the second emotion inference unit 232, and outputs the emotion determination result to the output unit 250. The emotion determination unit 233 is an example of the determination unit of the present invention.

[0064] In the first embodiment, the first emotion inference unit 231 and the second emotion inference unit 232 calculate the probability, i.e., a score, of the user having each of the multiple emotions, and output the score of each emotion as the inference result. Furthermore, the emotion determination unit 233 determines the user's emotion based on the scores of each emotion calculated by the first emotion inference unit 231 and the scores of each emotion calculated by the second emotion inference unit 232.

[0065] Here, in both emotion inference using facial images (emotion inference by the first emotion inference unit 231) and emotion inference using temperature (emotion inference by the second emotion inference unit 232), the scores are normalized using the same benchmark, and the score ranges are shared. For example, in both emotion inference using facial images and emotion inference using temperature, the maximum score is shared by a predetermined value such as 100. In the first embodiment, in each emotion inference using facial images and emotion inference using temperature, the score of each emotion is calculated such that the sum of the multiple scores corresponding to the multiple emotions is 100. The number and types of the multiple emotions are not particularly limited, but in the first embodiment, four emotions are used: normal, angry, sad, and happy.

[0066] If the emotion for which the maximum score is calculated by the first emotion inference unit 231 matches the emotion for which the maximum score is calculated by the second emotion inference unit 232, it is considered that the user is highly likely to have that emotion. Therefore, in such a case, the emotion determination unit 233 determines the emotion for which the maximum score is calculated by the first emotion inference unit 231 and the second emotion inference unit 232 as the user's emotion.

[0067] Sometimes, the emotion for which the maximum score is calculated by the first emotion inference unit 231 is inconsistent with the emotion for which the maximum score is calculated by the second emotion inference unit 232. In such cases, the emotion determination unit 233 determines the user's emotion based on the scores of each emotion calculated by the first emotion inference unit 231 and the second emotion inference unit 232. Specifically, the emotion determination unit 233 determines the emotion with the higher score between the emotions for which the maximum score is calculated by the first emotion inference unit 231 and the emotions for which the maximum score is calculated by the second emotion inference unit 232 as the user's emotion. Since a higher score is considered to indicate a higher level of trust (the probability that the user holds the corresponding emotion), this allows the determination of the emotion with higher trust to be obtained.

[0068] (action)

[0069] Figure 3 This is a flowchart illustrating an example of the processing flow in the first embodiment. PC200 executes repeatedly. Figure 3 The processing flow. Figure 3The processing cycle is not specifically limited, but it is repeated at the frame rate (e.g., 30fps) captured by camera 10. Figure 3 The processing flow.

[0070] First, the image acquisition unit 210 acquires a facial image of the user from the camera 10 (step S301). Then, the first emotion inference unit 231 detects the user's facial expressions from the facial image acquired in step S301, and calculates scores for normal, angry, sad, and joy based on the detected changes in facial expressions (changes in facial expressions from the past to the present) (step S302).

[0071] Next, the temperature acquisition unit 220 acquires the user's temperature information (the user's temperature) from the temperature measuring device 20 (step S303). Furthermore, the second emotion inference unit 232 calculates scores for normal, angry, sad, and joy based on changes in the user's temperature (temperature changes from the past to the present) (step S304).

[0072] Next, the emotion determination unit 233 determines whether the emotion E1, which has the highest score calculated by the first emotion inference unit 231, and the emotion E2, which has the highest score calculated by the second emotion inference unit 232, are consistent (step S305). Emotion E1 is the emotion with the highest score among the four scores calculated based on facial expression changes in step S302 (corresponding to normal, anger, sadness, and joy, respectively). Emotion E2 is the emotion with the highest score among the four scores calculated based on temperature changes in step S304 (corresponding to normal, anger, sadness, and joy, respectively). If emotion E1 and emotion E2 are consistent (step S305: Yes), the process proceeds to step S306; if emotion E1 and emotion E2 are inconsistent (step S305: No), the process proceeds to step S307.

[0073] In step S306, the emotion determination unit 233 determines the emotion E1 = E2 as the user's emotion Eresult (the final determination result). Furthermore, PC200 ends. Figure 3 The processing flow.

[0074] In step S307, the emotion determination unit 233 determines whether the score of emotion E1 (based on facial expression changes) is consistent with the score of emotion E2 (based on temperature changes). If the scores of emotion E1 and emotion E2 are inconsistent (step S307: No), the process proceeds to step S308. If the scores of emotion E1 and emotion E2 are consistent (step S307: Yes), PC200 ends. Figure 3 The processing flow is as follows. At this point, the same emotion as before is used as the user's Eresult.

[0075] In step S308, the emotion determination unit 233 determines whether the score of emotion E1 is higher than the score of emotion E2. If the score of emotion E1 is higher than the score of emotion E2 (step S308: Yes), the process proceeds to step S309; ​​if the score of emotion E1 is lower than the score of emotion E2 (step S308: No), the process proceeds to step S310.

[0076] In step S309, the emotion determination unit 233 determines the emotion E1 (the emotion inferred from facial expression changes) as the user's emotion Eresult. Furthermore, PC200 ends. Figure 3 The processing flow.

[0077] In step S310, the emotion determination unit 233 determines the emotion E2 (the emotion inferred based on temperature changes) as the user's emotion Eresult. Furthermore, PC200 ends. Figure 3 The processing flow.

[0078] (Specific examples of actions)

[0079] Figure 4 (A) to (C) show examples of the operation of the first embodiment. Figure 4 (A) shows an example of the time variation of the scores of each emotion (based on the scores of facial expression changes) calculated by the first emotion inference unit 231. Figure 4 In (A), emotion E1 (the emotion inferred from changes in facial expression) is shown in bold. Figure 4 (B) shows an example of the time-varying score (based on temperature change) of each emotion calculated by the second emotion inference unit 232. Figure 4 In (B), emotion E2 (emotion inferred from temperature changes) is shown in thick box. Figure 4 (C) shows an example of the temporal change of the emotion (Eresult) determined by the emotion determination unit 233.

[0080] exist Figure 4 In examples (A) to (C), during frames 0 to 2, both emotions E1 and E2 are normal, and "normal" is determined as the emotion Eresult. In frame 3, emotion E1 is anger, and emotion E2 is normal; emotions E1 and E2 are different. Here, the score for emotion E1 (anger) is 55, and the score for emotion E2 (normal) is 52. Since the score for emotion E1 (anger) (55) is higher than the score for emotion E2 (normal) (52), anger is determined as the emotion Eresult. The same applies to other frames, where the emotion Eresult is determined based on the consistency / inconsistency, magnitude, etc., between the scores of emotion E1 and emotion E2.

[0081] (Summarize)

[0082] As described above, according to the first embodiment, without using a lot of biological information, emotions can be determined with high accuracy using a simple structure based on the results of emotion inference from facial images and the results of emotion inference from temperature. Furthermore, since both facial images and temperature (detection values) are obtained non-contactly, the inconvenience of users wearing wearable devices is avoided, achieving high usability. Additionally, by determining the emotion with the higher calculated score from the emotion inferred from facial images and the emotion inferred from temperature as the user's emotion, a more reliable emotion can be obtained as the determination result.

[0083] <Second Implementation>

[0084] The second embodiment of the present invention will be described.

[0085] (structure)

[0086] The structure of the communication system and the structure of the PC200 in the second embodiment are almost identical to those in the first embodiment. However, in the second embodiment, the processing by the emotion determination unit 233 when the emotion with the highest score calculated by the first emotion inference unit 231 and the emotion with the highest score calculated by the second emotion inference unit 232 are inconsistent differs from that in the first embodiment. In the second embodiment, in such a case, the emotion determination unit 233 determines the emotion with the larger sum of the scores calculated by the first emotion inference unit 231 and the second emotion inference unit 232 as the user's emotion. This allows for the determination of an emotion with higher reliability.

[0087] (action)

[0088] Figure 5 This is a flowchart illustrating an example of the processing flow in the second embodiment. PC200 executes repeatedly. Figure 5 The processing flow. Figure 5 The processing cycle is not specifically limited, but it is repeated at the frame rate (e.g., 30fps) captured by camera 10. Figure 5 The processing flow.

[0089] The processing of steps S501 to S507 is the same as that of steps S301 to S307 in the first embodiment. If the score of emotion E1 (based on facial expression change) is inconsistent with the score of emotion E2 (based on temperature change) (step S507: No), the processing proceeds to step S508.

[0090] In step S508, the emotion determination unit 233 calculates the total score S1 corresponding to emotion E1. The total score S1 is calculated by adding the score of emotion E1 (the score calculated by the first emotion inference unit 231 and the score based on facial expression changes) and the score calculated by the second emotion inference unit 232 for the same emotion as emotion E1 (the score based on temperature changes).

[0091] In step S509, the emotion determination unit 233 calculates the total score S2 corresponding to emotion E2. The total score S2 is calculated by adding the score of emotion E2 (the score calculated by the second emotion inference unit 232 and the score based on temperature change) and the score calculated by the first emotion inference unit 231 for the same emotion as emotion E2 (the score based on facial expression change).

[0092] In step S510, the emotion determination unit 233 determines whether the total score S1 calculated in step S508 is consistent with the total score S2 calculated in step S509. If the total score S1 and the total score S2 are inconsistent (step S510: No), the process proceeds to step S511. If the total score S1 and the total score S2 are consistent (step S510: Yes), PC200 ends. Figure 5 The processing flow is as follows. At this point, the same emotion as before is used as the user's emotion result.

[0093] In step S511, the emotion determination unit 233 determines whether the total score S1 calculated in step S508 is higher than the total score S2 calculated in step S509. If the total score S1 is higher than the total score S2 (step S511: Yes), the process proceeds to step S512; if the total score S1 is lower than the total score S2 (step S511: No), the process proceeds to step S513.

[0094] In step S512, the emotion determination unit 233 determines the emotion E1 corresponding to the total score S1 as the user's emotion Eresult. Furthermore, PC200 ends. Figure 5 The processing flow.

[0095] In step S513, the emotion determination unit 233 determines the emotion E2 corresponding to the total score S2 as the user's emotion Eresult. Furthermore, PC200 ends. Figure 5 The processing flow.

[0096] (Specific examples of actions)

[0097] Figure 6 (A) to (C) show examples of the operation of the second embodiment. Figure 6(A) shows an example of the time variation of the scores of each emotion (based on the scores of facial expression changes) calculated by the first emotion inference unit 231. Figure 6 In (A), emotion E1 (the emotion inferred from changes in facial expression) is shown in bold. Figure 6 (B) shows an example of the time-varying score (based on temperature change) of each emotion calculated by the second emotion inference unit 232. Figure 6 In (B), emotion E2 (emotion inferred from temperature changes) is shown in thick box. Figure 6 (C) shows an example of the temporal change of the emotion (Eresult) determined by the emotion determination unit 233.

[0098] exist Figure 6 In examples (A) to 6 (C), during the period from frame 0 to frame 2, both emotions E1 and E2 are normal, so normal is also determined as the emotion Eresult. In frame 3, emotion E1 is anger, and emotion E2 is normal, so emotion E1 is different from emotion E2. Here, for emotion E1 (anger), the total score (55) calculated by the first emotion inference unit 231 and the total score (20) calculated by the second emotion inference unit 232 (total score S1) is 75. For emotion E2 (normal), the total score (22) calculated by the first emotion inference unit 231 and the total score (52) calculated by the second emotion inference unit 232 (total score S2) is 74. Since the total score S1 (75) corresponding to emotion E1 (anger) is higher than the total score S2 (74) corresponding to emotion E2 (normal), anger is determined as the emotion Eresult. The same applies to other frames, where the emotion Eresult is determined based on the consistency / inconsistency between the scores of emotion E1 and emotion E2, and the relationship between the total score S1 and the total score S2.

[0099] (Summarize)

[0100] As described above, even in the second embodiment, the same as in the first embodiment, it is possible to achieve the effect of accurately determining emotions using a simple structure with high usability. Furthermore, by determining the user's emotion as the emotion with the higher sum of the facial image-based score and the temperature-based score in both the emotion inferred from facial images and the emotion inferred from temperature, it is possible to obtain a more reliable emotion as the determination result.

[0101] <Third Implementation>

[0102] The third embodiment of the present invention will be described.

[0103] (structure)

[0104] The structure of the communication system and the structure of the PC200 in the third embodiment are almost identical to those in the first embodiment. However, in the third embodiment, the processing by the emotion determination unit 233 when the emotion with the maximum score calculated by the first emotion inference unit 231 is inconsistent with the emotion with the maximum score calculated by the second emotion inference unit 232 differs from that in the first embodiment. In the third embodiment, in such cases, the emotion determination unit 233 determines the emotion with the largest change from the previous score as the user's emotion. This allows for earlier detection of emotion changes.

[0105] (action)

[0106] Figure 7 This is a flowchart illustrating an example of the processing flow in the third embodiment. PC200 executes repeatedly. Figure 7 The processing flow. Figure 7 The processing cycle is not specifically limited, but it is repeated at the frame rate (e.g., 30fps) captured by camera 10. Figure 7 The processing flow.

[0107] The processing of steps S701 to S707 is the same as that of steps S301 to S307 in the first embodiment. If the score of emotion E1 (based on facial expression change) is inconsistent with the score of emotion E2 (based on temperature change) (step S707: No), the processing proceeds to step S708.

[0108] In step S708, the emotion determination unit 233 calculates the change D1 from the previous score (based on facial expression changes) of emotion E1 to the current score (based on facial expression changes) of emotion E1. The change D1 is the difference obtained by subtracting the previous score of emotion E1 from the current score of emotion E1.

[0109] In step S709, the emotion determination unit 233 calculates the change D2 from the previous score (score based on temperature change) of emotion E2 to the current score (score based on temperature change) of emotion E2. The change D2 is the difference obtained by subtracting the previous score of emotion E2 from the current score of emotion E2.

[0110] In step S710, the emotion determination unit 233 determines whether the change amount D1 calculated in step S708 is consistent with the change amount D2 calculated in step S709. If the change amount D1 and the change amount D2 are inconsistent (step S710: No), the process proceeds to step S711. If the change amount D1 and the change amount D2 are consistent (step S710: Yes), PC200 ends. Figure 7 The processing flow is as follows. At this point, the same emotion as before is used as the user's emotion result.

[0111] In step S711, the emotion determination unit 233 determines whether the change amount D1 calculated in step S708 is greater than the change amount D2 calculated in step S709. If the change amount D1 is greater than the change amount D2 (step S711: Yes), the process proceeds to step S712; if the change amount D1 is less than the change amount D2 (step S711: No), the process proceeds to step S713.

[0112] In step S712, the emotion determination unit 233 determines the emotion E1 corresponding to the change D1 as the user's emotion Eresult. Furthermore, PC200 ends. Figure 7 The processing flow.

[0113] In step S713, the emotion determination unit 233 determines the emotion E2 corresponding to the change D2 as the user's emotion Eresult. Furthermore, PC200 ends. Figure 7 The processing flow.

[0114] (Specific examples of actions)

[0115] Figure 8 (A) to (C) show examples of the operation of the third embodiment. Figure 8 (A) shows an example of the time variation of the scores of each emotion (based on the scores of facial expression changes) calculated by the first emotion inference unit 231. Figure 8 In (A), emotion E1 (the emotion inferred from changes in facial expression) is shown in bold. Figure 8 (B) shows an example of the time-varying score (based on temperature change) of each emotion calculated by the second emotion inference unit 232. Figure 8 In (B), emotion E2 (emotion inferred from temperature changes) is shown in thick box. Figure 8 (C) shows an example of the temporal change of the emotion (Eresult) determined by the emotion determination unit 233.

[0116] exist Figure 8In examples (A) to 8(C), during frames 0 to 2, both emotions E1 and E2 are normal, and "normal" is determined as the emotion Eresult. In frame 3, emotion E1 is anger, and emotion E2 is normal; therefore, emotions E1 and E2 are different. Here, for emotion E1 (anger), the change D1 from the score previously calculated by the first emotion inference unit 231 (23; score in frame 2) to the score currently calculated by the first emotion inference unit 231 (55; score in frame 3) is 32 (=55-23). For emotion E2 (normal), the change D2 from the score previously calculated by the second emotion inference unit 232 (61; score in frame 2) to the score currently calculated by the second emotion inference unit 232 (52; score in frame 3) is -9 (=52-61). Since the change in magnitude D1 (32) corresponding to emotion E1 (anger) is greater than the change in magnitude D2 (-9) corresponding to emotion E2 (normal), anger is determined as the emotion Eresult. The same applies to other frames, where the emotion Eresult is determined based on the consistency / inconsistency between the scores of emotion E1 and emotion E2, the magnitude relationship between the changes in magnitude D1 and magnitude D2, etc.

[0117] (Summarize)

[0118] As described above, even in the third embodiment, the same as in the first embodiment, it is possible to achieve the effect of accurately determining emotions using a simple structure with high usability. Furthermore, by identifying emotions with larger changes in scores from previous readings—those inferred from facial images and those inferred from temperature—as the user's emotions, changes in emotions can be captured earlier.

[0119] <Fourth Implementation>

[0120] The fourth embodiment of the present invention will be described.

[0121] (structure)

[0122] The structure of the communication system and the structure of the PC200 in the fourth embodiment are almost the same as those in the first embodiment. However, in the fourth embodiment, the processing by the emotion determination unit 233 when the emotion with the highest score calculated by the first emotion inference unit 231 is inconsistent with the emotion with the highest score calculated by the second emotion inference unit 232 differs from that in the first embodiment. In the fourth embodiment, in such cases, the emotion determination unit 233 determines the emotion with the larger variance value among the scores of these emotions as the user's emotion. Here, the variance value can also be described as the degree of score prominence. Since it is believed that the larger the variance value among the scores of multiple emotions, in other words, the more prominent the highest score, the higher the confidence level (the probability that the user holds the corresponding emotion), it is possible to obtain an emotion with higher confidence as the determination result. The variance value V of the n scores Xi (i is an integer greater than or equal to 1 and less than or equal to n) can be calculated, for example, by the following Equation 1. In Equation 1, Xave is the average of the n scores Xi.

[0123] [Number 1]

[0124]

[0125] (action)

[0126] Figure 9 This is a flowchart illustrating an example of the processing flow in the fourth embodiment. PC200 executes repeatedly. Figure 9 The processing flow. Figure 9 The processing cycle is not specifically limited, but it is repeated at the frame rate (e.g., 30fps) captured by camera 10. Figure 9 The processing flow.

[0127] The processing of steps S901 to S907 is the same as that of steps S301 to S307 in the first embodiment. If the score of emotion E1 (based on facial expression change) is inconsistent with the score of emotion E2 (based on temperature change) (step S907: No), the processing proceeds to step S908.

[0128] In step S908, the emotion determination unit 233 calculates the variance of multiple scores (multiple scores based on facial expression changes) calculated by the first emotion inference unit 231 for multiple emotions (normal, angry, sad, and joy), and uses it as the variance value V1 corresponding to the emotion E1.

[0129] In step S909, the emotion determination unit 233 calculates the variance of multiple scores (multiple scores based on temperature changes) calculated by the second emotion inference unit 232 for multiple emotions (normal, angry, sad, and joy), and uses it as the variance value V2 corresponding to the emotion E2.

[0130] In step S910, the emotion determination unit 233 determines whether the variance value V1 calculated in step S908 is consistent with the variance value V2 calculated in step S909. If the variance values ​​V1 and V2 are inconsistent (step S910: No), the process proceeds to step S911. If the variance values ​​V1 and V2 are consistent (step S910: Yes), PC200 ends. Figure 9 The processing flow is as follows. At this point, the same emotion as before is used as the user's emotional result.

[0131] In step S911, the emotion determination unit 233 determines whether the variance value V1 calculated in step S908 is greater than the variance value V2 calculated in step S909. If the variance value V1 is greater than the variance value V2 (step S911: Yes), the process proceeds to step S912; if the variance value V1 is less than the variance value V2 (step S911: No), the process proceeds to step S913.

[0132] In step S912, the emotion determination unit 233 determines the emotion E1 corresponding to the variance value V1 as the user's emotion Eresult. Furthermore, PC200 ends. Figure 9 The processing flow.

[0133] In step S913, the emotion determination unit 233 determines the emotion E2 corresponding to the variance value V2 as the user's emotion Eresult. Furthermore, PC200 ends. Figure 9 The processing flow.

[0134] (Specific examples of actions)

[0135] Figure 10 (A) to 10 (C) show examples of the operation of the fourth embodiment. Figure 10 (A) shows an example of the time variation of the scores of each emotion (based on the scores of facial expression changes) calculated by the first emotion inference unit 231. Figure 10 In (A), emotion E1 (the emotion inferred from changes in facial expression) is shown in bold. Figure 10 (B) shows an example of the time-varying score (based on temperature change) of each emotion calculated by the second emotion inference unit 232. Figure 10 In (B), emotion E2 (emotion inferred from temperature changes) is shown in thick box. Figure 10 (C) shows an example of the temporal change of the emotion (Eresult) determined by the emotion determination unit 233.

[0136] exist Figure 10In examples (A) to 10(C), during frames 0 to 2, both emotions E1 and E2 are normal, so "normal" is also determined as the emotion Eresult. In frame 3, emotion E1 is anger, and emotion E2 is normal; therefore, emotions E1 and E2 are different. Here, for emotion E1 (anger), the variance V1 calculated based on the four scores (22, 55, 12, 11) calculated by the first emotion inference unit 231 is 319. For emotion E2 (normal), the variance V2 calculated based on the four scores (52, 20, 8, 20) calculated by the second emotion inference unit 232 is 267. Since the variance V1 (319) corresponding to emotion E1 (anger) is greater than the variance V2 (267) corresponding to emotion E2 (normal), anger is determined as the emotion Eresult. The same applies to other frames, where the emotion Eresult is determined based on the consistency / inconsistency between the scores of emotion E1 and emotion E2, the relationship between the variance values ​​V1 and V2, and so on.

[0137] (Summarize)

[0138] As described above, even in the fourth embodiment, similar to the first embodiment, it is possible to achieve the effect of accurately determining emotions using a simple structure with high usability. Furthermore, by identifying the emotion with the larger variance value among the scores of multiple emotions inferred from facial images and emotions inferred from temperature as the user's emotion, it is possible to obtain a more reliable emotion as the determination result.

[0139] <Fifth Implementation>

[0140] The fifth embodiment of the present invention will be described.

[0141] (structure)

[0142] The structure of the AC system in the fifth embodiment is almost the same as that in the first embodiment. Figure 11 This is a block diagram illustrating a configuration example of the PC200 according to the fifth embodiment. The PC200 of the fifth embodiment also includes, similarly to that of the first embodiment, an image acquisition unit 210, a temperature acquisition unit 220, a control unit 230, a storage unit 240, and an output unit 250. The control unit 230 of the fifth embodiment further includes an inference result modification unit 234. In the fifth embodiment, the processing other than that described later is the same as in the first embodiment.

[0143] In the fifth embodiment, the first emotion inference unit 231 and the second emotion inference unit 232 respectively store (record, accumulate) the inference results of the user's emotions in the storage unit 240.

[0144] The inference result modification unit 234 modifies the inference results of the first emotion inference unit 231 and the second emotion inference unit 232 stored in the storage unit 240. From an easily understood perspective, in Figure 11 Instead of arrows indicating writing / reading inference results to the storage unit 240, arrows are shown pointing from the first emotion inference unit 231 and the second emotion inference unit 232 toward the inference result modification unit 234. Specifically, the inference result modification unit 234 detects a time when the variance of the score (the inferred emotion score) during the period up to a predetermined time is less than a predetermined threshold. Such a time can be considered a stable time for the inference result of the user's emotion. Furthermore, the inference result modification unit 234 traces back from the first time when the variance value is less than the predetermined threshold and replaces the inference result up to the immediately following time when the variance value is less than the predetermined threshold with the inference result of the first time. The inference result modification unit 234 modifies the inference results of the first emotion inference unit 231 and the second emotion inference unit 232 to perform such time detection and inference result replacement. The inference result modification unit 234 is an example of a modified unit of the present invention.

[0145] The emotion determination unit 233 determines the user's emotion based on the inference result modified by the inference result modification unit 234. For example, the user's emotion can be determined using any of the methods in embodiments 1 to 4.

[0146] In this way, it is possible to smoothly capture changes in emotions and accurately capture the trends of these changes. As a result, it is possible to appropriately predict and control these emotional changes. For example, if a user's emotion is trending from anger to sadness, the actions (speech, etc.) of the chatbot 30 can be controlled to engage in conversation with the user who is currently experiencing anger, assuming the emotion is shifting towards sadness.

[0147] (Specific examples of actions)

[0148] Figure 12 (A) to 12 (C) show examples of operation in the fifth embodiment. Hereinafter, an example of changing the inference result of the first emotion inference unit 231 will be described. The inference result of the second emotion inference unit 232 will also be changed in the same way as described below.

[0149] Figure 12 (A) shows an example of the time change of the score before the change by the inference result change unit 234, and an example of the time change of the score (based on facial expression change) of each emotion calculated by the first emotion inference unit 231. Figure 12 In (A), emotion E1 (the emotion inferred from changes in facial expression) is shown in bold.

[0150] First, the inference result change unit 234 is based on Figure 12 (A) The variance of the emotion E1 inferred by the inference result change unit 234 up to a specified time is calculated based on the time variation. The specified time is not particularly limited, but here it is the time of 3 frames. For example, in frame 2, the emotion E1 (the emotion with the highest score) is normal, so the inference result change unit 234 calculates the variance (16.9) based on the normal score (56) of frame 2, the normal score (50) of frame 1, and the normal score (60) of frame 0. In frame 3, the emotion E1 (the emotion with the highest score) is anger, and the inference result change unit 234 calculates the variance (251) based on the anger score (55) of frame 3, the anger score (23) of frame 2, and the anger score (20) of frame 1. For other frames, the inference result change unit 234 calculates the variance in the same way. Figure 12 (B) shows an example of the time variation of the calculated variance value.

[0151] Next, the inference result change unit 234 detects a time (frame) when the variance value is less than a specified threshold. The specified threshold is not specifically defined, but here it is 50. Figure 12 In (B), variance values ​​less than the specified threshold (50) are shown in bold. Specifically, frames 2, 7, 13, and 14 were detected.

[0152] Finally, the inference result modification unit 234 traces back from the first timing point where the variance value is less than the predetermined threshold (50), and replaces the inference result up to the timing point immediately following the second timing point where the variance value is less than the predetermined threshold (50) with the inference result of the first timing point. As described above, in Figure 12 In (B), frames 2, 7, 13, and 14 are detected when the variance value is less than a predetermined threshold (50). Therefore, the inference result modification unit 234 performs a score substitution so that the scores of frames 0 to 2 become the scores of frame 2, the scores of frames 3 to 7 become the scores of frame 7, and the scores of frames 8 to 13 become the scores of frame 13. Figure 12 (C) shows an example of the time change of the score after the change of the inference result change section 234.

[0153] In addition, Figure 12 (C) Only the score for emotion E1 (inferred to be the user's emotion) is shown, but the above substitution can also be applied to scores other than emotion E1. Furthermore, when multiple frames share the same emotion E1, score substitution can be performed to unify the scores across those frames. For example, Figure 12 In cases (A) to 12 (C), the emotions E1 of frame 13 and frame 14 are both sadness. Therefore, the inference result change unit 234 can also perform score substitution so that the scores of frames 8 to 14 become the scores of frame 13 or frame 14.

[0154] (Summarize)

[0155] As described above, even in the fifth embodiment, similar to the first embodiment, it is possible to achieve the effect of accurately determining emotions using a simple structure with high usability. Furthermore, by modifying the emotion inference results to smoothly capture emotion shifts, it is possible to accurately capture the trends of emotion shifts.

[0156] <Sixth Implementation>

[0157] The sixth embodiment of the present invention will be described.

[0158] (structure)

[0159] The structure of the communication system and the structure of the PC200 in the sixth embodiment are almost identical to those in the first embodiment. In the sixth embodiment, the first emotion inference unit 231 and the second emotion inference unit 232 infer the user's emotion in a first cycle. The emotion determination unit 233 updates and uses the inference results of the first emotion inference unit 231 and the second emotion inference unit 232 in a second cycle that is longer than the first cycle. For example, the second cycle is three times the first cycle. If the frame rate captured by the camera 10 is 30fps, the first cycle corresponds to that frame rate, and the second cycle corresponds to one-third of that frame rate (10fps). Other than these, the structure (processing) is the same as any one of the first to fourth embodiments.

[0160] In this way, since the processing frequency of the emotion determination unit 233, which updates the inference results of the first emotion inference unit 231 and the second emotion inference unit 232, can be reduced, the processing load of emotion determination can be reduced. Furthermore, the emotion determination unit 233 can perform emotion determination and output of results in a first cycle, or it can perform them in a second cycle. If they are also performed in a second cycle, the processing load of the emotion determination unit 233 can be further reduced.

[0161] (Specific examples of actions)

[0162] Figure 13 (A) and (B) show examples of the operation of the sixth embodiment. Hereinafter, an example of the emotion determination unit 233 updating the inference result of the first emotion inference unit 231 will be described. The emotion determination unit 233 also updates the inference result of the second emotion inference unit 232 in the same way as described below.

[0163] Figure 13 (A) shows an example of the time variation of the scores of each emotion (based on the scores of facial expression changes) calculated by the first emotion inference unit 231. Figure 13 In (A), emotion E1 (the emotion inferred from changes in facial expression) is shown in bold.

[0164] Figure 13 (B) shows an example of the time variation of the scores (based on facial expression changes) of each emotion used by the emotion determination unit 233. Figure 13 (B) Emotion E1 (inferred from changes in facial expression) is also indicated by a thick box.

[0165] like Figure 13 As shown in (A), the first emotion inference unit 231 calculates and updates the score every frame. On the other hand, as... Figure 13 As shown in (B), the emotion determination unit 233 calculates and updates the score every 3 frames. Specifically, the emotion determination unit 233 uses the score of frame 0 as the score of frames 1 and 2. Therefore, the emotion determination unit 233 can perform processing related to frames 1 and 2 at a timing earlier than frames 1 and 2. In other words, the emotion determination result can be obtained earlier. When updating the score in frame 3, the score of frame 3 is also used as the score of frames 4 and 5. Thereafter, the score is updated in the same way.

[0166] (Summarize)

[0167] As described above, even in the sixth embodiment, the same as in the first embodiment, it is possible to achieve the effect of accurately determining emotions using a simple structure with high usability. Furthermore, by reducing the update frequency of emotion inference results used for emotion determination, the processing load of emotion determination can be reduced, or the processing speed of emotion determination can be increased (the time until the emotion determination result is obtained is shortened).

[0168] <Other>

[0169] The above embodiments are merely illustrative examples illustrating the structure of the present invention. The present invention is not limited to the specific embodiments described above, and various modifications can be made within the scope of its technical concept. The processing of embodiments 1 to 6 can also be appropriately combined.

[0170] <Postscript 1>

[0171] An emotion determination device (100, 200) is characterized by having:

[0172] The first inference unit (101, 231) infers the user's emotion based on changes in the user's facial expressions detected from the user's facial image;

[0173] The second inference unit (102, 232) infers the user's emotion based on changes in the user's temperature detected from the aforementioned user in a non-contact manner; and

[0174] The determination unit (103, 233) determines the user's emotion based on the inference results of the first inference unit and the second inference unit.

[0175] <Appendix 2>

[0176] An emotion determination method, characterized by having:

[0177] The first inference step (S302, S502, S702, S902) infers the user's emotion based on changes in the user's facial expressions detected from the user's facial image;

[0178] The second inference step (S304, S504, S704, S904) infers the user's emotion based on the temperature changes detected from the user in a non-contact manner; and

[0179] The determination steps (S305~S310, S505~S513, S705~S713, S905~S913) determine the user's emotions based on the inference results in the first inference step and the inference results in the second inference step.

[0180] Explanation of reference numerals in the attached figures

[0181] 100: Emotion Detection Device

[0182] 101: First Emotion Inference Department 102: Second Emotion Inference Department 103: Emotion Judgment Department

[0183] 10: Camera; 20: Temperature measuring device; 30: Communication robot

[0184] 200: PC (Emotion Detection Device)

[0185] 210: Image acquisition unit; 220: Temperature acquisition unit; 230: Control unit

[0186] 240: Storage unit; 250: Output unit

[0187] 231: First Emotion Inference Department 232: Second Emotion Inference Department

[0188] 233: Emotion Judgment Department; 234: Inference Result Modification Department

[0189] 241: First Dictionary of Emotion Inference 242: Second Dictionary of Emotion Inference

Claims

1. An emotion determination device, characterized in that, have: The first inference unit infers the user's emotion based on changes in the user's facial expressions detected from the user's facial image; The second inference unit infers the user's emotions based on changes in the user's temperature detected from the user in a non-contact manner; The determination unit determines the user's emotion based on the inference results of the first inference unit and the inference results of the second inference unit; as well as The modification unit detects when the variance of the user's inferred emotion (i.e., the score) during the period up to a specified time is less than a specified threshold, and modifies the inference results of the first inference unit and the second inference unit so that, tracing back from the first time when the variance value is less than the specified threshold, the inference results up to the immediately following time when the variance value is less than the specified threshold are replaced with the inference results of the first time. The determination unit determines the user's emotion based on the inference result after the change unit's modification.

2. The emotion determination device according to claim 1, characterized in that, The first inference unit and the second inference unit each calculate a score indicating the probability that the user is experiencing a particular emotion for each of the multiple emotions. The determination unit determines the user's emotions based on the scores of each emotion calculated by the first inference unit and the scores of each emotion calculated by the second inference unit.

3. The emotion determination device according to claim 2, characterized in that, If the emotion for which the maximum score is calculated by the first inference unit is consistent with the emotion for which the maximum score is calculated by the second inference unit, the determination unit determines that the emotion is the user's emotion. If the emotion with the highest score calculated by the first inference unit is inconsistent with the emotion with the highest score calculated by the second inference unit, the determination unit determines the user's emotion based on the scores of each emotion calculated by the first inference unit and the scores of each emotion calculated by the second inference unit.

4. The emotion determination device according to claim 3, characterized in that, If the emotion with the highest score calculated by the first inference unit is inconsistent with the emotion with the highest score calculated by the second inference unit, the determination unit determines the emotion with the higher calculated score among these emotions as the user's emotion.

5. The emotion determination device according to claim 3, characterized in that, If the emotion with the highest score calculated by the first inference unit is inconsistent with the emotion with the highest score calculated by the second inference unit, the determination unit determines the emotion with the larger sum of the scores calculated by the first inference unit and the scores calculated by the second inference unit as the user's emotion.

6. The emotion determination device according to claim 3, characterized in that, If the emotion for which the maximum score is calculated by the first inference unit is inconsistent with the emotion for which the maximum score is calculated by the second inference unit, the determination unit determines the emotion for which the score has changed more significantly from the previous one as the user's emotion.

7. The emotion determination device according to claim 3, characterized in that, If the emotion with the highest score calculated by the first inference unit is inconsistent with the emotion with the highest score calculated by the second inference unit, the determination unit determines the emotion with the larger variance value among the scores of these emotions as the user's emotion.

8. An emotion determination method, characterized in that, include: The first inference step infers the user's emotion based on changes in the user's facial expressions detected from the user's facial image; The second inference step infers the user's emotions based on changes in the user's temperature detected from the user in a non-contact manner; The determination step determines the user's emotion based on the inference results of the first inference step and the inference results of the second inference step; as well as The modification step involves detecting when the variance of the user's inferred emotion (i.e., the score) is less than a predetermined threshold during a specified period. Then, the inference results of the first and second inference steps are modified such that, tracing back from the first time point when the variance is less than the predetermined threshold, the inference results up to the immediately following time point when the variance is less than the predetermined threshold are replaced with the inference results from the first time point. In the determination step, the user's emotion is determined based on the inference result after the change step.

9. A program product for causing a computer to perform the steps of the emotion determination method of claim 8.

Images