Emotion determination system

Abstract

To provide a system capable of suppressing erroneous determination of an occupant's emotion.SOLUTION: An emotion determination system comprises a camera for imaging an expression of an occupant of a vehicle, a microphone for collecting voice of the occupant, and a controlling unit connected to the camera and the microphone. The controlling unit determines which of a positive emotion and a negative emotion the occupant's expression emotion is, on the basis of the occupant's expression imaged by the camera; determines which of a positive emotion and a negative emotion the occupant's voice emotion is, on the basis of the occupant's voice collected by the microphone; determines that occupant's emotion is negative, when at least one of the occupant's expression emotion and the occupant's voice emotion is negative; and determines that occupant's emotion is positive, when both of the occupant's expression emotion and the occupant's voice emotion are positive.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an emotion determination system. [Background technology]

[0002] Patent Document 1 discloses a device for estimating the emotions of an occupant. This device uses a camera that captures the field where the occupant is present, a microphone that records the sound in the field, and determines the behavior and emotions of the occupant based on the images captured by the camera and the sounds recorded by the microphone. Compared to a method of determining emotions from data sensed by a single sensor, this device can determine emotions two-dimensionally using images and sounds as evaluation axes, improving the accuracy of emotion determination. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-185117 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the device described in Patent Document 1, it is unclear how to evaluate the image and the sound to improve the accuracy of emotion determination. In order to improve the emotion determination accuracy, it is necessary to suppress erroneous determination. The present disclosure provides a system that can suppress erroneous determination of the emotion of an occupant. [Means for solving the problem]

[0005] An apparatus according to an embodiment of the present disclosure includes a camera, a microphone, and a control unit. The camera captures an image of a facial expression of a vehicle occupant. The microphone collects the voice of the occupant. The control unit is connected to the camera and the microphone. The control unit determines whether the facial expression emotion of the occupant is a positive emotion or a negative emotion based on the facial expression of the occupant captured by the camera. The control unit determines whether the vocal emotion of the occupant is a positive emotion or a negative emotion based on the voice of the occupant collected by the microphone. When at least one of the facial expression emotion of the occupant and the vocal emotion of the occupant is a negative emotion, the control unit determines that the emotion of the occupant is a negative emotion. When both the facial expression emotion of the occupant and the vocal emotion of the occupant are positive emotions, the control unit determines that the emotion of the occupant is a positive emotion. Effect of the Invention

[0006] According to the present disclosure, erroneous determination of the emotions of an occupant can be suppressed. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a vehicle equipped with an emotion determination system according to an embodiment. [Diagram 2] FIG. 2 is a flowchart showing an example of the operation of the emotion determination system according to an embodiment. [Diagram 3] FIG. 3 is a flowchart showing an example of the operation of the emotion determination system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated description will be omitted.

[0009] [Vehicle configuration] Fig. 1 is a block diagram showing an example of a configuration of a vehicle equipped with an emotion determination system according to an embodiment. As shown in Fig. 1, the emotion determination system 1 is mounted on a vehicle 2 as an example. The vehicle 2 is an autonomous vehicle as an example.

[0010] The vehicle 2 includes a camera 3, a microphone 4, a control unit 5, a content providing device 6, a user interface 7, and a storage device 8. The emotion determination system 1 includes the camera 3, the microphone 4, and the control unit 5.

[0011] The camera 3 is a device that captures images of facial expressions of occupants of the vehicle 2. The camera 3 outputs image data as an imaging result. The microphone 4 is a device that collects the voices of the occupants. The microphone 4 outputs audio data as an audio collection result. The camera 3 and the microphone 4 are installed, for example, inside the vehicle cabin.

[0012] The control unit 5 is, for example, an ECU (Electronic Control Unit). The ECU is an electronic control unit having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CAN (Controller Area Network) communication circuit, etc. The control unit 5 is connected to the camera 3 and the microphone 4.

[0013] The control unit 5 determines whether the facial expression of the occupant is a positive emotion or a negative emotion based on the facial expression of the occupant captured by the camera 3. For example, the control unit 5 inputs image data output from the camera 3 and recognizes the facial expression of the occupant by image analysis. The control unit 5 determines that the facial expression of the occupant is positive when the facial expression of the occupant is bright, and determines that the facial expression of the occupant is negative when the facial expression of the occupant is dark. Alternatively, the control unit 5 may determine the facial expression of the occupant using a model that has learned the relationship between image data and facial expressions.

[0014] The control unit 5 determines whether the occupant's voice emotion is a positive emotion or a negative emotion based on the occupant's voice collected by the microphone 4. For example, the control unit 5 inputs voice data output from the microphone 4 and recognizes the occupant's voice by voice analysis. If the occupant's voice is cheerful, the control unit 5 determines that the emotion is positive, and if the occupant's voice is gloomy, the control unit 5 determines that the emotion is negative. Alternatively, the control unit 5 may determine the occupant's emotion using a model that has learned the relationship between voice data and voice emotion.

[0015] The control unit 5 determines that the occupant's emotion is a negative emotion when at least one of the occupant's facial expression emotion and the occupant's vocal emotion is a negative emotion. In other words, the control unit 5 determines that the occupant's emotion is a negative emotion when the facial expression emotion is a negative emotion, when the vocal emotion is a negative emotion, or when both the facial expression emotion and the vocal emotion are negative emotions.

[0016] When the emotion of an expressionless occupant is determined only from the results of the camera 3, it may be determined that the emotion is positive, at least not negative. However, there are occupants whose facial expressions do not show negative emotions. Alternatively, there are occupants whose facial expressions are always positive, regardless of their own emotions. The same applies to voice. The control unit 5 combines the determination of facial expressions and voice emotions, and determines that the emotion of the occupant is negative if at least one of them is negative, thereby preventing an occupant whose facial expression or voice does not show negative emotions from being erroneously determined to be positive. If both the facial expression and voice emotions of the occupant are positive, the control unit 5 determines that the emotion of the occupant is positive.

[0017] The content providing device 6 provides content to the occupants of the vehicle 2. The content providing device 6 is connected to a user interface 7 and provides the content via the user interface 7. The content is a service provided to the occupants, and examples of the content include providing video or music, enabling a massage function installed in the occupant's seat, enabling an automatic driving function, and stopping at a service area during automatic driving. The content providing device 6 may have a function of suggesting content in an interactive format via the user interface 7 before providing the content.

[0018] The control unit 5 may determine the emotion of the occupant in cooperation with the content providing device 6. For example, the content providing device 6 may propose content in an interactive format, and the control unit 5 may determine the emotion of the occupant based on the response to the proposal.

[0019] Alternatively, the control unit 5 may determine the facial expression of the occupant for a predetermined period of time after the content providing device 6 starts providing the content. The control unit 5 may then determine the facial expression of the occupant related to the provided content according to the ratio of the period during which the facial expression of the occupant is determined to be a positive emotion to the period during which the facial expression of the occupant is determined to be a negative emotion during the predetermined period. The control unit 5 may similarly determine the vocal emotion. With regard to voice, the control unit 5 may determine the vocal emotion of the occupant from the dialogue after the content is provided. This allows the control unit 5 to appropriately determine the facial expression or vocal emotion corresponding to the provided content.

[0020] The control unit 5 may store the ratio of facial expressions and vocal emotions of the occupant for the provided content in the storage device 8. This allows the storage device 8 to store the relationship between the multiple provided contents and the ratio of facial expressions and vocal emotions of the occupant for each content.

[0021] The content providing device 6 may determine the order of suggesting the contents based on the information stored in the storage device 8. The content providing device 6 may determine the order of suggesting the contents based on the history of the passenger's emotions at the time of suggestion, the history of the passenger's emotions after the suggestion, and the like. The history of the passenger's emotions after the suggestion may be determined based on a dialogue, or may be determined from the passenger's reaction for a predetermined period of time as shown below. For example, the content providing device 6 may select a plurality of candidate contents from a plurality of contents, acquire a ratio of facial expression emotions and a ratio of voice emotions corresponding to each of the plurality of candidate contents by referring to the storage device 8, and determine the order of suggesting the plurality of candidate contents based on the acquired ratio of facial expression emotions and ratio of voice emotions. For example, when there are three candidate contents to be provided after detecting drowsiness of the passenger, namely, "provision of video or music," "activation of massage function," and "stopping at a service area," the content providing device 6 calculates the overall ratio of the passenger's positive emotions based on the acquired ratio of facial expression emotions and ratio of voice emotions based on the history, and determines the order of suggesting the contents in descending order of the calculated overall ratio. In this way, the contents are proposed in an order according to the passenger's preferences. The content providing device 6 may also determine whether or not to propose content using a similar method and threshold value. In this way, whether or not to provide content is determined in accordance with the preferences of the passenger.

[0022] The emotion determination system 1 may transmit the information stored in the storage device 8 to an external server. This allows the server to aggregate the contents and scenes that caused positive or negative emotions. The aggregated information can be used to plan measures against negative emotions. In other words, minor passenger dissatisfaction can be fed back early and reflected in improvements to the vehicle. In addition, the passenger dissatisfaction and satisfaction can be fed back to determine the order of development of vehicle functions.

[0023] Fig. 2 is a flowchart showing an example of the operation of the emotion determination system according to an embodiment. The flowchart shown in Fig. 2 starts when the emotion determination system 1 receives a start instruction operation.

[0024] As shown in FIG. 2, first, in step S10, the control unit 5 of the emotion determination system 1 determines whether the facial emotion of the occupant is a positive emotion or a negative emotion based on the facial expression of the occupant captured by the camera 3.

[0025] At step S12, the control unit 5 determines whether the voice emotion of the occupant is a positive emotion or a negative emotion based on the voice of the occupant collected by the microphone 4.

[0026] At Step S14, the control unit 5 determines whether or not at least one of the facial expression of the occupant determined at Step S10 and the vocal emotion of the occupant determined at Step S12 is a negative emotion.

[0027] If at least one of the occupant's facial expression and voice is a negative emotion (step S14: YES), the control unit 5 determines, in step S16, that the occupant's emotion is a negative emotion. If at least one of the occupant's facial expression and voice is not a negative emotion (step S14: NO), that is, if both the occupant's facial expression and voice are positive emotions, the control unit 5 determines, in step S18, that the occupant's emotion is a positive emotion.

[0028] When steps S16 and S18 are completed, the flow chart shown in Fig. 2 ends. By executing the flow chart shown in Fig. 2, erroneous determination of the emotion of the occupant can be suppressed.

[0029] Next, an example of an operation related to the linkage between the emotion determination system 1 and the content providing device 6 will be described. Fig. 3 is a flowchart showing an example of an operation of the emotion determination system according to an embodiment. The flowchart shown in Fig. 3 starts when the emotion determination system 1 and the content providing device 6 receive a start instruction operation.

[0030] First, the content providing device 6 proposes content in step S20. For example, the content providing device 6 proposes "enable massage function" via the user interface 7 in an interactive format.

[0031] Next, in step S22, the camera 3 and the microphone 4 acquire the reaction of the occupant as image data and voice data.

[0032] Next, the control unit 5 determines the emotion in step S24. The process of step S24 is the process of the flowchart shown in Fig. 2 with step S22 as input. When step S24 ends, the flowchart shown in Fig. 3 ends. By executing the flowchart shown in Fig. 3, the proposed content and the emotion of the occupant can be associated and stored.

[0033] [Summary of the embodiment] According to the emotion determination system 1, it is determined whether the facial expression of the occupant is a positive emotion or a negative emotion based on the facial expression of the occupant captured by the camera 3, and it is determined whether the voice of the occupant is a positive emotion or a negative emotion based on the voice of the occupant collected by the microphone 4. If at least one of the facial expression and the voice of the occupant is a negative emotion, the emotion of the occupant is determined to be a negative emotion. If both the facial expression and the voice of the occupant are positive emotions, the emotion of the occupant is determined to be a positive emotion. This makes it possible to avoid erroneously determining that an occupant whose facial expression or voice does not show a negative emotion is a positive emotion. Therefore, the emotion determination system 1 can suppress erroneous determination of the emotion of the occupant.

[0034] Although exemplary embodiments have been described above, various omissions, substitutions, and modifications may be made without being limited to the above-described exemplary embodiments. [Explanation of symbols]

[0035] 1...emotion determination system, 2...vehicle, 3...camera, 4...microphone, 5...control unit.

Claims

[Claim 1] A camera that captures the facial expressions of vehicle occupants; A microphone for collecting the voice of the occupant; A control unit connected to the camera and the microphone; Equipped with The control unit is determining whether the facial expression of the occupant is a positive emotion or a negative emotion based on the facial expression of the occupant captured by the camera; determining whether a voice emotion of the occupant is the positive emotion or the negative emotion based on the voice of the occupant collected by the microphone; When at least one of the facial expression emotion of the occupant and the voice emotion of the occupant is the negative emotion, the emotion of the occupant is determined to be a negative emotion; When both the facial expression emotion of the occupant and the voice emotion of the occupant are the positive emotion, the emotion of the occupant is determined to be the positive emotion. Emotion detection system.

Images