Emotion detection device

Abstract

To provide an emotion determination device which can appropriately determine an emotion of an occupant on a vehicle.SOLUTION: An emotion determination device comprises: traveling state detection units 310, 320, 330 that detect a traveling state of a vehicle; an occupant monitoring unit that monitors a state of an occupant on the vehicle; and an emotion determination unit 400 that determines an emotion of the occupant on the basis of outputs of the traveling state detection units and the occupant monitoring unit. If predetermined conditions for determining a second emotion state is satisfied during a first caution period in which output histories of the traveling state detection units and the occupant monitoring unit are monitored for a first period in order to determine a first feeling state, the emotion determination device starts a second caution period for monitoring the output histories of the traveling state detection units and the occupant monitoring unit for a second period in order to determine the second emotion state.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an emotion discrimination device that estimates the emotions of passengers in vehicles such as automobiles.

Background Art

[0002] For example, in automobiles such as passenger cars, it has been proposed to estimate the emotions of passengers such as drivers and reflect the emotional state in the control of functions related to the vehicle. As a technology related to estimating the emotions of vehicle passengers, for example, in Patent Document 1, in an information providing device that provides information to vehicle passengers, an emotion estimation unit estimates the emotions of a user, sets a base emotion that serves as a basis for virtual emotions, repeatedly estimates the emotions of a target user until a predetermined period elapses, and after the predetermined period has elapsed, sets the base emotion in consideration of the estimated emotions. Patent Document 2 describes acquiring the biometric information of a driver and calculating an arousal level value indicating the degree of arousal state and a time change rate of the arousal level value. Further, it describes comparing the arousal level value with a plurality of threshold values to determine whether the driver is in a distracted state or an excited state. It also describes holding the arousal level value when the driver's arousal state is an appropriate state as a reference value. Patent Document 3 describes acquiring biometric information and calculating a feature amount necessary for estimating, for example, the θ-wave content rate, which is the state of the driver's brain waves, and estimating the driver's concentration. Further, in the tendency of the estimated driver's concentration to decrease over time, it is determined whether or not the driver's concentration falls below a threshold value that is the lower limit of the concentration required for driving after an arbitrary time, and this arbitrary time is set as the maximum time during which the driver can drive the vehicle safely. Furthermore, when the driver's concentration falls below the threshold value, it describes performing driving support in which the vehicle performs constant-speed driving and following driving regardless of the driver's driving operation.

Prior Art Documents

Patent Documents

[0003] [Patent Document 1] Japanese Patent Publication No. 2019-139354 [Patent Document 2] International Publication No. WO2021 / 014637 [Patent Document 3] International Publication No. WO2014 / 147828 [Overview of the project] [Problems that the invention aims to solve]

[0004] In recent years, devices that issue warnings or provide driving assistance to prevent dangerous driving situations have become widespread in vehicles such as automobiles. However, depending on the emotional state of the driver or other occupants, there are concerns that the intervention of warnings and driver assistance controls may be perceived as bothersome, further worsening the occupants' emotions and hindering safe driving. Therefore, there is a need to appropriately estimate the emotions of the occupants and reflect this in the vehicle's control system.

[0005] However, there is an upper limit to how long a person's concentration can last, and some sources suggest it's around 15 minutes. In driving a vehicle, which requires complex abilities such as cognition, judgment, and operation, it is thought that some kind of change in the occupants' emotions may occur after driving for, for example, about 10 minutes (about 5 km on a typical Japanese road). In particular, when occupants are exposed to high levels of stress, there is a need to appropriately detect the occupants' emotions, which change rapidly in a short period of time, and to provide optimal driving support control to suppress distracted driving and dangerous driving that could lead to accidents. In view of the above-mentioned problems, the object of the present invention is to provide an emotion discrimination device capable of appropriately discriminating the emotions of vehicle occupants. [Means for solving the problem]

[0006] To solve the above-mentioned problems, the present invention provides an emotion discrimination device comprising: a driving state detection unit for detecting the driving state of a vehicle; an occupant monitoring unit for monitoring the state of the occupants of the vehicle; and an emotion discrimination unit for discriminating the emotions of the occupants based on the outputs of the driving state detection unit and the occupant monitoring unit, wherein during a first alert period in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a first period in order to discriminate a first emotional state, if predetermined conditions for discriminating a second emotional state are met, the driving state detection unit and the occupant monitoring unit are monitored over a second period in order to discriminate a second emotional state. This is characterized by initiating a second warning period in which the output history of the tarring section is monitored. According to this, if an event occurs during the first alert period for identifying the first emotional state that suggests a second emotional state different from the first emotional state, a second alert period for identifying the second emotional state can be initiated. This allows for the appropriate identification of multiple emotional states of crew members, which often occur sequentially over time. Generally, when occupants become anxious or confused, they exhibit characteristics such as a change in their seating posture (typically leaning forward) or gripping the steering wheel more tightly, reducing the gap between their fingers and the steering wheel rim. According to the present invention, the occupant's seating posture and the positional relationship between the steering wheel and the occupant's fingers can be appropriately determined to indicate the occupant's state of anxiety or confusion.

[0007] In the present invention, the first emotional state is a state of distraction of the occupant, and the emotional determination unit can be configured to determine the state of distraction based on at least one of the state of fixation of the occupant's gaze detected by the occupant monitoring unit during the first alert period and the variation in the vehicle's driving state detected by the driving state detection unit. According to this, by utilizing images captured by a typical occupant monitoring device and the output of sensors commonly used for vehicle control, it is possible to appropriately identify a state of distraction with a simple device configuration.

[0009] In the present invention, if predetermined conditions for determining a third emotional state are met during the second alert period, a third alert period can be initiated in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a third period in order to determine the third emotional state. According to this, when a second alert period is initiated to identify the second emotional state, an alert period is also initiated to identify the third emotional state, which represents a further deterioration of the emotional state, thereby enabling the appropriate identification of the third emotional state.

[0010] In the present invention, the third emotional state is the occupant's state of impatience and anger, the driving state detection unit has a steering angle detection unit for detecting the steering angle of the vehicle's steering device and an acceleration / deceleration detection unit for detecting the acceleration / deceleration state of the vehicle, and the emotional determination unit can be configured to determine the impatience and anger state when the steering angle of the steering device increases and the acceleration / deceleration state occurs simultaneously during the third alert period. According to this, by utilizing sensors that are normally installed in typical vehicles, it is possible to appropriately determine a state of agitation and anger with a simple device configuration. To solve the above-mentioned problems, the present invention provides an emotion discrimination device comprising: a driving state detection unit for detecting the driving state of a vehicle; an occupant monitoring unit for monitoring the state of the occupants of the vehicle; and an emotion discrimination unit for discriminating the emotions of the occupants based on the outputs of the driving state detection unit and the occupant monitoring unit, wherein during a first alert period in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a first period in order to discriminate a first emotional state, if predetermined conditions for discriminating a second emotional state are met, the device monitors the output history of the driving state detection unit and the occupant monitoring unit over a second period in order to discriminate a second emotional state. The system includes commencing a second alert period, and if predetermined conditions for determining a third emotional state are met during the second alert period, commencing a third alert period for which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a third period in order to determine the third emotional state, wherein the third emotional state is a state of impatience and anger of the occupant, the driving state detection unit has a steering angle detection unit for detecting the steering angle of the vehicle's steering device and an acceleration / deceleration detection unit for detecting the acceleration / deceleration state of the vehicle, and the emotional determination unit determines the state of impatience and anger when an increase in the steering angle of the steering device and the acceleration / deceleration state occur simultaneously during the third alert period. According to this, if an event occurs during the first alert period for identifying the first emotional state that suggests a second emotional state different from the first emotional state, a second alert period for identifying the second emotional state can be initiated. This allows for the appropriate identification of multiple emotional states of crew members, which often occur sequentially over time. According to this, when a second alert period is initiated to identify the second emotional state, an alert period is also initiated to identify the third emotional state, which represents a further deterioration of the emotional state, thereby enabling the appropriate identification of the third emotional state. According to this, by utilizing sensors that are normally installed in typical vehicles, it is possible to appropriately determine a state of agitation and anger with a simple device configuration. [Effects of the Invention]

[0011] As described above, the present invention provides an emotion discrimination device capable of appropriately discriminating the emotions of vehicle occupants. [Brief explanation of the drawing]

[0012] [Figure 1] This figure schematically shows the system configuration of a vehicle having a driving assistance device that includes an embodiment of an emotion discrimination device to which the present invention is applied. [Figure 2] This diagram schematically shows the arrangement of the driver monitoring camera and seat pressure sensor in the emotion discrimination device of the embodiment. [Figure 3] It is a flowchart showing the basic configuration of passenger emotion discrimination in the emotion discrimination device of the embodiment. [Figure 4] It is a diagram showing an example of the state of the fingers of a passenger holding the steering wheel. [Figure 5] It is a flowchart showing the content of basic control in the driving support device of the embodiment. [Figure 6] It is a diagram showing the change in the content of driving support control according to the emotional state of the passenger in the driving support device of the embodiment.

Mode for Carrying Out the Invention

[0013] Hereinafter, an embodiment of an emotion discrimination device to which the present invention is applied will be described. The emotion discrimination device of the embodiment is provided, for example, in a four-wheeled automobile such as a passenger car. FIG. 1 is a diagram schematically showing the system configuration of a vehicle having a driving support device including the emotion discrimination device of the embodiment.

[0014] The vehicle 1 includes an environment recognition unit 100, a driving support control unit 200, an electric power steering (EPS) control unit 310, a power unit control unit 320, a brake control unit 330, a passenger state discrimination unit 400, and the like. Each of these units can be configured as a microcomputer having an information processing unit such as a CPU, a storage unit such as a RAM and a ROM, an input / output interface, and a bus connecting these. Also, each unit can communicate with each other via an in-vehicle LAN such as a CAN communication system, or directly.

[0015] The environment recognition unit 100 recognizes the environment around the host vehicle based on the output of various sensors and the like. The environment surrounding the vehicle that is to be recognized includes, for example, the lane shape of the road on which vehicle 1 is traveling, the relative position and relative speed of various obstacles with respect to the vehicle, and information regarding the route that vehicle 1 will travel in the future, as well as information regarding accidents, congestion, and regulations. The environmental recognition sensor 100 is connected to a visible light camera 110, a millimeter-wave radar device 120, a laser scanner device 130, a high-precision map database 140, a positioning device 150, and the like.

[0016] The visible light camera device 110 is an imaging device that captures images of the area around the vehicle (front, rear, sides, etc.) using visible light cameras such as stereo cameras and single-lens cameras. The visible light camera device 110 has a function to perform image processing on the captured image and detect the presence or absence of objects around the vehicle, the relative position and relative speed of the objects with respect to the vehicle, the lane shape, and so on. The millimeter-wave radar system 120 is a radar system that uses radio waves in a frequency band of, for example, 30 to 300 GHz, and has the function of detecting the presence or absence of an object and the relative position of the object with respect to the vehicle. The laser scanner device 130 has a function to scan the area around the vehicle by, for example, irradiating it with pulsed near-infrared laser light, and to detect the presence or absence of an object, the relative position of the object to the vehicle, the shape of the object, etc., based on the presence or absence of reflected light and the time difference until the reflected light returns.

[0017] The high-precision map database 140 stores data related to high-precision 3D map data (HD map) within the range where vehicle 1 is expected to travel. This data, for example, includes 3D data of lane markings, shoulder edges, and lane dividers (so-called white lines), with a resolution of, for example, centimeters, and contains information on latitude, longitude, and altitude. The positioning device 150 includes, for example, a receiver for a quasi-zenith satellite system such as GPS, a vehicle-to-infrastructure communication device, a gyro sensor for autonomous navigation, and detects the current position of vehicle 1. The high-precision map database 140 and positioning device 150 work in cooperation with the environment recognition unit 100 and the driver assistance control unit 200 to function as a navigation system that provides route guidance to a predetermined destination and displays information such as lane changes and right / left turns. Such navigation system functionality is also included in the driver assistance control of the present invention.

[0018] The driver assistance control unit 200 performs driver assistance control based on the environment around the vehicle recognized by the environment recognition unit 100, the driving state of the vehicle 1 recognized based on the output of each unit and sensor described later, and the emotional state of the occupant P. Driving assistance control includes, for example, information presentation control that provides information to the occupant P through images, sounds, vibrations, etc., and vehicle driving assistance control such as braking force control, power output control of the driving power source, and steering control.

[0019] The driver assistance control unit 200 is connected to an input / output device 210 and a communication device 220. The input / output device 210 includes, for example, an image display device that also serves as an input device such as a touch panel display, output devices such as a sound speaker, a vibration generator installed in a part that comes into contact with the occupant such as a seat, and input devices such as a physical switch and a voice microphone. The occupant P can use the input / output device 210 to make various settings related to driver assistance control. The communication device 220 communicates with a base station located outside the vehicle, for example, using a wireless communication line, and transmits and receives various types of data.

[0020] The electric power steering control unit 310 controls the steering device (not shown) that steers the steering wheels (typically the front wheels) of the vehicle 1, providing assist force in response to the steering operation of the occupant P, and steering force during automatic steering. The electric power steering control unit 310 is connected to a steering angle sensor 311, a torque sensor 312, a motor 313, and the like.

[0021] The steering angle sensor 311 is a sensor (steering angle detection unit) that detects the steering angle in the steering system. The torque sensor 312 is a sensor that detects the torque applied to the steering shaft to which a steering wheel (not shown) is connected, which is operated by the occupant P. The electric power steering control unit 310 generates assist force according to the torque detected by the torque sensor 312. Motor 313 is an electric actuator that provides assist force and steering force to the steering system and generates rack thrust. The output of the motor 313 is controlled by the electric power steering control unit 310.

[0022] The power unit control unit 320 comprehensively controls the vehicle 1's power source and its auxiliary equipment. As a power source for driving, for example, an internal combustion engine (ICE), an electric motor, or an engine-electric motor hybrid system can be used. The power unit control unit 320 sets the required torque based on, for example, the amount of accelerator pedal operation (not shown), and controls the driving power source so that the actual torque generated by the driving power source matches the required torque.

[0023] The brake control unit 330 controls the braking force of the brake devices provided on each of the front, rear, left, and right wheels of the vehicle individually (for each wheel). The braking system can be configured, for example, to include a hydraulic disc brake. The brake control unit 330 is connected to the hydraulic control unit 331, vehicle speed sensor 332, acceleration sensor 333, yaw rate sensor 334, and the like.

[0024] The hydraulic control unit 331 is a hydraulic pressure control device that individually adjusts the brake fluid pressure of the wheel cylinders (not shown) of each wheel. The hydraulic control unit 331 includes an electric pump for pressurizing the brake fluid, and pressure boosting valves, pressure reducing valves, and pressure holding valves for controlling the brake fluid pressure in each wheel cylinder.

[0025] The hydraulic control unit 331 is connected to a master cylinder, wheel cylinders, etc. (not shown) via brake fluid piping. The master cylinder pressurizes the brake fluid in response to the driver's operation of the brake pedal (not shown). The brake fluid pressure generated by the master cylinder is transmitted to the wheel cylinder via the hydraulic control unit 331. The hydraulic control unit 331 has the function of overriding the brake fluid pressure generated by the master cylinder to increase or decrease the brake fluid pressure of each wheel cylinder. A wheel cylinder is installed on each wheel and, for example, presses the brake pad against the disc rotor, generating a frictional force (braking force) corresponding to the brake fluid pressure.

[0026] The vehicle speed sensor 332 is installed in the hub portion that rotatably supports each wheel, and is a sensor that generates a vehicle speed signal corresponding to the rotational speed of each wheel. The acceleration sensor 333 is a sensor (acceleration / deceleration detection unit) that detects acceleration acting on the vehicle body in the longitudinal direction and the lateral direction (vehicle width direction). The yaw rate sensor 334 is a sensor that detects the yaw rate, which is the rotational (spinning) angular velocity of the vehicle body around its vertical axis. The electric power steering control unit 310, power unit control unit 320, and brake control unit 330 described above function as a driving state detection unit that detects the driving state of vehicle 1 in cooperation with the environment recognition unit 100.

[0027] The occupant status determination unit 400 determines the conscious state, emotional state, health state, etc., of occupant P (typically the driver). The occupant status group unit 400 is connected to a driver monitoring camera 401, a seat pressure sensor 402, and the like. The driver monitoring camera 401 and seat pressure sensor 402 work in cooperation with the occupant state determination unit 400 to function as an occupant monitoring unit.

[0028] Figure 2 is a schematic diagram showing the arrangement of the driver monitoring camera 401 and the seat pressure sensor 402 in the driving assistance device of the embodiment. Vehicle 1 has a seat S in which occupant P is seated. The seat S includes a seat cushion S1, a backrest S2, a headrest S3, and the like. The seat cushion S1 is the seat surface on which the occupant P's thighs, buttocks, etc., rest. The backrest S2 is the part that contacts the occupant's back and supports the occupant's upper body. The backrest S2 protrudes upward and diagonally backward from near the rear of the seat cushion S1. The headrest S3 is positioned at the rear of the occupant's head and is the part that holds the head in place when the head is reclined. The headrest S3 is positioned to protrude upward from the upper edge of the backrest S2.

[0029] The driver monitoring camera 401 is an imaging device that captures images of the occupant P from the front of the vehicle. The driver monitoring camera 401 is positioned to include the occupant P's face and their hands, while they are gripping the steering wheel SW, within its imaging field of view. The driver monitoring camera 401 includes, for example, a solid-state image sensor such as a CMOS or CCD, an optical system such as a lens group that forms an image of a subject on the solid-state image sensor, a drive circuit for the solid-state image sensor, and an output processing circuit.

[0030] The seat pressure sensor 402 is installed on the seat cushion S1 and measures the distribution of surface pressure received by the upper surface of the seat cushion S1 from the occupant P. Furthermore, the occupant status determination unit 400 may also be configured to include sensors other than those mentioned above. For example, sensors may be provided to acquire various biological information (so-called vital signs) such as the crew's heart rate, body temperature, blood pressure, and blood oxygen saturation.

[0031] The occupant state determination unit 400 has the function of determining the emotions of occupant P based on information about the environment around the vehicle recognized by the environment recognition unit 100, the driving state of the vehicle 1 including information detected by various sensors, and the occupant state detected by the driver monitoring camera 401, seat pressure sensor 402, etc. This point will be explained in detail below. Specifically, the occupant state determination unit 400 has the function of determining unsuitable driving emotions, such as distracted state, anxious / confused state, and agitated / angry state. Figure 3 is a flowchart showing the basic configuration for determining the emotions of the occupants in the emotion determination device of the embodiment. The following explains each step in order.

[0032] <Step S101: Start monitoring driving and occupant conditions> The occupant status determination unit 400 starts monitoring the driving status and occupant status as described above after a predetermined initialization process. The initialization process is a process that initializes various parameters used for determining the emotions of the occupants, and is usually performed at the start of the driving cycle of vehicle 1 (a series of periods from the main power of vehicle 1 being turned on, the start of driving, arrival at the destination, and the main power being turned off). For example, the occupant status determination unit 400 can determine the start of a driving cycle when the main power of the vehicle 1 is turned on (ignition on), occupant P is wearing a seatbelt, and the vehicle is ready to drive (for example, by shifting the transmission to the driving range).

[0033] For example, the occupant state determination unit 400 sets the initialization values ​​of parameters used for emotion determination, such as the distribution of seat pressure detected by the seat pressure sensor 402 and the distance g between the steering wheel SW and the occupant's fingers (see Figure 4), which is obtained by image processing from images captured by the driver monitoring camera 401, based on the average value over a predetermined period (for example, 5 minutes) when driving at a low speed of 10 km / h or less or when stopped. Furthermore, parameters used for emotion discrimination include, for example, the amount of eye movement and the amount of head (face) orientation movement during normal times. These parameters are also initialized (initialized values ​​are obtained) in a similar manner. The initialization values ​​set here serve as baseline values ​​for parameters indicating the occupant's physical condition. The emotion discrimination described below is basically performed by comparing the difference between the initialization value (baseline value) of each parameter and the threshold value with a predetermined threshold.

[0034] If a predetermined amount of data cannot be collected within a predetermined period, or if measurement cannot be performed when the vehicle is in a horizontal plane, the system will correct or reacquire initialization values ​​to improve accuracy, provided that the predetermined conditions are met. The predetermined conditions include, for example, that the traffic light in front of the vehicle is stopped (red light on), that the vehicle speed is 0 km / h (including extremely low speeds where vehicle speed detection is impossible), that the steering angle detected by the steering angle sensor 311 is approximately 0, and that the acceleration in the longitudinal and lateral directions detected by the acceleration sensor 333 is 0 (meaning the vehicle is stationary on a horizontal plane). Note that this type of parameter initialization may be performed multiple times during a single driving cycle. After the monitoring of the driving condition and occupant condition begins, proceed to step S02.

[0035] <Step S102: Beware of Distracted Attention> The crew state determination unit 400 initiates distraction alert, focusing on monitoring parameters used to determine crew member P's distracted state (first emotional state). Focused monitoring includes increasing the amount of data, for example, by improving the temporal resolution of data acquisition compared to normal (non-alert) conditions. Distraction can be identified, for example, based on factors such as the fixation of the occupant P's field of vision and fluctuations (variations) in the distance to the vehicle ahead. Field of view fixation can be determined, for example, based on the driver's gaze direction detected from images captured by the driver monitoring camera 401.

[0036] For example, if a state in which the direction of gaze does not change by more than a predetermined amount (typically, a state in which the amount of displacement in the direction of gaze is less than or equal to a small threshold) continues for a predetermined period of time or longer, it can be determined that the person is in a state of distraction. Fluctuations in the distance between your vehicle and the vehicle ahead can be determined, for example, based on the change in the relative distance of the vehicle ahead to your vehicle, as detected by the environmental recognition unit 100. For example, if the following distance fluctuates by more than a predetermined value within a predetermined period, it can be determined that the driver is in a state of distraction. Then proceed to step S103.

[0037] <Step S103: Determining the state of distractedness> The crew state determination unit 400 detects whether or not crew member P is in a state of distraction within a predetermined distraction alert period (for example, within 10 minutes) using at least one of the determination methods described above. After determining whether or not the patient is in a state of distraction, the process proceeds to step S104.

[0038] <Step S104: Determining the conditions for warning about anxiety and confusion> The crew state determination unit 400 determines whether predetermined conditions for determining crew member P's state of anxiety and confusion (second emotional state) have been met while the crew member is on alert for distracted state. Conditions for identifying a state of anxiety and confusion (events that suggest a high probability of an anxious and confused state) can be used to determine this, for example, based on the frequency of up-and-down movement (shaking) of crew member P's head (face direction) and fluctuations in the direction of their gaze.

[0039] If the head is moving up and down frequently, it is thought that occupant P is frequently looking at things like the navigation screen or mirrors, and it is presumed that they are not fully aware of their surroundings or are feeling anxious about their surroundings. For example, if head shaking (e.g., reversal of direction of movement) occurs more than a predetermined number of times (e.g., 3 times) within a predetermined time (e.g., 3 seconds), a state of anxiety and confusion should be identified. Similarly, if there are frequent changes in the direction of the crew's gaze, it can be inferred that the crew is either not fully aware of their surroundings or is feeling anxious about their surroundings. For example, if the blurring of the central visual field (the range scanned by the central visual field) is more than three times wider than when waiting at a traffic light, a state of anxiety and confusion should be identified.

[0040] Furthermore, the conditions for identifying a state of anxiety and confusion can be determined based on the driving conditions. For example, if the driver assistance control unit 200 determines that the vehicle is swerving based on the change in the vehicle's lateral position within the lane as recognized by the environmental recognition unit 100, it should also determine that the vehicle is in a state of anxiety and confusion if the change in the distance between the vehicle and the vehicle in front exceeds a predetermined value within a predetermined period. If it is determined that a state of anxiety and confusion should be identified, proceed to step S105; otherwise, proceed to step S107.

[0041] <Step S105: Beware of anxiety and confusion> The crew state determination unit 400 initiates an anxiety / confusion state alert, focusing its monitoring on parameters used to determine the anxiety / confusion state of crew member P. The determination of an anxious or confused state can be made, for example, based on the seat pressure distribution of the seat cushion S1 detected by the seat pressure sensor 402 (a parameter indicating the seating posture of the occupant P), and the distance between the rim of the steering wheel SW and the fingers of the occupant P, which is detected based on the image captured by the driver monitoring camera 401. Regarding seat pressure, for example, if there are fluctuations in the seat pressure distribution (changes in sitting posture) that exceed a predetermined level at a predetermined frequency (e.g., three or more times in 10 minutes), it can be determined that the person is in a state of anxiety and confusion. Furthermore, if the rearward shift of the occupants' center of gravity exceeds a predetermined amount (for example, 2% or more relative to the most recent initialization value), it can be determined that they are in a state of tension and a kind of anxious or confused state.

[0042] Figure 4 shows an example of the position of the occupant's fingers while holding the steering wheel. Figure 4(a) shows a situation where the crew is not under stress. Generally, when not under tension and in a relaxed state, occupant P does not grip the rim of the steering wheel SW tightly. In such cases, a gap g is often observed between the steering wheel SW and the thumb T. Note that while the thumb (T) is used here, the distance between other fingers and the steering wheel switches (SW) could also be used as a parameter.

[0043] Figure 4(b) shows the situation when the crew is under stress. Generally, when under stress (including anxiety, confusion, or anger), the occupant P grips the rim of the steering wheel SW tightly, so the distance g becomes smaller or disappears compared to when the occupant is not under stress. The occupant state determination unit 400 can determine that occupant P is gripping the steering wheel SW tightly and is in a state of anxiety and confusion if the difference between the current value of the gap g detected based on the image captured by the driver monitoring camera 401 and the initial value of the gap g becomes small enough to exceed a preset threshold. Then proceed to step S106.

[0044] <Step S106: Discrimination of Anxiety and Confusion> The crew state determination unit 400 determines whether or not crew member P is in an anxious or confused state within a predetermined anxiety / confusion state alert period (for example, within 10 minutes) using at least one of the determination methods described above. After determining whether or not the person is in a state of anxiety and confusion, proceed to step S107.

[0045] <Step S107: Determining the conditions for alerting about a state of impatience and anger> The crew state determination unit 400 determines whether predetermined conditions for determining crew member P's state of agitation and anger (third emotional state) have been met while the system is on alert for anxiety and confusion. For example, if the gap g between the steering wheel SW and the thumb T falls below a predetermined value, the system should determine that the patient is in a state of impatience and anger, and proceed to step S108; otherwise, the series of processes is terminated.

[0046] <Step S108: Beware of state of impatience and anger> The crew state determination unit 400 initiates an impatience and rage state alert, focusing its monitoring on parameters used to determine the impatience and rage state of crew member P. The state of impatience and anger can be determined, for example, based on the simultaneous occurrence of rapid acceleration / deceleration and steering operations exceeding a predetermined level in vehicle 1 (a typical example of reckless driving). The occupant state determination unit 400 can determine that the occupant is in a state of agitation or anger if, for example, the steering angle sensor 311 detects a change in steering angle exceeding a predetermined threshold, and at the same time, the acceleration sensor 333 detects a change in longitudinal acceleration (sudden acceleration or sudden deceleration) exceeding a predetermined threshold. Then proceed to step S109.

[0047] <Step S109: Determining the state of impatience and anger> The crew state determination unit 400 determines, using the determination method described above, whether or not crew member P is in a state of agitation or anger within a predetermined period of alert for anxiety and confusion (for example, within 10 minutes). After determining whether or not the user is in a state of anxiety or anger, the series of processes is terminated. In addition, in the emotion discrimination of this embodiment, it is possible that multiple emotional states from among distracted state, anxious state, and impatient / angry state may be detected simultaneously.

[0048] Generally speaking, it is said that a person's attention span can last for about 15 minutes. In driving a car, which requires the use of multiple abilities, a 10-minute drive equivalent to a 5km journey on a public road often results in some kind of change in emotional state. In particular, during periods of high stress, there is a concern that drivers may become distracted, leading to frequent oversights while driving, or that their driving may deteriorate, resulting in accidents. In contrast, the crew status determination unit 400 manages the sensing system in units of a predetermined warning period (for example, 10 minutes).

[0049] Specifically, if suspicion of a state of anxiety and confusion arises while vigilance for a state of distraction is being maintained for a predetermined period (for example, 10 minutes), vigilance for the state of anxiety and confusion is initiated for a new period (for example, 10 minutes). Subsequently, if suspicion of a state of agitation and anger arises, vigilance for the state of agitation and anger is initiated for a new period (for example, 10 minutes). This allows for accurate determination (estimation) of the emotional state of crew member P, even if their emotions shift in a direction that is inappropriate for the driving operation.

[0050] In addition to the emotion discrimination described above, the occupant state discrimination unit 400 determines that if the frequency of the driver's brake operations detected by the brake control unit 330 is below a predetermined value and extremely infrequent, and if the deceleration of the vehicle 1 detected by the acceleration sensor 333 is below a predetermined value and no deceleration operation is substantially performed, then the occupant P's driving ability is significantly reduced, and the driver support control unit 200 performs a forced stop control to bring the vehicle 1 to a safe location such as the shoulder of the road by automatic driving.

[0051] Furthermore, if the status determination unit 400 detects any of the following events, it will suspect that the occupant P is unable to drive due to an abnormal physical condition (such as a deterioration in physical condition that leads to a decrease in the level of consciousness, or intoxication due to alcohol or drugs), and will execute the aforementioned forced stop control, as well as use the communication device 220 to notify servers of designated organizations such as the police, ambulance services, and medical facilities. (a) Crew member P's field of vision is fixed (e.g., for 3 seconds or more) (b) If, based on the images captured by the driver monitoring camera 401, it is determined that the occupant P has closed eyes (i.e., is deemed unconscious) (c) When the driver assistance control unit 200 detects a predetermined meandering motion (d) When the change in the distance between the vehicle and the vehicle in front exceeds a predetermined threshold for detecting poor physical condition. (e) When the amount of backward or downward movement of the occupant P's center of gravity detected by the seat pressure sensor 402 is greater than or equal to a predetermined threshold for determining poor physical condition.

[0052] Next, we will explain the content of the driver assistance control that corresponds to the results of the emotion discrimination (estimation) of occupant P described above. Figure 5 is a flowchart showing the basic control content of the driver assistance system in the embodiment. The following explains each step in order.

[0053] <Step S201: Initialize emotion discrimination parameters> The occupant state determination unit 400 initializes each parameter used for emotion determination (estimation) in response to the start of the driving cycle of vehicle 1 (for example, when the main power of vehicle 1 is turned on (ignition on)). Then proceed to step S202.

[0054] <Step S202: Start monitoring driving conditions and emotional state> The driver assistance control unit 200 starts monitoring the driving conditions of vehicle 1, including vehicle speed, longitudinal acceleration, distance to the vehicle in front, lateral position within the lane, presence or absence of obstacles, and, if there are obstacles, their relative position and speed relative to vehicle 1. The occupant status determination unit 400 begins monitoring the emotional state of the occupant P (driver) as described above. Next, proceed to step S203.

[0055] <Step S203: Driving condition sensing judgment> The driver assistance control unit 200 senses events that require driver assistance regarding the driving state, and determines whether or not the occupant state determination unit 400 sensed any events related to emotional state (determination of at least one of distracted state, anxious state, or agitated / angry state). Examples of events that require driver assistance regarding driving conditions include the following: (a) Vehicle 1 swerving or wobbling (a deviation of the lateral position within the lane that exceeds a specified limit) (b) Lane departure of Vehicle 1 (c) Approaching the distance between vehicle 1 and the vehicle in front of it to a predetermined value or less (d) Detection of obstacles (risk objects) around the vehicle Here, the risk objects can include, for example, other vehicles, pedestrians, bicycles, and various other objects that obstruct the passage of vehicle 1. Furthermore, the objects of risk include other vehicles present to the rear and sides when changing lanes. (e) Vehicle speed exceeds the speed limit (f) Various information regarding traffic congestion, restrictions, accidents, etc. along the route that Vehicle 1 is scheduled to travel. If an event occurs that requires driver assistance only in relation to the driving conditions, proceed to step S210; otherwise, proceed to step S204.

[0056] <Step S204: Emotional State Sensing Judgment> The driver assistance control unit 200 determines whether the occupant state determination unit 400 sensed an event related to emotional state (at least one of distracted state, anxious state, or irritable state) and whether it sensed an event requiring driver assistance regarding the driving state. If only events related to emotional states are sensed, proceed to step S220; otherwise, proceed to step S205.

[0057] <Step S205: Driving condition / emotional state sensing and judgment> The driver assistance control unit 200 senses events that require driver assistance due to the driving conditions, and determines whether the occupant state determination unit 400 has sensed events related to emotional states unsuitable for driving (distracted state, anxious state, restless state). If events related to both driving conditions and emotional states are sensed, the process proceeds to step S230; otherwise, the series of processes ends (returns).

[0058] <Step S210: Warning display output> The driver assistance control unit 200 causes the input / output device 210 to output a display to alert the occupant P regarding the identified events among the above-mentioned events that require driver assistance in relation to driving conditions. After that, the series of processes will be terminated.

[0059] <Step S220: Sensing threshold update and warning display output> The driver assistance control unit 200 updates the threshold for sensing events requiring driver assistance regarding the driving state in a way that makes it easier to make a judgment (lower the threshold). Furthermore, the driver assistance control unit 200 causes the input / output device 210 to output a display to alert the occupant P regarding the identified events among the above-mentioned events that require driver assistance in relation to driving conditions. Then proceed to step S221.

[0060] <Step S221: Re-sensing decision based on update threshold> The driver assistance control unit 200 determines, based on the updated threshold in step S220, whether or not an event requiring driver assistance regarding the driving state has been sensed again. If an event requiring driver assistance is sensed again regarding the driving conditions, proceed to step S222; otherwise, terminate the series of processes.

[0061] <Step S222: Forced Stop Warning Display Output> The driver assistance control unit 200 outputs (presents) a warning to the occupant P via the input / output device 210 that the emotional state and driving condition have deteriorated and that the vehicle 1 should be stopped. Then proceed to step S223.

[0062] <Step S223: Re-sensing decision based on the re-update threshold> The driver assistance control unit 200 determines whether an event requiring driver assistance has been sensed again regarding the driving state, using a threshold that has been updated in step S220, further updated (threshold lowered) in a direction that makes the determination easier to obtain. If an event requiring driver assistance is sensed again regarding the driving conditions, proceed to step S224; otherwise, terminate the series of processes.

[0063] <Step S224: Execute forced stop control> The driver assistance control unit 200 performs a forced stop control to bring the vehicle 1 to a safe location such as the shoulder of the road through automatic driving, where the driver assistance control unit 200 controls steering and acceleration / deceleration. After that, the series of processes will be terminated.

[0064] <Step S230: Sensing threshold update / Forced stop warning display output> The driver assistance control unit 200 updates the threshold for sensing events that require driver assistance due to driving conditions in a way that makes it easier to make a judgment (lower the threshold). Furthermore, the driver assistance control unit 200 outputs (presents) a warning to the occupant P via the input / output device 210 that the emotional state and driving condition have deteriorated and that the vehicle 1 should be stopped. Then proceed to step S231.

[0065] <Step S231: Re-sensing decision based on update threshold> The driver assistance control unit 200 determines, based on the updated threshold in step S220, whether or not an event requiring driver assistance control intervention has been sensed again regarding the driving conditions. If an event requiring intervention by driver assistance control is sensed again regarding the driving conditions, proceed to step S232; otherwise, terminate the series of processes.

[0066] <Step S232: Execute forced stop control> The driver assistance control unit 200 performs a forced stop control to bring the vehicle 1 to a safe location such as the shoulder of the road through autonomous driving. After that, the series of processes will be terminated.

[0067] If occupant P stops vehicle 1 in response to the aforementioned forced stop warning, the driver assistance control unit 200 prohibits driving vehicle 1 until it is estimated that occupant P's emotional state has recovered to a state suitable for driving, and then permits driving vehicle 1 thereafter. For example, if the vehicle speed is 0 km / h and at least one of the following is detected: the door or side door glass is open, the seat S is reclined, or the transmission is kept in a non-driving range for a predetermined time (for example, 5 minutes) or longer, it is assumed that occupant P has taken a rest for a predetermined period of time or longer and that occupant P's emotional state has recovered (refreshed), and driving of vehicle 1 is permitted.

[0068] Furthermore, the same process is performed when the driver assistance control unit 200 forcibly stops the vehicle 1 (automatic stop). Furthermore, if the emotional state deteriorates beyond a predetermined level, the communication device 220 will be used to notify relevant parties such as family members, medical institutions, and specialized organizations.

[0069] Furthermore, in the driving assistance device of the embodiment, different driving assistance is provided depending on the type of emotional state that is identified. Figure 6 is a diagram showing how the content of the driver assistance control in the driver assistance device of the embodiment changes according to the emotional state of the occupant. The following explains each step in order.

[0070] <Step S301: Start monitoring based on the initialization value> The crew state determination unit 400 initializes the parameters related to the determination of crew member P's emotions as described above, and then begins monitoring each parameter. Then proceed to step S302.

[0071] <Step S302: Assessment of Distracted State> The driver assistance control unit 200 determines whether the occupant state determination unit 400 has determined that occupant P is in a state of distraction (step S103). If a state of distraction is identified, proceed to step S310; otherwise, proceed to step S303.

[0072] <Step S303: Assessment of Anxiety and Confusion> The driver assistance control unit 200 determines whether the occupant state determination unit 400 has determined (step S106) that occupant P is in a state of anxiety or confusion. If a state of distraction is identified, proceed to step S320; otherwise, proceed to step S304.

[0073] <Step S304: Assessment of the state of impatience and anger> The driver assistance control unit 200 determines whether the occupant state determination unit 400 has determined (step S106) that occupant P is in a state of agitation or anger. If a state of distraction is detected, proceed to step S330; otherwise, proceed to step S305.

[0074] <Step S305: Normal control> The driver assistance control unit 200 performs normal driver assistance control, assuming that the occupant P's emotional state is normal (suitable for driving). After that, the series of processes will be terminated.

[0075] <Step S310: Information display control - Specialized for pop-up information> The driver assistance control unit 200 performs information presentation control, which presents various types of information to the occupant, using the input / output device 210. The information presented to the crew may include, for example, the following: (a) Warning of the distance to the vehicle in front (warning when the distance is excessively small or fluctuating) (b) Speed ​​control warning (warning in case of speeding) (c) Driving lane information (warnings in case of weaving or lane departure, and guidance to encourage lane changes) (d) Sudden obstacle information (warning when a risk object with a specified or higher probability of collision appears in front of the vehicle) (e) Route environment information (information on road surface conditions including curvature, gradient, position of the sun, and temperature drop on the route the vehicle is scheduled to travel) The driver assistance control unit 200 reduces the output frequency of information related to (a) to (c) and (e) compared to normal operation, and focuses on outputting information related to (d) (specializing in sudden appearance information related to the environment in front of the vehicle and environmental information presentation control). Then proceed to step S311.

[0076] <Step S311: Lowering the intervention threshold for driving support control and vehicle behavior> The driver assistance control unit 200 performs driving assistance control by intervening in the control of the vehicle's steering system, power unit, and brake system according to the environment around the vehicle detected by the environment recognition unit 100 and the driving state of the vehicle detected by sensors. Examples of driving assistance control include the following: (a) Collision damage mitigation braking control when a risk object with a predetermined or higher probability of collision appears in front of the vehicle. (b) Lane departure prevention control that generates a yaw moment in the direction of restoration when the vehicle deviates from its lane or is highly likely to deviate from its lane. The driver assistance control unit 200 lowers the control intervention threshold for lane departure prevention control, which is primarily a control mechanism related to the vehicle's behavior. (It changes the threshold so that lane departure prevention control intervenes earlier.) After that, the series of processes is terminated (return).

[0077] <Step S320: Specialized for information display control and driving line information> The driver assistance control unit 200 reduces the output frequency of information related to (a), (b), and (d) from the information related to (a) to (d) described in step S310 above, compared to normal operation, and focuses on displaying information related to (c) (specializing in driving line information). Then proceed to step S321.

[0078] <Step S321: Driving support control - Decreasing the control intervention threshold for the surrounding environment> The driver assistance control unit 200 lowers the intervention threshold of the collision damage mitigation braking control described in step S311 above, which is primarily a control (environment-responsive driver assistance control) caused by changes in the environment surrounding the vehicle. (The threshold is changed to allow the collision damage mitigation braking control to intervene earlier.) After that, the series of processes will be terminated.

[0079] <Step S330: Specialized for information presentation control and route environment information> The driver assistance control unit 200 reduces the output frequency of information (a) to (c) from the information (a) to (e) described in step S310 above compared to normal operation, and outputs route environment information (e) (typically the situation at a distance of 1 km or more ahead) at a higher frequency and with greater emphasis compared to normal operation (specializing in route environment information). Furthermore, the information regarding the sudden appearance of an object (d) will be output in the same way as under normal circumstances. In this case, the driving assistance control will be performed in the same way as under normal circumstances. After that, the series of processes will be terminated.

[0080] According to the embodiments described above, the following effects can be obtained. (1) If an event suggesting a state of anxiety or confusion occurs during a distraction alert period for identifying a state of distraction, a new anxiety or confusion alert period can be initiated to identify the state of anxiety or confusion. This allows for the appropriate identification of the crew's state of distraction and state of anxiety or confusion, which often occur sequentially over time. (2) By detecting a state of distraction based on the fixed gaze of the occupant detected from the image captured by the driver monitoring camera 401 and the variation in the distance to the vehicle ahead detected by the environmental recognition unit 100, it is possible to appropriately determine a state of distraction with a simple device configuration by utilizing the output of sensors installed in a typical vehicle. (3) Generally, when occupant P falls into a state of anxiety and confusion, their seating posture relative to the seat S changes (typically leaning forward), or they grip the steering wheel SW more tightly, and the gap between the rim of the steering wheel SW and their fingers (thumb T, etc.) becomes smaller. According to this embodiment, the occupant's state of anxiety and confusion can be appropriately determined by the surface pressure distribution on the seat surface indicating the occupant P's seating posture on the seat S, and the positional relationship between the steering wheel and the occupant's fingers. (4) When an anxiety-confusion alert is initiated to identify an anxiety-confusion state, an agitation-anger alert is initiated to identify an agitation-anger state, which is a further deterioration of the emotional state, thereby enabling appropriate identification of the agitation-anger state. By determining the state of impatience and anger based on the steering angle sensor 311 and the acceleration sensor 333, it is possible to appropriately determine the state of impatience and anger with a simple device configuration using sensors that are normally installed in a typical vehicle.

[0081] (modified version) The present invention is not limited to the embodiments described above, and various modifications and changes are possible, all of which fall within the technical scope of the present invention. (1) The configuration of the ring road discrimination device, the driving assistance device, and the vehicle on which they are installed is not limited to the embodiments described above and can be modified as appropriate. For example, the functions of each unit, the types of sensors installed, and the types of events to be sensed are not particularly limited and can be changed as appropriate. (2) The specific details of the driver assistance control are not limited to those of the embodiment, and other functions may be added. Also, some functions in the embodiment may be omitted. (3) The parameters and discrimination logic used to determine the emotional state of the crew members are not limited to the configuration of the embodiment and can be changed as appropriate. For example, in this embodiment, the state of anxiety, confusion, and agitation are determined based on the distance between the occupant's thumb and the rim of the steering wheel. However, the finger used as the reference for measuring the distance is not limited to the thumb; other fingers may also be used. Alternatively, the distance between the palm and the rim may be used as a substitute. Alternatively, instead of such intervals, a contact pressure sensor may be installed on the rim of the steering wheel to estimate the degree to which the occupant grips the steering wheel based on the gripping force (contact pressure) of the occupant's fingers and palm. (4) In this embodiment, the acceleration and deceleration of the vehicle to determine the state of impatience and anger are detected using an acceleration sensor, but the acceleration and deceleration of the vehicle may be detected or estimated by other methods. For example, instead of deceleration, parameters such as braking force, braking amount, hydraulic fluid pressure in hydraulic brakes, absorption torque in regenerative braking, vehicle speed reduction amount, and reduction rate may be used. Alternatively, instead of acceleration, parameters such as accelerator pedal input, target torque of the driving power source, or estimated actual torque may be used. [Explanation of Symbols]

[0082] 1 Vehicle 100 Environmental Recognition Units 110 Stereo camera system 120 Millimeter-wave radar system 130 Laser scanner equipment 140 High-precision map database 150 Positioning device 200 Driver support control unit 210 Input / Output Devices 220 Communication Devices 310 Electric Power Steering Control Unit 311 Steering angle sensor 312 Steering angle sensor 313 Motor 320 Power Unit Control Unit 330 Brake control unit 331 Hydraulic Control Unit 332 Vehicle speed sensor 333 Acceleration sensor 334 Yaw rate sensor 400 Crew status determination unit 401 Driver monitoring camera 402 Seat pressure sensor P Passenger T Thumb SW steering wheel g interval S Seat S1 Seat Cushion S2 Backrest S3 Headrest

Claims

1. A driving state detection unit that detects the driving state of the vehicle, A passenger monitoring unit that monitors the condition of the occupants of the vehicle, Based on the output of the driving state detection unit and the occupant monitoring unit, an emotion determination unit determines the emotions of the occupant. An emotion discrimination device equipped with, During a first alert period in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a first period in order to determine a first emotional state, if predetermined conditions for determining a second emotional state are met, a second alert period is initiated in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a second period in order to determine a second emotional state. Includes, The second emotional state described above is the state of anxiety and confusion of the crew member. The emotion determination unit determines the state of anxiety and confusion based on at least one of the seating posture of the occupant in the seat detected by the occupant monitoring unit during the second alert period, and the positional relationship between the steering wheel and the occupant's fingers detected by the occupant monitoring unit. An emotion discrimination device characterized by the following.

2. The first emotional state is a state of distraction in the crew member, The emotion determination unit determines the state of distraction based on at least one of the fixed gaze state of the occupant detected by the occupant monitoring unit during the first alert period and the variability of the vehicle's driving state detected by the driving state detection unit. The emotion discrimination device according to claim 1, characterized by the following:

3. If, during the second alert period, predetermined conditions for determining a third emotional state are met, a third alert period is initiated in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a third period in order to determine the third emotional state. The emotion discrimination device according to claim 1, characterized by the following:

4. The third emotional state described above is the state of agitation and anger of the crew member. The aforementioned driving state detection unit includes a steering angle detection unit for detecting the steering angle of the vehicle's steering device, and an acceleration / deceleration detection unit for detecting the acceleration / deceleration state of the vehicle. The emotion discrimination unit determines the state of impatience and anger when the steering angle of the steering device increases and the acceleration / deceleration state occurs simultaneously during the third alert period. The emotion discrimination device according to claim 3, characterized by the following:

5. A driving state detection unit that detects the driving state of the vehicle, A passenger monitoring unit that monitors the condition of the occupants of the vehicle, Based on the output of the driving state detection unit and the occupant monitoring unit, an emotion determination unit determines the emotions of the occupant. An emotion discrimination device equipped with, During a first alert period in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a first period in order to determine a first emotional state, if predetermined conditions for determining a second emotional state are met, a second alert period is initiated in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a second period in order to determine a second emotional state. If predetermined conditions for determining a third emotional state are met during the second alert period, a third alert period is initiated in which the output history of the driving state detection unit and the occupant monitoring unit is monitored over a third period in order to determine the third emotional state. Includes, The third emotional state described above is the state of agitation and anger of the crew member. The aforementioned driving state detection unit includes a steering angle detection unit for detecting the steering angle of the vehicle's steering device, and an acceleration / deceleration detection unit for detecting the acceleration / deceleration state of the vehicle. The emotion discrimination unit determines the state of impatience and anger when the steering angle of the steering device increases and the acceleration / deceleration state occurs simultaneously during the third alert period. An emotion discrimination device characterized by the following.

Images