Driver status determination device, driver status determination method, and program

Abstract

To provide a driver status determination device, a driver status determination method, and a program that can more appropriately determine the driver's condition. [Solution] The driver status determination device of the embodiment comprises a determination unit that determines the status of the driver of a moving object, and a notification unit that notifies the driver based on the determination result of the determination unit, wherein the notification unit makes a first notification when the determination unit determines that the driver's status is a first status, and makes a second notification different from the first notification when the determination unit determines that the driver's status is a second status different from the first status.

Description

Technical Field

[0001] The present invention relates to a driver state determination device, a driver state determination method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development related to preventive safety technology have been focused on further improving traffic safety and convenience through research and development. In this context, conventionally, a technique for performing a drowsiness alarm based on a drowsiness level that matches the driver's feeling is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in conventional preventive safety technology, in the case of determining the driver's state based on the drowsiness level, it has sometimes been difficult to determine whether the driver is drowsy or just has a momentary glance with the eyes closed. Therefore, there has been a problem that the driver's state may not be appropriately determined.

[0005] One of the objects of the present application is to provide a driver state determination device, a driver state determination method, and a program that can more appropriately determine the driver's state in order to solve the above problems. And by extension, it contributes to the development of a sustainable transportation system.

Means for Solving the Problems

[0006] The driver status determination device, driver status determination method, and program according to this invention employ the following configuration. (1) A driver status determination device according to one aspect of the present invention comprises a determination unit that determines the status of the driver of a moving body, and a notification unit that notifies the driver based on the determination result of the determination unit, wherein the notification unit gives a first notification when the determination unit determines that the driver's status is a first status, and gives a second notification different from the first notification when the determination unit determines that the driver's status is a second status different from the first status.

[0007] (2) In the embodiment of (1) above, the first state is distracted looking, and the second state is dozing off.

[0008] (3) In the embodiment of (2) above, if the determination unit determines that the driver's condition is the second condition, the notification unit will provide notification corresponding to the drowsiness, and then provide notification urging the driver to take a break.

[0009] (4) In the embodiment of (1) above, the determination unit determines that the driver's state is the second state when the driver keeps their eyes closed continuously and the driver has been driving the mobile body for a predetermined time or longer.

[0010] (5) In the embodiment of (1) above, the determination unit determines that the driver's state is the first state if the driver keeps their eyes closed continuously and the speed at which the driver closes their eyes is equal to or greater than a predetermined speed, and determines that the driver's state is the second state if the speed at which the driver closes their eyes is less than a predetermined speed.

[0011] (6): In the embodiment of (1) above, the determination unit determines that the driver's state is the first state if the driver keeps their eyes closed continuously, the speed at which the driver closes their eyes is equal to or greater than a predetermined speed, and the driver's mouth is closed, and determines that the driver's state is the second state if the speed at which the driver closes their eyes is less than a predetermined speed, and the driver's mouth is open.

[0012] (7) In the embodiment of (1) above, the determination unit determines that the driver's state is the second state when the driver has their eyes closed continuously and the behavior of the moving body is not stable.

[0013] (8) In the embodiment of (1) above, the determination unit determines that the driver's state is the first state when the driver has their eyes closed continuously, the duration of driving the moving body is less than a predetermined time, the driver's drowsiness level is less than a predetermined value, and the behavior of the moving body is stable.

[0014] (9): A driver status determination method according to another aspect of the present invention is a driver status determination method in which a computer determines the status of the driver of a moving object, notifies the driver based on the determination result, provides a first notification if it is determined that the driver's status is a first status, and provides a second notification different from the first notification if it is determined that the driver's status is a second status different from the first status.

[0015] (10): A program according to another aspect of the present invention is a program that causes a computer to determine the state of the driver of a mobile body, to notify the driver based on the determined result, to give a first notification if it is determined that the driver's state is a first state, and to give a second notification different from the first notification if it is determined that the driver's state is a second state different from the first state. [Effects of the Invention]

[0016] According to the aspects (1) to (10) above, the driver's state can be determined more appropriately.

Brief Description of Drawings

[0017] [Figure 1] It is a configuration diagram of a vehicle system 1 including a driver state determination device according to an embodiment. [Figure 2] It is a diagram schematically showing the content of processing by the state recognition unit 124. [Figure 3] It is a diagram (Part 1) for explaining an example of a method for recognizing the eye opening rate. [Figure 4] It is a diagram (Part 2) for explaining an example of a method for recognizing the eye opening rate. [Figure 5] It is a flowchart showing an example of processing executed by the driving support device 100 in an embodiment.

Modes for Carrying Out the Invention

[0018] Hereinafter, referring to the drawings, embodiments of the driver state determination device, driver state determination method, and program of the present invention will be described. Hereinafter, an example in which the driver state determination device is applied to a moving body will be described. Also, as an example of the moving body, a vehicle will be used. The moving body may include, in addition to vehicles, for example, ships that can move on the ground (on the road) like hovercrafts, flying bodies that can travel on roads, standing vehicles having a power unit, micromobility such as electric kick scooters, and the like.

[0019] [Overall Configuration] Figure 1 is a diagram showing the configuration of a vehicle system 1 including a driver status determination device according to an embodiment. The vehicle on which the vehicle system 1 is installed (hereinafter referred to as vehicle M) is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle or a micromobility, and its power source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a battery (storage battery) such as a secondary battery or a fuel cell.

[0020] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an in-cabin camera 70, a driver control unit 80, a driver assistance device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Note that the configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. Camera 10, radar device 12, and LiDAR 14 are examples of "detection devices DD". Driver assistance device 100 is an example of a "driver state determination device".

[0021] The camera 10 is, for example, a digital camera that uses a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to an arbitrary location on the vehicle M on which the vehicle system 1 is mounted. When imaging the front, the camera 10 is attached to, for example, the upper part of the front windshield, the back surface of the rearview mirror, the front head part of the vehicle body, etc. When imaging the rear, the camera 10 is attached to the upper part of the rear windshield, the back door, etc. When imaging the side, the camera 10 is attached to the left and right door mirrors, etc. The camera 10, for example, periodically repeats imaging the periphery of the vehicle M. The camera 10 may be a stereo camera.

[0022] The radar device 12 emits radio waves (radar) such as millimeter waves to the periphery of the vehicle M, and detects the radio waves (reflected waves) reflected by surrounding objects to detect at least the position (distance and azimuth) of the objects. The radar device 12 is attached to an arbitrary location on the vehicle M. The radar device 12 may detect the position and speed of an object by the FM-CW (Frequency Modulated Continuous Wave) method.

[0023] [[ID=​​​​​​​ The HMI 30 outputs various information to the occupants of the vehicle M (including the driver) and accepts input operations from the occupants. The HMI 30 includes, for example, a display unit 32, a speaker 34, and a microphone 36. The display unit 32 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit 32 displays various images (including video) in the embodiment. The display unit 32 may be configured integrally with the input unit as a touch panel. The speaker 34 outputs predetermined sounds (for example, notification sounds or message sounds). The microphone 36 accepts the voices of the occupants, including the driver. The HMI 30 may also include a buzzer, touch panel, switches, keys, etc. The switches may include switches that execute or terminate predetermined driving controls that can be executed by the driving control unit described later, and switches that approve (permit) or reject driving control recommendations (suggestions) from the system (vehicle system 1). Furthermore, the switches may include switches for operating the turn signals (turn signal switches), etc.

[0026] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, and a yaw rate sensor for detecting yaw rate (for example, the angular velocity of rotation around the vertical axis passing through the center of gravity of the vehicle M). The vehicle sensor 40 may also include a lateral acceleration sensor (lateral G sensor) for detecting the lateral acceleration (lateral G) of the vehicle M, a steering angle sensor for detecting the steering angle of the vehicle M (which may be the angle of the steering wheels or the operating angle of the steering wheel), a steering angular velocity sensor for detecting the steering angular velocity, and a compass sensor for detecting the orientation of the vehicle M. Furthermore, the vehicle sensor 40 may include sensors for detecting the operating state of the vehicle M (whether or not the vehicle system 1 is operating) or for detecting the distance traveled by the vehicle M.

[0027] Furthermore, the vehicle sensor 40 may include a position sensor that detects the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that acquires position information using, for example, a GNSS (Global Navigation Satellite System) receiver of a navigation device 50. The vehicle sensor 40 may derive the speed of the vehicle M from the difference (i.e., distance) of position information at a predetermined time from the position sensor. The results detected by the vehicle sensor 40 are output to the driver assistance device 100.

[0028] The navigation device 50 includes, for example, a GNSS receiver, a navigation HMI, and a route determination unit. The navigation device 50 may store map information in a storage device such as an HDD (Hard Disk Drive) or flash memory, or it may acquire map information 192 stored in a storage unit 190, which will be described later. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI includes a display device, speaker, touch panel, keys, etc. The GNSS receiver may be provided on the vehicle sensor 40. The navigation HMI may be partially or completely shared with the HMI 30 described above. The route determination unit determines, for example, a route (hereinafter referred to as a route on a map) from the position of the vehicle M determined by the GNSS receiver (or any input position) to a destination input by the occupant using the navigation HMI, by referring to, for example, map information 192, etc. Furthermore, the navigation device 50 provides route guidance using the navigation HMI based on the determined route on the map. The navigation device 50 may also transmit its current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0029] Here, map information 192 is information in which the road shape is represented by links indicating roads (an example of a travel route) and nodes connected by those links. Map information 192 may also include POI (Point of Interest) information, etc. Map information 192 also includes, for example, the number of lanes (number of travel routes), the type and shape of road markings, information on the center of the lanes, or information on road boundaries. Map information 192 may also include information on whether the road boundary is a boundary (physical boundary) that includes structures that vehicles cannot pass through (including crossing and contact). Physical boundaries include, for example, guardrails, curbs, median strips, fences, etc. Map information 192 may also include road shape information, traffic regulation information, address information (address and postal code), facility information, parking information, telephone number information, etc. Road shape information includes, for example, the curvature of the road (which may be rephrased as radius of curvature; the same applies below), width, road surface gradient, branching and merging points, intersections, T-junctions, etc. Map information 192 may be updated as needed by the communication device 20 communicating with an external device.

[0030] The in-vehicle camera 70 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The in-vehicle camera 70 is mounted at any location in the vehicle M in a position and orientation that allows it to capture the head of the driver seated in the driver's seat of the vehicle M from the front. For example, the in-vehicle camera 70 is mounted near (e.g., above or below) the display device located in the center of the instrument panel of the vehicle M. Alternatively, the in-vehicle camera 70 may capture images of the interior of the vehicle by illuminating the interior with infrared light. The in-vehicle camera 70 also captures images of the interior of the vehicle M including the occupant (passenger) seated in the passenger seat of the vehicle M.

[0031] The driver control unit 80 includes, for example, a steering wheel, an accelerator pedal, and a brake pedal. The driver control unit 80 may also include a shift lever, a modified steering wheel, a joystick, or other controls. Each control of the driver control unit 80 is equipped with an operation detection unit that detects, for example, the amount of operation performed by the driver on the control or whether or not an operation has been performed. The operation detection unit detects, for example, the steering angle and steering torque of the steering wheel (for example, the amount of steering due to the driver's driving operation (steering input torque)), the rate of change of the steering torque, the amount of depression of the accelerator pedal and brake pedal, etc. The operation detection unit then outputs the detection results to the driver assistance device 100, or to one or both of the driving force output device 200, the brake device 210, and the steering device 220. The driver control unit 80 may also include a turn signal control unit (for example, a turn signal lever, a turn signal switch). When the turn signal control unit is operated, the turn signal of the vehicle M corresponding to the operation will flash, and the operation details (including, for example, the detection result that the operation was performed by the driver) will be output to the driver assistance device 100.

[0032] The driver assistance device 100 performs various controls to assist the driver of the vehicle M. The driver assistance device 100 includes, for example, a recognition unit 120, a determination unit 140, an HMI control unit 160, a driving control unit 180, and a storage unit 190. The recognition unit 120, the determination unit 140, the HMI control unit 160, and the driving control unit 180 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The above-mentioned program may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the driver assistance device 100, or it may be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the driver assistance device 100 when the storage medium (non-transient storage medium) is inserted into a drive device or card slot. The driving control unit 180 is an example of a "mobility control unit". The HMI 30 and HMI control unit 160 are examples of "notification units".

[0033] The storage unit 190 may be implemented using the various storage devices described above, or an EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 190 stores, for example, map information 192, various information in the embodiment, programs, etc. The storage unit 190 may also store various setting information used in the processing in this embodiment.

[0034] The recognition unit 120 includes, for example, a surrounding recognition unit 122, a state recognition unit 124, and a behavior recognition unit 126. The surrounding recognition unit 122 recognizes the surrounding conditions of the vehicle M based on, for example, the detection results of the detection device DD (information input from the camera 10, radar device 12, and LIDAR 14). For example, the surrounding recognition unit 122 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to recognize the state of objects present around the vehicle M (within a predetermined distance from the vehicle M), such as their position (relative position), size, speed (relative velocity), and acceleration. Objects recognized by the surrounding recognition unit 122 may include, for example, physical boundaries that demarcate roads (travel paths), as well as other vehicles, pedestrians, bicycles, and other traffic participants (examples of obstacles). The position of an object is recognized as a position on an absolute coordinate system with the vehicle M's representative point (such as the center of gravity or drive axis center) as the origin, and is used for control. The position of an object may be represented by its center of gravity, a representative point such as a corner, or by the represented region. The "state" of an object may include, for example, the acceleration or jerk of another vehicle, or its "action state" (for example, whether the other vehicle is changing lanes or is about to change lanes).

[0035] Furthermore, the surrounding area recognition unit 122 may recognize, for example, stop lines, red lights, toll booths, other road events, road signs, and markings drawn on the road (e.g., speed limits). The surrounding area recognition unit 122 may also recognize the curvature of the vehicle M's lane (road) based on the detection results of the detection device DD or the map information 192. The surrounding area recognition unit 122 may also recognize the road surface conditions (e.g., whether the road surface is slippery, such as being frozen) based on the detection results of the detection device DD.

[0036] Furthermore, the surrounding recognition unit 122 recognizes, for example, the lane in which the vehicle M is traveling (driving lane) and other surrounding lanes (for example, adjacent lanes). For example, the surrounding recognition unit 122 recognizes road markings from images captured by the camera 10 and recognizes the driving lane and other lanes based on the positional relationship of the road markings as seen from the recognized vehicle M. Alternatively, the surrounding recognition unit 122 may refer to map information 192 based on the position information of the vehicle M obtained from the vehicle sensor 40, etc., to recognize the lane in which the vehicle M is traveling and other lanes.

[0037] The state recognition unit 124 recognizes the state of the occupants of vehicle M using images captured by the in-vehicle camera 70. For example, the state recognition unit 124 performs known image analysis processing on the images captured by the in-vehicle camera 70 and recognizes the open / closed state of the driver's eyes based on the analysis results. The state recognition unit 124 may also recognize the open / closed state of the driver's mouth. Furthermore, the state recognition unit 124 may perform known voice analysis processing on the voice emitted by the driver acquired by the microphone 36 and recognize the content of the voice based on the analysis results, or it may recognize the driver's gaze and the direction of their face. Details of the functions of the state recognition unit 124 will be described later.

[0038] The behavior recognition unit 126 recognizes the behavior of vehicle M based on the detection results of the vehicle sensor 40, the detection results of the operation detection unit of the driver control device 80, and the control content executed by the driving control unit 180. The behavior of vehicle M includes the behavior due to manual driving by the driver and the behavior due to driving control executed by the driving control unit 180. The behavior recognition unit 126 may also recognize that the behavior of vehicle M is stable (or unstable).

[0039] For example, the behavior recognition unit 126 recognizes the lateral position (position in the lane width direction) of the vehicle M relative to the driving lane and the attitude (orientation) of the vehicle M relative to the direction of extension of the driving lane, based on the positional relationship of the vehicle M with respect to the driving lane. Alternatively, the behavior recognition unit 126 may recognize the deviation of the vehicle M's reference point from the center of the lane and the angle it makes with a line connecting the centers of the lanes in the direction of travel of the vehicle M as the relative position and attitude of the vehicle M relative to the driving lane. Alternatively, the behavior recognition unit 126 may recognize the position of the vehicle M's reference point relative to any side edge of the driving lane (road marking or road boundary) as the relative position (lateral position) of the vehicle M relative to the driving lane. Furthermore, the behavior recognition unit 126 may recognize the lateral behavior of the vehicle M (for example, whether or not it has moved laterally by a predetermined distance or more in a predetermined time) from the amount of change in the lateral position and orientation (yaw rate) of the vehicle M as described above.

[0040] Furthermore, the behavior recognition unit 126 may detect the behavior of the vehicle M from the amount of steering wheel operation (e.g., steering angle, steering torque, steering torque change rate), the amount of depression of the accelerator pedal and brake pedal, etc., obtained by the operation detection unit when the vehicle M is being driven manually, or it may recognize whether the behavior is stable or not from the amount of change in the behavior over a predetermined period of time.

[0041] Furthermore, the behavior recognition unit 126 may recognize that the behavior of vehicle M is stable if the amount of change in the lateral position or orientation of vehicle M over a predetermined time is less than a threshold, and recognize that the behavior is unstable if it is greater than or equal to the threshold. Also, the behavior recognition unit 126 may recognize that the behavior of vehicle M is stable if the amount of change in the acceleration or deceleration of vehicle M over a predetermined time is less than a threshold, and recognize that the behavior is unstable if it is greater than or equal to the threshold.

[0042] Furthermore, the behavior recognition unit 126 may recognize the behavior of the vehicle M based on the content of the driving control performed by the driving control unit 180. Driving control is, for example, a control that drives the vehicle M by controlling at least one of the steering and speed of the vehicle M, either without driver operation or by accepting some instructions. Driving control includes, for example, ACC (Adaptive Cruise Control System), LKAS (Lane Keeping Assistance System), ALC (Auto Lane Changing), etc. Driving control may also include control that stops the vehicle M in a safe position such as the shoulder of the road based on the determination result of the determination unit 140, or control that controls the steering and speed to avoid contact between the vehicle M and obstacles recognized by the surrounding recognition unit 122. Driving control may also include, for example, VSA (Vehicle Stability Assist). VSA is a driving control that automatically stabilizes the behavior of the vehicle M when a skid or similar event occurs in the vehicle M. VSA includes, for example, safety devices such as ABS (Anti-lock Braking System) which reduces skidding due to wheel lock during sudden deceleration or deceleration on low-friction roads, TCS (Traction Control System) which prevents wheelspin during starting and acceleration, and a skid suppression device, or a system that comprehensively controls these safety devices. For example, the behavior recognition unit 126 may recognize that the behavior of vehicle M is unstable when VSA is being executed by the driving control unit 180, and may recognize that the behavior of vehicle M is stable after the execution of VSA has finished.

[0043] Furthermore, the behavior recognition unit 126 may recognize the driving duration by the driver based on the detection results of the vehicle sensor 40, etc. The driving duration may be the time from when the vehicle system 1 is activated (for example, from when the ignition switch of vehicle M is turned ON) until the vehicle system 1 is shut down (for example, when the ignition switch is turned OFF), or it may be the time while vehicle M is moving (excluding stopping time). The behavior recognition unit 126 may also recognize the distance traveled by vehicle M due to the driver's driving (driving distance). The driving distance is, for example, the distance traveled from when the vehicle system 1 is activated until it is shut down.

[0044] The determination unit 140 includes, for example, a state determination unit 142. The state determination unit 142 determines the state of the driver of the vehicle M. For example, the state determination unit 142 determines whether the driver is in a predetermined state based on the recognition result of the state recognition unit 124. The predetermined state may be at least a first state, a second state, or any other state. The first state is, for example, the driver being distracted. The second state is, for example, the driver falling asleep at the wheel. Details of the functions of the state determination unit 142 will be described later.

[0045] The HMI control unit 160 notifies the occupants (including the driver) of predetermined information via the HMI 30 and receives information input by the HMI 30. The predetermined information includes, for example, information related to the driving of vehicle M, such as information regarding the status of vehicle M and information regarding driving control. Information regarding the status of vehicle M includes, for example, the speed of vehicle M, engine speed, and shift position. Information regarding driving control includes, for example, whether or not driving control is being performed by the driving control unit 180, information regarding the status of driving control, information regarding driving control recommendations (suggestions) from the system, and notification information to the driver (warnings, etc.). The predetermined information may also include information regarding the surrounding conditions recognized by the detection device DD. The predetermined information may also include information unrelated to the driving of vehicle M, such as content (e.g., video) stored on a storage medium such as a television program or DVD. The predetermined information may also include, for example, information regarding the current location and destination of vehicle M, and the remaining fuel level of vehicle M. The HMI control unit 160 may output the information received by the HMI 30 to the communication device 20, navigation device 50, recognition unit 120, determination unit 140, driving control unit 180, etc.

[0046] Furthermore, the HMI control unit 160 may generate inquiry information and recommendation information for the occupants, recognition results from the recognition unit 120, judgment results from the judgment unit 140, notification information (the first notification and the second notification described later), etc., and output the generated information to the HMI 30. The generated information may include images and sounds (including notification sounds, etc.). In addition, the HMI control unit 160 may transmit the various information to be output to the HMI 30 to terminal devices used by the occupants of the vehicle M via the communication device 20.

[0047] The driving control unit 180 controls the driving of the vehicle M (movement of the moving body). For example, the driving control unit 180 performs driving control on the vehicle M based on the recognition results from the recognition unit 120 and the determination results from the determination unit 140. Driving control may be performed in response to an execution instruction from the driver via the HMI 30, or it may be performed based on the recognition results from the recognition unit 120 without any instruction from the driver. When performing driving control, the driving control unit 180 generates a future target trajectory for the vehicle M according to the content of the driving control based on the recognition results from the recognition unit 120, and controls at least one of the steering and speed of the vehicle M so that the vehicle M travels along the generated target trajectory.

[0048] For example, the driving control unit 180 performs driving control such as ACC, LKAS, ALC, and VSA. Furthermore, if the driver's monitoring direction does not improve even after a predetermined time has elapsed since the HMI control unit 160 has outputted notification information indicating that the driver's monitoring direction is inappropriate, the driving control unit 180 will perform control to stop the vehicle M in a safe position such as the shoulder of the road, or control to avoid contact between the vehicle M and an obstacle.

[0049] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the driving control unit 180 or information input from the accelerator pedal of the driver control unit 80.

[0050] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 180 or from the brake pedal of the driver control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may be equipped with a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the driving control unit 180 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0051] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 180 or from the steering wheel of the driver control unit 80.

[0052] [State Recognition Unit] Next, the functions of the state recognition unit 124 will be described in detail. The state recognition unit 124 performs image analysis processing, such as edge extraction, on images captured by the in-vehicle camera 70, and uses the facial feature information obtained as a result of the analysis to recognize the eye opening rate and mouth opening rate as examples of the open / closed state of the driver's eyes and mouth.

[0053] Figure 2 is a schematic diagram illustrating the processing performed by the state recognition unit 124. In the figure, IM represents an image in which an edge EG is superimposed on the image captured by the in-vehicle camera 70. This figure focuses exclusively on the driver seated in the driver's seat. First, as shown in the upper part of Figure 2, the state recognition unit 124 extracts the driver's facial contour CT by fitting a model such as an ellipse or egg shape to the edge EG. Next, as shown in the middle part of Figure 2, the state recognition unit 124 sets a nose detection window NM based on the facial contour CT and detects the position and shape of the nasal bridge BN, which is a part where edges are easily extracted clearly, within the nose detection window NM. Next, as shown in the lower part of Figure 2, the state recognition unit 124 sets predetermined-sized eye detection windows EW1 and EW2 on the right and left sides of the nasal bridge BN, where the driver's eyes (right eye, left eye) are expected to be located, based on the position and shape of the nasal bridge BN, and detects at least a part of the eye in each of the eye detection windows EW1 and EW2.

[0054] Furthermore, the state recognition unit 124 may, based on the position and shape of the contour CT and nasal bridge BN, set a mouth detection window MW of a predetermined size below the nasal bridge BN where the driver's mouth is predicted to be located, and detect at least a portion of the driver's mouth within the mouth detection window MW. When detecting the contours of the driver's eyes and mouth, the state recognition unit 124 detects the contours by, for example, fitting a curve model to the distribution of edges EG within each window, but other methods may also be used.

[0055] Next, the system recognizes the eye opening rate and mouth opening rate of the occupant based on the positional relationship of multiple feature points in the eye and mouth contours detected by the state recognition unit 124. The multiple feature points include, for example, in the case of the eye contour, a first feature point which is the end closer to the in-cabin camera 70 in the lateral direction (corresponding to the outer corner of the eye), a second feature point which is the upper end, and a third feature point which is the lower end. In the case of the mouth contour, the first feature point may be the end closer to the in-cabin camera 70 in the lateral direction (corresponding to the mouth), the second feature point the upper end, and the third feature point the lower end. Below, as an example, the method for recognizing the eye opening rate of the right eye within the eye detection window EW1 will be described.

[0056] Figure 3 is a diagram (part 1) illustrating an example of a method for recognizing the eye-opening rate. Figure 4 is a diagram (part 2) illustrating an example of a method for recognizing the eye-opening rate. In the diagrams, P1 represents the first feature point, P2 represents the second feature point, and P3 represents the third feature point. The state recognition unit 124 sets a vertical line within the eye detection window EW1, virtually moves this vertical line from the right end of the eye detection window EW1 to the left, and defines the intersection point where it first intersects with the eye contour ECT as the first feature point P1. The state recognition unit 124 also sets a horizontal line within the eye detection window EW1, virtually moves the set horizontal line from the top end of the eye detection window EW1 to the bottom, and defines the intersection point where it first intersects with the eye contour ECT as the second feature point P2. Furthermore, the state recognition unit 124 virtually moves a horizontal line upward from the lower end of the eye detection window EW1 within the eye detection window EW1, and defines the intersection point where it first intersects with the eye contour ECT as the third feature point P3.

[0057] Then, as shown in Figure 4, the state recognition unit 124 acquires the angle θ between the first straight line L1 connecting the first feature point P1 and the second feature point P2, and the second straight line L2 connecting the first feature point P1 and the third feature point P3, and recognizes the driver's eye-opening rate α based on the acquired angle θ. For example, the state recognition unit 124 defines a reference angle θ0, which is the average of the angles θ acquired based on images taken for the first few minutes after the driver gets into the vehicle M, as the state where the eye-opening rate α is 100%. Subsequently, it divides the acquired angles θ by the reference angle θ0 and multiplies the divided value by 100 to recognize the eye-opening rate α during driving (for example, eye-opening rate α = (θ / θ0) × 100).

[0058] The reference angle θ0 may be acquired in advance and stored in the memory unit 190, and the state recognition unit 124 may read the reference angle θ0 from the memory unit 190 and use it for calculations when recognizing the eye-opening rate α. The reference angle θ0 may also be a fixed value set in advance, or it may be adjusted as needed to match the average angle of the driver while driving, obtained from time-series captured images.

[0059] Furthermore, the state recognition unit 124 also recognizes the driver's left eye and mouth using the method described above, specifically the left eye opening rate and the mouth opening rate. The state recognition unit 124 may also recognize the eye closing rate and mouth closing rate instead of the eye opening rate and mouth opening rate. In this case, for example, the state recognition unit 124 recognizes the eye closing rate as 1 minus the eye opening rate α, and the mouth closing rate as 1 minus the mouth opening rate.

[0060] Furthermore, the state recognition unit 124 may recognize the driver's gaze (the direction the driver is looking) or the orientation of the driver's face based on the results of image analysis processing of the images captured by the in-vehicle camera 70. For example, the state recognition unit 124 uses methods such as template matching to detect combinations of reference points (the stationary part of the eye) and moving points (the moving part of the eye) of the driver's eyes from the image. Examples of combinations of reference points and moving points include combinations of the inner corner of the eye and the iris, and combinations of the corneal reflection region and the pupil. The corneal reflection region is, for example, the infrared light reflection region on the cornea when the in-vehicle camera 70 shines infrared light towards the driver. The state recognition unit 124 then recognizes the driver's gaze by performing coordinate transformations from the image plane to real space based on the position of the moving points relative to the reference points. The state recognition unit 124 also recognizes the orientation of the driver's face based on positional information of the eyes, nose, mouth, etc. within the face region obtained from the image analysis results.

[0061] [State determination unit] Next, the functions of the state determination unit 142 will be described in detail. The state determination unit 142 determines the driver's state based on information recognized by the state recognition unit 124, the behavior recognition unit 126, the driving control unit 180, etc. In addition to the first state (e.g., distracted driving) and the second state (e.g., falling asleep) described above, the driver's state may also include a third state, etc. The third state is, for example, being inattentive.

[0062] Here, distraction refers to a state in which attention is not paid to the direction of travel of the vehicle M. Distraction is a state in which the driver is distracted by the surrounding scenery, the display image of the navigation device 50, conversation with passengers, etc., and is not able to concentrate on driving, and is referred to as "external forward inattention." Distraction may also include a state in which the driver's gaze or the direction of their face is not within the pre-set monitoring range (a predetermined range including the direction of travel) with respect to the direction of travel of the vehicle M. In the case of distraction, it may also be recognized that the driver's eyes are closed based on the analysis results using the images captured by the in-vehicle camera 70 (for example, when the driver is looking down or has their eyes closed and is thinking). Drowsiness refers to a state in which the driver's eyes are continuously closed. "Distracted driving" refers to a state in which the driver's eyes are open but they are not concentrating. For example, their gaze and face may be directed towards the vehicle M's monitoring range, but they may be daydreaming or thinking about things other than driving, causing them to overlook danger. This is known as "intrinsic forward inattention."

[0063] The state determination unit 142 determines that the driver has their eyes closed if the eye opening rate of the driver's left and right eyes, as recognized by the state recognition unit 124, is below a first threshold, and determines that the driver has their eyes open if it is above the first threshold. By using the eye opening rates of both the left and right eyes for the determination, it is possible to determine whether or not the driver is dozing off in subsequent processing more accurately. If only the eye opening rate of one eye can be recognized from the image captured by the in-vehicle camera 70, the determination of whether or not the driver has their eyes closed may be made using only the eye opening rate of that one eye. This makes it possible to determine whether or not the driver has their eyes closed even if only one eye is captured because the driver has turned to the side. In the following explanation, "eyes closed" means that both the left and right eyes are closed. In the following explanation, the driver state determination method will be described in several patterns.

[0064] <First judgment pattern> In the first determination pattern, the state determination unit 142 determines that the driver's state is the second state (e.g., drowsiness) if the driver has their eyes closed continuously and the duration of driving the vehicle M by the driver, as recognized by the behavior recognition unit 126, is the first predetermined time or longer. "The driver has their eyes closed continuously" means, for example, that the state determined by the state determination unit 142 to be the driver having their eyes closed continues for the second predetermined time or longer. Furthermore, in the first determination pattern, the state determination unit 142 may determine that the driver's state is the first state (e.g., distracted) or not the second state if the duration of driving the vehicle M by the driver is less than the first predetermined time. The first predetermined time may also be adjusted according to the vehicle M's speed and surrounding conditions (e.g., distance to the preceding vehicle). In this case, the higher the speed or the shorter the distance between vehicles, the smaller the first predetermined time.

[0065] <Second judgment pattern> In the second determination pattern, the state determination unit 142 determines that the driver's state is the second state if the driver has their eyes closed continuously and the driving distance recognized by the behavior recognition unit 126 is equal to or greater than the first predetermined distance, and determines that the driver's state is the first state or not the second state if it is less than the first predetermined distance.

[0066] <Third Judgment Pattern> In the third determination pattern, the state determination unit 142 determines that the driver's state is the first state if the driver keeps their eyes closed continuously and the speed at which the driver closes their eyes (eye-closing speed) is equal to or greater than the first predetermined speed, and determines that the driver's state is the second state if the eye-closing speed is less than the first predetermined speed. The eye-closing speed may be derived, for example, based on the change in the eye-opening rate during the third predetermined time, or it may be derived based on the change in the distance between the second feature point P2 and the third feature point P3 shown in Figures 3 and 4 during the third predetermined time (d1-d2). The difference distance is the difference (d1-d2) between the distance d1 between the second feature point P2 and the third feature point P3 at a certain time and the distance d2 between the second feature point P2 and the third feature point P3 at a time three predetermined times after that time. The third predetermined time is, for example, a short time less than or equal to the time it takes to go from an open eye state to a closed eye state. The first predetermined speed may be a fixed speed based on the average eye-closing speed at which drowsiness generally begins, or it may be a speed based on the eye-closing speed during blinking for each driver while driving. This allows for a more accurate determination of the driver's condition using the eye-closing speed.

[0067] <Fourth Judgment Pattern> In the fourth determination pattern, the state determination unit 142 determines that the driver's state is the first state (or not the second state) if the driver keeps their eyes closed continuously, the eye-closing speed is equal to or greater than the first predetermined speed, and the driver's mouth is closed. A closed mouth means, for example, that the opening ratio of the driver's mouth, as recognized by the state recognition unit 124, is less than the second threshold. The state determination unit 142 also determines that the driver's state is the second state (or not the first state) if the driver keeps their eyes closed continuously, the eye-closing speed is less than the first predetermined speed, and the driver's mouth is open. An open mouth means that the opening ratio of the driver's mouth, as recognized by the state recognition unit 124, is equal to or greater than the third threshold. The third threshold may be the same value as the second threshold. The second and third thresholds may be fixed values ​​or variable values ​​depending on the position and shape of each driver's mouth.

[0068] <Fifth Judgment Pattern> In the fifth determination pattern, the state determination unit 142 determines that the driver's state is the second state (or not the first state) if the driver has their eyes closed continuously and the behavior recognition unit 126 recognizes that the behavior of the vehicle M is unstable. This allows for a more appropriate determination of the driver's state using not only image information but also vehicle M behavior information.

[0069] <6th Judgment Pattern> In the sixth determination pattern, the state determination unit 142 determines that the driver's state is the first state (or not the second state) if the driver keeps their eyes closed continuously, the driving duration of the vehicle M is less than the first predetermined time, the driver's drowsiness level is less than a predetermined value, and the behavior of the vehicle M is stable. The driver's drowsiness level is an index value indicating the degree of drowsiness; for example, the higher the drowsiness level, the higher the value. The drowsiness level is estimated, for example, by how the eyes are closed, with a slower eye-closing speed resulting in a higher drowsiness level. The state determination unit 142 may also obtain a drowsiness level with a lower value the fewer times the eyes blink during the fourth predetermined time obtained from the images captured by the in-vehicle camera 70. Furthermore, the drowsiness level may also be estimated by the driver's mouth movements. In this case, the state determination unit 142 obtains a drowsiness level with a higher value the more times the driver yawns during the fifth predetermined time obtained from the images captured by the in-vehicle camera 70.

[0070] <7th Judgment Pattern> In the seventh determination pattern, the state determination unit 142 determines that the driver's state is the second state if the driver has their eyes closed continuously and the audio input by the microphone 36 of the HMI 30 includes a specific sound. The specific sound is, for example, a sound related to breathing or snoring. This allows for a more accurate determination of whether the driver is in the second state (drowsy) using audio information as well as image information. If the image captured by the in-vehicle camera 70 determines that there is an occupant other than the driver (passenger) in the vehicle M, the state determination unit 142 may choose not to perform the determination process according to the seventh determination pattern. This prevents the driver's state from being determined by audio (snoring, etc.) emitted by a passenger, thus allowing for a more appropriate determination of the driver's state.

[0071] <8th Judgment Pattern> In the eighth determination pattern, the state determination unit 142 determines that the driver's state is the second state if the driver has their eyes closed continuously and the elapsed time since the switch from driving control by the driving control unit 180 to manual driving by the driver is less than the sixth predetermined time, and determines that the driver's state is the first state if the elapsed time is the sixth predetermined time or longer. For example, if the driver fell asleep while the vehicle M was under driving control (e.g., fully automated driving), and then the system switched to manual driving, the driver may not have fully recovered from drowsiness, and subsequent continuous eye closures strongly suggest that the driver is still asleep. Therefore, by including information on the status and history of the vehicle M's driving control by the driving control unit 180, in addition to images, the driver's state can be determined more appropriately.

[0072] <9th Judgment Pattern> Furthermore, the state determination unit 142 determines that the driver's state is the third state (for example, distracted) if the driver is not continuously closing their eyes (i.e., their eyes are open) and the behavior recognition unit 126 recognizes that the behavior of the vehicle M is unstable. The state determination unit 142 may also determine that the driver's state is the third state if the driver is not continuously closing their eyes and the vehicle M has deviated from the driving lane for a seventh predetermined time or longer. The state determination unit 142 may also determine that the driver's state is the third state if the driver is not continuously closing their eyes and the driver's gaze has not changed for an eighth predetermined time or longer. The seventh and eighth predetermined times may be fixed times or variable times depending on the driver and driving conditions.

[0073] Furthermore, each of the first to ninth determination patterns described above may be combined with at least some or more of the other determination patterns, and some of the processing may be replaced with the processing of other determination patterns. When multiple determination patterns are combined, the state determination unit 142 may determine the driver's state if the determination conditions of at least one of the multiple determination patterns are met, or it may determine the driver's state if all of the determination conditions are met. In addition, in this embodiment, the state determination unit 142 may select a determination pattern according to the condition of the vehicle M. In this case, for example, a predetermined determination pattern is set according to the time of day the driver is driving (daytime or nighttime, etc.), the weather in the surrounding area, whether there are passengers or not, the road conditions, etc. The state determination unit 142 may also pre-set priorities for the first to ninth determination patterns and determine the driver's state using the determination pattern with the highest execution priority. Furthermore, the state determination unit 142 may select the number and types of determination patterns to combine based on the driver's age (or whether they are elderly or not), the number of times they have fallen asleep, looked away, or driven carelessly in the past, etc. This allows for a more appropriate assessment method to determine the driver's condition, depending on the individual driver.

[0074] [Content of the report] Next, the notification content corresponding to the driver's state will be described. For example, if the state determination unit 142 determines that the driver's state is a first state (e.g., distracted driving), the HMI control unit 160 generates distracted driving notification information (first notification information) and outputs the generated distracted driving notification information to the HMI 30 (an example of the first notification). The distracted driving notification information may be, for example, information indicating that the driver has been determined to be distracted, or it may be information urging the driver not to be distracted (or to pay attention to the surroundings). The distracted driving notification information includes at least one of an image and sound (warning sound).

[0075] Furthermore, if the state determination unit 142 determines that the driver's state is a second state (for example, drowsiness), the HMI control unit 160 generates drowsiness notification information (second notification information) that is different from the distraction notification information, and outputs the generated drowsiness notification information to the HMI 30 (an example of second notification). Drowsiness notification information may be, for example, information indicating that the driver has been determined to be drowsy, or it may be information urging the driver not to fall asleep (or to wake up). Drowsiness notification information includes at least one of an image and sound (warning sound). Note that drowsiness notification information may have a higher degree of notification (warning degree) than distraction notification information. In this case, for example, if the distraction notification information is either an image or sound, the HMI control unit 160 will include both an image and sound in the drowsiness notification information. Furthermore, the HMI control unit 160 increases the output volume of the drowsiness warning information compared to the distraction warning information, makes the drowsiness warning information sound more easily noticeable to the driver, and displays the drowsiness warning information image in a more easily noticeable color and font than the distraction warning information. This makes it easier for the driver to recognize the importance of the drowsiness warning.

[0076] Furthermore, the HMI control unit 160 may, after outputting drowsiness warning information from the HMI 30, generate rest reminder information to prompt the driver to take a break and output the generated rest reminder information to the HMI 30. The rest reminder information includes at least one of an image and / or sound. In addition to information prompting a break, the rest reminder information may also include driving information such as driving duration and driving distance, and may also include information about nearby rest spots obtained from map information 192 based on the vehicle M's location information. The rest reminder information may also be a strong warning information commanding a break. Since continuing to drive is inappropriate when drowsy, the driver can be immediately prompted to take a break.

[0077] Furthermore, if the state determination unit 142 determines that the driver's state is a third state (for example, distracted), the HMI control unit 160 generates distracted driver notification information (third notification information) that is different from the distracted driver notification information and the drowsy driver notification information, and outputs the generated distracted driver notification information to the HMI 30 (an example of the third notification). The distracted driver notification information may be information indicating that the driver has been determined to be driving distractedly, or it may be information urging the driver not to drive distractedly. The distracted driver notification information is information with a lower degree of notification than the drowsy driver notification information. The first notification, second notification, and third notification described above are all different notifications.

[0078] The driving control unit 180 may execute predetermined driving control if the driver's state remains the same (no change in state) even after the HMI control unit 160 has output the above-mentioned distracted driving warning information, drowsiness warning information (rest reminder information), and inattentive driving warning information. For example, if the driver's state remains in state 2 even after the 9th predetermined time has elapsed since the HMI control unit 160 outputted the drowsiness warning information, the driving control unit 180 will execute driving control to move the vehicle M to a safe position and stop it. Also, if the driver's state remains in state 1 or 3 even after the 10th predetermined time has elapsed since the HMI control unit 160 outputted the distracted driving warning information or inattentive driving warning information, the driving control unit 180 may perform driving control to move the vehicle M to a safe position and stop it, or it may perform driving control such as ACC or LKAS, predicting that the distracted driving will improve in the near future. Note that the above-mentioned 9th predetermined time is shorter than the 10th predetermined time. In this way, by performing driving control based on the driver's state after the notification information is received, more appropriate driving control can be performed according to the driver's state.

[0079] [Processing flow] The following describes the processes performed by the driver assistance device 100 of this embodiment. The following description will primarily focus on processes performed by the driver assistance device 100 that include determining the driver's condition and providing notification according to that condition.

[0080] Figure 5 is a flowchart showing an example of processing performed by the driver assistance device 100 in the embodiment. In the example in Figure 5, the state recognition unit 124 recognizes the driver's state (step S100). Next, the behavior recognition unit 126 recognizes the behavior of the vehicle M (step S110). Next, the state determination unit 142 determines whether the driver has been continuously closing their eyes (step S120). If it is determined that the driver has been continuously closing their eyes, the state determination unit 142 determines whether the driver's driving duration is longer than or equal to a predetermined time (first predetermined time) (step S130). If it is determined that the driving duration is not longer than or equal to the predetermined time, the state determination unit 142 determines whether the driver's drowsiness level is greater than or equal to a predetermined value (step S140). If it is determined that the drowsiness level is not greater than or equal to a predetermined value, the state determination unit 142 determines whether the behavior of the vehicle M is stable. If it is determined that the behavior of the vehicle M is stable, it determines that the driver's state is the first information (distracted driving) (step S160). Next, the HMI control unit 160 causes the HMI 30 to output distraction notification information corresponding to the first state (performs the first notification) (step S170).

[0081] Furthermore, if it is determined in step S130 that the driver's driving time is longer than a predetermined time, if it is determined in step S140 that the drowsiness level is longer than a predetermined value, or if it is determined in step S150 that the behavior of vehicle M is unstable, the HMI control unit 160 determines that the driver's state is the second state (drowsiness) (step S180). Next, the HMI control unit 160 outputs drowsiness notification information corresponding to the second state to the HMI 30 (performs a second notification) (step S190). Next, the HMI control unit 160 outputs rest reminder information to the HMI 30 (step S200). This completes the processing of this flowchart.

[0082] Furthermore, if it is determined in step S120 that the driver is not continuously closing their eyes, the process in this flowchart terminates.

[0083] Note that the process shown in Figure 5 is merely an example of an embodiment, and while it shows the process based on the sixth determination pattern among the first to ninth determination patterns described above, processing using other determination patterns may also be performed. Furthermore, each determination pattern may be combined with a part of another determination pattern, or some processing may be replaced with the processing of another determination pattern. For example, in the process of step S120, if it is determined that the driver is not continuously closing their eyes (they are open), the state determination unit 142 may perform a determination using the ninth determination pattern described above and determine whether the driver's state is distracted or not based on the determination result. Also, if the HMI control unit 160 determines that the driver's state is distracted, it may generate distracted driving notification information and output it to the HMI 30. Furthermore, in the process of step S130, the second determination pattern may be applied to determine whether the driving distance is greater than or equal to the first predetermined distance. Also, of the processes in Figure 5, at least one of the processes of step S140 and step S150 may be omitted.

[0084] According to the embodiment described above, the driver status determination device includes a determination unit 140 that determines the status of the driver of a vehicle M (an example of a moving object), and a notification unit (HMI control unit 160, HMI 30) that notifies the driver based on the determination result of the determination unit. The notification unit provides a first notification when the determination unit 140 determines that the driver's status is in a first status, and provides a second notification different from the first notification when the occupant's status is determined to be in a second status different from the first status, thereby enabling a more appropriate determination of the driver's status.

[0085] For example, according to the embodiment, even if the driver keeps their eyes closed continuously, drowsiness and distraction can be determined separately, and different notifications (warnings) can be output according to each. Also, according to the embodiment, when the driver is drowsy, a warning is issued, followed by a notification to encourage a break, thereby discouraging continued driving while drowsy and allowing the driver to take a break earlier. Furthermore, according to the embodiment, based on the first to ninth determination patterns described above, the driver's driving state, such as distraction, drowsiness, and inattentiveness, can be determined more accurately not only from images but also from voice, vehicle behavior, and the execution status of driving control.

[0086] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Determine the status of the driver of the moving vehicle, Based on the results of the determination, the driver will be notified. If it is determined that the driver's condition is in the first condition, the first notification is given. If it is determined that the driver's condition is a second condition different from the first condition, a second notification different from the first notification is given. Driver status determination device.

[0087] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0088] 1...Vehicle system, 10...Camera, 12...Radar device, 14...LIDAR, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 70...In-cabin camera, 80...Driver's control panel, 100...Driver's assistance device, 120...Recognition unit, 122...Surroundings recognition unit, 124...State recognition unit, 126...Behavior recognition unit, 140...Determination unit, 142...State determination unit, 160...HMI control unit, 180...Driving control unit, 190...Storage unit, 200...Driving force output device, 210...Brake device, 220...Steering device, M...Vehicle

Claims

1. A determination unit that determines the status of the driver of the moving vehicle, The system includes a notification unit that notifies the driver based on the determination result of the determination unit, The notification unit provides a first notification when the determination unit determines that the driver's state is a first state, and provides a second notification different from the first notification when the determination unit determines that the driver's state is a second state different from the first state. Driver status determination device.

2. The first state is distraction, and the second state is dozing off. The driver status determination device according to claim 1.

3. If the determination unit determines that the driver's condition is the second condition, the notification unit will, after issuing a notification corresponding to the drowsiness, issue a notification prompting the driver to take a break. The driver status determination device according to claim 2.

4. The determination unit determines that the driver's state is the second state if the driver keeps their eyes closed continuously and the driver has been operating the mobile body for a predetermined time or longer. The driver status determination device according to claim 1.

5. The determination unit determines that the driver has their eyes closed continuously, and If the speed at which the driver closes their eyes is equal to or greater than a predetermined speed, it is determined that the driver's state is the first state. If the speed at which the eyes are closed is less than a predetermined speed, the driver's condition is determined to be the second condition. The driver status determination device according to claim 1.

6. The determination unit determines that the driver has their eyes closed continuously, If the speed at which the driver closes their eyes is equal to or greater than a predetermined speed, and the driver's mouth is closed, it is determined that the driver's state is the first state. The driver's condition is determined to be the second state if the speed at which the eyes are closed is less than a predetermined speed and the driver's mouth is open. The driver status determination device according to claim 1.

7. The determination unit determines that the driver's state is the second state when the driver's eyes are continuously closed and the behavior of the moving object is unstable. The driver status determination device according to claim 1.

8. The determination unit determines that the driver has their eyes closed continuously, If the continuous driving time of the mobile body is less than a predetermined time, the driver's drowsiness level is less than a predetermined value, and the behavior of the mobile body is determined to be stable, then the driver's state is determined to be the first state. The driver status determination device according to claim 1.

9. Computers Determine the status of the driver of the moving vehicle, Based on the results of the determination, the driver will be notified. If it is determined that the driver's condition is in the first condition, the first notification is given. If it is determined that the driver's state is a second state different from the first state, a second notification different from the first notification is given. Driver status determination method.

10. On the computer, Determine the status of the driver of the moving vehicle. Based on the determined result, the driver will be notified. If it is determined that the driver's condition is in the first condition, the first notification is made. If it is determined that the driver's state is a second state different from the first state, a second notification different from the first notification is issued. program.

Images