Driver status detection device, driver status detection method, and program

Abstract

To more accurately detect the driver's condition. [Solution] The driver state detection device of the embodiment includes a recognition unit that recognizes at least one of the gaze or face direction of the driver of a moving object, and a determination unit that determines whether the driver is driving while distracted based on the recognition result of the recognition unit and a distraction determination condition. The determination unit makes a determination of distracted driving using a first distraction determination condition when the turning of the moving object is not predicted or the moving object is not turning, and makes a determination of distracted driving using a second distraction determination condition when the turning of the moving object is predicted or the moving object is turning. When the determination unit determines whether the moving object is turning, it changes the turning determination condition for determining whether the moving object is turning depending on whether the turning direction of the moving object is left or right.

Description

Technical Field

[0001] The present invention relates to a driver state detection device, a driver state detection method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development focusing on further improving traffic safety and convenience through research and development on preventive safety technologies have been carried out. In this regard, conventionally, first detection information indicating the driver's line of sight or face direction is acquired, and based on the driver's line of sight or face direction indicated by the first detection information and a first determination condition set for determining glance driving, it is determined whether the driver is glancing, second detection information indicating the rotational angular velocity around the vertical axis of the vehicle or the lateral acceleration is acquired, and during a period when a rotational angular velocity or lateral acceleration that does not satisfy a preset criterion is detected, the first determination condition is changed to a second determination condition different from the first determination condition (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in conventional preventive safety technologies, when the moving body wobbles left and right due to driving or when the moving body is traveling on a curved road or the like, the determination condition for the driver's glance is not changed. Therefore, depending on the moving situation of the moving body, an appropriate glance determination may not be made. Accordingly, there are cases where the driver's state cannot be appropriately detected.

[0005] One of the objectives of this application is to provide a driver status detection device, a driver status detection method, and a program that can more appropriately detect the driver's status according to the movement status of a moving object, in order to solve the above-mentioned problems. Ultimately, this will contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The driver status detection device, driver status detection method, and program according to this invention employ the following configuration. (1) A driver state detection device according to one aspect of the present invention comprises a recognition unit that recognizes at least one of the gaze or face direction of a driver of a moving body, and a determination unit that determines whether the driver is driving while distracted based on the recognition result of the recognition unit and a distraction determination condition, wherein the determination unit makes a determination of distracted driving using a first distraction determination condition when the turning of the moving body is not predicted or the moving body is not turning, and makes a determination of distracted driving using a second distraction determination condition when the turning of the moving body is predicted or the moving body is turning, and when the determination unit determines whether the moving body is turning, it changes the turning determination condition for determining whether the moving body is turning depending on whether the turning direction of the moving body is left or right.

[0007] (2) In the embodiment of (1) above, the turning includes turning the moving body right or left, and the determination unit sets the turning determination condition such that when the moving body turns right or left without crossing the oncoming lane opposite to the lane in which it is moving, the distraction determination condition is more likely to be changed than when the moving body turns right or left by crossing the oncoming lane.

[0008] (3) In the embodiment of (2) above, the determination unit sets the turning determination condition such that when turning right or left without crossing the oncoming lane, the distraction determination condition is changed when the operation of the direction indicator of the moving body by the driver is detected, and when turning right or left while crossing the oncoming lane, the distraction determination condition is changed when a driving operation that causes lateral movement of the moving body or lateral movement of the moving body is detected.

[0009] (4) In the embodiment of (1) above, whether or not the moving body is rotating is determined when the speed or deceleration of the moving body is less than a threshold.

[0010] (5) In the embodiment of (1) above, the determination unit changes the turning determination condition based on the reliability of the recognition result by the recognition unit.

[0011] (6) In the embodiment of (5) above, the determination unit relaxes the distraction determination condition earlier when the confidence level is less than the threshold, compared to when the confidence level is above the threshold.

[0012] (7) In the embodiment of (1) above, the determination unit relaxes the distraction determination condition if it determines that there is an object in the passenger seat of the moving body that obstructs the driver's view, or that the external environment makes it difficult for the driver to see.

[0013] (8) In the embodiment of (7) above, the object includes an occupant, and the determination unit does not relax the distraction determination condition when the occupant is in the passenger seat of the moving body and the inclination angle of the passenger seat is greater than or equal to a predetermined angle.

[0014] (9) In the embodiment of (1) above, the determination unit relaxes the distraction determination condition when the amount of change in the driver's posture is greater than a predetermined amount.

[0015] (10): In the embodiment of (1) above, the device further includes an alarm control unit that outputs an alarm when the amount of change in the driver's line of sight recognized by the recognition unit is less than a predetermined amount during the rotational movement of the moving body.

[0016] (11): A driver state detection method according to another aspect of the present invention is a driver state detection method in which a computer recognizes at least one of the gaze or face direction of the driver of a moving object, determines whether the driver is driving while distracted based on the recognition result and a distraction determination condition, makes a determination of distracted driving using a first distraction determination condition when the turning of the moving object is not predicted or the moving object is not turning, makes a determination of distracted driving using a second distraction determination condition when the turning of the moving object is predicted or the moving object is turning, and when determining whether the moving object is turning, changes the turning determination condition for determining whether the moving object is turning depending on whether the turning direction of the moving object is left or right.

[0017] (12): A program according to another aspect of the present invention causes a computer to recognize at least one of the gaze or face direction of the driver of a moving body, to determine whether the driver is driving while distracted based on the recognition result and a distraction determination condition, to determine whether the driver is driving while distracted using a first distraction determination condition when the turning of the moving body is not predicted or the moving body is not turning, to determine whether the driver is driving while distracted using a second distraction determination condition when the turning of the moving body is predicted or the moving body is turning, and to change the turning determination condition for determining whether the moving body is turning or not depending on whether the turning direction of the moving body is left or right. [Effects of the Invention]

[0018] According to the embodiments described in (1) to (12) above, the driver's condition can be detected more appropriately depending on the movement of the moving object. [Brief explanation of the drawing]

[0019] [Figure 1] It is a configuration diagram of a vehicle system 1 including a driver state detection device according to an embodiment. [Figure 2] It is a diagram showing the relationship between the driver's line of sight and the side glance determination condition. [Figure 3] It is a diagram for explaining the changed side glance determination area. [Figure 4] It is a diagram for explaining right / left turns and turning determination. [Figure 5] It is a flowchart showing an example of the processing executed by the driving support device 100 in the embodiment.

Mode for Carrying Out the Invention

[0020] Hereinafter, embodiments of the driver state detection device, the driver state detection method, and the program of the present invention will be described with reference to the drawings. Hereinafter, a vehicle will be used as an example of the moving body. The moving body may include, in addition to the vehicle, for example, ships that can move on the ground (road) like hovercrafts, flying objects that can travel on roads, standing vehicles with power units, and the like. Further, hereinafter, mainly, the case where the left-hand traffic regulation is applied will be mainly described. However, when the right-hand traffic regulation is applied, the left and right may be read in reverse.

[0021] [Overall Configuration] FIG. 1 is a configuration diagram of a vehicle system 1 including a driver state detection device according to an embodiment. The vehicle (hereinafter referred to as vehicle M) on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, four-wheeled vehicle, or micromobility, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using the electric power generated by a generator connected to the internal combustion engine, or the discharge power of a battery (storage battery) such as a secondary battery or a fuel cell.

[0022] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an in-cabin camera 70, a driver control unit 80, a driver assistance device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Note that the configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. A combination of camera 10, radar device 12, and LiDAR 14 is an example of a "detection device DD". HMI 30 is an example of a "warning unit".

[0023] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle M on which the vehicle system 1 is installed. When imaging the front, camera 10 is mounted on the top of the front windshield, behind the rearview mirror, or on the front of the vehicle body. When imaging the rear, camera 10 is mounted on the top of the rear windshield or on the tailgate. When imaging the side, camera 10 is mounted on the door mirror or the like. Camera 10 periodically and repeatedly images the area around vehicle M. Camera 10 may also be a stereo camera.

[0024] The radar device 12 emits radio waves (radar) such as millimeter waves around the vehicle M and detects radio waves (reflected waves) reflected by surrounding objects to detect at least the position (distance and bearing) of an object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0025] The LIDAR 14 illuminates the area around the vehicle M with light and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time from emission to reception. The emitted light is, for example, pulsed laser light. The LIDAR 14 can be mounted at any location on the vehicle M.

[0026] The communication device 20 communicates with other vehicles in the vicinity of vehicle M, terminal devices of users using vehicle M, or various server devices, for example, by utilizing networks such as cellular networks, Wi-Fi networks, Bluetooth®, DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet.

[0027] The HMI 30 outputs various information to the occupants of the vehicle M (including the driver) and accepts input operations from the occupants. The HMI 30 includes, for example, a display unit 32 and a speaker 34. The display unit 32 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit 32 displays various images (including video) in the embodiment. The display unit 32 may be configured integrally with the input unit as a touch panel. The speaker 34 outputs predetermined sounds (for example, alarm sounds). In addition to (or instead of) the display unit 32 and speaker 34, the HMI 30 may also include a microphone, buzzer, touch panel, switch, key, etc.

[0028] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, and a yaw rate sensor for detecting yaw rate (for example, the rotational angular velocity around the vertical axis passing through the center of gravity of the vehicle M). The vehicle sensor 40 may also include a lateral acceleration sensor (lateral G sensor) for detecting the lateral acceleration (lateral G) of the vehicle M, a steering angle sensor for detecting the steering angle of the vehicle M (which may be the angle of the steering wheels or the operating angle of the steering wheel), a steering angular velocity sensor for detecting the steering angular velocity, and a compass sensor for detecting the orientation of the vehicle M.

[0029] Furthermore, the vehicle sensor 40 may include a position sensor that detects the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that acquires position information using, for example, a GNSS (Global Navigation Satellite System) receiver of a navigation device 50. The vehicle sensor 40 may derive the speed of the vehicle M from the difference (i.e., distance) of position information at a predetermined time from the position sensor.

[0030] The vehicle sensor 40 may also include an illuminance sensor for detecting the illuminance (brightness) inside the vehicle and a weather sensor for detecting the weather around the vehicle M. The weather sensor may, for example, detect the humidity and temperature outside the vehicle and predict the weather based on the detection results and the operation status of the vehicle M's wipers, or it may acquire weather information around the vehicle relative to the position of the vehicle M, acquired by a position sensor, etc., from an external device connected via the communication device 20. The vehicle sensor 40 may also include a seat sensor for detecting the position of the seats inside the vehicle M and the tilt angle of the seatback (the angle between the seat and the backrest). The results detected by the vehicle sensor 40 are output to the driver assistance device 100.

[0031] The navigation device 50 includes, for example, a GNSS receiver, a navigation HMI, and a route determination unit. The navigation device 50 may store map information in a storage device such as an HDD (Hard Disk Drive) or flash memory, or it may acquire map information 192 stored in a storage unit 190, which will be described later. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI includes a display device, speaker, touch panel, keys, etc. The GNSS receiver may be provided on the vehicle sensor 40. The navigation HMI may be partially or completely shared with the HMI 30 described above. The route determination unit determines, for example, a route (hereinafter referred to as a route on a map) from the position of the vehicle M determined by the GNSS receiver (or any input location) to a destination input by the occupant using the navigation HMI, by referring to, for example, map information 192. Furthermore, the navigation device 50 provides route guidance using the navigation HMI based on the determined route on the map. The navigation device 50 may also transmit its current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0032] Here, map information 192 is information in which the road shape is represented by links indicating roads (an example of a travel route) and nodes connected by those links. Map information 192 may also include POI (Point of Interest) information, etc. Map information 192 also includes, for example, the number of lanes (number of travel routes), the type and shape of road markings, information on the center of the lanes, or information on road boundaries. Map information 192 may also include information on whether the road boundary is a boundary (physical boundary) that includes structures that vehicles cannot pass through (including crossing and contact). Physical boundaries include, for example, guardrails, curbs, median strips, fences, etc. Map information 192 may also include road shape information, traffic regulation information, address information (address and postal code), facility information, parking information, telephone number information, etc. Road shape information includes, for example, the curvature of the road (which may be rephrased as radius of curvature; the same applies below), width, road surface gradient, branching and merging points, intersections, T-junctions, etc. Map information 192 may be updated as needed by the communication device 20 communicating with an external device.

[0033] The in-vehicle camera 70 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The in-vehicle camera 70 is mounted at any location in the vehicle M in a position and orientation that allows it to capture the head of the driver seated in the driver's seat of the vehicle M from the front (in a direction that captures the face). For example, the in-vehicle camera 70 is mounted above the display device located in the center of the instrument panel of the vehicle M. The in-vehicle camera 70 also captures images of the interior of the vehicle M, including the area where the occupant (passenger) is seated in the front passenger seat of the vehicle M is located.

[0034] The driver control unit 80 includes, for example, a steering wheel, an accelerator pedal, and a brake pedal. The driver control unit 80 may also include a shift lever, a modified steering wheel, a joystick, or other controls. Each control of the driver control unit 80 is equipped with an operation detection unit that detects, for example, the amount of operation performed by the driver on the control or whether or not an operation has been performed. The operation detection unit detects, for example, the steering angle and steering torque of the steering wheel (for example, the amount of steering due to the driver's driving operation (steering input torque)), the rate of change of the steering torque, the amount of depression of the accelerator pedal and brake pedal, etc. The operation detection unit then outputs the detection results to the driver assistance device 100, or to one or both of the driving force output device 200, the brake device 210, and the steering device 220. The driver control unit 80 may also include a turn signal unit (turn signal lever, turn lever). When the turn signal unit is operated, the turn signal of the vehicle M corresponding to the operation will flash, and the operation details (including, for example, the detection result that the operation was performed by the driver) will be output to the driver assistance device 100.

[0035] The driver assistance device 100 performs various controls to assist the driver of the vehicle M. The driver assistance device 100 includes, for example, a recognition unit 120, a determination unit 140, an HMI control unit 160, a driving control unit 180, and a storage unit 190. The recognition unit 120, the determination unit 140, the HMI control unit 160, and the driving control unit 180 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The above-mentioned program may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the driver assistance device 100, or it may be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the driver assistance device 100 when the storage medium (non-transient storage medium) is inserted into a drive device or card slot. The HMI control unit 160 is an example of an "alarm control unit". The recognition unit 120, the determination unit 140, and the HMI control unit 160 are an example of a "driver status detection device".

[0036] The storage unit 190 may be implemented using the various storage devices described above, or an EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 190 stores, for example, map information 192, various information in the embodiment, programs, etc. The storage unit 190 may also store setting information used in the processing in this embodiment.

[0037] The recognition unit 120 includes, for example, a surrounding area recognition unit 122, a state recognition unit 124, a reliability recognition unit 126, and a behavior recognition unit 128.

[0038] The surrounding recognition unit 122 recognizes the surrounding conditions of vehicle M based on, for example, the detection results of the detection device DD (information input from camera 10, radar device 12, and LIDAR 14). For example, the surrounding recognition unit 122 performs sensor fusion processing on some or all of the detection results from camera 10, radar device 12, and LIDAR 14 to recognize the position (relative position), size, speed (relative speed), acceleration, and other states of objects present around vehicle M (within a predetermined distance). Objects recognized by the surrounding recognition unit 122 may include, for example, physical boundaries that demarcate roads (travel paths), as well as other vehicles, pedestrians, bicycles, and other traffic participants (examples of obstacles). The position of an object is recognized as a position on an absolute coordinate system with a representative point of vehicle M (such as the center of gravity or the center of the drive axis) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or by a represented region. The "state" of an object may include, for example, the acceleration or jerk of a moving object such as another vehicle, or its "action state" (for example, whether or not the other vehicle is changing lanes or is about to change lanes).

[0039] Furthermore, the surrounding area recognition unit 122 may recognize, for example, stop lines, red lights, toll booths, other road events, road signs, and markings drawn on the road (for example, speed limits).

[0040] Furthermore, the surrounding recognition unit 122 recognizes, for example, the lane in which the vehicle M is traveling (driving lane) and other surrounding lanes (for example, oncoming lanes). For example, the surrounding recognition unit 122 recognizes road markings from images captured by the camera 10 and recognizes the driving lane and other lanes based on the positional relationship of the road markings as seen from the recognized vehicle M. Alternatively, the surrounding recognition unit 122 may refer to map information 192 based on the position information of the vehicle M obtained from the vehicle sensor 40, etc., to recognize the lane in which the vehicle M is traveling and other lanes.

[0041] The state recognition unit 124 recognizes the state of the occupants of vehicle M using images captured by the in-vehicle camera 70. For example, the state recognition unit 124 performs known image analysis processing on the images captured by the in-vehicle camera 70 and, based on the analysis results, recognizes the driver's gaze (the direction the driver is looking) or the direction of the driver's face.

[0042] For example, the state recognition unit 124 uses methods such as template matching to detect combinations of reference points (the stationary part of the eye) and moving points (the moving part of the eye) of the driver's eyes from the image. Examples of combinations of reference points and moving points include the inner corner of the eye and the iris, and the corneal reflection region and the pupil. The corneal reflection region is the area of ​​infrared light reflection on the cornea when the in-vehicle camera 70 or the like shines infrared light towards the driver. The state recognition unit 124 then performs coordinate transformations from the image plane to real space based on the position of the moving points relative to the reference points to recognize the driver's line of sight. The state recognition unit 124 also recognizes the orientation of the driver's face based on positional information of the eyes, nose, mouth, etc. within the face region obtained from the image analysis results (relative positional information of each part, etc.).

[0043] Furthermore, the state recognition unit 124 may recognize the shape of the driver's eyes from the image analysis results, or recognize whether the driver is wearing sunglasses or glasses through template matching or the like. The state recognition unit 124 may also determine whether the driver is monitoring the surroundings of the vehicle M based on the driver's gaze and the direction of their face. The state recognition unit 124 may also recognize the driver's posture (movement) and the amount of change (amount of action) over a predetermined period of time.

[0044] Furthermore, the state recognition unit 124 may recognize the position, gaze, and face orientation of any occupants other than the driver (for example, a passenger sitting in the front passenger seat) if such occupants are present. In recognizing each piece of information using the images described above, a pre-trained model, for example, that has been trained in advance by machine learning, may be used.

[0045] The reliability recognition unit 126 recognizes the reliability of the recognition result obtained by the state recognition unit 124. For example, if the state recognition unit 124 recognizes the direction of the driver's face but not their gaze, the reliability recognition unit 126 will reduce the reliability of the recognition result compared to when the gaze is also recognized. Also, if the reliability recognition unit 126 recognizes that the driver is wearing sunglasses or glasses, or that their eyes are narrowed, the reliability of the recognition result will be reduced even if the gaze is recognized, because the movement point of the eyes may not be correctly recognized. Furthermore, the state recognition unit 124 may reduce the reliability of the recognition as the illuminance decreases (the darker it is) according to the illuminance obtained by the vehicle sensor 40. In addition, the reliability recognition unit 126 may set the reliability of the recognition result for nighttime hours to be lower than the reliability of the recognition result for daytime hours (hours other than nighttime). Furthermore, the reliability recognition unit 126 may reduce the reliability of the recognition result in certain weather conditions (e.g., fog, rain) based on the weather conditions around the vehicle M obtained by the vehicle sensor 40, because the captured image becomes darker compared to other weather conditions, making it difficult to recognize the gaze and face direction. The reliability recognition unit 126 may also recognize the reliability by combining two or more of the above-described methods.

[0046] The behavior recognition unit 128 recognizes the behavior of vehicle M based on the detection results of the vehicle sensor 40. For example, the behavior recognition unit 128 recognizes the lateral position of vehicle M relative to the driving lane (position in the direction of the lane width) and the attitude (orientation) of vehicle M relative to the direction of extension of the driving lane, based on the positional relationship of vehicle M with respect to the driving lane. For example, the behavior recognition unit 128 may recognize the deviation of the vehicle M's reference point from the center of the lane, and the angle it makes with a line connecting the centers of the lanes in the direction of vehicle M's travel, as the relative position and attitude of vehicle M with respect to the driving lane. Alternatively, the behavior recognition unit 128 may recognize the position of the vehicle M's reference point relative to either the left or right side edge (road marking or road boundary) of the driving lane as the relative position (lateral position) of vehicle M with respect to the driving lane. Furthermore, the behavior recognition unit 128 may recognize the lateral behavior of vehicle M (for example, whether or not it has moved laterally by a predetermined distance or more) from the amount of change in the lateral position and orientation of vehicle M as described above. Furthermore, the behavior recognition unit 128 may recognize the lateral behavior of the vehicle M based on at least one value obtained from the vehicle sensor 40, such as the steering angle, steering angular velocity, and yaw rate of the vehicle M, or from the steering torque and steering torque change rate obtained from the driver control unit 80.

[0047] The determination unit 140 includes, for example, a distraction determination unit 142 and a turning determination unit 144. The distraction determination unit 142 determines whether the driver is driving while distracted, based on at least one of the driver's gaze or face direction recognized by the state recognition unit 124 and a preset distraction determination condition. If it determines that the driver is driving while distracted, the distraction determination unit 142 causes the HMI control unit 160 to output warning information about distracted driving and notifies the occupants, including the driver. The warning information is, for example, distraction warning information such as information notifying that distraction has been detected or information cautioning against distracted driving, and includes at least one of an image or sound (warning sound). In addition, the distraction determination unit 142 changes the distraction determination condition if, for example, the turning determination unit 144 determines that the vehicle M is turning.

[0048] The turning determination unit 144 determines whether or not the vehicle M is turning based on the detection results from the vehicle sensor 40 and the pre-set turning determination conditions. In addition, the turning determination unit 144 may use information obtained from the driver control unit 80 instead of (or in addition to) the detection results from the vehicle sensor 40, or it may use information regarding the behavior of the vehicle M recognized by the behavior recognition unit 128. Turning includes, for example, turning right or left, driving on a curved road, and driving with swaying from side to side. Details of the processing of the distraction determination unit 142 and the turning determination unit 144 will be described later.

[0049] The HMI control unit 160 notifies the occupant of predetermined information via the HMI 30 and receives information input by the HMI 30. The predetermined information includes, for example, information related to the driving of vehicle M, such as information regarding the status of vehicle M and information regarding driving control. Information regarding the status of vehicle M includes, for example, the speed of vehicle M, engine speed, and shift position. Information regarding driving control includes, for example, whether or not driving control is being performed by the driving control unit 180 and information regarding the status of driving control. The predetermined information may also include information regarding the surrounding conditions recognized by the detection device DD. The predetermined information may also include information unrelated to the driving of vehicle M, such as content stored on a storage medium such as a television program or DVD (for example, a movie). The predetermined information may also include, for example, information regarding the current location and destination of vehicle M, and the remaining fuel level of vehicle M. The HMI control unit 160 may output the information received by the HMI 30 to the communication device 20, recognition unit 120, determination unit 140, navigation device 50, etc.

[0050] Furthermore, the HMI control unit 160 may output to the HMI 30 information such as inquiry information for the occupant, recognition results from the recognition unit 120, and determination results from the determination unit 140. In addition, the HMI control unit 160 may transmit various information to be output to the HMI 30 to a terminal device used by the occupant of the vehicle M via the communication device 20.

[0051] The driving control unit 180 performs driving control that controls at least one of the steering or speed of the vehicle M based on the recognition results from the recognition unit 120 and the determination results from the determination unit 140. For example, if the determination unit 140 determines that the driver is distracted and this condition continues for a predetermined time or longer, the driving control unit 180 performs control to stop the vehicle M in a safe position such as the shoulder of the road. The driving control unit 180 may also perform the above-mentioned driving control to avoid contact between the vehicle M and obstacles recognized by the surrounding recognition unit 122. Furthermore, the driving control unit 180 may control at least one of the steering or speed of the vehicle M in response to occupant instructions input from the HMI 30 to perform driving control such as ACC (Adaptive Cruise Control System), LKAS (Lane Keeping Assistance System), or LCA (Lane Change Assist).

[0052] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the driving control unit 180 or information input from the accelerator pedal of the driver control unit 80.

[0053] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 180 or from the brake pedal of the driver control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may be equipped with a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the driving control unit 180 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0054] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 180 or from the steering wheel of the driver control unit 80.

[0055] [Distraction detection unit and turning detection unit] Next, the functions of the distraction detection unit 142 and the turning detection unit 144 will be explained in detail. The distraction detection unit 142 compares the driver's gaze or the direction of their face with the distraction detection area (an example of a distraction detection condition) to determine whether or not the driver is driving while distracted.

[0056] Figure 2 is a diagram illustrating the relationship between the driver's line of sight and the distraction detection conditions. In the example in Figure 2, driver D is seated in the driver's seat ST1 of vehicle M and is performing manual driving of vehicle M by operating the steering wheel SW and other driving controls. In the example in Figure 2, the display units 32-1 and 32-2 and the passenger seat ST2 are shown. For example, when vehicle M is traveling in a straight direction (X-axis direction in the figure), the distraction detection unit 142 sets a distraction detection area AR1 that has a predetermined angle to the left and right with respect to the direction of travel V of vehicle M from the position of driver D's head, as shown in Figure 2. The distraction detection area AR1 is an example of the "first distraction detection condition". The distraction detection area AR1 may be adjusted according to the speed of vehicle M, the shape of the road, etc. In this case, for example, the angle (radians) θ1 indicating the size of the arc of the distraction detection area AR1 may be set to decrease in proportion to the speed, or the angle θ1 may be set to increase as the road width increases.

[0057] In this situation, the distraction detection unit 142 determines, for example, that driver D is not driving distractedly if the gaze of driver D recognized by the state recognition unit 124 is within the angle θ1 of the distraction detection area AR1, and determines that driver D is driving distractedly if the gaze is not within the angle θ1 (or if this state continues for a predetermined time or longer). Alternatively, the distraction detection unit 142 may perform the above-mentioned distraction determination by comparing the orientation of driver D's head, recognized by the state recognition unit 124, with the distraction detection area AR1, instead of (or in addition to) the gaze of driver D. For example, if the state recognition unit 124 could not recognize driver D's gaze (or the confidence level of the gaze was below a threshold), but the orientation of the face was recognized (or the confidence level of the face orientation was above a threshold), the distraction detection unit 142 may perform the distraction determination using the orientation of the face. The case where at least one of driver D's line of sight or face direction is not within the distraction detection area AR1 is an example of "the case where the first distraction detection condition is met."

[0058] If the distraction detection unit 142 determines that the driver is distracted, the HMI control unit 160 generates information indicating that distracted driving has been detected (distraction warning information) and outputs it to the HMI 30. The distraction warning information may be a warning image displayed on at least one of the display units 32-1 or 32-2, or a warning sound output to the speaker 34.

[0059] Furthermore, the distraction detection unit 142 changes the distraction detection area AR1 depending on whether or not the vehicle M is turning. Figure 3 is a diagram illustrating the changed distraction detection area. In the example in Figure 3, the change (transformation) of the distraction detection area is shown when it is determined that the vehicle M is turning to the right, but for the case where it is turning to the left, the following explanation should be read in reverse. For example, if it is determined that the vehicle M is turning (in other words, if the turning determination conditions described later are met), the distraction detection unit 142 changes the distraction detection area AR1 to the distraction detection area AR2 according to the turning direction and degree of turning. The distraction detection area AR2 is an example of the "second distraction detection condition". The distraction detection area AR2 is changed according to the turning direction and degree of turning of the vehicle M. For example, if at least one of the driver D's line of sight or face direction is not in the distraction detection area AR2 during turning, this is an example of "satisfying the second distraction detection condition".

[0060] In the example shown in Figure 3, since vehicle M is turning to the right, the distraction detection unit 142 changes the distraction detection area AR2, which is located at a position rotated to the right by an angle △θ around the position of driver D's head, from the distraction detection area AR1, which is associated with vehicle M when it is moving straight, to the distraction detection area AR2, and performs distraction detection. The angle △θ may be changed according to the degree of turning, and may also be adjusted according to the speed of vehicle M. In the example shown in Figure 3, the distraction detection area AR1 and the distraction detection area AR2 are the same size (in other words, the angle (arc) θ1 indicating the size of the distraction detection area AR1 and the angle θ2 indicating the size of the distraction detection area AR2 are the same), but they may be different sizes (for example, angles θ1 and θ2 are different angles). The angles △θ, θ1, and θ2 described above may be adjusted according to the road shape (e.g., width, curvature, road connection angles at intersections, etc.) on which vehicle M is traveling (or the road shape that is expected to be traveled depending on the degree of turning and the route to the destination).

[0061] For example, suppose driver D's line of sight A1 is in the situation shown in Figure 3. In this case, the distraction detection unit 142 determines that driver D is not distracted if vehicle M is not expected to turn or is not turning, and driver D's line of sight A1 is within the distraction detection area AR1, and determines that driver D is distracted if A1 is not within the distraction detection area AR1. Furthermore, the distraction detection unit 142 determines that driver D is not distracted if vehicle M is expected to turn or is turning, and driver D's line of sight A1 is within the distraction detection area AR2, and determines that driver D is distracted if A1 is not within the distraction detection area AR2. In the example in Figure 3, if vehicle M is not expected to turn or is not turning, driver D is determined to be distracted, and if vehicle M is expected to turn or is turning, driver D is determined to be distracted. In this way, by changing the distraction detection conditions based on whether or not vehicle M is turning (or whether or not a turn is predicted), distraction detection can be performed more appropriately according to the situation of vehicle M.

[0062] Next, the turning determination conditions of the turning determination unit 144 will be explained. For example, when determining whether a vehicle M is turning, the turning determination unit 144 determines whether it is turning or not depending on whether the turning direction is left or right. For example, the turning determination unit 144 determines that a vehicle M is turning if at least one of the steering angle, steering angular velocity, steering torque, and yaw rate of the vehicle M or steering wheel SW detected by the vehicle sensor 40 is greater than or equal to a threshold set in correspondence with each piece of information. The turning determination unit 144 may also determine whether a turning of the vehicle M is predicted (whether the vehicle M will turn in the near future) based on whether predetermined conditions are met. For example, the turning determination unit 144 determines that a turning of the vehicle M is predicted when the driver D operates the turn indicator unit (turn signal lever, turn lever). Furthermore, the turning determination unit 144 may determine that vehicle M is turning if at least one of the following is below a threshold set in association with each piece of information, but the amount of change is increasing (increasing for a predetermined period of time or longer). In addition, the turning determination unit 144 changes the left and right thresholds depending on whether the direction of vehicle M's turn is to the right or to the left. As a result, it is determined that the vehicle is turning at different timings on the left and right, and consequently, the distraction detection conditions are changed at different timings on the left and right.

[0063] Furthermore, if the turning determination unit 144 uses the same threshold value on both the left and right sides to make a determination, and after it determines that the vehicle M is turning or has turned, it controls the system to change the distraction determination conditions after a predetermined time has elapsed or after the vehicle M has traveled a predetermined distance, the predetermined time or predetermined distance may be different on the left and right sides. This case also corresponds to an example of "changing the turning determination conditions."

[0064] For example, the turning determination unit 144 makes it easier to change the distraction determination conditions when the vehicle M makes a left turn without crossing (passing over) an oncoming lane that is opposite to the vehicle's lane, compared to when the vehicle M makes a right or left turn while crossing (passing over) an oncoming lane, among the turning actions in the embodiment (right or left turn of the vehicle M, driving on a curved road, driving with lateral swaying, etc.). Whether or not the vehicle M is crossing an oncoming lane is determined, for example, based on the recognition result of the surrounding recognition unit 122. Furthermore, whether or not a right or left turn is being made may be determined based on the road shape recognized by the surrounding recognition unit 122 and the direction of travel of the vehicle M (driving route to the destination), or it may be determined by whether or not at least one value among the steering angle, steering angular velocity, steering torque, and yaw rate falls within a preset range for right or left turns. These determinations are made, for example, by the turning determination unit 144 or the distraction determination unit 142. "Making it easier to change the distraction detection conditions" can be rephrased as, for example, "changing the turning detection conditions so that it is easier to detect that the vehicle is turning."

[0065] Figure 4 is a diagram illustrating the determination of right and left turns and turns. In the example in Figure 4, road R1 with lanes L1 and L2, road R2 with lanes L3 and L4, road R3 with lanes L5 and L6, and road R4 with lanes L7 and L8 are connected to intersection CR1 in a cross shape. Also in the example in Figure 4, vehicle M is traveling in lane L1 toward intersection CR1 at speed VM. Lane L1 is the driving lane for vehicle M (the path on which the moving object travels), and lane L2 is the opposing lane to lane L1. In the example in Figure 4, lane L3 exists in the direction of extension of lane L1 in the X-axis direction in the figure, and lane L4 exists in the direction of extension of lane L2 in the X-axis direction in the figure. Therefore, on intersection CR1, the area from lane L1 to lane L3 may be called the driving lane, and the area from lane L4 to lane L2 may be called the opposing lane. Furthermore, in the example shown in Figure 4, a pedestrian crossing (CW) and a stop line (SL) exist near each intersection (CR1) of each road R1 to R4.

[0066] In the road conditions shown in Figure 4, vehicle M turning left at intersection CR1 and entering lane L6 is an example of "vehicle M turning left without crossing the oncoming lane," and vehicle M turning right at intersection CR1 and entering lane L7 is an example of "vehicle M turning right while crossing the oncoming lane." Turns are not limited to intersection CR1, but also include entering movable areas such as parking lots and vacant lots along the road. Also, Figure 4 shows the case of driving on a road where left-hand traffic regulations apply, but if right-hand traffic regulations apply, left turns and right turns should be reversed. Furthermore, if the road is a single-lane road, it is possible to turn left or right without crossing the oncoming lane.

[0067] For example, as shown in Figure 4, the turning determination unit 144 modifies the turning determination conditions so that when vehicle M turns left along the travel path K1, the distraction determination conditions are more easily changed from the first distraction determination conditions to the second distraction determination conditions than when vehicle M turns right along the travel path K2. Specifically, for example, the steering angle threshold when determining that vehicle M is turning to the left may be made smaller than the steering angle threshold when turning to the right, and the determination conditions themselves may be made different for left turns and right turns so that it is easier to determine that a left turn is occurring.

[0068] For example, when vehicle M turns without crossing the oncoming lane (turning left at intersection CR1 shown in Figure 4), driver D needs to check the rear and side of vehicle M early on. On the other hand, when turning while crossing the oncoming lane (turning right at intersection CR1 shown in Figure 4), driver D moves vehicle M to near the center of intersection CR1 before entering lane L7 by performing a turning maneuver exceeding a threshold. Until the vehicle M is near the center of intersection CR1, driver D's gaze is directed towards oncoming traffic to check for approaching obstacles such as oncoming vehicles, and then, if it is determined that it is safe to proceed towards lane L7, the driver's gaze moves to check the pedestrian crossing CW ahead or any vehicles ahead. Therefore, it is preferable to change the distraction detection condition according to this action. Thus, since the situation differs when vehicle M turns left or right, in this embodiment, by changing the turning detection condition according to whether vehicle M is turning left or right, the distraction detection condition can be changed at a more appropriate timing, and the driver's state (for example, whether or not they are distracted) can be detected more appropriately. As a result, the occurrence of alarms that seem inappropriate can be suppressed.

[0069] Furthermore, the turning determination unit 144 may set the turning determination conditions so that, for example, when a vehicle M makes a right or left turn without crossing an oncoming lane, the distraction determination conditions are changed when the operation of the turn signal unit by the driver D is detected. When making a right or left turn without crossing an oncoming lane, the distance to the shoulder or sidewalk is short when entering intersection CR1, and the driver D needs to check the rear and side before lateral movement occurs. Therefore, by setting the turning determination conditions so that the distraction determination conditions are changed at the timing when the operation of the turn signal unit is detected, more appropriate distraction determination can be performed.

[0070] Furthermore, the turning determination unit 144 may set the turning determination conditions so that, in the case of a right or left turn that crosses the oncoming lane, the distraction determination conditions are changed when a driving operation that causes lateral movement of the vehicle M or lateral movement of the vehicle M is detected. Lateral movement refers to, for example, movement (lateral movement) of the vehicle M traveling in lane L1 in the lane width direction (Y direction in Figure 4) of a predetermined distance or more, and is obtained from the recognition result of the behavior recognition unit 128. Driving operation refers to, for example, steering operation using the steering wheel SW. As a result, for example, when moving to near the center of intersection CR1, no lateral movement of a predetermined distance or more occurs, so the judgment conditions for straight driving can be applied to perform a distraction determination, and when the actual right turn begins, judgment conditions for right turns can be provided to perform a distraction determination.

[0071] In this embodiment, the distraction detection unit 142 performs the distraction detection when the vehicle M is moving (when the speed VM is greater than 0), but does not need to perform it when the vehicle is stopped. Also, in this embodiment, the turning detection unit 144 may perform the turning detection when the speed VM of the moving vehicle M is less than a threshold (first threshold), or when the amount of deceleration over a predetermined time is less than a threshold (second threshold). This makes it possible to change the distraction detection conditions to be limited to behaviors such as turning by reducing the speed VM in order to make a right or left turn.

[0072] Furthermore, the turning determination unit 144 may change the turning determination conditions based on the reliability recognized by the reliability recognition unit 126. In this case, the turning determination unit 144 makes it easier to determine that a vehicle is turning as the reliability decreases. For example, the turning determination unit 144 may switch the turning determination conditions depending on whether the reliability is below a threshold or above a threshold, and may linearly change the turning determination conditions so that it is easier to determine that a vehicle is turning as the reliability decreases. When the reliability is low, there is a possibility that the correct gaze or face direction cannot be recognized, so there is a higher possibility that the first distracted driving determination condition will be met and the vehicle will be misidentified as distracted driving. Therefore, in situations where the vehicle M's behavior makes the distracted driving warning bothersome, such as during a turn, the bothersomeness can be reduced by switching the turning determination conditions to switch the distracted driving determination condition earlier.

[0073] Furthermore, in the embodiment, the distraction detection conditions may be relaxed if there is an object obstructing the driver D's view in the passenger seat of vehicle M (for example, passenger seat ST2 in Figure 2), or if the external environment makes it difficult for driver D to see. An object obstructing driver D's view may include not only objects that actually obstruct at least a part of the view, but also objects that are not actually obstructing the view but are presumed to potentially obstruct it. Objects include, for example, passengers, pets (animals), luggage, etc. Luggage is defined as luggage larger than a predetermined size. Whether or not the above object is present in the passenger seat of vehicle M is obtained, for example, from images captured by the in-vehicle camera 70. An external environment that makes it difficult for driver D to see includes, for example, when the area around vehicle M is in a weather condition such as fog or rain, or when there are many traffic participants such as pedestrians or cyclists in the surrounding area (more than a predetermined number). Whether or not the external environment makes it difficult for driver D to see may be determined, for example, by the detection results of the vehicle sensor 40 (e.g., a weather sensor), or by the recognition results of the surrounding recognition unit 122.

[0074] Relaxing (easing) the distraction detection criteria means making it more difficult for the system to determine that the distraction detection criteria are met. In this case, the distraction detection unit 142, for example, when there is an occupant in the passenger seat, increases the angles θ1, which indicate the size of the distraction detection area AR1, and θ2, which indicates the size of the distraction detection area AR2, compared to when there is no occupant in the passenger seat. Also, the distraction detection unit 142, for example, when the external environment makes it difficult to see the driver D, increases the angles θ1 and θ2 compared to when the external environment does not make it difficult to see the driver D. In this way, by relaxing the distraction detection criteria in environments where it is expected that the driver will move their field of vision more to see, the annoyance of the distraction warning can be reduced.

[0075] Furthermore, the distraction detection unit 142 may not loosen the distraction detection conditions if there is an occupant (passenger) in the passenger seat of the vehicle M (for example, passenger seat ST2 in Figure 2), which is an example of an object that obstructs the driver D's view, and the inclination angle of the backrest (seat back) obtained from the vehicle sensor 40 (for example, seat sensor) is greater than or equal to a predetermined angle. When the backrest of the passenger seat is inclined greater than or equal to a predetermined angle, the view toward the passenger side as seen from the driver D is not obstructed, and by not changing the distraction detection conditions based on other occupants, distraction detection can be performed more appropriately.

[0076] Furthermore, in this embodiment, the distraction detection unit 142 may relax the distraction detection condition if the amount of change in the driver D's posture (behavior) over a predetermined period of time obtained from the state recognition unit 124 is greater than a predetermined amount. For example, if the driver D is making a large movement that involves moving their entire body, there is a possibility that they are intentionally changing their gaze direction. In this case, relaxing the distraction detection condition (making it less likely to be judged as distracted driving) can reduce the annoyance of the distraction warning.

[0077] Furthermore, in this embodiment, the HMI control unit 160 may output an alarm from the HMI 30 if, during the turning movement of the vehicle M, the movement (amount of change) of the driver D's gaze over a predetermined period of time, as recognized by the state recognition unit 124, is less than a predetermined amount. In the above situation, it is assumed that the driver D is not monitoring the area around the vehicle, so issuing an alarm separately from the distraction detection can provide more appropriate support for the driver D's driving.

[0078] [Processing flow] The following describes the processes performed by the driver assistance device 100 of this embodiment. The following description will primarily focus on the processes related to driver D's distraction detection among those performed by the driver assistance device 100. Figure 5 is a flowchart showing an example of the processes performed by the driver assistance device 100 in this embodiment. Note that the processes shown in Figure 5 may be repeatedly executed at predetermined timings or predetermined cycles.

[0079] In the example shown in Figure 5, the surrounding area recognition unit 122 recognizes the surrounding conditions of vehicle M based on the output from the detection device DD (step S100). Next, the state recognition unit 124 recognizes at least one of the following: the driver's gaze or the direction of their face (step S110). In step S110, the degree to which the recognized driver's gaze or the direction of their face is recognized may also be recognized. Next, the behavior recognition unit 128 recognizes the behavior of vehicle M (step S120).

[0080] Next, the turning determination unit 144 changes the turning determination conditions according to whether the turning direction of vehicle M is left or right (step S130). Next, the distraction determination unit 142 determines whether vehicle M is predicted to turn or is actually turning based on the turning determination conditions (step S140). If it is determined that vehicle M is predicted to turn or is actually turning, the distraction determination unit 142 performs a distraction determination using the second distraction determination conditions described above (step S150). Also, if it is determined in the process of step S140 that vehicle M is not predicted to turn or is not turning, the distraction determination unit 142 performs a distraction determination using the first distraction determination conditions described above (step S160).

[0081] After processing in step S150 or step S160, the distraction detection unit 142 determines whether the driver is distracted (step S170). If it is determined that the driver is distracted, the HMI control unit 160 outputs a warning information regarding distraction (distraction warning information) to the HMI 30 (step S180). This completes the processing of this flowchart. If it is determined in step S170 that the driver is not distracted, the processing of this flowchart also completes.

[0082] [Differentiation] In this embodiment, when the distraction detection unit 142 determines that the driver is driving while distracted, instead of (or in addition to) outputting a warning about distraction, the driving control unit 180 may be instructed to execute driving control that moves the vehicle M to a safe position and stops it. This allows, for example, if the driver is distracted due to feeling unwell, the vehicle M can be stopped in a safe position, thereby more effectively ensuring the driver's safety.

[0083] In addition, in this embodiment, the HMI control unit 160 may output information indicating the distraction detection conditions (for example, the distraction detection area) and information indicating that the distraction detection conditions have been changed from the HMI 30. This allows the driver to be notified of the distraction detection conditions according to the status of the vehicle M, thereby suppressing driver distraction, and as a result, warnings related to distraction can also be suppressed.

[0084] According to the embodiments described above, the driver state detection device includes a recognition unit 120 that recognizes at least one of the driver's gaze or face direction of a moving object, and a determination unit 140 that determines whether the driver is driving while distracted based on the recognition result of the recognition unit 120 and a distraction determination condition. The determination unit 140 uses a first distraction determination condition to determine whether the driver is driving while distracted when a turn of the moving object is not expected or the moving object is not turning, and uses a second distraction determination condition to determine whether the driver is driving while distracted when a turn of the moving object is expected or the moving object is turning. When determining whether the moving object is turning, the determination unit 140 changes the turn determination condition to determine whether the moving object is turning depending on whether the turning direction of the moving object is left or right, thereby enabling the driver's state to be detected more appropriately according to the movement of the moving object.

[0085] For example, according to the embodiment, the distraction detection conditions can be appropriately changed according to the turning direction of the vehicle M, thereby suppressing the occurrence of unnatural warnings. Furthermore, according to the embodiment, for example, while early rear and side checks are necessary for right and left turns that do cross oncoming lanes, when heading towards the center of the intersection, the eye movement will be such that the driver checks the pedestrian crossing at the turn destination if it is determined that it is possible to proceed while checking the oncoming vehicle side. Thus, the timing of the change in the distraction detection conditions can be appropriately controlled according to whether it is a left or right turn.

[0086] Furthermore, according to the embodiment, when turning right or left without crossing the oncoming lane, the distance to the shoulder and sidewalk is short when entering the intersecting road, and the driver may check the rear and side before lateral movement occurs, so the distraction detection conditions can be changed early by changing them according to the operation of the turn signal. Furthermore, according to the embodiment, the distraction detection conditions can be changed to focus on turning behaviors such as turning right or left at reduced speed. Furthermore, according to the embodiment, the timing of switching the distraction detection conditions is changed according to reliability, so distraction detection can be performed more appropriately. Furthermore, according to the embodiment, there is a possibility of misjudgment when the reliability of the recognition result is low, so the hassle can be reduced by switching the distraction detection conditions early in behavior situations such as turning where the distraction warning becomes bothersome. Furthermore, according to the embodiment, the hassle of the warning can be reduced by loosening (relaxing) the distraction detection conditions in environments in which the driver is expected to move their field of vision more to check the surroundings.

[0087] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Recognize at least one of the following: the driver's gaze or the direction of their face. Based on the recognized results and the distraction detection criteria, it is determined whether or not the driver is driving while distracted. If the turning of the moving body is not predicted and the moving body is not turning, the first distraction determination condition is used to determine whether the driver is distracted. If the turning of the moving body is predicted or the moving body is turning, the second distraction determination condition is used to determine whether the driver is distracted. When determining whether the moving body is rotating, the rotation determination condition is changed according to whether the direction of rotation of the moving body is left or right. Driver status detection device.

[0088] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0089] 1...Vehicle system, 10...Camera, 12...Radar device, 14...LIDAR, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 70...In-cabin camera, 80...Driver's control panel, 100...Driver's assistance device, 120...Recognition unit, 122...Surroundings recognition unit, 124...State recognition unit, 126...Reliability recognition unit, 128...Behavior recognition unit, 140...Decision unit, 142...Distraction detection unit, 144...Turning detection unit, 160...HMI control unit, 180...Driving control unit, 190...Memory unit, 200...Driving force output device, 210...Brake device, 220...Steering device, M...Vehicle

Claims

1. A recognition unit that recognizes at least one of the following: the gaze or face direction of the driver of a moving vehicle, The system includes a determination unit that determines whether or not the driver is driving while distracted, based on the recognition result of the recognition unit and the distraction determination conditions. The determination unit, If the turning of the moving body is not predicted and the moving body is not turning, the first distraction determination condition is used to determine whether the driver is distracted. If the turning of the moving body is predicted or the moving body is turning, the second distraction determination condition is used to determine whether the driver is distracted. When the determination unit determines whether the moving body is rotating, it changes the rotation determination condition for determining whether the moving body is rotating, depending on whether the rotation direction of the moving body is left or right. Driver status detection device.

2. The aforementioned turning includes turning the moving body to the right or left, The determination unit sets the turning determination conditions such that when the moving body makes a right or left turn without crossing an oncoming lane opposite to the lane in which it is moving, the distraction determination conditions are more likely to be changed than when the moving body makes a right or left turn while crossing the oncoming lane. The driver status detection device according to claim 1.

3. The determination unit sets the turning determination condition such that, when turning right or left without crossing the oncoming lane, the distraction determination condition is changed when the operation of the direction indicator of the moving body by the driver is detected, and when turning right or left while crossing the oncoming lane, the distraction determination condition is changed when a driving operation that causes lateral movement of the moving body or lateral movement of the moving body is detected. The driver status detection device according to claim 2.

4. Whether the moving body is rotating is determined when the speed or deceleration of the moving body is below a threshold. The driver status detection device according to claim 1.

5. The determination unit modifies the turning determination conditions based on the reliability of the recognition result by the recognition unit. The driver status detection device according to claim 1.

6. The determination unit, when the confidence level is below the threshold, loosens the distraction determination condition earlier than when the confidence level is above the threshold. The driver status detection device according to claim 5.

7. The determination unit relaxes the distraction detection conditions if it determines that there is an object obstructing the driver's view in the passenger seat of the moving vehicle, or that the external environment makes it difficult for the driver to see. The driver status detection device according to claim 1.

8. The aforementioned object includes the occupants, The determination unit shall not relax the distraction determination condition if the occupant is in the passenger seat of the moving body and the tilt angle of the passenger seat is greater than or equal to a predetermined angle. The driver status detection device according to claim 7.

9. The determination unit relaxes the distraction detection condition if the amount of change in the driver's posture is greater than a predetermined amount. The driver status detection device according to claim 1.

10. The system further includes an alarm control unit that outputs an alarm when the amount of change in the driver's line of sight, as recognized by the recognition unit, is less than a predetermined amount during the rotational movement of the moving body. The driver status detection device according to claim 1.

11. Computers Recognize at least one of the following: the driver's gaze or the direction of their face. Based on the recognized results and the distraction detection criteria, it is determined whether or not the driver is driving while distracted. If the turning of the moving body is not predicted and the moving body is not turning, the first distraction determination condition is used to determine whether the driver is distracted. If the turning of the moving body is predicted or the moving body is turning, the second distraction determination condition is used to determine whether the driver is distracted. When determining whether the moving body is rotating, the rotation determination condition is changed according to whether the direction of rotation of the moving body is left or right. Driver status detection method.

12. On the computer, To recognize at least one of the following: the driver's gaze or the direction of their face. Based on the recognized results and the distraction detection conditions, the system determines whether or not the driver is driving while distracted. If the turning of the moving body is not predicted and the moving body is not turning, the first distraction determination condition is used to determine whether the driver is distracted. If the turning of the moving body is predicted or the moving body is turning, the second distraction determination condition is used to determine whether the driver is distracted. When determining whether the moving body is rotating, the rotation determination condition is changed according to whether the direction of rotation of the moving body is left or right. program.

Images