Driver status detection device, driver status detection method, and program

Abstract

To more accurately detect the driver's condition. [Solution] The driver state detection device of the embodiment includes a recognition unit that recognizes at least one of the gaze and face direction of the driver of a moving object; a determination unit that determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit; and a lighting control unit that controls the lighting of the turn signal in response to the driver's operation of an operator that lights up the turn signal of the moving object, and controls the lighting to continue even if the operator returns to the neutral position, wherein the determination unit is less likely to determine that the driver's monitoring direction is inappropriate compared to before the operator was operated, until predetermined conditions are met after the operator returns to the neutral position.

Description

Technical Field

[0001] The present invention relates to a driver state detection device, a driver state detection method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development focusing on further improving traffic safety and convenience through research and development of preventive safety technologies has been carried out. In this context, conventionally, first detection information indicating the driver's line of sight or face direction is acquired as a change pattern, and when the change pattern corresponds to a reference pattern preset according to the driver's lane confirmation operation, a technique for temporarily switching the determination conditions set for determining side glances is known (see, for example, Patent Document 1). Furthermore, in the technique of Patent Document 1, it is shown that the determination conditions are temporarily switched during the period when the direction indicator is operating. An operator such as a wiper lever for operating the direction indicator is maintained in a state of having moved to the position indicated by the operation when operated by the driver from the neutral position (neutral position), and the direction indicator corresponding to the indicated direction is lit. Thereafter, when a predetermined steering operation or the like is executed by operating the steering wheel, it returns to the neutral position again and ends the operation (lighting) of the direction indicator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Incidentally, in recent years, there are mobile devices that have a function (so-called one-touch turn signal function) in which, after the driver releases the control unit, the control unit automatically returns to the neutral position, and the turn signal continues to illuminate a predetermined number of times or for a predetermined time after returning. The driver uses this function to notify those around them of the lane change destination and to change lanes, including after the indicator light has illuminated. However, with conventional technology, because the operation (illumination) of the turn signal ends before the lane change is completed due to such a function, the judgment conditions also revert to the original state, and the driver's gaze and head movements during the subsequent lane change may be judged as distracted driving. Thus, conventional technology sometimes could not properly detect the driver's state.

[0005] One of the objectives of this application is to provide a driver status detection device, a driver status detection method, and a program that can more appropriately detect the driver's condition in order to solve the above-mentioned problems. Ultimately, this will contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The driver status detection device, driver status detection method, and program according to this invention employ the following configuration. (1) A driver state detection device according to one aspect of the present invention comprises: a recognition unit that recognizes at least one of the gaze and face direction of the driver of a moving object; a determination unit that determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit; and a lighting control unit that controls the lighting of the turn signal in response to the driver's operation of an operator that lights up the turn signal of the moving object, and controls the lighting to continue even if the operator returns to the neutral position, wherein the determination unit is less likely to determine that the driver's monitoring direction is inappropriate compared to before the operator was operated, from the time the operator returns to the neutral position until predetermined conditions are met.

[0007] (2) In the embodiment of (1) above, the predetermined conditions include a predetermined time elapsed since the operator was operated or since the operator returned to the neutral position, or the moving body moving a predetermined distance or more.

[0008] (3) In the embodiment of (1) above, the predetermined conditions include the fact that a predetermined time has elapsed since the turn signal stopped lighting up, or that the moving body has moved a predetermined distance or more.

[0009] (4) In the embodiment of (2) or (3) above, if the determination unit determines that the direction of the driver's monitoring is inappropriate before the predetermined time has elapsed or when the movement is less than the predetermined distance, if the movement of the moving body or the operation that causes the movement occurs, it will be less likely to determine that the direction of the driver's monitoring is inappropriate until the movement of the moving body or the operation that causes the movement is completed, even if the predetermined time has elapsed or the body has moved more than the predetermined distance.

[0010] (5) In the embodiment of (2) or (3) above, if the determination unit determines that the driver's monitoring direction is inappropriate when the moving body moves laterally or when an operation that causes the moving body moves laterally occurs before the predetermined time has elapsed or when the moving body has moved more than the predetermined distance, even if the predetermined time has elapsed or the moving body has moved more than the predetermined distance, until the moving body crosses the lane markings that define the lane in which it is moving, or until the lane markings to the left and right of the moving body are recognized after the moving body moves laterally.

[0011] (6) In the embodiment of (1) above, the operation of the operator initiates the lane change of the moving body, and the predetermined condition includes the completion of the lane change.

[0012] (7) In the embodiment of (1) above, the predetermined conditions include that the moving body is moving on a designated private road or that the moving body is moving at a predetermined speed or higher.

[0013] (8) In the embodiment of (1) above, the predetermined conditions include a first predetermined condition used when the moving body is moving on a designated private road or when the moving body is moving at a predetermined speed or higher, and a second predetermined condition used when the moving body is moving on a road other than the designated private road or when the moving body is moving at a speed less than the predetermined speed, wherein the first predetermined condition is less likely to be satisfied than the second predetermined condition.

[0014] (9): A driver state detection method according to another aspect of the present invention is a driver state detection method in which a computer recognizes at least one of the gaze and face orientation of the driver of a moving object, determines whether the driver's monitoring direction is appropriate based on the recognized result, controls the illumination of the turn signals in response to the driver's operation of an operator for illuminating the turn signals of the moving object, controls the lights to continue to illuminate even if the operator returns to the neutral position, and makes it less likely to determine that the driver's monitoring direction is inappropriate compared to before the operator was operated, from the time the operator returns to the neutral position until predetermined conditions are met.

[0015] (10): A program according to another aspect of the present invention causes a computer to recognize at least one of the gaze and face orientation of the driver of a moving body, to determine whether the driver's direction of observation is appropriate based on the recognized result, to control the illumination of the turn signals of the moving body in response to the driver's operation of an operator for illuminating the turn signals of the moving body, to control the illumination to continue even if the operator returns to the neutral position, and to make it less likely that the computer will determine that the driver's direction of observation is inappropriate compared to before the operator was operated, from the time the operator returns to the neutral position until predetermined conditions are met. [Effects of the Invention]

[0016] According to the embodiments described in (1) to (10) above, the driver's condition can be detected more appropriately. [Brief explanation of the drawing]

[0017] [Figure 1] It is a configuration diagram of a vehicle system 1 including a driver state detection device according to an embodiment. [Figure 2] It is a diagram showing the relationship between the driver's line of sight (face direction) and the side glance determination condition. [Figure 3] It is a diagram for explaining the changed side glance determination area. [Figure 4] It is a diagram for explaining the operation position of the wiper lever 84. [Figure 5] It is a diagram for explaining the wiper operation and determination process in the first embodiment. [Figure 6] It is a diagram for explaining the wiper operation and determination process in the second embodiment. [Figure 7] It is a flowchart showing an example of the process executed by the driving support device 100 in the embodiment. [Figure 8] It is a flowchart showing an example of the side glance determination process.

Embodiments for Carrying Out the Invention

[0018] Hereinafter, embodiments of the driver state detection device, driver state detection method, and program of the present invention will be described with reference to the drawings. Hereinafter, a vehicle will be used as an example of a moving body. The moving body may include, in addition to vehicles, for example, ships that can move on the ground (roads) such as hovercrafts, flying bodies that can travel on roads, and standing vehicles with power units.

[0019] [Overall Configuration] FIG. 1 is a configuration diagram of a vehicle system 1 including a driver state detection device according to an embodiment. The vehicle (hereinafter referred to as vehicle M) on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, four-wheeled vehicle or micromobility, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using the electric power generated by a generator connected to the internal combustion engine or the discharge power of a battery (storage battery) such as a secondary battery or a fuel cell.

[0020] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an in-cabin camera 60, a turn signal device 70, a driver control unit 80, a driver assistance device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Note that the configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The combination of camera 10, radar device 12, and LiDAR 14 is an example of a "detection device DD". HMI 30 is an example of a "warning unit". Turn signal device 70 is an example of a "direction indicator".

[0021] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle M on which the vehicle system 1 is installed. When imaging the front, camera 10 is mounted on the top of the front windshield, behind the rearview mirror, or on the front of the vehicle body. When imaging the rear, camera 10 is mounted on the top of the rear windshield or on the tailgate. When imaging the side, camera 10 is mounted on the door mirror or the like. Camera 10 periodically and repeatedly images the area around vehicle M. Camera 10 may also be a stereo camera.

[0022] The radar device 12 emits radio waves (radar) such as millimeter waves around the vehicle M and detects radio waves (reflected waves) reflected by surrounding objects to detect at least the position (distance and bearing) of an object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0023] The LIDAR 14 illuminates the area around the vehicle M with light and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time from emission to reception. The emitted light is, for example, pulsed laser light. The LIDAR 14 can be mounted at any location on the vehicle M.

[0024] The communication device 20 communicates with other vehicles in the vicinity of vehicle M, terminal devices of users using vehicle M, or various server devices, for example, by utilizing networks such as cellular networks, Wi-Fi networks, Bluetooth®, DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet.

[0025] The HMI 30 outputs various information to the occupants of the vehicle M (including the driver) and accepts input operations from the occupants. The HMI 30 includes, for example, a display unit 32 and a speaker 34. The display unit 32 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit 32 displays various images (including video) in the embodiment. The display unit 32 may be configured integrally with the input unit as a touch panel. The speaker 34 outputs predetermined sounds (for example, alarm sounds). In addition to (or instead of) the display unit 32 and speaker 34, the HMI 30 may also include a microphone, buzzer, touch panel, switch, key, etc.

[0026] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, and a yaw rate sensor for detecting yaw rate (for example, the angular velocity of rotation around the vertical axis passing through the center of gravity of the vehicle M). The vehicle sensor 40 may also include a lateral acceleration sensor (lateral G sensor) for detecting the lateral velocity (lateral G) of the vehicle M, a steering angle sensor for detecting the steering angle (which may be the angle of the steering wheels or the operating angle of the steering wheel) and steering torque of the vehicle M, a steering angular velocity sensor for detecting the steering angular velocity, and a compass sensor for detecting the orientation of the vehicle M.

[0027] The vehicle sensor 40 may also include a position sensor that detects the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that acquires position information using, for example, a GNSS (Global Navigation Satellite System) receiver of a navigation device 50. The vehicle sensor 40 may derive the speed of the vehicle M from the difference (i.e., distance) of position information at a predetermined time from the position sensor. The results detected by the vehicle sensor 40 are output to the driver assistance device 100.

[0028] The navigation device 50 includes, for example, a GNSS receiver, a navigation HMI, and a route determination unit. The navigation device 50 may store map information in a storage device such as an HDD (Hard Disk Drive) or flash memory, or it may acquire map information 162 stored in a storage unit 160, which will be described later. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI includes a display device, speaker, touch panel, keys, etc. The GNSS receiver may be provided on the vehicle sensor 40. The navigation HMI may be partially or completely shared with the HMI 30 described above. The route determination unit determines, for example, a route (hereinafter referred to as a route on the map) from the position of the vehicle M determined by the GNSS receiver (or any input location) to a destination input by the occupant using the navigation HMI, by referring to, for example, map information 162, etc. Furthermore, the navigation device 50 provides route guidance using the navigation HMI based on the determined route on the map. The navigation device 50 may also transmit its current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0029] Here, map information 162 is information in which the road shape is represented by links indicating roads (an example of a travel route) and nodes connected by those links. Map information 162 may also include POI (Point of Interest) information, etc. Map information 162 also includes, for example, the number of lanes (number of travel routes), the type and shape of road markings, information on the center of the lanes, or information on road boundaries. Map information 162 may also include information on whether the road boundary is a boundary (physical boundary) that includes structures that vehicles cannot pass through (including crossing and contact). Physical boundaries include, for example, guardrails, curbs, median strips, fences, etc. Map information 162 may also include road shape information, traffic regulation information, address information (address and postal code), facility information, parking information, telephone number information, etc. Road shape information includes, for example, information on the curvature of the road (which may be rephrased as radius of curvature; the same applies below), width, road surface gradient, branching and merging points, intersections, T-junctions, etc. Map information 162 may be updated as needed by the communication device 20 communicating with an external device.

[0030] The in-vehicle camera 60 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The in-vehicle camera 60 is mounted at any location in the vehicle M in a position and orientation that allows it to capture the head of the driver seated in the driver's seat of the vehicle M from the front (in a direction that captures the face). For example, the in-vehicle camera 60 is mounted above the display device located in the center of the instrument panel of the vehicle M. The in-vehicle camera 60 also captures images of the interior of the vehicle M, including the area where the occupant (passenger) is seated in the front passenger seat of the vehicle M is located.

[0031] The turn signal device 70 controls the operation of the turn signal, such as starting and stopping it, according to information input from the driver assistance device 100 or from the driver control unit 80. A turn signal is a lamp that indicates the direction of a right or left turn or a lane change to the surroundings by lighting up (flashing). When the driver operates the turn signal lever 84, which will be described later, the turn signal device 70 activates the turn signal corresponding to the lever position detected by the lever position detection unit 84a. Furthermore, while the turn signal is activated, the turn signal device 70 outputs information to the driver assistance device 100 indicating that the turn signal is activated.

[0032] The driver control unit 80 includes, for example, a steering wheel 82 and a turn signal lever 84. The turn signal lever 84 is an example of an "operator that illuminates the turn signal of a moving object (vehicle M)". The driver control unit 80 also includes an accelerator pedal, a brake pedal, a shift lever, and other controls. The driver control unit 80 is equipped with a sensor that detects the amount of operation or whether or not an operation is being performed.

[0033] The steering wheel 82 is an example of an "operator that accepts steering input from the driver." The steering wheel 82 does not necessarily have to be ring-shaped; it may take the form of an irregularly shaped steering wheel, a joystick, buttons, etc. A steering grip sensor (not shown) is attached to the steering wheel 82. The steering grip sensor is implemented using a capacitive sensor or the like, and outputs a signal to the driver assistance device 100 that can detect whether or not the occupant (driver) is gripping the steering wheel 82 (meaning making contact with it in a state where force can be applied).

[0034] The turn signal lever 84 is provided with a lever position detection unit 84a. The lever position detection unit 84a detects the position of the turn signal lever 84. Details of the position detection will be described later. In addition, the turn signal lever 84 has a one-touch turn signal function in which it automatically returns to the neutral position when the driver releases their hand after being operated, and the turn signal continues to illuminate a predetermined number of times or for a predetermined time after returning.

[0035] Detection units provided on the accelerator pedal and brake pedal detect the amount of pedal depression, and a detection unit provided on the steering wheel detects the steering angle and steering torque of the steering wheel 82. Each detection unit (including the lever position detection unit 84a) then outputs a detection signal indicating the detection result to the driver assistance device 100, or to one or both of the driving force output device 200, the brake device 210, and the steering device 220.

[0036] The driver assistance device 100 performs various controls to assist the driver of the vehicle M. The driver assistance device 100 includes, for example, a recognition unit 110, a determination unit 120, a lighting control unit 130, an HMI control unit 140, a driving control unit 150, and a storage unit 160. The recognition unit 110, the determination unit 120, the HMI control unit 140, and the driving control unit 150 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The above-mentioned program may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the driver assistance device 100, or it may be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the driver assistance device 100 when the storage medium (non-transient storage medium) is inserted into a drive device or card slot. The HMI control unit 140 is an example of an "alarm control unit". The driver control unit 80, recognition unit 110, determination unit 120, lighting control unit 130, and HMI control unit 140 are an example of a "driver status detection device".

[0037] The storage unit 160 may be implemented using the various storage devices described above, or an EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. For example, the storage unit 160 stores map information 162, various information in this embodiment, programs, etc. The storage unit 160 may also store various setting information used in the processing in this embodiment.

[0038] The recognition unit 110 recognizes the surrounding conditions of vehicle M based on, for example, the detection results of the detection device DD (information input from camera 10, radar device 12, and LIDAR 14). For example, the surrounding recognition unit 122 performs sensor fusion processing on some or all of the detection results from camera 10, radar device 12, and LIDAR 14 to recognize the position (relative position), size, speed (relative speed), acceleration, and other states of objects present around vehicle M (within a predetermined distance). Objects recognized by the recognition unit 110 may include, for example, physical boundaries that demarcate roads (travel paths), as well as other vehicles, pedestrians, bicycles, and other traffic participants (examples of obstacles). The position of an object is recognized as a position on an absolute coordinate system with a representative point of vehicle M (such as the center of gravity or the center of the drive axis) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or by a represented region. The "state" of an object may include, for example, the acceleration or jerk of a moving object such as another vehicle, or its "action state" (for example, whether or not the other vehicle is changing lanes or is about to change lanes).

[0039] Furthermore, the recognition unit 110 recognizes, for example, the lane in which the vehicle M is traveling (driving lane) and other surrounding lanes (for example, oncoming lanes). For example, the recognition unit 110 recognizes road markings from images captured by the camera 10 and recognizes the driving lane and other lanes based on the positional relationship of the road markings as seen from the recognized vehicle M. Alternatively, the recognition unit 110 may refer to map information 162 based on the position information of the vehicle M obtained from the vehicle sensor 40, etc., to recognize the lane in which the vehicle M is traveling and other lanes. The recognition unit 110 may also recognize, for example, stop lines, red lights, toll booths, other road events, road signs, and markings drawn on the road (for example, speed limits).

[0040] Furthermore, the recognition unit 110 recognizes the state of the occupants of vehicle M using images captured by the in-vehicle camera 60. For example, the recognition unit 110 performs known image analysis processing on the images captured by the in-vehicle camera 60 and, based on the analysis results, recognizes the driver's gaze (the direction the driver is looking) and the orientation of the driver's face.

[0041] For example, the recognition unit 110 uses methods such as template matching to detect combinations of reference points (the stationary part of the eye) and moving points (the moving part of the eye) of the driver's eyes from the image. Examples of combinations of reference points and moving points include the inner corner of the eye and the iris, and the corneal reflection region and the pupil. The corneal reflection region is the region of infrared light reflection on the cornea when the in-vehicle camera 60 or the like shines infrared light towards the driver. The recognition unit 110 then performs coordinate transformations from the image plane to real space based on the position of the moving points relative to the reference points to recognize the driver's line of sight. The recognition unit 110 also recognizes the orientation of the driver's face based on positional information of the eyes, nose, mouth, etc. within the face region obtained from the image analysis results (relative positional information of each part, etc.).

[0042] Furthermore, the recognition unit 110 may recognize the position, gaze, and face orientation of any occupants other than the driver (for example, a passenger sitting in the front passenger seat) if such occupants are present. In recognizing each piece of information using the images described above, a pre-trained model, such as one trained in advance by machine learning, may be used.

[0043] Furthermore, the recognition unit 110 recognizes the behavior of vehicle M based on the detection results of the vehicle sensor 40. For example, the recognition unit 110 recognizes the lateral position of vehicle M relative to the driving lane (position in the lane width direction) and the attitude (orientation) of vehicle M relative to the direction of extension of the driving lane, based on the positional relationship of vehicle M with respect to the driving lane. For example, the recognition unit 110 may recognize the deviation of the vehicle M's reference point from the center of the lane, and the angle it makes with a line connecting the centers of the lanes in the direction of vehicle M's travel, as the relative position and attitude of vehicle M with respect to the driving lane. Alternatively, the recognition unit 110 may recognize the position of the vehicle M's reference point relative to any side edge of the driving lane (road marking or road boundary), etc., as the relative position (lateral position) of vehicle M with respect to the driving lane. Furthermore, the recognition unit 110 may recognize the lateral behavior of vehicle M (for example, whether or not it has moved laterally by a predetermined distance or more) from the amount of change in the lateral position and orientation of vehicle M as described above. Furthermore, the recognition unit 110 may recognize the lateral behavior of the vehicle M based on at least one of the following values ​​obtained from the vehicle sensor 40, such as the steering angle, steering torque, steering angular velocity (derivative value of steering angle), and yaw rate, or the steering torque and steering torque change rate obtained from the driver control unit 80.

[0044] The determination unit 120 determines whether the driver is driving while distracted, based on at least one of the driver's gaze or face direction recognized by the recognition unit 110 and a preset distraction determination condition. If it determines that the driver is driving while distracted, the determination unit 120 causes the HMI control unit 140 to output warning information about distracted driving and notifies the occupants, including the driver. The warning information is distraction warning information, such as information notifying that distraction has been detected or information cautioning against distracted driving, and includes at least one of an image or sound (warning sound).

[0045] Furthermore, the determination unit 120 determines whether a predetermined steering maneuver is being performed on the vehicle M based on the detection results from the vehicle sensor 40 and information regarding the behavior of the vehicle M recognized by the recognition unit 110. The predetermined steering maneuver is, for example, steering maneuver associated with a lane change of the vehicle M. Alternatively, the predetermined steering maneuver may be, for example, steering maneuver for a right or left turn of the vehicle M, or steering maneuver during curved road driving or swaying from side to side. If the determination unit 120 determines that a predetermined steering maneuver is being performed, it may change the distraction detection conditions.

[0046] Furthermore, the determination unit 120 determines whether the driver's monitoring direction is appropriate based on the recognition result from the recognition unit 110. An appropriate monitoring direction is, for example, a direction that has a predetermined range (angle) based on the direction of travel of the vehicle M or the shape of the surrounding roads. For example, as one example of determining whether the monitoring direction is appropriate, the determination unit 120 determines whether the driver is driving while distracted (driving the vehicle M while looking away). Details of the processing by the determination unit 120 will be described later.

[0047] The lighting control unit 130 controls the illumination of the turn signals (direction indicators) of the vehicle M in response to the driver's operation of the turn signal lever 84. For example, the lighting control unit 130 illuminates (flashes) the turn signal corresponding to the operated position (direction) while the turn signal lever 84 is held in the position operated by the driver. The lighting control unit 130 also stops the flashing of the turn signal when the turn signal lever 84 returns to the neutral position. Furthermore, if the turn signal lever 84 has a one-touch turn signal function, the lighting control unit 130 may control the turn signal to continue illuminating until a predetermined lighting condition is met, even if the turn signal lever 84 returns to the neutral position when the driver releases their hand from the turn signal lever 84. The lighting condition is, for example, that a predetermined time has elapsed since the hand was released, or that the turn signal has flashed a predetermined number of times.

[0048] The HMI control unit 140 notifies the occupant of predetermined information via the HMI 30 and receives information input by the HMI 30. The predetermined information includes, for example, information related to the driving of vehicle M, such as information regarding the status of vehicle M and information regarding driving control. Information regarding the status of vehicle M includes, for example, the speed of vehicle M, engine speed, shift position, and the content of the lighting control performed by the lighting control unit 130. Information regarding driving control includes, for example, whether or not the driving control unit 150 is performing driving control and information regarding the status of the driving control. The predetermined information may also include information regarding the surrounding conditions recognized by the detection device DD. The predetermined information may also include information unrelated to the driving of vehicle M, such as content stored on a storage medium such as a television program or DVD (for example, a movie). The predetermined information may also include, for example, the current location and destination of vehicle M, and information regarding the remaining fuel level of vehicle M. The HMI control unit 140 may output the information received by the HMI 30 to the communication device 20, recognition unit 110, determination unit 120, navigation device 50, etc.

[0049] Furthermore, the HMI control unit 140 may output to the HMI 30 information such as inquiry information for the occupant, recognition results from the recognition unit 110, and determination results from the determination unit 120. The HMI control unit 140 may also transmit various information to be output to the HMI 30 to a terminal device used by the occupant of the vehicle M via the communication device 20.

[0050] The driving control unit 150 performs driving control that controls at least one of the steering or speed of the vehicle M based on the recognition results from the recognition unit 110 and the determination results from the determination unit 120. For example, if the determination unit 120 determines that the driver is distracted and this condition continues for a predetermined time or longer, the driving control unit 150 performs control to stop the vehicle M in a safe position such as the shoulder of the road. The driving control unit 150 may also perform the above-mentioned driving control to avoid contact between the vehicle M and obstacles recognized by the surrounding recognition unit 122. Furthermore, the driving control unit 150 may control at least one of the steering or speed of the vehicle M in response to occupant instructions input from the HMI 30 to perform driving control such as ACC (Adaptive Cruise Control System), LKAS (Lane Keeping Assistance System), or ALC (Auto Lane Change). ALC is a driving control that performs a lane change of the vehicle M by operating at least the steering of the vehicle M on the system side.

[0051] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the driving control unit 150 or information input from the accelerator pedal of the driver control unit 80.

[0052] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 150 or from the brake pedal of the driver control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may be equipped with a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the driving control unit 150 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0053] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 150 or from the steering wheel of the driver control unit 80.

[0054] [Judgment section] Next, the details of the processing performed by the determination unit 120 will be explained. In the following, the determination unit will compare at least one of the driver's gaze and face direction of the vehicle M recognized by the recognition unit 110 with a distraction determination area (an example of a distraction determination condition) to determine whether or not the driver is driving while distracted.

[0055] Figure 2 is a diagram illustrating the relationship between the driver's gaze (direction of the face) and the distraction detection conditions. In the example in Figure 2, driver D is seated in the driver's seat ST1 of vehicle M and is performing manual driving of vehicle M by operating the driving controls 80 such as the steering wheel 82. In the example in Figure 2, display units 32-1 and 32-2 are also shown. For example, when vehicle M is traveling in a straight direction (X-axis direction in the figure), the detection unit 120 sets a distraction detection area AR1 that has a predetermined angle to the left and right with respect to the direction of travel V of vehicle M from the position of driver D's head, as shown in Figure 2. The distraction detection area AR1 is an example of the "first distraction detection condition". The distraction detection area AR1 may be adjusted according to the speed of vehicle M, the shape of the road, etc. In this case, for example, the angle (radians) θ1 indicating the size of the arc of the distraction detection area AR1 may be set to decrease in proportion to the speed, or the angle θ1 may be set to increase as the road width increases.

[0056] In this situation, the determination unit 120 determines, for example, that if the gaze of driver D recognized by the recognition unit 110 is within the angle θ1 of the distraction determination area AR1, driver D is not driving distractedly. If the gaze is not within the angle θ1 (or if this state continues for a predetermined time or longer), driver D is driving distractedly. Alternatively, the determination unit 120 may perform the above-mentioned distraction determination by comparing the orientation of driver D's head, recognized by the recognition unit 110, with the distraction determination area AR1, instead of (or in addition to) the gaze of driver D. For example, if the recognition unit 110 could not recognize the gaze of driver D but could recognize the orientation of the face, the determination unit 120 may use the orientation of the face to perform the distraction determination.

[0057] If the determination unit 120 determines that the user is distracted, the HMI control unit 140 generates distraction warning information and outputs it to the HMI 30. The distraction warning information may be a warning image displayed on at least one of the display units 32-1 or 32-2, or a warning sound output to the speaker 34.

[0058] Furthermore, the determination unit 120 may change the distraction detection area AR1 when the turn signal lever 84 is operated by the driver D. Figure 3 is a diagram illustrating the changed distraction detection area. In the example in Figure 3, the change (transformation) of the distraction detection area is shown when the turn signal lever 84 is operated to turn on the right turn signal of the vehicle M. However, if the turn signal lever 84 is operated to turn on the left turn signal, the left and right should be reversed.

[0059] For example, if the turn signal lever 84 is operated to turn on the left turn signal, the determination unit 120 sets a distraction determination area AR2, which is the same as the distraction determination area AR1 rotated to the right around the driver D's head, as the new distraction determination area AR1. The distraction determination area AR2 is an example of a "second distraction determination condition".

[0060] In the example shown in Figure 3, since vehicle M is turning to the right, the determination unit 120 changes the distraction determination area AR2 from the distraction determination area AR1, which is associated with vehicle M when it is moving straight, to a position rotated to the right by an angle △θ around the position of driver D's head, and performs a distraction determination. The angle △θ may be changed according to the degree of turning, and may also be adjusted according to the speed of vehicle M. In the example shown in Figure 3, the distraction determination areas AR1 and AR2 are the same size (in other words, the angle (arc) θ1 indicating the size of the distraction determination area AR1 and the angle θ2 indicating the size of the distraction determination area AR2 are the same), but they may be different sizes (for example, angles θ1 and θ2 are different angles). The angles △θ, θ1, and θ2 described above may be adjusted according to the shape of the road on which vehicle M is traveling (for example, width, curvature, road connection angle at intersections, etc.).

[0061] For example, suppose driver D's line of sight A1 is in the situation shown in Figure 3. In this case, if the turn signal lever 84 is not operated, the determination unit 120 determines that driver D is driving while distracted because their line of sight A1 is not within the distraction determination area AR1. However, if the turn signal lever 84 is operated, the determination unit 120 determines that driver D is not driving while distracted because their line of sight A1 is within the distraction determination area AR2. This makes it difficult to determine that driver D is driving while distracted (making it difficult to determine that driver D's monitoring direction is inappropriate), even if driver D's line of sight A1 moves to the right to check the situation at the lane change destination.

[0062] Furthermore, when the turn signal lever 84 is operated, the determination unit 120 may also use the area including the distraction determination area AR1 and the distraction determination area AR2 as the determination area, and may choose not to perform a distraction determination (or not output distraction warning information even if it is determined that the driver is distracted). This also makes it less likely that the driver D's monitoring direction will be determined to be inappropriate. Furthermore, when the turn signal device 70 is activated by the operation of the turn signal lever 84, and then the operation of the turn signal device 70 ends (when the turn signal stops lighting up), the determination unit 120 determines the distracted driving determination condition based on the original condition (distracted driving determination area AR1).

[0063] In this embodiment, the turn signal device 70 and the turn signal lever 84 of the lighting control unit 130 are controlled by the operation of the one-touch turn signal function. Therefore, the determination unit 120 performs determination control for the operation of the one-touch turn signal function.

[0064] [Turn signal lever] Here, we will explain the operating position of the turn signal lever. Figure 4 is a diagram illustrating the operating position of the turn signal lever 84. In the figure, the X-axis represents the forward and backward direction (front and back) of the vehicle M, the Y-axis represents the width direction (side to side) of the vehicle M, and the Z-axis represents the up and down direction (vertical direction) of the vehicle M. For example, one end of the turn signal lever 84 is supported at a predetermined location (e.g., the steering column). When the driver operates the turn signal lever 84, it rotates in the up and down direction (Z direction) starting from the support point at one end.

[0065] The neutral position P0 is the position in which the turn signal device 70 is not activated, and this position is maintained when the turn signal lever 84 is not operated. Furthermore, when the driver D operates the turn signal lever 84 and rotates it upward relative to the neutral position P0 to the half-press position (left first operating position) P1 or the fully-press position (left second operating position) P2, the turn signal device 70 is activated. "Activation" refers to the action of lighting (flashing) the lamp (turn lamp) that functions as a turn signal.

[0066] The half-press position P1 is the position in which the left-side turn signal device 70 of the vehicle M is activated. This position is maintained while the turn signal lever 84 is being operated, and when the turn signal lever 84 is no longer being operated, the turn signal lever 84 moves to the neutral position P0. When the driver D pushes the turn signal lever 84 up to the half-press position P1 by hand, the turn signal device 70 is activated, and the left turn signal lights up (flashes) a predetermined number of times (e.g., 3 times) or for a predetermined time (e.g., a few seconds) before ending the operation. If the driver D releases their hand while the turn signal is operating, the turn signal lever 84 moves to the neutral position P0 on its own, but the turn signal continues to light up until the aforementioned flashing ends. The half-press position P1 is set, for example, when changing lanes from the driving lane to the adjacent left lane of the vehicle M.

[0067] The fully pressed position P2 is the position in which the left turn signal of vehicle M is activated, and this position is maintained when the turn signal lever 84 is not operated. In other words, once driver D pushes the turn signal lever 84 up to the fully pressed position P2, the turn signal device 70 continues to operate and the left turn signal remains illuminated until driver D pushes the turn signal lever 84 down. The fully pressed position P2 is set, for example, when turning vehicle M left.

[0068] Furthermore, as shown in the illustrated example, when the turn signal lever 84 rotates downward from the neutral position P0 to the half-press position P1# or the fully-press position P2#, the turn signal device 70 activates and illuminates the right turn signal of the vehicle M. The half-press position (right first operating position) P1# is the position that activates the right turn signal device 70 of the vehicle M. This position is maintained while the turn signal lever 84 is being operated, and when the turn signal lever 84 is no longer being operated, the position of the turn signal lever 84 moves back to the neutral position P0. For example, when driver D manually pushes the turn signal lever 84 down to the half-press position P1#, the turn signal device 70 activates and illuminates (flashes) the right turn signal a predetermined number of times (e.g., 3 times) or for a predetermined period (e.g., a few seconds) before ending the operation. In this state, when driver D releases their hand, the turn signal lever 84 moves on its own to the neutral position P0, and the activated right turn signal device 70 stops. The half-press position P1# is used, for example, when changing lanes from the driving lane to the adjacent lane to the right.

[0069] The fully pressed position (right second operating position) P2# is the position in which the right-side turn signal of the vehicle M is activated, and this position is maintained when the turn signal lever 84 is not operated. In other words, once the driver D pushes the turn signal lever 84 down to the fully pressed position P2#, the right-side turn signal device 70 will continue to operate until the driver pushes the turn signal lever 84 up. The fully pressed position P2# is set, for example, when turning the vehicle M to the right.

[0070] The lever position detection unit 84a detects, for example, which position the turn signal lever 84 is located in among the neutral position P0, half-press position P1, fully-press position P2, half-press position P1#, and fully-press position P2#. The turn signal lever 84 may, after moving to the fully-press position P2 or P2#, return to the neutral position P0 on its own when the steering wheel returns to the neutral position by utilizing the rotation of the steering wheel shaft (rotation axis). When the vehicle M changes lanes, the driver D positions the lever to the half-press position P1 or half-press position P1# corresponding to the direction of the lane change. In other words, the function (control) when positioned to the half-press position P1 or half-press position P1# corresponds to the one-touch turn signal function (control).

[0071] For example, when the turn signal lever 84 is positioned in the half-press position P1 or P1#, the determination unit 120 automatically returns to the neutral position P0 and, until predetermined conditions are met, makes it less likely to determine that the driver D's monitoring direction is inappropriate compared to before the turn signal lever 84 was operated (in other words, it makes it less likely to determine that the driver D is driving while distracted).

[0072] [Operation and judgment control for the turn signal lever] Next, the relationship between the operation of the turn signal lever 84 in the embodiment (hereinafter referred to as "turn signal operation") and the determination process by the determination unit 120 will be explained. In the following, the process when the turn signal lever 84 is positioned in the half-press position P1 or P1# by the driver D will be mainly explained. Furthermore, the following will be explained separately as a first embodiment in which there is lateral movement of the vehicle M (movement in the lane width direction) after the turn signal operation, and a second embodiment in which there is no such movement.

[0073] <First Example> Figure 5 is a diagram illustrating the turn signal operation and determination process in the first embodiment. In the following description, the position and speed of vehicle M at time T* will be represented as M(T*) and VM(T*), respectively. In the following description, T0 is the earliest, and the times T1, T2, T3, T4, and T5 are in the order of decreasing speed.

[0074] In the example in Figure 5, two lanes L1 and L2 are shown, both capable of traveling in the same direction (X-axis direction in the figure). Lane L1 is demarcated by road markings LL and CL, and lane L2 is demarcated by road markings CL and RL. From the perspective of lane L1, lane L2 is an adjacent lane from which lane changes are possible. In the example in Figure 5, vehicle M is traveling in lane L1 at speed VM. In the example in Figure 5, the position or change in position of the turn signal lever 84 detected by the lever position detection unit 84a is shown at each timing from T0 to T5.

[0075] For example, when the turn signal lever 84 is positioned in the half-press position P1 or P1#, the determination unit 120 makes it less likely to determine that driver D is distracted driving compared to before operating the turn signal lever 84, until predetermined conditions are met after the turn signal lever 84 returns to the neutral position P0. Making it less likely to determine distracted driving may, for example, make it appear as if distracted driving has been determined, or it may be done by loosening the distracted driving determination conditions (making the area where distracted driving is determined smaller / making the area where distracted driving is not determined larger).

[0076] In the first embodiment, for example, driver D positioned the turn signal lever 84 in the half-press position P1# to activate the turn signal in order to change the vehicle M traveling in lane L1 to lane L2. However, due to factors such as the presence of other vehicles or obstacles in the adjacent lane L2, the vehicle did not change lanes to L2 and continued to travel on lane L1 without any lateral movement.

[0077] In this situation, at time T0, the turn signal lever 84 is in the neutral position P0. At time T1, the driver D moves the turn signal lever 84 from the neutral position P0 to the half-press position P1#, which is used to activate the right turn signal. At time T2, the driver D releases the turn signal lever 84, causing it to move from the half-press position P1# back to the neutral position P0. In the example in Figure 5, it is assumed that the turn signal lever 84 is in the neutral position P0 from time T2 onward.

[0078] The lighting control unit 130 keeps the turn signal lit (flashing) until a predetermined time has elapsed from the moment the turn signal lever 84 is instructed to be in the half-press position P1#, or until the turn signal has flashed a predetermined number of times. Therefore, at time T3 before the predetermined time has elapsed or before the predetermined number of flashes has been reached, the turn signal remains lit.

[0079] After moving the turn signal lever 84 to the half-press position P1#, driver D shifts their gaze and rotates their head to check the right side (including the front right and rear right) of vehicle M, where lane L2 is located, in order to confirm the safety of the lane L2 in which they will change lanes. However, if a distracted driving detection is performed at this timing, the driver will be judged as distracted driving (the direction of the driver's monitoring is inappropriate) due to the gaze and head direction required for the lane change.

[0080] Therefore, the determination unit 120 makes it less likely to determine that driver D is driving distractedly compared to before the turn lever 84 was operated, until a first predetermined time has elapsed from the time T1 when the turn lever 84 was operated, or until vehicle M has traveled (moved) a first predetermined distance or more. For example, in the example of Figure 5, if the time after the first predetermined time has elapsed from time T1 is time T4, the determination unit 120 makes it less likely to determine that driver D is driving distractedly until time T4. Alternatively, the determination unit 120 may make it less likely to determine that driver D is driving distractedly until vehicle M reaches point E1, which is a first predetermined distance D1 traveled from point S1 where vehicle M (T1) was located when the turn lever 84 was operated.

[0081] Furthermore, in the above example, the time T1 when the turn lever 84 was operated was used as the reference point. However, instead, the determination unit 120 may be less likely to determine that driver D is driving distractedly compared to before the turn lever 84 was operated, until a second predetermined time has elapsed from the time T2 when the turn lever 84 returned to the neutral position P0, or until vehicle M has traveled (moved) a second predetermined distance or more. In this case, if the time after the second predetermined time has elapsed from time T2 is time T5, the determination unit 120 will be less likely to determine that driver D is driving distractedly until time T5. Also, the determination unit 120 may be less likely to determine that driver D is driving distractedly until vehicle M reaches point E2, which is a second predetermined distance D2 from point S2 where vehicle M (T2) was located when the turn lever 84 returned to the neutral position P0.

[0082] Furthermore, in the above example, instead of times T1 and T2, the determination unit 120 may make it less likely to determine that driver D is driving distractedly compared to before operating the turn signal lever 84, until a third predetermined time has elapsed from time T4 when the turn signal stopped lighting up, or until vehicle M has traveled (moved) a third predetermined distance or more. In this case, if the time after the third predetermined time has elapsed from time T4 is time T5, the determination unit 120 will make it less likely to determine that driver D is driving distractedly until time T5. Also, the determination unit 120 may make it less likely to determine that driver D is driving distractedly until vehicle M reaches point E3, which is a third predetermined distance D3 away from point S3 where vehicle M (T4) was located when the turn signal stopped lighting up.

[0083] The first to third predetermined times described above may be the same, or at least one of them may be different. Similarly, the first to third predetermined distances D1 to D3 described above may be the same, or at least one of them may be different. When different times or distances are used, for example, in the case of time, the first predetermined time, second predetermined time, and third predetermined time should be made smaller in that order, and in the case of distance, the first predetermined distance D1, second predetermined distance D2, and third predetermined distance D3 should be made smaller in that order. As time elapses since the turn signal is activated, or as the distance traveled increases, the likelihood of driver D no longer looking in the direction of the lane change increases. Therefore, by reducing the predetermined time and predetermined distance based on the time of turn signal activation, the state of driver D (distracted driving) can be detected more appropriately.

[0084] According to the first embodiment described above, in the case of a turn signal lever 84 that returns to the neutral position P0 when the hand is released after operation, such as a one-touch turn signal, the distraction detection can be suppressed for a predetermined period and distance even after returning to the neutral position P0. Therefore, it is possible to suppress the detection of distraction during surrounding monitoring when changing lanes with a one-touch turn signal, and to suppress the occupant feeling annoyed by unnecessary distraction warnings, etc.

[0085] <Second Example> Figure 6 is a diagram illustrating the turn signal operation and determination process in the second embodiment. The example in Figure 6 differs from the example in Figure 5 in that, after operating the turn signal lever 84, the vehicle M changes lanes from lane L1 to lane L2. Also, among the times Ta, Tb, and Tc shown in Figure 6, time Ta is the earliest, followed by times Tb and Tc in that order, and so on. Furthermore, time Ta is later than the time T2 mentioned above, and the position of the turn signal lever 84 after time T2 is assumed to be the neutral position P0. The behavior of the vehicle M and the operation of the turn signal lever 84 at times T0 to T2 in the second embodiment are the same as at times T0 to T2 in the first embodiment described above, so the explanation is omitted here.

[0086] At time Ta, if, for example, a lateral movement of the vehicle M or an operation that causes a lateral movement occurs before a predetermined time has elapsed or while the vehicle is traveling less than a predetermined distance, the determination unit 120 makes it less likely to determine that the driver D is distracted until the lateral movement or the operation that causes a lateral movement is completed, even if the predetermined time has elapsed or the vehicle has traveled a predetermined distance or more.

[0087] For example, in the example shown in Figure 6, the turn signal lever 84 is operated by the driver at time T1 until the vehicle travels a predetermined distance D1. In this state, since lateral movement of the vehicle M is detected during the time Ta until the vehicle M travels the distance D1, the determination unit 120 is less likely to determine that the driver D is distracted even after the vehicle M reaches point E1.

[0088] Here, we will explain how to determine whether the vehicle M is moving laterally or whether an operation that causes lateral movement is occurring. For example, the determination unit 120 determines whether the vehicle M is moving laterally or whether an operation that causes lateral movement is occurring, based on information obtained from the vehicle sensor 40, information obtained from the driver control unit, and the recognition result of the recognition unit 110.

[0089] For example, the determination unit 120 determines that vehicle M is moving laterally if the lateral movement distance of vehicle M (distance moved in the lane width direction) or the amount of movement per unit time is greater than or equal to a first threshold, and determines that vehicle M is not moving laterally if it is less than the first threshold. The determination unit 120 also determines that vehicle M is moving laterally if the orientation of vehicle M with respect to the extension direction of the driving lane, or the amount of change in orientation, is greater than or equal to a second threshold, and determines that vehicle M is not moving laterally if it is less than the second threshold. The determination unit 120 may also determine that vehicle M is moving laterally if the lateral speed (lateral G) is greater than or equal to a third threshold, and determines that vehicle M is not moving laterally if it is less than the third threshold.

[0090] Furthermore, the determination unit 120 determines that an operation causing lateral movement of the vehicle M has occurred if the steering torque is equal to or greater than the fourth threshold, and determines that no operation has occurred if it is less than the fourth threshold. The determination unit 120 may also determine that an operation causing lateral movement of the vehicle M has occurred if the steering angle derivative is equal to or greater than the fifth threshold, and determines that no operation has occurred if it is less than the fifth threshold. The determination unit 120 may also determine that an operation causing lateral movement of the vehicle has occurred if the steering angle is equal to or greater than the sixth threshold, and determines that no operation has occurred if it is less than the sixth threshold.

[0091] The determination unit 120 may perform the determination process using at least one of the various determination conditions related to lateral movement described above, and may also include the condition that the speed VM of the vehicle M is 7th threshold or higher as a condition for determining that the vehicle M is moving laterally or that an operation that causes lateral movement is occurring.

[0092] In the example in Figure 6, the determination unit 120 makes it less likely to determine that driver D is distracted until vehicle M reaches point E1, which is a first predetermined distance D1 away from point S1 where vehicle M (T1) is located at time T1 when the turn signal lever 84 is operated. In this case, it is determined that lateral movement of vehicle M or an operation that causes lateral movement occurs on the way to point E1.

[0093] Furthermore, instead of determining when the lateral movement or the operation that generates the lateral movement is completed, the determination unit 120 may make it less likely to determine that the driver D is driving while distracted until the vehicle M crosses the lane markings that define the lane L1 it is traveling in, or until the presence of lane markings (road markings) to the left and right of the vehicle M is recognized after the lateral movement of the vehicle M. "The vehicle M crosses the lane markings that define the lane L1 it is traveling in" means, for example, that a predetermined point on the vehicle M (e.g., center of gravity, center, front end) is on the road marking CL that defines the lane L1 and the adjacent lane L2, or that the entire vehicle M is on lane L2 beyond the road markings. Also, "the presence of road markings to the left and right of the vehicle M is recognized" means, for example, that the vehicle M is recognized to be traveling in the center of the lane to which it is changing lanes (including a predetermined allowable range), or that the orientation of the vehicle M is facing the direction of lane extension (including a predetermined allowable range).

[0094] In the example in Figure 6, if the determination unit 120 determines that there is lateral movement before the vehicle M reaches point E1, it may be less likely to determine that the driver D is driving distractedly until time Tb, when the entire vehicle M is on lane L2. Alternatively, in the example in Figure 6, if the determination unit 120 determines that there is lateral movement before the vehicle M reaches point E1, it may be less likely to determine that the driver D is driving distractedly until time Tc, when the vehicle M is traveling in the center of lane L2.

[0095] According to the second embodiment described above, the suppression of distraction detection can be continued until lateral movements such as lane changes based on the operation of the turn signal lever 84 are completed. Furthermore, according to the second embodiment, the suppression of distraction detection can be released when the lane change converges. Therefore, the driver's state can be detected more appropriately.

[0096] In this embodiment, the vehicle M may also be subjected to driving control by the driving control unit 150. In this case, the driving control unit 150 starts ALC triggered by the operation of the turn signal lever 84. Therefore, the operation of the turn signal lever 84 includes an operation to start ALC on the vehicle M. Furthermore, the predetermined conditions for making it less likely that the driver D's monitoring direction is deemed inappropriate (making it less likely that the driver D is being distracted) from the time the turn signal lever 84 returns to the neutral position until predetermined conditions are met may include the completion of ALC. In this way, by loosening the distraction detection when ALC is executed, it is possible to suppress the forcible cancellation of driving control such as ALC due to the detection of distracted driving.

[0097] Furthermore, in the embodiment, the above predetermined conditions may also include that the vehicle M is traveling on a designated dedicated road (for example, an expressway or toll road), or that the vehicle M's speed VM is above a predetermined speed (in this case, a speed based on the legal speed limit of the designated dedicated road). This allows for appropriate driving control, for example, because when the vehicle M is traveling at high speed, it is likely to change lanes using the one-touch turn signal, and in such cases, the detection of distracted driving can be suppressed.

[0098] Furthermore, in the embodiment, the above-mentioned predetermined conditions may include a first predetermined condition used when the vehicle M is traveling on a designated dedicated road or when the speed VM of the vehicle M is traveling at or above a predetermined speed, and a second predetermined condition used when the vehicle M is traveling on a road other than a designated dedicated road (e.g., a general road) or when the speed of the vehicle M is below a predetermined speed. In this case, the first predetermined condition is made more difficult to satisfy than the second predetermined condition. Making the condition more difficult to satisfy includes increasing the number of conditions required to satisfy it, or adjusting the values ​​or ranges so that the conditions are less likely to be satisfied. For example, an example of making the condition more difficult to satisfy is making the predetermined time or distance for widening the distraction detection area longer than that of the second predetermined condition so that it is less likely to be detected as distraction. This makes it possible to make a more appropriate distraction detection according to the road conditions under which the vehicle is traveling.

[0099] In addition, in this embodiment, when the lane change is completed, if the determination unit 120 changes from a state where a predetermined condition is not met to a state where it is met, it resets the distraction determination condition and performs a distraction determination. If it is determined that the driver is distracted, it causes the HMI control unit 140 to output distraction warning information. The HMI control unit 140 may also output information from the HMI 30 indicating that the driver D's monitoring direction is less likely to be determined to be inappropriate (less likely to be determined to be distracted driving) during the driving state when changing lanes, and notify the driver. This allows the driver to move their gaze and face direction to check the situation in the lane ahead without hesitation.

[0100] [Processing flow] The following describes the processes performed by the driver assistance device 100 of this embodiment. The following description will primarily focus on the processes related to driver D's distraction detection among those performed by the driver assistance device 100. Figure 7 is a flowchart showing an example of the processes performed by the driver assistance device 100 in this embodiment. Note that the processes shown in Figure 7 may be repeatedly executed at predetermined timings or cycles.

[0101] In the example shown in Figure 7, the recognition unit 110 recognizes the surrounding conditions of the vehicle M based on the output from the detection device DD (step S100). Next, the recognition unit 110 recognizes at least one of the driver D's gaze or the direction of their face P (step S200). Next, the determination unit 120 performs a distraction determination process using at least one of the recognized driver D's gaze or face direction (step S300). Details of the distraction determination process will be described later.

[0102] Next, the HMI control unit 140 determines whether or not driver D is driving while distracted, based on the result of the distraction detection by the determination unit 120 (step S400). If it is determined that driver D is driving while distracted, the HMI control unit 140 outputs a warning about distracted driving to driver D (step S500). This completes the processing of this flowchart. If it is determined in step S400 that driver D is not driving while distracted, the process ends there.

[0103] Figure 8 is a flowchart showing an example of the distraction detection process. In the example in Figure 8, the process corresponds to the process in S300 described above. In the example in Figure 8, the determination unit 120 determines whether or not the turn signal lever 84 has been operated by the driver D (step S310). If it is determined that the turn signal lever 84 has been operated, the determination unit 120 determines whether or not the turn signal lever 84 has returned to the neutral position (step S320). If it is determined that the turn signal lever 84 has returned to the neutral position, the determination unit 120 determines whether or not the turn signal continues to light up (flashing) even after returning to the neutral position by the one-touch turn signal function (step S330). If it is determined that the turn signal does not continue to light up, the determination unit 120 determines whether or not a predetermined condition has been met (step S340).

[0104] If, in step S340, it is determined that the predetermined conditions are not met, in step S320, it is determined that the lever has not returned to the neutral position, or in step S330, it is determined that the turn signal remains lit, the determination unit 120 changes the conditions to make it less likely that driver D is being judged as distracted compared to before the turn signal lever 84 was operated (step S350). Changing the conditions to make it less likely that driver D is being judged as distracted includes, for example, widening the distraction judgment area (the range in which distraction is not judged), not judging it as distraction, or controlling it so that a distraction warning is not issued even if distraction is judged.

[0105] Next, if it is determined in step S310 that the turn signal lever 84 has not been operated, if it is determined in step S340 that a predetermined condition has been met, or after the processing of step S350, the determination unit 120 determines whether the driver's gaze or the direction of their face is within a predetermined distraction determination area (step S360). If it is determined that the driver is within the distraction determination area, the determination unit 120 determines that the driver is not driving while distracted (step S370). If it is determined that the driver is not within the distraction determination area, the determination unit 120 determines that the driver is driving while distracted (step S380). This completes the processing of this flowchart.

[0106] In addition, instead of changing the conditions in step S350 of Figure 8 to make it less likely that the driver is being judged as distracted, the process in the subsequent step S360 may be adjusted so that even if the direction of the gaze or face is determined to be within the distraction judgment area, the driver is not judged as being distracted.

[0107] [Differentiation] In this embodiment, if the determination unit 120 determines that the driver is driving while distracted, instead of (or in addition to) outputting a warning about distracted driving, the driving control unit 150 may be instructed to execute driving control that moves the vehicle M to a safe position and stops it. This allows, for example, if the driver is distracted due to feeling unwell, the vehicle M can be stopped in a safe position, thereby more effectively ensuring the driver's safety.

[0108] Furthermore, the operator for illuminating the turn signal in this embodiment may be a mechanical switch provided on the steering wheel 82 instead of the turn signal lever 84, or a GUI (Graphical User Interface) switch provided on the display unit 32 of the HMI 30, etc.

[0109] According to the embodiment described above, the driver state detection device includes a recognition unit 110 that recognizes at least one of the driver's gaze and face direction in a vehicle M (an example of a moving object), a determination unit 120 that determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit 110, and a lighting control unit 130 that controls the lighting of the turn signals in response to the driver's operation of an operator that lights up the turn signals of the vehicle M, and controls the lights to continue to be lit even if the operator returns to the neutral position. The determination unit 120 is less likely to determine that the driver's monitoring direction is inappropriate compared to before the operator was operated, from the time the operator returns to the neutral position until predetermined conditions are met, thereby enabling more appropriate detection of the driver's state.

[0110] For example, according to the embodiment, even if the turn signal lever 84 (operator) returns to the neutral position after one-touch turn signal operation, the suppression of distracted driving detection can be continued for a predetermined period. Specifically, even if the turn signal lever 84 automatically returns to the neutral position, distracted driving detection can be continued by waiting for a predetermined distance or time of the vehicle. The return of the detection condition is controlled based on the determination of a predetermined time or distance, and convergence information of the vehicle's behavior, including steering angle. Therefore, the detection condition can be switched more appropriately according to the situation of the vehicle M, and the driver's state (distracted driving, etc.) can be detected more appropriately using this detection condition.

[0111] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Recognize at least one of the driver's gaze and the direction of their face, Based on the recognized results, it is determined whether the driver's monitoring direction is appropriate. The system controls the illumination of the turn signals of the moving body in response to the driver's operation of the control unit that illuminates the turn signals, and controls the system to keep the turn signals illuminated even if the control unit returns to the neutral position. From the time the operator returns to the neutral position until predetermined conditions are met, it becomes less likely that the operator's monitoring direction will be deemed inappropriate compared to before the operator was operated. Driver status detection device.

[0112] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0113] 1...Vehicle system, 10...Camera, 12...Radar device, 14...LIDAR, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 60...In-cabin camera, 70...Turn signal device, 80...Driver's control panel, 82...Steering wheel, 84...Turn signal lever, 100...Driver's assistance device, 110...Recognition unit, 120...Determination unit, 130...Lighting control unit, 140...HMI control unit, 150...Driving control unit, 160...Storage unit, 200...Driving force output device, 210...Brake device, 220...Steering device, M...Vehicle

Claims

1. A recognition unit that recognizes at least one of the driver's gaze and the direction of their face, A determination unit determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit, The system includes a lighting control unit that controls the illumination of the turn signal in response to the driver's operation on an operator that illuminates the turn signal of the moving body, and controls the illumination to continue even if the operator returns to the neutral position, The determination unit makes it less likely to determine that the driver's monitoring direction is inappropriate compared to before the operator was operated, from the time the operator returns to the neutral position until predetermined conditions are met. Driver status detection device.

2. The aforementioned predetermined conditions include the elapsed time since the operator was operated or since the operator returned to the neutral position, or the moving body moving a predetermined distance or more. The driver status detection device according to claim 1.

3. The aforementioned predetermined conditions include the fact that a predetermined time has elapsed since the turn signal stopped illuminating, or that the moving object has moved a predetermined distance or more. The driver status detection device according to claim 1.

4. The determination unit, if a lateral movement of the moving body or an operation that causes the lateral movement occurs before the predetermined time has elapsed or while the body has moved less than the predetermined distance, makes it less likely to determine that the driver's monitoring direction is inappropriate until the lateral movement or the operation that causes the lateral movement is completed, even if the predetermined time has elapsed or the body has moved more than the predetermined distance. The driver status detection device according to claim 2 or 3.

5. The determination unit, when a lateral movement of the moving body or an operation that causes such lateral movement occurs before the predetermined time has elapsed or when the body has moved less than the predetermined distance, makes it less likely to determine that the driver's monitoring direction is inappropriate, even if the predetermined time has elapsed or the body has moved more than the predetermined distance, until the moving body crosses the lane markings that define the lane it is moving in, or until the presence of lane markings to the left and right of the moving body is recognized after the lateral movement of the moving body. The driver status detection device according to claim 2 or 3.

6. The operation of the aforementioned control device initiates the vehicle's lane change, The aforementioned predetermined conditions include the completion of the vehicle lane change, The driver status detection device according to claim 1.

7. The aforementioned predetermined conditions include the fact that the moving object is moving on a designated dedicated road, or that the moving object is moving at a predetermined speed or higher. The driver status detection device according to claim 1.

8. The aforementioned predetermined conditions include a first predetermined condition used when the moving body is traveling on a designated dedicated road or when the moving body is traveling at a predetermined speed or higher, and a second predetermined condition used when the moving body is traveling on a road other than the designated dedicated road or when the moving body is traveling at a speed lower than the predetermined speed. The first predetermined condition is less likely to be satisfied than the second predetermined condition. The driver status detection device according to claim 1.

9. Computers Recognize at least one of the driver's gaze and the direction of their face, Based on the recognized results, it is determined whether the driver's monitoring direction is appropriate. The system controls the illumination of the turn signals of the moving body in response to the driver's operation of the control unit that illuminates the turn signals, and controls the system to keep the turn signals illuminated even if the control unit returns to the neutral position. From the time the operator returns to the neutral position until predetermined conditions are met, it becomes less likely that the operator's monitoring direction will be deemed inappropriate compared to before the operator was operated. Driver status detection method.

10. On the computer, To recognize at least one of the driver's gaze and the direction of their face, Based on the recognized results, the system determines whether the driver's monitoring direction is appropriate. The operation of the operator that controls the movement of the movement to illuminate the turn signal will be controlled in response to the driver's operation of the operator that controls the illumination of the turn signal, and the operation will be controlled to keep the turn signal illuminated even if the operator returns to the neutral position. From the time the operator returns to the neutral position until predetermined conditions are met, it is made less likely that the operator's monitoring direction will be judged as inappropriate compared to before the operator was operated. program.

Images