Vehicle control system

Abstract

The system will appropriately detect driver abnormalities according to the driving situation and minimize delays in notifying the driver. [Solution] The vehicle control device 1 includes a surrounding situation recognition unit 51 that recognizes the surrounding situation of the vehicle 2, a driving support unit 52 that provides driving support for the vehicle 2 based on the recognition results of the surrounding situation recognition unit 51, and a driver state determination unit 56 that determines whether the driver's condition is normal or abnormal based on images captured by an in-vehicle camera 25 that captures images of the driver of the vehicle 2. The driver state determination unit 56 is configured to make it easier to determine that the driver's condition is abnormal when driving support is being performed by the driving support unit 52 compared to when driving support is not being performed.

Description

Technical Field

[0001] The present invention relates to a vehicle control device that controls the running of a vehicle to assist driving, and more particularly to a vehicle control device that performs driving assistance based on the state of a driver of the vehicle.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. In order to achieve this, efforts to further improve traffic safety and convenience through research and development of driving assistance technologies have attracted attention.

[0003] Conventionally, a running control device including a peripheral detection unit that detects the surrounding situation of a vehicle and a lane keeping control unit that performs lane keeping control based on the detection result of lane marks by the peripheral detection unit is known (see, for example, Patent Document 1). This running control device determines whether it is possible to change lanes according to the driver's intention to change lanes, and notifies the driver of the determination result by a display and / or a speaker.

[0004] Patent Document 2 discloses a driver state detection device that can accurately determine whether a driver is in a distracted state by using changes in the expression of the driver's face. This device detects changes in specific parts of the driver's face imaged by an imaging device, and determines that the driver is in a distracted state when the duration of a temporary change in the specific part is longer than a predetermined time set in advance.

[0005] Patent Document 3 discloses a concentration determination device that takes into account both driving safety and driving comfort. This device includes a monitoring data acquisition unit that acquires monitoring data from a sensor that monitors the driver of the vehicle, and a concentration estimation unit that estimates the driving concentration of the driver based on a predetermined index from the monitoring data. When the driving concentration does not meet the standard, an instruction signal for instructing assistance to the driver is output.

Prior Art Document

[0006] [Patent Document 1] Japanese Patent Publication No. 2017-146817 [Patent Document 2] Japanese Patent Publication No. 2016-71577 [Patent Document 2] Japanese Patent Publication No. 2018-149941 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, with conventional technology, the accuracy of the driver camera used to image the driver is low, and it takes time to determine if the driver's condition is abnormal. Therefore, even in truly dangerous situations, there is a risk that notification to the driver will be delayed due to the time it takes to detect the driver's abnormality.

[0008] In view of the above background, the present invention aims to appropriately determine abnormal driver conditions according to the driving scenario and suppress delays in notifying the driver. Ultimately, this will contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0009] To solve the above problems, one aspect of the present invention comprises: a surrounding situation recognition unit that recognizes the surrounding situation of a vehicle; a driving support unit that provides driving support for the vehicle based on the recognition results of the surrounding situation recognition unit; and a driver state determination unit that determines whether the driver's state is normal or abnormal based on an image captured by an in-vehicle camera that captures the driver of the vehicle, wherein the driver state determination unit is configured to more easily determine the driver's state as abnormal when the driving support unit is performing the driving support compared to when the driving support is not performing. [Effects of the Invention]

[0010] According to the above configuration, abnormal driver conditions are more easily detected when the driver assistance unit performs driver assistance, allowing for appropriate detection of driver conditions according to the driving scene and suppressing delays in notifying the driver. [Brief explanation of the drawing]

[0011] [Figure 1] Configuration diagram of a vehicle system according to an embodiment of the present invention [Figure 2] Diagram showing images from an in-car camera. [Figure 3] An explanatory diagram illustrating an example of unsuitable driving behavior as seen in images from an in-car camera. [Figure 4] Flowchart showing the procedure for driving assistance control by the control device. [Figure 5] A time chart of an example of driving assistance control when driving control is not being performed. [Figure 6] A time chart of an example of driving assistance control when driving control is being performed. [Modes for carrying out the invention]

[0012] As shown in Figure 1, the vehicle control device 1 is installed in the vehicle 2. The vehicle 2 may be, for example, a four-wheeled automobile. The vehicle 2 is an autonomous vehicle or a vehicle with driver assistance functions.

[0013] Vehicle 2 has a propulsion system 3, a braking system 4, and a steering system 5. The propulsion system 3 is a device that provides driving force to vehicle 2 and includes, for example, a power source and a transmission. The power source has at least one of an internal combustion engine such as a gasoline engine or a diesel engine and an electric motor. The braking system 4 is a device that provides braking force to vehicle 2 and includes, for example, a brake caliper that presses pads against a brake rotor and an electric cylinder that supplies hydraulic pressure to the brake caliper. The steering system 5 is a device for changing the steering angle of the wheels and includes, for example, a rack and pinion mechanism that steers the wheels and an electric motor that drives the rack and pinion mechanism. The propulsion system 3, the braking system 4, and the steering system 5 are controlled by a vehicle control device 1.

[0014] The vehicle 2 has an external recognition device 7. The external recognition device 7 is a device that detects objects outside the vehicle and the like. The external recognition device 7 is a sensor that captures electromagnetic waves and light from the surroundings of the vehicle 2 to detect objects outside the vehicle and the like. The external recognition device 7 includes, for example, a radar 8, a lidar 9 (LIDAR), and an external camera 10.

[0015] The vehicle 2 has a vehicle sensor 12. The vehicle sensor 12 includes a vehicle speed sensor 13 that detects the speed of the vehicle 2, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, an azimuth sensor that detects the orientation of the vehicle 2, and the like.

[0016] The vehicle 2 has a communication device 15, a navigation device 16, a driving operation device 17, an occupant monitoring device 18, and an HMI 19 (Human Machine Interface). The communication device 15 mediates communication between the vehicle control device 1 and the navigation device 16 and peripheral vehicles and servers located outside the vehicle.

[0017] The navigation device 16 is a device that acquires the current position of the vehicle 2 and provides route guidance to the destination and the like. The navigation device 16 preferably has a GNSS receiving unit, a map storage unit, a navigation interface, and a route determination unit. The GNSS receiving unit specifies the position (latitude and longitude) of the vehicle 2 based on signals received from artificial satellites (positioning satellites). The map storage unit is composed of a known storage device such as a flash memory or a hard disk and stores map information. The navigation interface receives inputs such as the destination from the occupant and presents various information to the occupant by display and voice. The navigation interface may include, for example, a touch panel display, a speaker 32, and the like.

[0018] The driving operation device 17 receives an input operation performed by an occupant (driver) to control the vehicle 2. The driving operation device 17 includes a steering wheel 21, an accelerator pedal 22, and a brake pedal 23. Further, the driving operation device 17 may include a shift lever, a parking brake lever, and the like. A sensor for detecting an operation amount is attached to each driving operation device 17. The driving operation device 17 outputs a signal indicating the operation amount to the vehicle control device 1.

[0019] The occupant monitoring device 18 monitors the state of the occupant in the vehicle interior. The occupant monitoring device 18 has, for example, an in-vehicle camera 25 that images the occupant sitting on the seat in the vehicle interior, and a grip sensor 26 provided on the steering wheel 21. The in-vehicle camera 25 is a digital camera that uses a solid-state imaging device such as a CCD or a CMOS. The grip sensor 26 detects whether or not the driver is gripping the steering wheel 21. The grip sensor 26 may be formed by, for example, a capacitance sensor or a piezoelectric element provided on the steering wheel 21.

[0020] The HMI 19 notifies various information to the occupant by display and voice, and receives an input operation by the occupant. The HMI 19 includes a display device 31 and a speaker 32. The display device 31 may be a touch panel display including liquid crystal or organic EL. The display device 31 may also serve as a navigation interface. The display device 31 and the speaker 32 function as notification devices for notifying the occupant by image and voice. Here, the image may be a moving image including a plurality of consecutive frames. Further, the HMI 19 may include various actuators. For example, the actuator may be one that acts on the driver's six senses. For example, the actuator may be a vibration actuator built into the steering wheel 21 or a belt tightening device built into the seat belt retractor.

[0021] Vehicle 2 has a first operation switch 35 and a second operation switch 36. The first operation switch 35 and the second operation switch 36 are switches that can be operated by the occupants. The first operation switch 35 and the second operation switch 36 may be mechanical switches or GUI switches displayed on a touch panel, and are arranged in appropriate places in the vehicle interior. The first operation switch 35 and the second operation switch 36 may be configured as a display device 31 or a navigation interface. The first operation switch 35 may be a switch for changing the on / off state of driver assistance control. The second operation switch 36 may be a switch for selecting a driver assistance control to be executed from among several driver assistance controls, and for setting the level of automated driving. The second operation switch 36 may be, for example, a rotary switch. The first operation switch 35 and the second operation switch 36 may be configured as a single unit as a common switch.

[0022] The vehicle control device 1 is a computer having a processor 41 and a memory 42 that is communicatively connected to the processor 41. The processor 41 may include at least one of the following as its core: a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a RISC (Reduced Instruction Set Computer). The memory 42 stores control programs executed by the processor 41 and various data. The memory 42 may include at least one of volatile memory and non-volatile memory. The volatile memory may be, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The non-volatile memory may be an SSD (Solid State Drive), flash memory, magnetic disk storage device, or optical disk storage device. At least a part of the vehicle control device 1 may be implemented by hardware such as an LSI (Large Scale Integration), ASIC (application specific integrated circuit), or FPGA (field-programmable gate array), or by a combination of software and hardware. The vehicle control device 1 may be composed of a single piece of hardware, or it may be composed of multiple pieces of hardware that can communicate with each other. Part of the vehicle control device 1 may be composed of an external server located outside the vehicle 2.

[0023] The processor 41 implements various applications by executing control programs stored in memory 42. The control programs may be stored on removable recordable media such as DVDs or CD-ROMs, and installed in memory 42 when the recordable media is read by a reader. Alternatively, the control programs may be downloaded to and installed in memory 42 via a communication network such as the Internet.

[0024] The processor 41 functions as the surrounding situation recognition unit 51, the driving control unit 52, the mode setting unit 53, the emergency control unit 54, the driver status recognition unit 55, the driver status determination unit 56, the vehicle position recognition unit 57, the operation detection unit 58, and the notification unit 59 by executing the control program stored in the memory 42. As the processor 41 executes the control program, the vehicle control device 1, which is a computer, executes the vehicle control method.

[0025] The surrounding environment recognition unit 51 recognizes the surrounding environment of the vehicle 2. Based on the detection results of the external environment recognition device 7, the surrounding environment recognition unit 51 recognizes the surrounding environment (external environment), including obstacles located around the vehicle 2, the shape of the road, the presence or absence of sidewalks, road markings, etc. Obstacles include, for example, guardrails, utility poles, surrounding vehicles, and people such as pedestrians. From the detection results of the external environment recognition device 7, the surrounding environment recognition unit 51 can acquire the status of surrounding vehicles, such as their position, speed, and acceleration.

[0026] The vehicle position recognition unit 57 recognizes the driving lane in which the vehicle 2 is traveling, as well as the relative position and angle of the vehicle 2 with respect to the driving lane. The vehicle position recognition unit 57 may recognize the driving lane based on map information and the position of the vehicle 2 acquired by the GNSS receiver. In addition, the vehicle position recognition unit 57 may extract the lane markings around the vehicle 2 drawn on the road surface from the map information and compare them with the shape of the lane markings captured by the external camera 10 to recognize the relative position and angle of the vehicle 2 with respect to the driving lane.

[0027] The driver state recognition unit 55 recognizes the driver's status. Based on the detection results of the occupant monitoring device 18, the driver state recognition unit 55 recognizes the driver's status and the driver's holding status of the steering wheel 21. The driver's status includes the driver's head position, face orientation, eye opening / closing state, and gaze direction. The driver state recognition unit 55 can acquire this driver status information from the detection results of the in-vehicle camera 25.

[0028] The driver state determination unit 56 determines the driver's state. Based on the signal from the occupant monitoring device 18, the driver state determination unit 56 determines whether the driver is in a state of surrounding monitoring. The state of surrounding monitoring is a state suitable for driving in which the driver is monitoring the surroundings of the vehicle 2 and meets predetermined suitability criteria, and is required in automated driving of level 2 or lower, where surrounding monitoring is mandatory. In the state of surrounding monitoring, it is preferable that the driver is in a state in which they can quickly start manual driving. For example, in the state of surrounding monitoring, the driver is seated facing forward in the seat and is monitoring the area in front of the vehicle 2. The driver state determination unit 56 may, for example, acquire the driver's posture or head orientation based on the image from the in-vehicle camera 25, and if the driver's posture or head orientation matches a predetermined surrounding monitoring posture, determine that the driver is in a state of surrounding monitoring. Alternatively, the driver state determination unit 56 may, for example, acquire the driver's gaze based on the image from the in-vehicle camera 25, and if the driver's gaze is directed forward, determine that the driver is in a state of surrounding monitoring that meets the suitability criteria. Furthermore, the driver state determination unit 56 may determine that the driver's state is in a state of surroundings monitoring when the driver is gripping the steering wheel 21 based on the signal from the grip sensor 26. Also, the driver state determination unit 56 may determine that the driver's state is in a state of surroundings monitoring when the driver's gaze is directed forward and the steering wheel 21 is being gripped by the driver.

[0029] In Level 3 and above automated driving, where there is no obligation to monitor the surroundings, an abnormal state means a situation in which the driver is unable to promptly take over driving when a request for a driver change occurs. A state in which a driver change is not possible means that the driver is unable to monitor the screen displaying warnings, etc., and includes situations where the driver is asleep or looking behind. In this embodiment, the suitability criteria for Level 3 automated driving include conditions to ensure that the driver fulfills their obligation to monitor the surroundings of the vehicle when notified to do so.

[0030] At each level of autonomous driving, the driver state determination unit 56 determines the driver's state to be abnormal if the pre-set appropriate criteria are not met, as the driver is not in a normal state corresponding to the level of autonomous driving.

[0031] Figure 2 shows an image from the in-vehicle camera 25. As shown in Figure 2, the driver state determination unit 56 sets a predetermined region 37 in the image of the driver's seat captured by the in-vehicle camera 25 as the area where the driver's head should be while driving. In this embodiment, the driver state determination unit 56 sets the predetermined region 37 as a vertically elongated rectangle. In other embodiments, the predetermined region 37 may be set to a shape other than a rectangle. The position and size of the predetermined region 37 should be appropriately set according to the position and size of the driver's head when seated in the driver's seat. Alternatively, the shape of the predetermined region 37 may be set according to the shape of the driver's head.

[0032] The driver status determination unit 56 analyzes the images from the in-vehicle camera 25 and sequentially detects the driver's head. Specifically, the driver status determination unit 56 extracts an edge representing the outline of the driver's face from the driver's seat image, and detects the rectangular area enclosed by vertical lines passing through the left and right ends of the extracted edge and horizontal lines passing through the top and bottom ends of the edge as the head (face). The predetermined area 37 is set to a size that adds predetermined margins to the left, right, top, and bottom of the head, and when the driver is in a state of surrounding monitoring, the driver's face is located approximately in the center of the predetermined area 37. The driver status determination unit 56 obtains the trajectory of the driver's head from the sequentially detected position of the driver's head.

[0033] The driver state determination unit 56 also detects characteristic features such as the eyes, nose, and mouth from the driver's face. Based on the contour of the driver's face and the positions of these characteristic features, the driver state determination unit 56 detects the orientation of the face. The orientation of the face is calculated as the direction of deviation and the degree of deviation (angle) relative to the forward direction in which the driver's face should be facing while driving. The driver state determination unit 56 acquires changes in the orientation of the driver's face from the sequentially detected orientations of the driver's face.

[0034] Figure 3 is an explanatory diagram of changes in head position in the image from the in-vehicle camera 25. A driver in the surrounding monitoring state shown in Figure 2 may tilt their head forward significantly, as shown in Figure 3(A). This may occur, for example, if the driver suddenly becomes ill. In this case, the driver state determination unit 56 detects that the driver's head has moved downward out of the frame from the predetermined area 37 and determines that the driver's state has become unsuitable, no longer meeting the suitability criteria according to the autonomous driving level.

[0035] On the other hand, as shown in Figure 3(B), the driver may move their head from side to side while driving. For example, this may occur when visually checking the white lines through the side window or when visually checking the rear of the vehicle 2. In this case as well, as shown in Figure 3(B2), the driver state determination unit 56 detects that the driver's head has moved out of the frame in the vehicle width direction from the predetermined area 37, and determines that the driver's state has become an unsuitable state that does not meet the predetermined suitability criteria.

[0036] In this manner, the driver state determination unit 56 determines that the driver state is in an unsuitable state when at least one of the predetermined conditions set as appropriate criteria is not met. If the unsuitable state continues for a predetermined time threshold, the driver state determination unit 56 determines that the driver state is abnormal. The driver state determination unit 56 determines that the driver state is normal unless the unsuitable state continues for a predetermined time threshold. Furthermore, even if the driver state has been determined to be abnormal, if the driver state returns from an unsuitable state to an appropriate state such as an ambient monitoring state, the driver state determination unit 56 determines that the driver state is normal.

[0037] Returning to Figure 1, the explanation continues. The operation detection unit 58 acquires the amount of operation of the driving control device 17 based on the signal from the driving control device 17. The operation detection unit 58 acquires the amount of operation of the brake pedal 23, accelerator pedal 22, and steering wheel 21 based on signals from sensors provided on the brake pedal 23, accelerator pedal 22, and steering wheel 21. In other words, the operation detection unit 58 is configured to detect brake pedal operation, accelerator pedal operation, and steering wheel operation by the driver.

[0038] The operation detection unit 58 is configured to detect the operation of the first operation switch 35 and the second operation switch 36 by the occupant based on signals from the first operation switch 35 and the second operation switch 36.

[0039] The driving control unit 52 is configured to control the acceleration and deceleration of the vehicle 2 according to the driving mode. In addition to controlling the acceleration and deceleration of the vehicle 2, the driving control unit 52 may also be configured to control the steering of the vehicle 2. The driving control unit 52 performs automatic driving control, including adaptive cruise control (hereinafter referred to as ACC) and lane keeping assist control (hereinafter referred to as LKAS). Adaptive cruise control is an example of vehicle speed control that controls the speed of the vehicle 2 by controlling the acceleration and deceleration of the vehicle 2. The driving control unit 52 controls the acceleration and deceleration of the vehicle 2 by controlling the propulsion system 3 and the braking system 4, thereby assisting the driver's driving. In addition, lane keeping assist control is an example of steering control that controls the steering of the vehicle 2 by controlling the steering system 5. The driving control unit 52 controls the trajectory of the vehicle 2 by controlling the steering system 5, thereby assisting the driver's driving.

[0040] In addition to the above-mentioned controls that support driving during normal driving, the driving control unit 52 also performs accident reduction system driving support controls to prevent accidents, such as collision mitigation braking (hereinafter referred to as CMBS) and lane departure prevention function. CMBS is implemented by controlling the braking system 4 after issuing a warning to decelerate the vehicle 2 in order to avoid or mitigate a collision when there is a risk of the vehicle 2 colliding with a preceding vehicle or pedestrian. The lane departure prevention function is implemented by controlling the steering system 5 after issuing a warning to change the direction of travel of the vehicle 2 when there is a risk of the vehicle 2 deviating from the driving lane.

[0041] In addition, accident reduction driver assistance controls may include various controls such as blind spot information (hereinafter referred to as BSI) for collision avoidance, accident reduction steering control for collision avoidance, error prevention to prevent driver errors such as sudden starts and rapid acceleration, and warnings for traffic rule violations. BSI is implemented by a control that notifies the driver of the presence of another vehicle when it is detected to be in the driver's blind spot diagonally behind, by displaying a warning mark on the door mirror in that direction. The accident reduction steering function is implemented by providing a warning with sound and a display when the vehicle deviates from the lane on the pedestrian side and there is a risk of collision with a pedestrian, and further assisting steering operation toward the roadway. The error prevention function is implemented by preventing sudden starts and acceleration when the driver presses the accelerator pedal 22 while the vehicle 2 is in motion and an obstacle such as a wall or a vehicle in front is detected at close range, and by providing a warning with sound and a display. The traffic rule violation warning function is implemented by issuing warnings with sound and on the display when traffic rule violations such as speeding, entering an intersection on a red light, entering a no-entry area, or driving the wrong way are occurring.

[0042] These accident reduction driver assistance controls issue warnings to the driver to encourage intervention in driving operations when the driving conditions of vehicle 2 are not in accordance with traffic rules. In other words, accident reduction driver assistance controls are operation request controls that ask the driver to perform driving operations that more appropriately change the driving conditions of vehicle 2.

[0043] These accident reduction driver assistance controls are set to be always executable while the vehicle 2 is in motion, regardless of the operation of the first operation switch 35 and the second operation switch 36. That is, while the vehicle 2 is in motion, the driving control unit 52 constantly determines whether the execution conditions for each control have been met based on the surrounding conditions recognized by the surrounding conditions recognition unit 51. If it determines that the execution conditions for each control have been met, the driving control unit 52 executes the corresponding control and activates the braking system 4, steering system 5, HMI 19, etc.

[0044] When the operation detection unit 58 detects an ON signal from the first operation switch 35, the driving control unit 52 starts driving assistance control. When the operation detection unit 58 detects an OFF signal from the first operation switch 35, the driving control unit 52 terminates driving assistance control. When the operation detection unit 58 detects a selection signal from the second operation switch 36, the driving control unit 52 switches the driving assistance control to be executed or currently executed to the driving assistance control selected by the selection signal. The selection signal from the second operation switch 36 may, for example, select ACC and LKAS, or select LKAS only.

[0045] The mode setting unit 53 is configured to allow the driving mode to be changed. The driving modes include a manual driving mode, a driving assistance mode, and a stationary driving assistance mode. The driving assistance mode includes an automated driving level 1 mode, an automated driving level 2 mode, an automated driving level 3 mode, an accelerator pedal override mode (hereinafter referred to as APOR mode), etc. The automated driving level 1 mode is a mode in which the degree of driving tasks imposed on the driver is greater than that of the automated driving level 2 mode, or in which the degree of assistance to the driver's driving is less. The automated driving level 2 mode is a mode in which the degree of driving tasks imposed on the driver is greater than that of the automated driving level 3 mode, or in which the degree of assistance to the driver's driving is less. The stationary driving assistance mode includes a start-up standby mode and a stop-hold mode. The driving mode transitions according to the occupant's operation, the state of the vehicle 2, and the occupant's state.

[0046] In manual driving mode, the driving control unit 52 controls the vehicle 2 based on the driver's driving operations. Specifically, in manual driving mode, the driving control unit 52 obtains the amount of operation of the brake pedal 23 from the brake pedal 23 and controls the braking system 4 based on the amount of operation of the brake pedal 23. In addition, in manual driving mode, the driving control unit 52 obtains the amount of operation of the accelerator pedal 22 from the accelerator pedal 22 and controls the propulsion system 3 based on the amount of operation of the accelerator pedal 22. In addition, in manual driving mode, the driving control unit 52 obtains the amount of operation of the steering wheel 21 from the steering wheel 21 and controls the steering system 5 based on the amount of operation of the steering wheel 21.

[0047] The driving assistance mode can be set while the vehicle 2 is in motion. When LKAS is selected in the driving assistance mode, the driving control unit 52 performs lane keeping assistance control to drive the vehicle 2 along the lane. Lane information is included in the surrounding conditions recognized by the surrounding conditions recognition unit 51.

[0048] Furthermore, when ACC is selected in the driving assistance mode, the driving control unit 52 controls the acceleration and deceleration of the vehicle 2 according to the surrounding conditions, in addition to controlling the LKAS as described above. In other embodiments, the control of ACC and LKAS may be performed independently. The surrounding conditions include, for example, a vehicle traveling in front of the vehicle 2. In the driving assistance mode, the driving control unit 52 controls the propulsion system 3 and the braking system 4 to maintain a distance of at least a predetermined value between the vehicle 2 and the vehicle traveling in front of the vehicle 2, and to maintain the vehicle speed at a target speed within a range in which the distance can be maintained. The driving control unit 52 acquires the position and speed of the vehicle ahead based on the surrounding conditions acquired by the surrounding conditions recognition unit 51. The target speed may be set by the driver. The target speed may be set by the driver's operation on the display device 31 or operation switches. In addition, in the driving assistance mode, the driving control unit 52 may control the propulsion system 3 and the braking system 4 based on the signal information of traffic lights and sign information acquired by the surrounding conditions recognition unit 51.

[0049] APOR mode is set when the driving mode is in driver assistance mode and the accelerator pedal 22 is pressed. In APOR mode, the driving control unit 52 controls the propulsion system 3 based on the amount the accelerator pedal 22 is pressed. This enables the vehicle 2 to accelerate in response to the driver's accelerator pedal operation.

[0050] In driving assistance mode, when the vehicle in front slows down and stops, the driving control unit 52 maintains a safe distance from the vehicle in front and stops the vehicle 2. In addition, in driving assistance mode, the driving control unit 52 may acquire traffic signal information and stop the vehicle 2 at the stop line according to the traffic signal information.

[0051] Furthermore, if LKAS is selected in the driver assistance mode, the driving control unit 52 performs lane keeping assistance control to drive the vehicle 2 along the lane, in addition to the ACC described above. The driving control unit 52 performs driving control, including driver assistance, based on the determination result of the driver state determination unit 56.

[0052] The notification unit 59 notifies the driver via the HMI 19 when the driver status determination unit 56 determines that the driver's status is abnormal.

[0053] The emergency control unit 54 executes emergency control to ensure the safety of the vehicle 2 and the driver when the driver status determination unit 56 determines that the driver is abnormal. Emergency control is executed regardless of the driving level or the driving state of the vehicle 2. For example, emergency control is executed when the vehicle 2 is stopped, driving, when ACC or LKAS is running, or when ACC and LKAS are running. Emergency control takes precedence over driving operations by the driver and driving assistance control performed by the driving control unit 52, and is performed as follows: When driver abnormality is determined, the emergency control unit 54 executes in-vehicle notification to the occupants, acceleration suppression of the vehicle 2, and lane keeping control. If the driver abnormality continues, the emergency control unit 54 confirms the driver abnormality and executes external notification to outside the vehicle, deceleration of the vehicle 2, and lane change control, bringing the vehicle 2 to a stop in the driving lane or on the shoulder of the road. After the vehicle 2 has stopped, the emergency control unit 54 executes control to contact a hospital or other facility and to maintain the vehicle's stop.

[0054] Next, an example of a vehicle control procedure performed by the vehicle control device 1 will be described. This example will describe the case where Level 1 or 2 automated driving control is being performed.

[0055] Figure 4 is a flowchart showing the procedure for driving assistance control by the control device. When the ignition switch of the vehicle 2 is turned on, the vehicle control device 1 starts the vehicle control shown in Figure 4. First, the vehicle control device 1 starts monitoring the driver using the occupant monitoring device 18 (ST1). In monitoring the driver, the vehicle control device 1 recognizes the driver's condition based on the occupant monitoring device 18 and determines whether the driver's condition is normal or abnormal.

[0056] Next, the vehicle control device 1 sets a preset first time T1 as the time threshold for an unsuitable state (ST2). The process in step ST2 may be performed before the process in step ST1.

[0057] After the processing in steps ST1 and ST2, the vehicle control device 1 determines whether the driver's unsuitable state continues for a first time T1 or longer (ST3). The count of unsuitable states starts from the time the driver state determination unit 56 determines that the driver's state is unsuitable and is reset when the driver's state returns to the surrounding monitoring state (appropriate state).

[0058] If the driver's unsuitable condition has not continued for a first time T1 or longer (ST3: No), the vehicle control device 1 proceeds to step ST4. If the driver's unsuitable condition has continued for a first time T1 or longer (ST3: Yes), the vehicle control device 1 proceeds to step ST8. In step ST4, the vehicle control device 1 determines whether a specific driving assistance, specifically an accident reduction system driving assistance, is being performed by the driving control unit 52. If the accident reduction system driving assistance is not being performed (ST4: No), the vehicle control device 1 returns to step ST3.

[0059] On the other hand, if the accident reduction system's driving assistance is being implemented (ST4: Yes), the vehicle control device 1 sets the second time T2 to the time threshold for an unsuitable state (ST5). The second time T2 is shorter than the first time T1. The second time T2 may be a constant value, or it may be a different value depending on the mode of automatic driving control by the driving control unit 52, and the control state such as the type and amount of driving assistance control of the accident reduction system. Details of the second time T2 will be explained later.

[0060] After the processing in step ST5, the vehicle control device 1 determines whether the driver's unsuitable state continues for a second time T2 or longer (ST6). Here again, the count of the unsuitable state starts from the time the driver state determination unit 56 determines that the driver's state is unsuitable, and is reset when the driver's state returns to the surrounding monitoring state (appropriate state).

[0061] However, if a driver's inappropriate condition occurs before the process proceeds to step ST5, that is, before the accident reduction system's driver assistance is implemented, the count that started before the driver assistance was implemented will continue as is.

[0062] If the count that started before the execution of the driving assistance is 2 hours T2 or longer when the process proceeds to step ST6, then it is determined that the driver's unsuitable state has continued for 2 hours T2 or longer (Yes) when the process proceeds to step ST6. In this case, the vehicle control device 1 immediately proceeds to step ST8.

[0063] If, in the process of step ST6, the driver's unsuitable state does not continue for a second time T2 or longer (No), the vehicle control device 1 determines whether a predetermined time has elapsed since the start of the accident reduction system's driving assistance (when ST4 becomes Yes) (ST7). The accident reduction system's driving assistance is not performed for a very long time, and the predetermined time is set to a value that exceeds the normal execution time of the accident reduction system's driving assistance.

[0064] If a predetermined time has not elapsed since the start of the accident reduction system's driver assistance (ST7: No), the vehicle control device 1 returns to step ST6. On the other hand, if a predetermined time has elapsed since the start of the accident reduction system's driver assistance (ST7: Yes), the vehicle control device 1 returns to step ST2 and sets the first time T1 as the time threshold for the unsuitable state.

[0065] If the driver's unsuitable condition persists for a first time T1 or longer in step ST3 (Yes), and if the driver's unsuitable condition persists for a second time T2 or longer in step ST6 (Yes), the vehicle control device 1 determines that the driver's condition is abnormal (ST8). Subsequently, the vehicle control device 1 executes the emergency control described above (ST9).

[0066] The second time T2, which is set as the time threshold in step ST5, is shorter than the first time T1, which is set as the time threshold in step ST2. In other words, if the accident reduction system's driving assistance is performed (ST4:Yes), the time threshold will be set to a smaller value than when the accident reduction system's driving assistance is not performed (ST4:No), unless a predetermined time has elapsed since then (ST7:No). As a result, the driver's condition is more likely to be judged as abnormal.

[0067] Next, the operation of the vehicle 2 when the vehicle control device 1 performs such vehicle control will be explained with reference to Figures 5 and 6.

[0068] Figure 5 is a time chart showing an example of driving assistance control when the vehicle control unit 1 does not perform accident reduction system driving assistance. At time t1, the vehicle control unit 1 determines that the driver's state has changed from an appropriate state (surround monitoring state) for Level 2 autonomous driving to an inappropriate state. Since the accident reduction system driving assistance is not performed, the time threshold is set to the first time T1. At time t2, when the inappropriate state has continued for the first time T1, the vehicle control unit 1 determines that the driver's state is abnormal. Subsequently, the vehicle control unit 1 performs in-vehicle notification to the occupants, acceleration suppression of the vehicle 2, and lane keeping control.

[0069] At point t3, if the driver's unsuitable condition persists for a predetermined period despite the in-vehicle notification being made, the vehicle control device 1 confirms that the driver's condition is abnormal. Upon confirming the abnormal condition, the vehicle control device 1 executes various emergency controls, including external notification to outside the vehicle, deceleration of vehicle 2, and lane changes. At point t4, when the vehicle control device 1 has brought vehicle 2 to a stop in the driving lane or on the shoulder, the vehicle control device 1 executes controls for contacting a hospital or other facility and maintaining the vehicle's stop.

[0070] Figure 6 is a time chart showing an example of driving assistance control when the vehicle control device 1 performs accident reduction system driving assistance. At time t11, the vehicle control device 1 determines that the driver's state has changed from an appropriate state (surround monitoring state) for Level 2 autonomous driving to an inappropriate state. At this time t11, the accident reduction system driving assistance is not being performed, so the time threshold is set to the first time T1.

[0071] At time t12, before the first time period T1 has elapsed from time t11, the accident reduction system's driver assistance is activated. Then, the time threshold is set to a second time period T2, which is shorter than the first time period T1. In the illustrated example, at time t12, when the accident reduction system's driver assistance is activated, the count of unsuitable states is shorter than the second time period T2. Therefore, the count that started before the accident reduction system's driver assistance is activated continues.

[0072] At time t13, when the unsuitable condition has persisted from time t11 to the second time period T2, the vehicle control device 1 determines that the driver's condition is abnormal. Subsequently, the vehicle control device 1 performs the following controls: in-vehicle notification to the occupants, acceleration suppression of the vehicle 2, and lane keeping.

[0073] If, for example, the accident reduction system's driver assistance is executed after time t13, which is two hours T2 after time t11, the count of the unsuitable state will have exceeded two hours T2 at the time the accident reduction system's driver assistance is executed, and the vehicle control device 1 will immediately determine that the driver's state is abnormal. In this case, immediately after the accident reduction system's driver assistance is executed, the vehicle control device 1 will perform in-vehicle notification to the occupants, acceleration suppression of the vehicle 2, and lane keeping control.

[0074] Alternatively, the vehicle control device 1 may determine that the driver's condition is abnormal immediately when the accident reduction system's driving assistance is implemented, that is, as soon as the driving assistance is implemented, independently of the count for determining driver abnormality.

[0075] At time t14, if the driver's unsuitable condition persists for a predetermined period despite the in-vehicle notification being made, the vehicle control device 1 confirms that the driver's condition is abnormal. Upon confirming the abnormal condition, the vehicle control device 1 executes external notification to the outside of the vehicle, deceleration of vehicle 2, and lane change controls. At time t14, when the vehicle control device 1 has stopped vehicle 2 in the driving lane or on the shoulder, the vehicle control device 1 executes emergency controls, including contacting a hospital or other emergency facility and maintaining the vehicle's stop.

[0076] Therefore, when the vehicle control device 1 performs the accident reduction system's driving support (Figure 6), it activates the notification (time t13) earlier than at time t2, and starts the deceleration of vehicle 2 (time t14) and stops vehicle 2 (time t15) earlier than at times t3 and t4, compared to when the accident reduction system's driving support is not performed (Figure 5).

[0077] As described above, the second time T2 is set to different values ​​depending on the mode of automatic driving control by the driving control unit 52, and the control state such as the type and amount of accident reduction system driving support control. For example, when the vehicle control device 1 is executing both CMBS and the road departure prevention function, it may set the second time T2 to a smaller value than when only the road departure prevention control is being executed. Also, when the vehicle control device 1 is executing either CMBS or the road departure prevention function, it may set the second time T2 to a smaller value than when the traffic rule violation warning control is being executed.

[0078] Furthermore, when the vehicle control device 1 is operating in autonomous driving level 2 mode, it may set the second time T2 to a smaller value compared to when it is operating in autonomous driving level 1 mode. In this case, when operating in autonomous driving level 2 mode, the driver's condition is more likely to be judged as abnormal compared to when operating in autonomous driving level 1 mode.

[0079] Thus, according to the vehicle control device 1 of this embodiment, abnormal driver conditions are more easily detected when the driver assistance unit performs driver assistance. Therefore, it is possible to appropriately detect abnormal driver conditions according to the driving scene and suppress delays in notifying the driver.

[0080] The above embodiment is configured as follows.

[0081] The vehicle control device 1 includes a surrounding environment recognition unit 51 that recognizes the surrounding environment of the vehicle 2, a driving support unit 52 that provides driving support for the vehicle 2 based on the recognition results of the surrounding environment recognition unit 51, and a driver state determination unit 56 that determines whether the driver's condition is normal or abnormal based on images captured by an in-vehicle camera 25 that captures images of the driver of the vehicle 2. The driver state determination unit 56 is configured to make it easier to determine that the driver's condition is abnormal when driving support is being performed by the driving support unit 52 compared to when driving support is not being performed.

[0082] In this embodiment, when the driving control unit 52 performs driving assistance, abnormal driver conditions are more easily detected by the driver condition determination unit 56. As a result, abnormal driver conditions are appropriately detected according to the driving scene, and delays in notifying the driver are suppressed. This improves safety while the driving control unit 52 is performing driving assistance.

[0083] The driver status determination unit 56 sets a time threshold based on the driving assistance provided by the driving control unit 52, and may determine the driver's status to be abnormal if the driver, as captured by the in-vehicle camera 25, remains in an unsuitable state that does not meet predetermined suitability criteria for a period of time (first time T1 or second time T2) or longer. The driver status determination unit 56 may set a smaller time threshold when driving assistance is being performed by the driving control unit 52 compared to when driving assistance is not being performed.

[0084] According to this embodiment, when driver assistance is being performed, the time threshold for driver abnormality detection is set to be short, thereby enabling early detection of driver abnormalities.

[0085] The vehicle control device 1 further includes a notification unit 59 that notifies the driver when the driver status determination unit 56 determines that the driver's status is abnormal. The notification unit 59 may activate the notification earlier when driving assistance is being performed by the driving control unit 52 compared to when driving assistance is not being performed.

[0086] According to this configuration, if a driver abnormality is detected while driver assistance is being performed, safety can be improved by providing earlier notification. Furthermore, when driver assistance is not being performed, notification is given later than when driver assistance is being performed, so the driver does not feel bothered.

[0087] The vehicle control device 1 further includes an emergency control unit 54 that decelerates or stops the vehicle 2 when the driver state determination unit 56 determines that the driver's state is abnormal. The emergency control unit 54 may start decelerating the vehicle 2 or stop the vehicle 2 earlier than when driving assistance is being performed by the driving control unit 52, compared to when driving assistance is not being performed.

[0088] According to this configuration, if a driver abnormality is detected during the operation of driver assistance, safety can be improved by accelerating the deceleration or stopping of vehicle 2. Furthermore, when driver assistance is not being performed, the deceleration or stopping of vehicle 2 occurs more slowly than when driver assistance is being performed, so the driver does not experience any inconvenience.

[0089] The driving control unit 52 is capable of executing multiple driver assistance controls for the vehicle 2, and the driver assistance being executed when the driver's condition is likely to be judged as abnormal may be a specific driver assistance control from among the multiple driver assistance controls.

[0090] According to this embodiment, when a specific driver assistance control is executed, the driver status determination unit 56 is more likely to detect abnormal driver conditions. As a result, abnormal driver conditions are appropriately determined according to the driving scene, and delays in notifying the driver are suppressed.

[0091] A specific driver assistance control may be a driver assistance control that includes operation request control that requests the driver to perform a driving operation that changes the driving conditions of vehicle 2.

[0092] According to this embodiment, when driver assistance control, including operation request control of an accident reduction system that requests such driving operations from the driver, is executed, abnormalities in the driver's condition are more easily detected. As a result, safety is further improved while such accident reduction system driver assistance is being executed.

[0093] The specific driver assistance control may be any of the following: collision avoidance control to avoid collision between the vehicle 2 and an obstacle recognized by the surrounding situation recognition unit 51; road departure prevention control to prevent the vehicle 2 from deviating from its own lane; or driver error prevention control to prevent driver error.

[0094] According to this embodiment, abnormal driver conditions are more easily detected when such accident reduction driver assistance systems are in operation. This further improves safety while these accident reduction driver assistance systems are running.

[0095] The driver status determination unit 56 may be configured to make it easier to determine the driver's status as abnormal when driving assistance is being performed, compared to when driving assistance is not being performed, if the driver, as captured by the in-vehicle camera 25, has been in an unsuitable state that does not meet predetermined suitability criteria since before the start of driving assistance execution.

[0096] According to this embodiment, if the driver's unsuitable condition persists from before the start of the driving assistance system, the driver's abnormal condition can be more easily determined based on the driver's condition before the start of the driving assistance system. This further improves safety while the driving assistance system is being operated by the driving control unit 52.

[0097] The driver state determination unit 56 determines that the driver's state is abnormal if the unsuitable state persists for a predetermined time threshold (first time T1 or second time T2) or longer. The driver state determination unit 56 may set the time threshold shorter when driving assistance is being performed by the driving control unit 52 compared to when driving assistance is not being performed.

[0098] According to this embodiment, if the driver's unsuitable condition persists before the start of driver assistance, the driver's abnormal condition can be determined earlier based on the driver's condition before the start of driver assistance. For example, it is possible to determine the driver's abnormal condition at the same time as the start of driver assistance.

[0099] If the driver state determination unit 56 determines that an unsuitable state that persisted before the start of driving assistance by the driving control unit 52 has disappeared after the start of driving assistance, it may return the shortened time threshold to its original value.

[0100] According to this embodiment, it is possible to determine the driver's abnormal condition earlier when driving assistance is performed, and it is possible to suppress the detection of the driver's abnormal condition excessively early after the unsuitable condition has been eliminated.

[0101] The driver status determination unit 56 may determine that the driver's status is abnormal if the driving assistance provided by the driving control unit 52 is performed.

[0102] According to this embodiment, abnormal driver conditions can be detected earlier when driver assistance is being performed. This further improves safety while driver assistance is being performed by the driving control unit 52.

[0103] This concludes the description of specific embodiments, but the present invention is not limited to the configuration of the above embodiments and can be broadly modified and implemented.

[0104] For example, in the above embodiment, the driver state determination unit 56 sets a time threshold based on driving assistance and determines the driver state to be abnormal if the unsuitable state continues for longer than the time threshold. However, based on driving assistance, the predetermined region 37 in the image shown in Figure 2 may be set to a different size and the time threshold may be fixed.

[0105] In addition, the specific configuration, arrangement, quantity, and materials of each component and part, as well as the specific content and order of each process, can be modified as appropriate, as long as they do not depart from the spirit of the present invention. Furthermore, not all of the components shown in the above embodiments are necessarily essential, and can be selected as appropriate. [Explanation of Symbols]

[0106] 1: Vehicle control system 2: Vehicles 25: In-car camera 51: Surrounding Situation Recognition Unit 52: Driving control unit 54: Emergency Control Unit 56: Driver status determination unit 59: Hochi Department T1: First hour T2: Second hour

Claims

1. A surrounding environment recognition unit that recognizes the surrounding conditions of the vehicle, A driving support unit that provides driving support for the vehicle based on the recognition results of the surrounding situation recognition unit, The system includes a driver status determination unit that determines whether the driver's condition is normal or abnormal based on an image captured by an in-vehicle camera that captures the driver of the vehicle. The driver state determination unit is configured to make it easier to determine the driver's state as abnormal when the driving assistance unit is performing the driving assistance compared to when the driving assistance is not performing, in a vehicle control device.

2. (Proposal 8) The driver status determination unit sets a time threshold based on the driving assistance provided by the driving assistance unit, and determines that the driver's status is abnormal if the driver, as captured by the in-vehicle camera, remains in an unsuitable state that does not meet predetermined suitability criteria for a period of time longer than the time threshold. The vehicle control device according to claim 1, wherein the driver state determination unit sets the time threshold smaller when the driving assistance is being performed by the driving assistance unit compared to when the driving assistance is not being performed.

3. The driver status determination unit further includes a notification unit that notifies the driver when it determines that the driver's status is abnormal. The vehicle control device according to claim 2, wherein the notification unit accelerates the operation of the notification when the driving assistance is being performed by the driving assistance unit compared to when the driving assistance is not being performed.

4. The vehicle is further provided with an emergency control unit that decelerates or stops the vehicle when the driver status determination unit determines that the driver's status is abnormal. The vehicle control device according to claim 3, wherein the emergency control unit accelerates the start of deceleration of the vehicle or the stopping of the vehicle when the driving assistance is being performed by the driving assistance unit, compared to when the driving assistance is not being performed.

5. The aforementioned driving support unit is capable of performing multiple driving support controls for the vehicle, The vehicle control device according to any one of claims 1 to 4, wherein the aforementioned driving assistance is provided by a specific driving assistance control among the plurality of driving assistance controls.

6. The vehicle control device according to claim 5, wherein the specific driving assistance control includes an operation request control that requests the driver to perform a driving operation that changes the driving conditions of the vehicle.

7. The vehicle control device according to claim 6, wherein the specific driving assistance control is any one of the following: collision avoidance control to avoid a collision between the vehicle and an obstacle recognized by the surrounding situation recognition unit; road departure prevention control to prevent the vehicle from deviating from the lane in which it is traveling; or error prevention control to prevent the driver from making a driving error.

8. The vehicle control device according to claim 1, wherein the driver status determination unit is configured to make it easier to determine the driver's status as abnormal when the driver is being driven, compared to when the driver is not being driven, if the driver, as captured by the in-vehicle camera, has been in an unsuitable state that does not meet predetermined suitability criteria since before the start of the driving assistance.

9. The driver state determination unit determines that the driver state is abnormal if the unsuitable state persists for a predetermined time threshold or longer. The vehicle control device according to claim 8, wherein the driver state determination unit sets the time threshold shorter when the driving assistance is being performed by the driving assistance unit compared to when the driving assistance is not being performed.

10. The vehicle control device according to claim 9, wherein the driver state determination unit returns the shortened time threshold to its original value if the unsuitable state that persisted from before the start of the driving assistance is eliminated after the driving assistance is performed.

11. The vehicle control device according to any one of claims 1 to 4, wherein the driver state determination unit determines that the driver's state is abnormal when the driving assistance unit performs the driving assistance.

Images