Vehicle control system

Abstract

To suppress over-judgment and over-notification during driving control, the system will appropriately detect driver abnormalities and improve the accuracy of driver detection. [Solution] The vehicle control device 1 includes a surrounding situation recognition unit 51 that recognizes the surrounding situation of the vehicle 2, a driving control unit 52 that performs driving control of the vehicle 2 based on the recognition results of the surrounding situation recognition unit 51, and a driver state determination unit 56 that determines whether the driver's state is normal or abnormal based on images captured by an in-vehicle camera 25 that captures images of the driver of the vehicle 2. The driver state determination unit 56 is configured to make it more difficult to determine the driver's state as abnormal when driving control is being performed by the driving control unit 52 compared to when driving control is not being performed.

Description

Technical Field

[0001] The present invention relates to a vehicle control device that controls the running of a vehicle, and more particularly to a vehicle control device that performs running control of a vehicle based on the state of a driver of the vehicle.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transport system that takes into account people who are in a vulnerable position among traffic participants. In order to achieve this, efforts to further improve traffic safety and convenience through research and development of driving support technologies have attracted attention.

[0003] Conventionally, a driving control device including a peripheral detection unit that detects the surrounding situation of a vehicle and a lane keeping control unit that performs lane keeping control based on the detection result of lane marks by the peripheral detection unit is known (see, for example, Patent Document 1). This driving control device determines whether or not a lane change can be made according to the driver's intention to change lanes, and notifies the driver of the determination result by a display and / or a speaker. In such a vehicle in which such driving support is performed, while the driving load of the driver is reduced, the concentration of the driver on driving is likely to decrease.

[0004] As a device for detecting the driver's distracted state, Patent Document 2 discloses a driver state detection device that can accurately determine whether or not the driver is in a distracted state by using the change in the expression of the driver's face. This device detects a change in a specific part of the driver's face, and determines that the driver is in a distracted state when the duration of the temporary change in the specific part is longer than a predetermined time set in advance.

[0005] Patent Document 3 discloses a concentration level determination device that takes into consideration both driving safety and driving comfort. This device comprises a monitoring data acquisition unit that acquires monitoring data from sensors that monitor the vehicle driver, and a concentration level estimation unit that estimates the driver's level of driving concentration based on predetermined indicators from the monitoring data. If the level of driving concentration does not meet the standard, an instruction signal is output to instruct the driver to provide assistance. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2017-146817 [Patent Document 2] Japanese Patent Publication No. 2016-71577 [Patent Document 2] Japanese Patent Publication No. 2018-149941 [Overview of the project] [Problems that the invention aims to solve]

[0007] Incidentally, when the vehicle's driving control system is in operation, the driver's driving posture is more likely to become unstable compared to when driving manually. Therefore, if an abnormal condition of the driver is determined based on the driver's driving posture, and the same criteria are used for determination during vehicle driving control as during manual driving, the determination of abnormal conditions may become excessive. Furthermore, if the driver is notified each time an abnormal condition is determined, the notifications during driving control may become excessively frequent, potentially causing annoyance to the driver.

[0008] In view of the above background, the present invention aims to improve the accuracy of driver judgment by appropriately detecting driver abnormalities in order to suppress over-judgment and over-notification during driving control. Ultimately, this will contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0009] To solve the above problems, one aspect of the present invention provides a vehicle control device comprising: a surrounding situation recognition unit that recognizes the surrounding situation of a vehicle; a driving control unit that performs driving control of the vehicle based on the recognition results of the surrounding situation recognition unit; and a driver state determination unit that determines whether the driver's state is normal or abnormal based on an image captured by an in-vehicle camera that captures the driver of the vehicle, wherein the driver state determination unit is configured to make it more difficult to determine the driver's state as abnormal when the driving control is being performed by the driving control unit compared to when the driving control is not being performed. [Effects of the Invention]

[0010] According to the above embodiment, it is possible to appropriately determine driver abnormalities and improve the accuracy of driver detection. [Brief explanation of the drawing]

[0011] [Figure 1] Configuration diagram of a vehicle system according to an embodiment of the present invention [Figure 2] Diagram showing images from an in-car camera. [Figure 3] An explanatory diagram illustrating an example of unsuitable driving behavior as seen in images from an in-car camera. [Figure 4] Flowchart showing the procedure for driving assistance control by the control device. [Figure 5] A time chart of an example of driving assistance control when driving control is not being performed. [Figure 6] A time chart of an example of driving assistance control when driving control is being performed. [Modes for carrying out the invention]

[0012] As shown in Figure 1, the vehicle control device 1 is installed in the vehicle 2. The vehicle 2 may be, for example, a four-wheeled automobile. The vehicle 2 is an autonomous vehicle or a vehicle with driver assistance functions.

[0013] Vehicle 2 has a propulsion system 3, a braking system 4, and a steering system 5. The propulsion system 3 is a device that provides driving force to vehicle 2 and includes, for example, a power source and a transmission. The power source has at least one of an internal combustion engine such as a gasoline engine or a diesel engine and an electric motor. The braking system 4 is a device that provides braking force to vehicle 2 and includes, for example, a brake caliper that presses pads against a brake rotor and an electric cylinder that supplies hydraulic pressure to the brake caliper. The steering system 5 is a device for changing the steering angle of the wheels and includes, for example, a rack and pinion mechanism that steers the wheels and an electric motor that drives the rack and pinion mechanism. The propulsion system 3, the braking system 4, and the steering system 5 are controlled by a vehicle control device 1.

[0014] Vehicle 2 has an external environment recognition device 7. The external environment recognition device 7 is a device that detects objects outside the vehicle. The external environment recognition device 7 is a sensor that detects objects outside the vehicle by capturing electromagnetic waves and light from the surroundings of Vehicle 2. The external environment recognition device 7 includes, for example, a radar 8, a LiDAR 9, and an external camera 10.

[0015] Vehicle 2 has vehicle sensors 12. Vehicle sensors 12 include a vehicle speed sensor 13 for detecting the speed of vehicle 2, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting angular velocity around the vertical axis, and an orientation sensor for detecting the orientation of vehicle 2.

[0016] Vehicle 2 is equipped with a communication device 15, a navigation device 16, a driving control device 17, an occupant monitoring device 18, and an HMI 19 (Human Machine Interface). The communication device 15 mediates communication between the vehicle control device 1 and the navigation device 16 and surrounding vehicles and servers located outside the vehicle.

[0017] The navigation device 16 is a device that acquires the current position of the vehicle 2 and provides route guidance to the destination. The navigation device 16 preferably includes a GNSS receiver, a map storage unit, a navigation interface, and a route determination unit. The GNSS receiver identifies the position (latitude and longitude) of the vehicle 2 based on signals received from artificial satellites (positioning satellites). The map storage unit is composed of a known storage device such as a flash memory or a hard disk and stores map information. The navigation interface accepts inputs such as the destination from the occupant and presents various information to the occupant by display and voice. The navigation interface preferably includes, for example, a touch panel display and a speaker 32, etc.

[0018] The driving operation device 17 accepts input operations performed by the occupant (driver) to control the vehicle 2. The driving operation device 17 includes a steering wheel 21, an accelerator pedal 22, and a brake pedal 23. Further, the driving operation device 17 may include a shift lever, a parking brake lever, etc. A sensor for detecting the operation amount is attached to each driving operation device 17. The driving operation device 17 outputs a signal indicating the operation amount to the vehicle control device 1.

[0019] The occupant monitoring device 18 monitors the state of the occupants in the vehicle interior. The occupant monitoring device 18 preferably has, for example, an in-vehicle camera 25 that images the occupants sitting on the seats in the vehicle interior and a gripping sensor 26 provided on the steering wheel 21. The in-vehicle camera 25 is a digital camera that uses a solid-state imaging device such as a CCD or a CMOS, etc. The gripping sensor 26 detects whether the driver is gripping the steering wheel 21. The gripping sensor 26 is preferably formed by, for example, a capacitance sensor or a piezoelectric element provided on the steering wheel 21.

[0020] The HMI 19 provides the occupant with various information through displays and sounds, and also accepts input operations from the occupant. The HMI 19 includes a display device 31 and a speaker 32. The display device 31 may be a touch panel display including liquid crystal or organic EL. The display device 31 may also serve as a navigation interface. The display device 31 and speaker 32 function as notification devices for providing information to the occupant through images and sounds. Here, the images may be videos containing multiple consecutive frames. The HMI 19 may also include various actuators. For example, the actuators may be those that act on the driver's six senses, such as a vibration actuator built into the steering wheel 21 or a belt tightening device built into the seat belt retractor.

[0021] Vehicle 2 has a first operation switch 35 and a second operation switch 36. The first operation switch 35 and the second operation switch 36 are switches that can be operated by the occupants. The first operation switch 35 and the second operation switch 36 may be mechanical switches or GUI switches displayed on a touch panel, and are arranged in appropriate places in the vehicle interior. The first operation switch 35 and the second operation switch 36 may be configured as a display device 31 or a navigation interface. The first operation switch 35 may be a switch for changing the on / off state of driver assistance control. The second operation switch 36 may be a switch for selecting a driver assistance control to be executed from among several driver assistance controls, and for setting the level of automated driving. The second operation switch 36 may be, for example, a rotary switch. The first operation switch 35 and the second operation switch 36 may be configured as a single unit as a common switch.

[0022] The vehicle control device 1 is a computer having a processor 41 and a memory 42 that is communicatively connected to the processor 41. The processor 41 may include at least one of the following as its core: a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a RISC (Reduced Instruction Set Computer). The memory 42 stores control programs executed by the processor 41 and various data. The memory 42 may include at least one of volatile memory and non-volatile memory. The volatile memory may be, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The non-volatile memory may be an SSD (Solid State Drive), flash memory, magnetic disk storage device, or optical disk storage device. At least a part of the vehicle control device 1 may be implemented by hardware such as an LSI (Large Scale Integration), ASIC (application specific integrated circuit), or FPGA (field-programmable gate array), or by a combination of software and hardware. The vehicle control device 1 may be composed of a single piece of hardware, or it may be composed of multiple pieces of hardware that can communicate with each other. Part of the vehicle control device 1 may be composed of an external server located outside the vehicle 2.

[0023] The processor 41 implements various applications by executing control programs stored in memory 42. The control programs may be stored on removable recordable media such as DVDs or CD-ROMs, and installed in memory 42 when the recordable media is read by a reader. Alternatively, the control programs may be downloaded to and installed in memory 42 via a communication network such as the Internet.

[0024] The processor 41 functions as the surrounding situation recognition unit 51, the driving control unit 52, the mode setting unit 53, the emergency control unit 54, the driver status recognition unit 55, the driver status determination unit 56, the vehicle position recognition unit 57, the operation detection unit 58, and the notification unit 59 by executing the control program stored in the memory 42. As the processor 41 executes the control program, the vehicle control device 1, which is a computer, executes the vehicle control method.

[0025] The surrounding environment recognition unit 51 recognizes the surrounding environment of the vehicle 2. Based on the detection results of the external environment recognition device 7, the surrounding environment recognition unit 51 recognizes the surrounding environment (external environment), including obstacles located around the vehicle 2, the shape of the road, the presence or absence of sidewalks, road markings, etc. Obstacles include, for example, guardrails, utility poles, surrounding vehicles, and people such as pedestrians. From the detection results of the external environment recognition device 7, the surrounding environment recognition unit 51 can acquire the status of surrounding vehicles, such as their position, speed, and acceleration.

[0026] The vehicle position recognition unit 57 recognizes the driving lane in which the vehicle 2 is traveling, as well as the relative position and angle of the vehicle 2 with respect to the driving lane. The vehicle position recognition unit 57 may recognize the driving lane based on map information and the position of the vehicle 2 acquired by the GNSS receiver. In addition, the vehicle position recognition unit 57 may extract the lane markings around the vehicle 2 drawn on the road surface from the map information and compare them with the shape of the lane markings captured by the external camera 10 to recognize the relative position and angle of the vehicle 2 with respect to the driving lane.

[0027] The driver state recognition unit 55 recognizes the driver's status. Based on the detection results of the occupant monitoring device 18, the driver state recognition unit 55 recognizes the driver's status and the driver's holding status of the steering wheel 21. The driver's status includes the driver's head position, face orientation, eye opening / closing state, and gaze direction. The driver state recognition unit 55 can acquire this driver status information from the detection results of the in-vehicle camera 25.

[0028] The driver state determination unit 56 determines the driver's state. Based on the signal from the occupant monitoring device 18, the driver state determination unit 56 determines whether the driver is in a state of surrounding monitoring. The state of surrounding monitoring is a state suitable for driving in which the driver is monitoring the surroundings of the vehicle 2 and meets predetermined suitability criteria, and is required in automated driving of level 2 or lower, where surrounding monitoring is mandatory. In the state of surrounding monitoring, it is preferable that the driver is in a state in which they can quickly start manual driving. For example, in the state of surrounding monitoring, the driver is seated facing forward in the seat and is monitoring the area in front of the vehicle 2. The driver state determination unit 56 may, for example, acquire the driver's posture or head orientation based on the image from the in-vehicle camera 25, and if the driver's posture or head orientation matches a predetermined surrounding monitoring posture, determine that the driver is in a state of surrounding monitoring. Alternatively, the driver state determination unit 56 may, for example, acquire the driver's gaze based on the image from the in-vehicle camera 25, and if the driver's gaze is directed forward, determine that the driver is in a state of surrounding monitoring that meets the suitability criteria. Furthermore, the driver state determination unit 56 may determine that the driver's state is in a state of surroundings monitoring when the driver is gripping the steering wheel 21 based on the signal from the grip sensor 26. Also, the driver state determination unit 56 may determine that the driver's state is in a state of surroundings monitoring when the driver's gaze is directed forward and the steering wheel 21 is being gripped by the driver.

[0029] In Level 3 and above automated driving, where there is no obligation to monitor the surroundings, an abnormal state means a situation in which the driver is unable to promptly take over driving when a request for a driver change occurs. A state in which a driver change is not possible means that the driver is unable to monitor the screen displaying warnings, etc., and includes situations where the driver is asleep or looking behind. In this embodiment, the suitability criteria for Level 3 automated driving include conditions to ensure that the driver fulfills their obligation to monitor the surroundings of the vehicle when notified to do so.

[0030] At each level of autonomous driving, the driver state determination unit 56 determines the driver's state to be abnormal if the pre-set appropriate criteria are not met, as the driver is not in a normal state corresponding to the level of autonomous driving.

[0031] Figure 2 shows an image from the in-vehicle camera 25. As shown in Figure 2, the driver state determination unit 56 sets a predetermined region 37 in the image of the driver's seat captured by the in-vehicle camera 25 as the area where the driver's head should be while driving. In this embodiment, the driver state determination unit 56 sets the predetermined region 37 as a vertically elongated rectangle. In other embodiments, the predetermined region 37 may be set to a shape other than a rectangle. The position and size of the predetermined region 37 should be appropriately set according to the position and size of the driver's head when seated in the driver's seat. Alternatively, the shape of the predetermined region 37 may be set according to the shape of the driver's head.

[0032] The driver status determination unit 56 analyzes the images from the in-vehicle camera 25 and sequentially detects the driver's head. Specifically, the driver status determination unit 56 extracts an edge representing the outline of the driver's face from the driver's seat image, and detects the rectangular area enclosed by vertical lines passing through the left and right ends of the extracted edge and horizontal lines passing through the top and bottom ends of the edge as the head (face). The predetermined area 37 is set to a size that adds predetermined margins to the left, right, top, and bottom of the head, and when the driver is in a state of surrounding monitoring, the driver's face is located approximately in the center of the predetermined area 37. The driver status determination unit 56 obtains the trajectory of the driver's head from the sequentially detected position of the driver's head.

[0033] The driver state determination unit 56 also detects characteristic features such as the eyes, nose, and mouth from the driver's face. Based on the contour of the driver's face and the positions of these characteristic features, the driver state determination unit 56 detects the orientation of the face. The orientation of the face is calculated as the direction of deviation and the degree of deviation (angle) relative to the forward direction in which the driver's face should be facing while driving. The driver state determination unit 56 acquires changes in the orientation of the driver's face from the sequentially detected orientations of the driver's face.

[0034] Figure 3 is an explanatory diagram of changes in head position in the image from the in-vehicle camera 25. A driver in the surrounding monitoring state shown in Figure 2 may tilt their head forward significantly, as shown in Figure 3(A). This may occur, for example, if the driver suddenly becomes ill. In this case, the driver state determination unit 56 detects that the driver's head has moved downward out of the frame from the predetermined area 37 and determines that the driver's state has become unsuitable, no longer meeting the suitability criteria according to the autonomous driving level.

[0035] On the other hand, as shown in Figure 3(B), the driver may move their head from side to side while driving. For example, this may occur when visually checking the white lines through the side window or when visually checking the rear of the vehicle 2. In this case as well, as shown in Figure 3(B2), the driver state determination unit 56 detects that the driver's head has moved out of the frame in the vehicle width direction from the predetermined area 37, and determines that the driver's state has become an unsuitable state that does not meet the predetermined suitability criteria.

[0036] In this manner, the driver state determination unit 56 determines that the driver state is in an unsuitable state when at least one of the predetermined conditions set as appropriate criteria is not met. If the unsuitable state continues for a predetermined time threshold, the driver state determination unit 56 determines that the driver state is abnormal. The driver state determination unit 56 determines that the driver state is normal unless the unsuitable state continues for a predetermined time threshold. Furthermore, even if the driver state has been determined to be abnormal, if the driver state returns from an unsuitable state to an appropriate state such as an ambient monitoring state, the driver state determination unit 56 determines that the driver state is normal.

[0037] Returning to Figure 1, the explanation continues. The operation detection unit 58 acquires the amount of operation of the driving control device 17 based on the signal from the driving control device 17. The operation detection unit 58 acquires the amount of operation of the brake pedal 23, accelerator pedal 22, and steering wheel 21 based on signals from sensors provided on the brake pedal 23, accelerator pedal 22, and steering wheel 21. In other words, the operation detection unit 58 is configured to detect brake pedal operation, accelerator pedal operation, and steering wheel operation by the driver.

[0038] The operation detection unit 58 is configured to detect the operation of the first operation switch 35 and the second operation switch 36 by the occupant based on signals from the first operation switch 35 and the second operation switch 36.

[0039] The driving control unit 52 is configured to control the acceleration and deceleration of the vehicle 2 according to the driving mode. In addition to controlling the acceleration and deceleration of the vehicle 2, the driving control unit 52 may also be configured to control the steering of the vehicle 2. The driving control unit 52 performs automatic driving control, including adaptive cruise control (hereinafter referred to as ACC) and lane keeping assist control (hereinafter referred to as LKAS). Adaptive cruise control is an example of vehicle speed control that controls the speed of the vehicle 2 by controlling the acceleration and deceleration of the vehicle 2. The driving control unit 52 controls the acceleration and deceleration of the vehicle 2 by controlling the propulsion system 3 and the braking system 4, thereby assisting the driver's driving. In addition, lane keeping assist control is an example of steering control that controls the steering of the vehicle 2 by controlling the steering system 5. The driving control unit 52 controls the trajectory of the vehicle 2 by controlling the steering system 5, thereby assisting the driver's driving.

[0040] In addition to the above-mentioned controls that support driving during normal driving, the driving control unit 52 also performs automatic driving controls for accident prevention, such as collision mitigation braking (hereinafter referred to as CMBS) and road departure prevention function. CMBS is implemented by controlling the braking system 4 after issuing a warning to decelerate the vehicle 2 in order to avoid or mitigate a collision when there is a risk of the vehicle 2 colliding with a preceding vehicle or pedestrian. The road departure prevention function is implemented by controlling the steering system 5 after issuing a warning to change the direction of travel of the vehicle 2 when there is a risk of the vehicle 2 deviating from its driving lane.

[0041] When the operation detection unit 58 detects an ON signal from the first operation switch 35, the driving control unit 52 starts driving assistance control. When the operation detection unit 58 detects an OFF signal from the first operation switch 35, the driving control unit 52 terminates driving assistance control. When the operation detection unit 58 detects a selection signal from the second operation switch 36, the driving control unit 52 switches the driving assistance control to be executed or currently executed to the driving assistance control selected by the selection signal. The selection signal from the second operation switch 36 may, for example, select ACC and LKAS, or select LKAS only.

[0042] The mode setting unit 53 is configured to allow the driving mode to be changed. The driving modes include a manual driving mode, a driving assistance mode, and a stationary driving assistance mode. The driving assistance mode includes an automated driving level 1 mode, an automated driving level 2 mode, an automated driving level 3 mode, an accelerator pedal override mode (hereinafter referred to as APOR mode), etc. The automated driving level 1 mode is a mode in which the degree of driving tasks imposed on the driver is greater than that of the automated driving level 2 mode, or in which the degree of assistance to the driver's driving is less. The automated driving level 2 mode is a mode in which the degree of driving tasks imposed on the driver is greater than that of the automated driving level 3 mode, or in which the degree of assistance to the driver's driving is less. The stationary driving assistance mode includes a start-up standby mode and a stop-hold mode. The driving mode transitions according to the occupant's operation, the state of the vehicle 2, and the occupant's state.

[0043] In manual driving mode, the driving control unit 52 controls the vehicle 2 based on the driver's driving operations. Specifically, in manual driving mode, the driving control unit 52 obtains the amount of operation of the brake pedal 23 from the brake pedal 23 and controls the braking system 4 based on the amount of operation of the brake pedal 23. In addition, in manual driving mode, the driving control unit 52 obtains the amount of operation of the accelerator pedal 22 from the accelerator pedal 22 and controls the propulsion system 3 based on the amount of operation of the accelerator pedal 22. In addition, in manual driving mode, the driving control unit 52 obtains the amount of operation of the steering wheel 21 from the steering wheel 21 and controls the steering system 5 based on the amount of operation of the steering wheel 21.

[0044] The driving assistance mode can be set while the vehicle 2 is in motion. When LKAS is selected in the driving assistance mode, the driving control unit 52 performs lane keeping assistance control to drive the vehicle 2 along the lane. Lane information is included in the surrounding conditions recognized by the surrounding conditions recognition unit 51.

[0045] Furthermore, when ACC is selected in the driving assistance mode, the driving control unit 52 controls the acceleration and deceleration of the vehicle 2 according to the surrounding conditions, in addition to controlling the LKAS as described above. In other embodiments, the control of ACC and LKAS may be performed independently. The surrounding conditions include, for example, a vehicle traveling in front of the vehicle 2. In the driving assistance mode, the driving control unit 52 controls the propulsion system 3 and the braking system 4 to maintain a distance of at least a predetermined value between the vehicle 2 and the vehicle traveling in front of the vehicle 2, and to maintain the vehicle speed at a target speed within a range in which the distance can be maintained. The driving control unit 52 acquires the position and speed of the vehicle ahead based on the surrounding conditions acquired by the surrounding conditions recognition unit 51. The target speed may be set by the driver. The target speed may be set by the driver's operation on the display device 31 or operation switches. In addition, in the driving assistance mode, the driving control unit 52 may control the propulsion system 3 and the braking system 4 based on the signal information of traffic lights and sign information acquired by the surrounding conditions recognition unit 51.

[0046] APOR mode is set when the driving mode is in driver assistance mode and the accelerator pedal 22 is pressed. In APOR mode, the driving control unit 52 controls the propulsion system 3 based on the amount the accelerator pedal 22 is pressed. This enables the vehicle 2 to accelerate in response to the driver's accelerator pedal operation.

[0047] In driving assistance mode, when the vehicle in front slows down and stops, the driving control unit 52 maintains a safe distance from the vehicle in front and stops the vehicle 2. In addition, in driving assistance mode, the driving control unit 52 may acquire traffic signal information and stop the vehicle 2 at the stop line according to the traffic signal information.

[0048] Furthermore, if LKAS is selected in the driver assistance mode, the driving control unit 52 performs lane keeping assistance control to drive the vehicle 2 along the lane, in addition to the ACC described above. The driving control unit 52 performs driving control based on the determination result of the driver state determination unit 56.

[0049] The notification unit 59 notifies the driver via the HMI 19 when the driver status determination unit 56 determines that the driver's status is abnormal.

[0050] The emergency control unit 54 executes emergency control to ensure the safety of the vehicle 2 and the driver when the driver status determination unit 56 determines that the driver is abnormal. Emergency control is executed regardless of the driving level or the driving state of the vehicle 2. For example, emergency control is executed when the vehicle 2 is stopped, driving, when ACC or LKAS is running, or when ACC and LKAS are running. Emergency control takes precedence over driving operations by the driver and driving assistance control performed by the driving control unit 52, and is performed as follows: When driver abnormality is determined, the emergency control unit 54 executes in-vehicle notification to the occupants, acceleration suppression of the vehicle 2, and lane keeping control. If the driver abnormality continues, the emergency control unit 54 confirms the driver abnormality and executes external notification to outside the vehicle, deceleration of the vehicle 2, and lane change control, bringing the vehicle 2 to a stop in the driving lane or on the shoulder of the road. After the vehicle 2 has stopped, the emergency control unit 54 executes control to contact a hospital or other facility and to maintain the vehicle's stop.

[0051] Next, an example of a vehicle control procedure performed by the vehicle control device 1 will be described. This example will describe the case where Level 1 or 2 automated driving control is being performed.

[0052] Figure 4 is a flowchart showing the procedure for driving assistance control by the control device. When the ignition switch of the vehicle 2 is turned on, the vehicle control device 1 starts the vehicle control shown in Figure 4. First, the vehicle control device 1 starts monitoring the driver using the occupant monitoring device 18 (ST1). In monitoring the driver, the vehicle control device 1 recognizes the driver's condition based on the occupant monitoring device 18 and determines whether the driver's condition is normal or abnormal.

[0053] Subsequently, the vehicle control device 1 determines whether or not automatic driving control by the driving control unit 52 is being performed (ST2). If neither ACC nor LKAS is being performed (ST2: No), the vehicle control device 1 proceeds to step ST3. If either ACC or LKAS is being performed, or if both ACC and LKAS are being performed (ST2: Yes), the vehicle control device 1 proceeds to step ST7.

[0054] In step ST3, the vehicle control device 1 sets a first time T1 as the time threshold for an unsuitable state (ST3). On the other hand, in step ST7, the vehicle control device 1 sets a second time T2 as the time threshold for an unsuitable state (ST3). The second time T2 is a longer time than the first time T1 and may be a constant value, or it may be a different value depending on the mode of automatic driving control by the driving control unit 52, and the control state such as the type of control and the amount of control. Details of the second time T2 will be explained later.

[0055] After step ST3, the vehicle control device 1 determines whether the driver's unsuitable state continues for a first time T1 or longer (ST4). The count of unsuitable states starts from the time the driver state determination unit 56 determines that the driver's state is unsuitable, and is reset when the driver's state returns to the surrounding monitoring state (appropriate state).

[0056] If the driver's unsuitable condition persists for more than 1 hour T1 (ST4: Yes), the vehicle control device 1 determines that the driver's condition is abnormal (ST5). Subsequently, the vehicle control device 1 executes the emergency control described above (ST6).

[0057] On the other hand, if step ST2 determines Yes because ACC or LKAS is running, the vehicle control device 1 sets the second time T2 as the time threshold for the unsuitable state in step ST7, and then determines whether the driver's unsuitable state continues for the second time T2 or longer (ST8). The second time T2 is longer than the first time T1. In other words, when driving control is being performed (ST2:Yes), the time threshold is set to a larger value than when the driving control is not being performed (ST2:No). As a result, the driver's state is less likely to be judged as abnormal.

[0058] If the driver's unsuitable condition persists for two hours T2 or longer (ST8: Yes), the vehicle control device 1 determines that the driver's condition is abnormal (ST9). Subsequently, the vehicle control device 1 executes the emergency control described above (ST10).

[0059] Next, the operation of the vehicle 2 when the vehicle control device 1 performs such vehicle control will be explained with reference to Figures 5 and 6.

[0060] Figure 5 is a time chart showing an example of driving assistance control when the vehicle control device 1 is not performing driving control. At time t1, the vehicle control device 1 determines that the driver's state has changed from an appropriate state (surround monitoring state) for Level 2 autonomous driving to an inappropriate state. Since driving control is not being performed, the time threshold is set to the first time T1. At time t2, when the inappropriate state has continued for the first time T1, the vehicle control device 1 determines that the driver's state is abnormal. Subsequently, the vehicle control device 1 performs in-vehicle notification to the occupants, acceleration suppression of the vehicle 2, and lane keeping control.

[0061] At point t3, if the driver's unsuitable condition persists for a predetermined period despite the in-vehicle notification being made, the vehicle control device 1 confirms that the driver's condition is abnormal. Upon confirming the abnormal condition, the vehicle control device 1 executes external notification to the outside of the vehicle, deceleration of vehicle 2, and lane change controls. At point t4, when the vehicle control device 1 has stopped vehicle 2 in the driving lane or on the shoulder, the vehicle control device 1 executes contact with a hospital or other facility and controls the vehicle to maintain its stop.

[0062] Figure 6 is a time chart showing an example of driving assistance control when the vehicle control device 1 is performing driving control. At time t11, the vehicle control device 1 determines that the driver's state has changed from an appropriate state (surround monitoring state) for Level 2 autonomous driving to an inappropriate state. Since driving control is being performed, the time threshold is set to a second time T2 which is longer than the first time T1. At time t12, when the inappropriate state has continued for the second time T2, the vehicle control device 1 determines that the driver's state is abnormal. Subsequently, the vehicle control device 1 performs in-vehicle notification to the occupants, acceleration suppression of the vehicle 2, and lane keeping control.

[0063] Therefore, when the vehicle control device 1 is performing driving control (Figure 6), it delays the activation of the notification (time t12), the start of deceleration of vehicle 2 (time t13), and the stopping of vehicle 2 (time t14) compared to when it is not performing driving control (Figure 5).

[0064] At time t13, if the driver's unsuitable condition persists for a predetermined period despite the in-vehicle notification being made, the vehicle control device 1 confirms that the driver's condition is abnormal. Upon confirming the abnormal condition, the vehicle control device 1 executes various emergency control actions, including external notification to the outside of the vehicle, deceleration of vehicle 2, and lane changes. At time t14, when the vehicle control device 1 has stopped vehicle 2 in the driving lane or on the shoulder, the vehicle control device 1 executes various emergency control actions, including contacting a hospital or other facility and maintaining the vehicle's stop.

[0065] As described above, the second time T2 is set to different values ​​depending on the mode of automatic driving control by the driving control unit 52, and the control state such as the type of control and the amount of control. For example, the vehicle control device 1 may set the second time T2 to a larger value when ACC and LKAS are running compared to when only ACC or LKAS is running. Also, when LKAS is running, the vehicle control device 1 may set the second time T2 to a smaller value when driving on a curve where the amount of control by LKAS, i.e., the steering amount of the steering device 5, is large, compared to when driving on a straight line or a gentle curve where the steering amount of the steering device 5 is small.

[0066] When the vehicle control device 1 is operating in autonomous driving level 2 mode, it may set the second time T2 to a larger value than when it is operating in autonomous driving level 1 mode. In this case, when operating in autonomous driving level 2 mode, it becomes less likely that the driver's condition will be judged as abnormal compared to when operating in autonomous driving level 1 mode.

[0067] Thus, according to the vehicle control device 1 of this embodiment, it is possible to appropriately determine driver abnormalities based on whether or not driving control is being performed, thereby improving the accuracy of determining the driver's state.

[0068] The above embodiment is configured as follows.

[0069] The vehicle control device 1 includes a surrounding environment recognition unit 51 that recognizes the surrounding environment of the vehicle 2, a driving control unit 52 that performs driving control of the vehicle 2 based on the recognition results of the surrounding environment recognition unit 51, and a driver state determination unit 56 that determines whether the driver's state is normal or abnormal based on images captured by an in-vehicle camera 25 that captures images of the driver of the vehicle 2. The driver state determination unit 56 is configured to make it more difficult to determine the driver's state as abnormal when driving control is being performed by the driving control unit 52 compared to when driving control is not being performed.

[0070] According to this embodiment, it is possible to appropriately determine driver abnormalities based on whether or not driving control is being performed, thereby improving the accuracy of driver determination. As a result, over-determination and over-notification of driver abnormalities during driving control can be suppressed, improving usability.

[0071] The driver state determination unit 56 sets a time threshold based on the driving control by the driving control unit 52, and may determine the driver's state to be abnormal if the driver, as captured by the in-vehicle camera 25, remains in an unsuitable state that does not meet predetermined suitability criteria for a period of time longer than the time threshold. The driver state determination unit 56 may set a higher time threshold when driving control is being performed by the driving control unit 52 compared to when driving control is not being performed.

[0072] According to this embodiment, the time threshold for driver abnormality detection can be appropriately changed based on the driving control being executed, thereby enabling appropriate detection of driver abnormalities.

[0073] The vehicle control device 1 further includes a notification unit 59 that notifies the driver when the driver status determination unit 56 determines that the driver's status is abnormal. The notification unit 59 may delay its notification operation when driving control is being performed by the driving control unit 52 compared to when driving control is not being performed.

[0074] According to this configuration, excessive notifications due to abnormality detection during the execution of driving control can be suppressed. Therefore, drivers are less likely to feel annoyed. In addition, during manual driving when driving control is not in operation, warnings are issued earlier than when driving control is in operation, so safety is not compromised.

[0075] The vehicle control device 1 further includes an emergency control unit 54 that decelerates or stops the vehicle 2 when the driver state determination unit 56 determines that the driver's state is abnormal. The emergency control unit 54 may delay the start of deceleration of the vehicle 2 or the stopping of the vehicle 2 compared to when driving control is not being performed, when driving control is being performed by the driving control unit 52.

[0076] According to this embodiment, excessive deceleration due to abnormal detection during the execution of driving control can be suppressed. Therefore, driver discomfort is reduced. Furthermore, during manual driving when driving control is not performed, the vehicle control device 1 decelerates and stops the vehicle 2 earlier than during driving control, so safety is not compromised.

[0077] The driving control includes at least one of adaptive cruise control, which is one of the vehicle speed controls of vehicle 2, and lane keeping assist control, which is one of the steering controls, and the driver state determination unit 56 may change the time threshold when the driving control being performed is changed.

[0078] According to this embodiment, it is possible to suppress over-determination and over-notification of abnormal conditions of the driver when vehicle speed control and / or steering control of vehicle 2 is performed.

[0079] A change in the executed driving control may mean that the type of driving control to be executed among the executable driving controls of the driving control unit 52 is changed, or that the control amount of the driving control is changed.

[0080] According to this embodiment, driver abnormalities can be appropriately determined according to the degree (level) of driving assistance.

[0081] The driving control unit 52 may set a smaller time threshold when the amount of control for driving control is large compared to when it is small.

[0082] In this configuration, driving assistance is provided in a comprehensive manner, and delays in detecting abnormalities in the driver's condition are suppressed when the driver should be paying more attention to the surrounding conditions and driving.

[0083] The driving control includes a first driving mode (autonomous driving level 2 mode) and a second driving mode (autonomous driving level 1 mode) in which the degree of driving tasks imposed on the driver is greater or the degree of assistance to the driver is less than in the first driving mode. The driver state determination unit 56 may be configured to make it more difficult to determine the driver's state as abnormal when the first driving mode is being executed compared to when the second driving mode is not being executed.

[0084] According to this embodiment, when driving assistance is comprehensive, it is possible to further suppress over-judgment and over-notification of abnormal conditions in the driver.

[0085] The driver state determination unit 56 sets a time threshold based on the driving control by the driving control unit 52, and determines the driver's state to be abnormal if the driver, as captured by the in-vehicle camera 25, is in an unsuitable state that does not meet predetermined suitability criteria for a period of time longer than the time threshold. The driver state determination unit 56 may set the time threshold to be higher when the first driving mode is being executed compared to when the second driving mode is not being executed.

[0086] According to this embodiment, when driving assistance is comprehensive, it is possible to further suppress over-judgment and over-notification of abnormal conditions in the driver.

[0087] This concludes the description of specific embodiments, but the present invention is not limited to the configuration of the above embodiments and can be broadly modified and implemented.

[0088] For example, in the above embodiment, the driver state determination unit 56 sets a time threshold based on the driving control and determines the driver's state to be abnormal if the unsuitable state continues for longer than the time threshold. However, the predetermined region 37 in the image shown in Figure 2 may be set to a different size based on the driving control, and the time threshold may be fixed. In other words, the driver state determination unit 56 may make it more difficult to determine that the driver's state is abnormal by making the predetermined region 37 larger. Specifically, the driver state determination unit 56 may be configured to make the predetermined region 37 larger when the driving control unit 52 is executing driving control compared to when driving control is not executing. When driving control is not executing, the driver state determination unit 56 makes it easier to determine that the driver's state is abnormal by making the predetermined region smaller.

[0089] In addition, the specific configuration, arrangement, quantity, and materials of each component and part, as well as the specific content and order of each process, can be modified as appropriate, as long as they do not depart from the spirit of the present invention. Furthermore, not all of the components shown in the above embodiments are necessarily essential, and can be selected as appropriate. [Explanation of Symbols]

[0090] 1: Vehicle control system 2: Vehicles 25: In-car camera 51: Surrounding Situation Recognition Unit 52: Driving control unit 54: Emergency Control Unit 56: Driver status determination unit 59: Hochi Department T1: First hour T2: Second hour

Claims

1. A surrounding environment recognition unit that recognizes the surrounding conditions of the vehicle, A driving control unit that controls the driving of the vehicle based on the recognition results of the surrounding situation recognition unit, The system includes a driver status determination unit that determines whether the driver's condition is normal or abnormal based on an image captured by an in-vehicle camera that captures the driver of the vehicle. The driver state determination unit is configured to make it more difficult to determine the driver's state as abnormal when the driving control is being performed by the driving control unit compared to when the driving control is not being performed, in a vehicle control device.

2. The driver state determination unit sets a time threshold based on the driving control by the driving control unit, and determines that the driver's state is abnormal if the driver, as captured by the in-vehicle camera, remains in an unsuitable state that does not meet predetermined suitability criteria for a period of time longer than the time threshold. The vehicle control device according to claim 1, wherein the driver state determination unit sets the time threshold to be larger when the driving control is being performed by the driving control unit compared to when the driving control is not being performed.

3. The driver status determination unit further includes a notification unit that notifies the driver when it determines that the driver's status is abnormal. The vehicle control device according to claim 2, wherein the notification unit delays the operation of the notification when the driving control is being performed by the driving control unit compared to when the driving control is not being performed.

4. The vehicle is further provided with an emergency control unit that decelerates or stops the vehicle when the driver status determination unit determines that the driver's status is abnormal. The vehicle control device according to claim 3, wherein the emergency control unit delays the start of deceleration of the vehicle or the stopping of the vehicle when the driving control is being performed by the driving control unit, compared to when the driving control is not being performed.

5. The aforementioned driving control includes at least one of the vehicle speed control and steering control of the vehicle. The vehicle control device according to claim 4, wherein the driver state determination unit changes the time threshold when the driving control being performed is changed.

6. The vehicle control device according to claim 5, wherein the change in the executed driving control means that the type of the executed driving control is changed from among the executable driving controls of the driving control unit, or the control amount of the driving control is changed.

7. The vehicle control device according to claim 6, wherein the driving control unit sets the time threshold to be smaller when the control amount of the driving control is large compared to when it is small.

8. The driving control includes a first driving mode and a second driving mode in which the degree of driving tasks imposed on the driver is greater than in the first driving mode, or the degree of support for the driver's driving is less. The vehicle control device according to claim 1, wherein the driver state determination unit is configured to make it more difficult to determine the driver's state as abnormal when the first driving mode is being executed compared to when the second driving mode is not being executed.

9. The driver state determination unit sets a time threshold based on the driving control by the driving control unit, and determines that the driver's state is abnormal if the driver, as captured by the in-vehicle camera, remains in an unsuitable state that does not meet predetermined suitability criteria for a period of time longer than the time threshold. The vehicle control device according to claim 8, wherein the driver state determination unit sets the time threshold to be larger when the first driving mode is being executed compared to when the second driving mode is not being executed.

Images