Vehicle control device, vehicle control method, and vehicle control program

Abstract

This vehicle control device comprises: a driver ascertaining part (32) that ascertains the state of a driver of a host vehicle on the basis of information from a driver monitor (18) installed in the host vehicle; a control determination part (35) that detects an abnormal response of the driver on the basis of driver information from the driver ascertaining part and determines execution of vehicle control in accordance with the abnormal response; and execution parts (34, 39) that execute the vehicle control determined by the control determination part. The vehicle control is at least one from among speed control, steering control, guidance provided to an occupant, notification to the surroundings of the host vehicle, correction of an operation by the driver, and travel resumption assistance.

Description

Vehicle control device, vehicle control method, and vehicle control program CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Japanese Patent Application No. 2024-153918, filed on September 6, 2024, the contents of which are incorporated herein by reference.

[0002] The present disclosure relates to a vehicle control device, a vehicle control method, and a vehicle control program for controlling a vehicle.

[0003] Conventionally, a vehicle control device is known that acquires occupant information regarding the state of a vehicle driver, and controls the vehicle based on the occupant information when the driver faces an unexpected emergency, thereby improving safety (for example, Patent Document 1).

[0004] The vehicle control device described in Patent Document 1 includes a startle response detection unit that detects a driver's startle response based on the driver's grip pressure on the steering wheel and pupil dilation, and a control unit that performs vehicle control to decelerate the vehicle when a startle response is detected. This allows the vehicle control device to execute vehicle control to decelerate the vehicle in an emergency situation, improving the safety of the vehicle and its occupants.

[0005] Patent No. 4740399

[0006] In recent years, it has become an issue to prevent accidents that occur when a driver of a vehicle falls into a state where he or she is unable to take appropriate action due to some kind of panic factor (hereinafter referred to as an "abnormal state").

[0007] Although the vehicle control device described in Patent Document 1 is capable of vehicle control to decelerate the vehicle in an emergency, vehicle control other than deceleration may be necessary depending on the abnormal state of the driver, and there is still room for improvement. For example, reactions or behaviors exhibited by a driver in an abnormal state (hereinafter referred to as "abnormal responses") include performing an erroneous operation, performing an inappropriate amount of operation that is not an erroneous operation, or being unable to take any action. As such, there are various types of abnormal responses, and in order to prevent accidents such as those described above, it is necessary to be able to perform vehicle control other than deceleration in response to the abnormal response.

[0008] Furthermore, the vehicle control device described in Patent Document 1 does not anticipate what to do after the system has slowed or stopped the vehicle through vehicle control, and after executing vehicle control, the system leaves it up to the driver to decide what to do. For example, even if the vehicle is stopped through vehicle control, if the driver is in a state of high panic or if the driver continues to be mentally unsettled even after the direct cause of panic has disappeared, there is a risk that the driver will not be able to move the vehicle appropriately thereafter.

[0009] The present disclosure relates to a vehicle control device, a vehicle control method, and a vehicle control program capable of executing appropriate vehicle control in response to an abnormal response of a driver.

[0010] According to one aspect of the present disclosure, there is provided a vehicle control device comprising: a driver grasping unit that grasps the state of a driver of the vehicle based on information from a driver monitor mounted on the vehicle; a control decision unit that detects an abnormal response of the driver based on the driver information from the driver grasping unit and decides to execute vehicle control in accordance with the abnormal response; and an execution unit that executes the vehicle control decided by the control decision unit, wherein the vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of driver operation, and assistance in resuming driving.

[0011] In this vehicle control device, a driver recognition unit recognizes the state of the driver of the vehicle based on information from a driver monitor, and a control decision unit detects an abnormal response of the driver based on the recognized driver information and decides to execute vehicle control in response to the abnormal response. In this vehicle control device, an execution unit executes the vehicle control decided by the control decision unit. The vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of driver operation, and travel resumption assistance. This enables the vehicle control device to execute appropriate vehicle control in response to the driver's abnormal response.

[0012] According to one aspect of the present disclosure, there is provided a vehicle control method that can be used in a vehicle, comprising: grasping the state of a driver of the vehicle from information from a driver monitor mounted on the vehicle; detecting an abnormal response of the driver based on the grasped driver information; determining execution of vehicle control in accordance with the abnormal response; and executing the determined vehicle control, wherein the vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of driver operation, and assistance in resuming driving.

[0013] This vehicle control method includes: grasping the driver's state of the vehicle based on information from a driver monitor; detecting an abnormal response of the driver based on the grasped driver information; determining vehicle control in response to the detected abnormal response; and executing the determined vehicle control. The vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of driver operation, and assistance for resuming driving. This results in a vehicle control method that can execute appropriate vehicle control in response to the driver's abnormal response.

[0014] According to one aspect of the present disclosure, a vehicle control program used to control the driving state of a vehicle or on-board equipment includes: a process of grasping the state of a driver of the vehicle from information from a driver monitor mounted on the vehicle; a process of detecting an abnormal response of the driver based on the grasped driver information; a process of deciding to execute vehicle control in accordance with the abnormal response; and a process of executing the decided vehicle control, wherein the vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of driver operation, and driving resumption assistance.

[0015] This vehicle control program includes a process for grasping the driver's state of the vehicle based on information from a driver monitor, a process for detecting an abnormal response of the driver based on the grasped driver information, a process for determining vehicle control in response to the detected abnormal response, and a process for executing the determined vehicle control. The vehicle control program then causes at least one execution unit to execute processes including the above-described processes. The vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of driver operation, and travel resumption assistance. This results in a vehicle control program that can execute appropriate vehicle control in response to the driver's abnormal response.

[0016] 1 is a block diagram showing an example of a vehicle control system according to an embodiment; FIG. 2 is a block diagram showing a vehicle control unit and related components; FIG. 3 is an explanatory diagram showing detection of an abnormal response; FIG. 4 is a flowchart showing an example of a process of vehicle control by the vehicle control system; FIG. 5 is a flowchart showing an example of a process of vehicle control when a first abnormal response is detected; and FIG. 6 is a flowchart showing an example of a process of vehicle control when a second abnormal response is detected.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the following embodiments, identical or equivalent parts will be denoted by the same reference numerals.

[0018] (Embodiment) A vehicle control system 1 according to an embodiment will be described. The vehicle control system 1 is mounted on a vehicle such as a general private passenger car, and various functions described below are executed by a vehicle control unit 28 shown in FIG. 1 . For convenience of explanation, the vehicle on which the vehicle control system 1 is mounted will be referred to as the "host vehicle." The vehicle control system 1 is preferably applied to, for example, autonomous driving levels 2 and higher defined by the Society of Automotive Engineers. Hereinafter, in this specification, an autonomous driving mode of level 2 or lower, in which a human is the primary driver, will be referred to as the "manual driving mode," and an autonomous driving mode of level 3 or higher, in which a system is the primary driver, will be referred to as the "autonomous driving mode." The autonomous driving mode is a state in which the system autonomously drives the vehicle.

[0019] The vehicle on which the vehicle control system 1 is installed is not limited to a passenger car, but may be a rental car vehicle, a manned taxi vehicle, a ride-sharing vehicle, a freight vehicle, a bus, etc., and may be either a right-hand drive vehicle or a left-hand drive vehicle. Furthermore, each vehicle control described below according to the present disclosure can be optimized as appropriate according to the road traffic laws of each country and region, as well as the steering wheel position of the vehicle, etc.

[0020] [Basic Configuration] As shown in FIG. 1 , the vehicle control system 1 includes an HMI system 100 mounted on the vehicle, various sensors such as a perimeter monitoring sensor 14, various devices such as an on-board communication device 22, a cruise control ECU 27, and a vehicle control unit 28, all connected via a communication bus 40. HMI and ECU are abbreviations for Human Machine Interface and Electronic Control Unit, respectively. The various sensors constituting the vehicle control system 1 are capable of communicating with each other via the communication bus 40, for example. However, the present invention is not limited to this connection configuration, and some of the various sensors may be capable of communicating without the communication bus 40. Examples of the various sensors include the perimeter monitoring sensor 14, as well as a pressure sensor 20 and a cabin acoustic sensor 21 constituting the driver monitor 18. Examples of the various devices include a locator 23, a navigation device 24, an external alarm ECU 25, and a collision prevention device 26, in addition to the on-board communication device 22 and cruise control ECU 27.

[0021] The HMI system 100 has an input interface function that accepts operations by a vehicle occupant, such as a driver, and an output interface function that presents information to the occupant. The HMI system 100 is configured to include, for example, an HMI control device 10, an input device 11, a display device 12, and an audio device 13.

[0022] The HMI control device 10 is a computer that mainly includes a control circuit equipped with, for example, a CPU, recording media such as ROM and RAM, an input / output interface, and a bus connecting these (not shown). The CPU, ROM, and RAM are abbreviations for Central Processing Unit, Read Only Memory, and Random Access Memory, respectively. The HMI control device 10 functions as a presentation control device and comprehensively controls the presentation of information to occupants using the display device 12, audio device 13, etc. The HMI control device 10 presents information related to automatic driving by the system in cooperation with, for example, a vehicle control unit 28.

[0023] The input device 11 is an input unit that accepts operations by a user such as a driver. Various user operations, such as operations related to setting a destination for route guidance and operations related to setting air conditioning, are input to the input device 11. The input device 11 includes, for example, a steering switch provided on the spokes of the steering wheel, an operating lever provided on the steering column, and a voice input device that recognizes what the passengers are saying.

[0024] The display device 12 is a device that presents information to the occupant's vision by displaying an image or the like. Examples of the display device 12 include a meter display, a center information display (CID), and a head-up display (HUD). For example, in the case of a CID, the display device 12 has a touch panel function and is configured to allow the driver or the like to touch the display screen.

[0025] The audio device 13 has a plurality of speakers mounted in the vehicle interior, and reproduces various sounds, such as notification sounds and voice messages associated with vehicle control by a vehicle control unit 28 (described later) and voice messages associated with user operations on the input device 11, through the speakers.

[0026] The perimeter monitoring sensor 14 is an autonomous sensor that monitors the environment around the vehicle and is composed of, for example, a camera unit 15, an exterior acoustic sensor 16, and an object detection sensor 17 mounted on the vehicle. The perimeter monitoring sensor 14 detects, for example, targets such as moving objects and stationary objects around the vehicle, external vehicle sounds, etc., and outputs the detection information to the vehicle control unit 28, etc. The detection information by the perimeter monitoring sensor 14 includes, for example, when a target is detected, information on the direction of the target and the distance from the vehicle to the target, and when an external vehicle sound is detected, various information such as sound pressure and sound occurrence intervals.

[0027] The camera unit 15 is an imaging device that captures images of the outside of the vehicle. The camera unit 15 includes, for example, a front camera module, a rear camera module, a left side camera module, and a right side camera module, and is configured to capture images of the entire surroundings of the vehicle. The camera unit 15 analyzes image data captured by each camera module using a known image analysis technique and outputs the analysis information as detection information to the communication bus 40. The detection information from the camera unit 15 includes, for example, information on other objects around the vehicle, such as vehicles, light vehicles, people, obstacles, buildings, road signs, and trees; road surface information such as white lines and road markings painted on the road; and traffic information such as traffic light displays.

[0028] The exterior acoustic sensor 16 is mainly composed of a microphone element that collects sounds from outside the vehicle and converts the collected sounds into electrical signals. Examples of microphone elements include a capacitor microphone that outputs an electrical signal based on changes in capacitance caused by vibration of a thin diaphragm due to sound pressure, and a piezoelectric microphone that outputs sound as an electrical signal using a piezoelectric element. A plurality of exterior acoustic sensors 16 are provided, for example, on the front, rear, left and right sides, and ceiling of the vehicle. The exterior acoustic sensors 16 collect sounds from the environment surrounding the vehicle and output the collected sound data as sound information to the communication bus 40.

[0029] The object detection sensor 17 is a device that detects objects around the vehicle by, for example, transmitting a transmission wave to the outside of the vehicle, receiving a reflected wave generated when the transmission wave is reflected from the outside, and analyzing the reflected wave. The object detection sensor 17 may be, for example, a millimeter-wave radar, LiDAR, or sonar that transmits laser light, radio waves, or ultrasonic waves as a transmission wave. LiDAR is an abbreviation for Light Detection and Ranging. The object detection sensor 17 is mounted, for example, on the front, rear, left, or right side of the vehicle, and outputs, as detection information, information on the presence or absence of objects, such as other vehicles or obstacles, around the vehicle, as well as the distance and direction to the detected objects, to the communication bus 40.

[0030] The driver monitor 18 is an autonomous information acquisition device that acquires the state of the driver of the vehicle and is configured to have, for example, a driver imaging unit 19, a pressure sensor 20, and a vehicle interior acoustic sensor 21 mounted on the vehicle. The driver monitor 18 detects, for example, the driver's face, upper body movements, voice, the driver's gripping force on the steering wheel, etc., and the pressure applied to the driver's seat, and outputs the detected information to the vehicle control unit 28, etc. The detected information by the driver monitor 18 is used, for example, in the vehicle control unit 28 to grasp the state of the driver, in particular, abnormal responses.

[0031] The driver imaging unit 19 is, for example, an imaging device that captures an image of the upper body including the face of the driver of the vehicle. The driver imaging unit 19 captures, for example, an image of the face, arms, and torso of the driver of the vehicle, analyzes the captured image data using known image analysis technology, acquires information such as the driver's face, line of sight, facial expression, and movement of the arms and torso as detection information, and outputs the information to the communication bus 40. The driver imaging unit 19 is, for example, a Driver Status Monitor (registered trademark) manufactured by Denso Corporation, but is not limited to this, and other known imaging devices may be used.

[0032] The pressure sensor 20 outputs an electrical signal corresponding to the pressure generated at a part of the body that comes into contact with the driver. The pressure sensor 20 is installed on the steering wheel or the driver's seat, for example, and outputs pressure information such as the driver's gripping force on the steering wheel and the surface pressure on the seat as detection information to the communication bus 40.

[0033] The vehicle interior acoustic sensor 21 has the same configuration as the vehicle exterior acoustic sensor 16, but is installed inside the vehicle interior and collects sounds inside the vehicle interior. The vehicle interior acoustic sensor 21 collects the voice of the driver of the vehicle and outputs the voice information as detection information to the communication bus 40. The voice information from the vehicle interior acoustic sensor 21, together with the detection information from the driver imaging unit 19 and the pressure sensor 20, is used to understand the driver's condition.

[0034] The in-vehicle communication device 22 is an external communication unit mounted on the vehicle itself, and functions as, for example, a V2X (Vehicle to Everything) communication device. The in-vehicle communication device 22 receives various types of information, such as congestion information, signal information indicating the lighting patterns of traffic signals, and detection information of stopped vehicles, parked vehicles, pedestrians, etc., by transmitting and receiving information via wireless communication with roadside devices installed on the side of the road. The in-vehicle communication device 22 provides the received various types of information to the HMI control device 10, the navigation device 24, the vehicle control unit 28, etc.

[0035] The locator 23 includes a GNSS receiver, an inertial sensor, and the like. GNSS is an abbreviation for Global Navigation Satellite System. The locator 23 sequentially determines the position and traveling direction of the vehicle by combining positioning signals received from multiple positioning satellites by the GNSS receiver, measurement results from the inertial sensor, and vehicle speed information output to the communication bus 40. The locator 23 sequentially outputs position information and direction information of the vehicle based on the positioning results to the communication bus 40 as locator information.

[0036] The locator 23 further includes, for example, a map database storing map data. The map database is mainly composed of a large-capacity storage medium storing a large number of three-dimensional map data and two-dimensional map data. The three-dimensional map data is a so-called HD (High Definition) map and includes road information necessary for autonomous driving by the vehicle control system 1. Specifically, the three-dimensional map data includes three-dimensional shape information of roads and detailed information about each lane. The locator 23 can update the three-dimensional map data and two-dimensional map data to the latest information through external communication using the on-board communication device 22. For example, the locator 23 reads map data around the current location from the map database and provides the data together with locator information to the vehicle control unit 28, etc.

[0037] The navigation device 24 acquires information about a destination specified by a passenger such as a driver based on operation information acquired from the HMI control device 10. The navigation device 24 acquires vehicle position information and direction information from the locator 23, and sets a route from the current position to the destination. The navigation device 24 provides route information indicating the set route to the destination to the HMI control device 10, the vehicle control unit 28, etc. The navigation device 24 works in conjunction with the HMI system 100 to provide route guidance to the destination by combining screen displays and voice messages, etc., and notifying the driver of the direction in which the vehicle should travel at intersections, branching points, etc.

[0038] The exterior notification ECU 25 is an electronic control device that mainly includes a microcontroller that presents information to an area outside the vehicle. The exterior notification ECU 25 issues a notification to an area outside the vehicle, for example, in response to a control command from the vehicle control unit 28. The exterior notification ECU 25 notifies the surrounding area of ​​the vehicle of the approach of the vehicle or the occurrence of an abnormal situation by, for example, emitting an alarm sound or a voice message from an exterior speaker (not shown) that outputs sound to the outside of the vehicle. The exterior notification ECU 25 may, for example, not only notify the surrounding area of ​​the vehicle of the approach of the vehicle by the exterior speaker, but also by using an exterior display (not shown) that displays text or the like to the outside of the vehicle, or may notify the surrounding area of ​​the abnormal situation by controlling the turn signal and hazard lamps to be turned on.

[0039] The collision suppression device 26 is an electronic control device that performs driving control to avoid collisions between the host vehicle and other objects such as vehicles, people, and obstacles, or to mitigate collision damage. The collision suppression device 26 performs deceleration control of the host vehicle and issues a warning to the driver, for example, based on object information around the host vehicle from the perimeter monitoring sensor 14 and vehicle information such as the host vehicle's position and traveling direction from the locator 23. The collision suppression device 26 is, for example, a known collision suppression system such as a pre-crash safety system (PCS), and requests the cruise control ECU 27 to perform deceleration control and requests the display device 12, audio device 13, etc. to issue a warning in order to avoid a collision.

[0040] The cruise control ECU 27 is an electronic control device that mainly includes a microcontroller that controls the cruise of the host vehicle. Examples of cruise control include acceleration / deceleration control and steering control. Examples of the cruise control ECU 27 include a steering ECU that controls steering, a power unit control ECU that controls acceleration / deceleration, and a brake ECU. The cruise control ECU 27 controls the cruise by outputting control signals to each cruise control device installed in the host vehicle. Examples of cruise control devices include an electronically controlled throttle, a brake actuator, and an EPS motor. EPS is an abbreviation for Electric Power Steering.

[0041] The vehicle control unit 28 is configured as a microcomputer, primarily including a control circuit equipped with, for example, a CPU, ROM, RAM, and nonvolatile rewritable memory (not shown), an input / output interface, and a bus connecting these. The vehicle control unit 28 receives output data from the above-mentioned various sensors and on-board devices via a communication bus 40. The vehicle control unit 28 then reads and executes a computer program, i.e., a vehicle control program, stored in a non-transitory tangible recording medium, such as a ROM or nonvolatile rewritable memory. Execution of this computer program executes a method corresponding to the computer program, i.e., a vehicle control method. In other words, the vehicle control unit 28 executes various control processes, such as the control processes shown in FIGS. 4 to 6 , in accordance with the computer program. The vehicle control unit 28 corresponds to a vehicle control device that executes various vehicle controls of the host vehicle and is configured as a functional unit for realizing an automatic driving function and a vehicle control function. In vehicle control, for example, the vehicle control unit 28 controls various devices such as the display device 12, the audio device 13, the navigation device 24, the vehicle exterior alarm ECU 25, the collision suppression device 26, and the cruise control ECU 27. The vehicle control unit 28 can switch between a manual driving mode in which the driver is primarily responsible for driving and an automated driving mode in which the system is primarily responsible for driving, in response to, for example, an operation command from the input device 11 or the like by the driver or a control command from the system.

[0042] The above is the basic configuration of the vehicle control system 1. Note that the vehicle control system 1 may be configured to include other known in-vehicle sensors and in-vehicle devices in addition to the various sensors and devices described above.

[0043] 2, the vehicle control unit 28 includes an environment recognition unit 29, a device linkage unit 33, a control decision unit 35, and a control execution unit 39. The vehicle control unit 28 corresponds to an execution unit that reads and executes a vehicle control program for executing vehicle control, which will be described later.

[0044] The environment recognition unit 29 recognizes the surrounding environment, driving environment, and driver state of the vehicle based on, for example, output data from the perimeter monitoring sensor 14, the driver monitor 18, and the locator 23, and provides the results to the device linkage unit 33 and the control decision unit 35. The environment recognition unit 29 recognizes the driving environment of the vehicle by, for example, combining locator information and map data acquired from the locator 23 with detection information acquired from the perimeter monitoring sensor 14. The environment recognition unit 29 may acquire detection information received by the in-vehicle communication device 22 from the device linkage unit 33 and use it to recognize the driving environment. The environment recognition unit 29 acquires, for example, operation information of a pedal device, a steering wheel, etc. (not shown) from the communication bus 40, and recognizes the driving state of the vehicle. The environment recognition unit 29 includes, for example, a landmark recognition unit 30, a road recognition unit 31, and a driver recognition unit 32. The environment recognition unit 29 sequentially provides the control decision unit 35 with various pieces of information detected / recognized by the target recognition unit 30, the road recognition unit 31, and the driver recognition unit 32.

[0045] The target recognition unit 30 recognizes various information such as the size, relative position, and relative speed of various targets such as other vehicles, people, and buildings around the vehicle, based on, for example, analysis information of the image data from the camera unit 15 and detection information from the object detection sensor 17.

[0046] The road recognition unit 31 acquires road information related to the road the vehicle is traveling on or is scheduled to travel on, for example, based on locator information and map data. The road information includes whether the road the vehicle is traveling on has multiple lanes on each side, whether the lane the vehicle is traveling on is a shoulder lane adjacent to the shoulder, etc.

[0047] The driver recognition unit 32 recognizes the state of the driver based on, for example, analysis information from the driver monitor 18 and information on various vehicle operations by the driver acquired via the communication bus 40. The information on various vehicle operations includes, for example, operations of various in-vehicle devices such as the steering wheel, pedal device, shift lever, and paddle switches, as well as the amount of operation and the interval between operations. The driver recognition unit 32 recognizes the state of the driver based on, for example, the direction of the driver's face, the direction of the driver's eyes, the movements of the arms and upper body, the operation of various in-vehicle devices, and the amount of operation.

[0048] The device linking unit 33 enables information linking between various in-vehicle devices, such as the display device 12, the audio device 13, the in-vehicle communication device 22, the navigation device 24, the exterior alarm ECU 25, the collision suppression device 26, and the HMI control device 10, and the vehicle control unit 28. The device linking unit 33 has a device control unit 34 that controls the various in-vehicle devices in response to either an operation command from the occupant or a control command from the vehicle control unit 28. The device linking unit 33 provides the environment recognition unit 29 with various pieces of information received by the in-vehicle communication device 22, for example.

[0049] The device control unit 34 requests various in-vehicle devices to perform predetermined processing corresponding to the vehicle control determined by the control determination unit 35. The device control unit 34 executes, for example, a request to the display device 12 and / or the audio device 13 to notify the inside of the vehicle, and an operation request to the navigation device 24, the vehicle exterior alarm ECU 25, and the collision suppression device 26. The device control unit 34 corresponds to an execution unit that executes the vehicle control determined by the control determination unit 35.

[0050] The control decision unit 35 determines the control content of the vehicle based on information about objects around the vehicle, information about the vehicle's driving environment and driver, and operation information about various in-vehicle devices acquired from the environment recognition unit 29. For example, when the control decision unit 35 detects an abnormal driver response, it decides to execute at least one of vehicle control including speed control, steering control, alerting the outside of the vehicle, notifying and guiding the occupants, automatically operating in-vehicle devices, and assisting in resuming driving. Speed ​​control refers to control of acceleration, deceleration, and stopping of the vehicle, and includes automatic system-based control of acceleration / deceleration and stopping, as well as correction of the driver's acceleration / deceleration operation. Steering control also includes not only automatic system-based steering but also correction of the driver's steering operation. The control decision unit 35 outputs, for example, a request signal for the vehicle control that has been decided to be executed to the device control unit 34 or the control execution unit 39. The control decision unit 35 includes, for example, an abnormality detection unit 36, a driving assistance unit 37, and a restart assistance unit 38.

[0051] The abnormality detection unit 36 ​​detects an abnormal response by the driver of the vehicle based on the object information, road information, and driver information acquired from the environment recognition unit 29, and the operation information by the driver acquired from various in-vehicle sensors, etc. The abnormal response and its detection will be described in detail later.

[0052] When an abnormal response of the driver is detected, the driving assistance unit 37 performs driving assistance to avoid or resolve a dangerous situation of the host vehicle. The driving assistance unit 37 determines the execution of vehicle control that prioritizes ensuring the safety of the host vehicle, such as speed control, steering control, and notification to the outside of the vehicle, and requests the device control unit 34 or the control execution unit 39 to execute the determined vehicle control. When an abnormal response of the driver is detected, the driving assistance unit 37 switches from manual driving mode to automatic driving mode as necessary and stops the host vehicle at a safe position.

[0053] When the driving assistance unit 37 stops the host vehicle, the restart assistance unit 38 performs driving assistance for the driver or passenger who has responded abnormally, to return the host vehicle to a state in which it is possible to resume driving. The restart assistance unit 38 determines whether it is possible to resume driving of the host vehicle, for example, based on whether the driver still responds abnormally after the host vehicle has stopped. Then, when it is not possible to resume driving of the host vehicle, for example, the restart assistance unit 38 determines vehicle control, such as notification and guidance to the occupants, automatic operation of in-vehicle devices, and assistance in resuming driving, and requests the device control unit 34 or the control execution unit 39 to execute the determined vehicle control.

[0054] For example, when the control decision unit 35 requests the execution of at least one of speed control and steering control, the control execution unit 39 generates a control command signal corresponding to the speed control or steering control of the host vehicle and outputs it to the cruise control ECU 27. Similar to the device control unit 34, the control execution unit 39 corresponds to an execution unit that executes the vehicle control determined by the control decision unit 35.

[0055] The above is the basic configuration of the vehicle control unit 28. Note that, although the above description has been given as a representative example of a configuration in which the device control unit 34 or the control execution unit 39 outputs execution commands for vehicle control determined by the control determination unit 35 to each in-vehicle device, the present invention is not limited to this. For example, the vehicle control unit 28 may be configured such that only the control execution unit 39 outputs execution commands for each vehicle control to each in-vehicle device, or may be configured to include other execution units in addition to the device control unit 34 and the control execution unit 39. Furthermore, execution commands to each in-vehicle device may be issued by either the device control unit 34 or the control execution unit 39.

[0056] [Abnormal Response and Its Detection] Next, abnormal responses and its detection will be described. When some panic factor occurs and the driver falls into an abnormal state where he or she cannot take appropriate action, the driver will react or behave differently from normal, i.e., exhibit an abnormal response. Examples of panic factors include the driver causing an accident, the driver making an operating error, a passenger becoming ill or losing consciousness, luggage placed on a seat collapsing, or a disaster such as an earthquake or a major accident occurring while driving.

[0057] Abnormal responses are mainly classified into three states: abnormal behavior, hypertension, and inability to act. Abnormal behavior is a state in which a driver behaves inappropriately in accordance with the situation due to an error in the information processing process. Examples of abnormal behavior include steering the vehicle toward an obstacle, confusing the accelerator and brake, operating in-vehicle devices that do not need to be operated, staring in a specific direction, and not turning off malfunctioning in-vehicle devices. Hypertension is a state in which a driver becomes hypertension due to the influence of the sympathetic nervous system and operates the vehicle with excessive force. Examples of hypertension include gripping the steering wheel too tightly, pressing the accelerator or brake too hard, and not operating smoothly. In other words, hypertension is a state in which the driver's operation itself is appropriate for the situation, but the amount of control is inappropriate. Inability to act is a state in which a driver is unable to take any action due to the influence of the dorsal vagus nervous system.

[0058] The abnormality detection unit 36 ​​detects an abnormal response by the driver based on various information, for example, driver information, road information during travel, detected object information, and operation information such as steering amount and pedal depression amount.

[0059] In detecting an abnormal response of a driver, based on the three patterns of abnormal responses described above, detection methods are assumed that include an observation target, characteristic changes / reactions in the observation target, and characteristic behaviors, as shown in Fig. 3. In detecting an abnormal response, the observation target is divided into five categories, for example, face / gaze, arms, legs, torso, and voice, and a detection method is set for each category.

[0060] Characteristic changes and reactions in the face and gaze, for example, changes in facial direction, gaze direction, and facial expression, are set and detected based on the image analysis results of the driver imaging unit 19. Characteristic changes and reactions in the arms, for example, stiffening of the arms, changes in steering operation, etc. are set and detected based on the image analysis results of the driver imaging unit 19 and operation information of the vehicle. Characteristic changes and reactions in the legs, for example, improper operation of a pedal device or stiffening of the legs, etc. are set and detected based on operation information of the vehicle. Characteristic changes and reactions in the torso, for example, lifting of the shoulders or upper body, leaning backward, etc. are set and detected based on the image analysis results of the driver imaging unit 19. Characteristic changes and reactions in the voice, for example, slurred speech, meaningless speech, etc. are set and detected based on sound data from the cabin acoustic sensor 21. In addition, setting data for detection conditions for characteristic changes and reactions in abnormal responses is created in advance in the form of a data table or the like, and the setting data is stored in a recording medium (not shown) of the vehicle control unit 28.

[0061] Then, for example, the abnormality detection unit 36 ​​acquires various information such as driver information, road information during travel, detected object information, and operation information such as the amount of steering of the steering wheel, the amount of depression of the pedal device, etc. Thereafter, the abnormality detection unit 36 ​​reads setting data for detecting an abnormal response, and detects an abnormal response based on whether or not the acquired information matches the set detection conditions for an abnormal response.

[0062] An abnormal response is detected, for example, by the above-described method, and the vehicle control device is required to perform appropriate vehicle control depending on the type of abnormal response. For example, if the abnormal response is abnormal behavior and the driver performs an incorrect steering operation, the vehicle control device is required to not accept any operation by the driver, switch the driving subject to the system, and perform an appropriate steering operation. For example, if the abnormal response is excessive tension and the driver's operation itself is correct but the control amount is inappropriate, the vehicle control device is required to accept the driver's operation while correcting the control amount to an appropriate value. For example, if the abnormal response is incapacity to act and no operation is performed, the vehicle control device is required to switch the driving subject to the system and perform appropriate autonomous driving control. Therefore, the vehicle control system 1 is configured to perform deceleration control and other vehicle control depending on the type of abnormal response when an abnormal response is detected.

[0063] Furthermore, abnormal responses can be divided into two types: a first abnormal response that occurs immediately after some kind of panic factor occurs, and a second abnormal response that occurs when mental agitation continues after the direct panic factor has disappeared. For the sake of convenience, the first abnormal response will sometimes be referred to as the "immediate response" and the second abnormal response as the "after-effect." Furthermore, hereinafter, after-effects will sometimes be abbreviated as "AE."

[0064] The vehicle control device described in Patent Document 1 detects an immediate response based on the grip pressure of the steering wheel and the dilation of the pupils and executes deceleration control, but does not handle vehicle control other than deceleration control, and does not anticipate detecting and responding to after-effects. For example, this vehicle control device cannot handle a situation where the vehicle needs to resume traveling after being stopped by the driver's operation or system control.

[0065] In contrast, in the vehicle control system 1 according to the embodiment, for example, the abnormality detection unit 36 ​​detects each of the immediate response and the after-effect, the control decision unit 35 decides vehicle control appropriate for the abnormal response, and the execution unit executes the decided vehicle control. The vehicle control executed is at least one of speed control, steering control, notification to the occupants of the vehicle, notification to those around the vehicle, correction of the driver's operation, and assistance in starting to drive the vehicle. As a result, the vehicle control system 1 executes appropriate vehicle control when there is an immediate response or an after-effect by the driver, and can also safely resume driving the vehicle that has stopped.

[0066] [Vehicle Control] Next, a description will be given of vehicle control by the vehicle control system 1. The vehicle control system 1 executes the control flow shown in Fig. 4 when a predetermined start condition is satisfied, for example, when the ignition of the host vehicle is turned on.

[0067] In step S10, for example, the environment recognition unit 29 acquires external sounds, target object information, and road information around the vehicle from the perimeter monitoring sensor 14, and acquires road information from the in-vehicle communication device 22 and the locator 23. The environment recognition unit 29 also acquires operation information of the steering wheel, pedal device, etc. of the vehicle via the communication bus 40. The environment recognition unit 29 then grasps the driving environment and driving state of the vehicle based on the acquired information, and sequentially provides the grasped driving information to the control decision unit 35.

[0068] In the following step S11, for example, the environment recognition unit 29 acquires driver information from the driver monitor 18 and sequentially provides the driver information to the control decision unit 35.

[0069] Next, in step S12, for example, the abnormality detection unit 36 ​​determines whether or not a first abnormal response from the driver of the vehicle has been detected based on the various information acquired in steps S10 and S11. Then, for example, the vehicle control unit 28 proceeds to step S20 if the determination in step S12 is affirmative, and returns to step S10 if the determination in step S12 is negative.

[0070] In step S20, for example, the vehicle control unit 28 executes a first vehicle control process. The first vehicle control is at least one of speed control, steering control, notification and guidance to the vehicle's occupants, notification to those around the vehicle, correction of the driver's operation, and assistance in starting to drive after the vehicle has stopped, and is determined by the control decision unit 35 in response to an abnormal response by the driver. As a result, if the driver exhibits an abnormal response that differs from normal due to some panic factor, the vehicle control system 1 executes appropriate vehicle control in response to the abnormal response, thereby improving the safety of the vehicle and its occupants. After executing the first vehicle control, the vehicle control unit 28 proceeds to step S30. Details of the first vehicle control will be described later.

[0071] In step S30, for example, the abnormality detection unit 36 ​​determines whether or not a second abnormal response, i.e., an after-effect (AE), of the driver of the vehicle has been detected based on the driving environment information and the driver information acquired after the execution of the first vehicle control. Then, for example, the vehicle control unit 28 proceeds to step S40 if the determination in step S30 is affirmative, and skips step S40 if the determination in step S20 is negative.

[0072] In step S40, for example, the vehicle control unit 28 executes a second vehicle control process. The second vehicle control is, for example, similar to the first vehicle control and is determined by the control determination unit 35 in accordance with the driver's AE. As a result, if the driver exhibits AE after any panic factor has disappeared, the vehicle control system 1 executes appropriate vehicle control in accordance with the AE, thereby improving the safety of the vehicle and its occupants. Details of the second vehicle control will be described later.

[0073] The vehicle control unit 28 then repeats the above series of processes until a predetermined termination condition is met, such as when the shift lever is in the parking position. This allows the vehicle control system 1 to execute deceleration control and other appropriate vehicle control in response to the driver's abnormal response, as well as assist the driver in resuming driving after the vehicle has come to a stop. Furthermore, the vehicle control system 1 can execute appropriate vehicle control not only immediately after the driver experiences some kind of panic factor, but also in a state in which the driver continues to be mentally unsettled and is unable to take appropriate action even after the panic factor has disappeared.

[0074] [First Vehicle Control] Next, an example of the first vehicle control in step S20 will be described. For example, the vehicle control unit 28 executes the first vehicle control according to the control flow shown in FIG.

[0075] In step S200 of the first vehicle control, for example, the abnormality detection unit 36 ​​estimates a panic factor based on various information such as object information from the perimeter monitoring sensor 14, driver information from the driver monitor 18, road information from the locator 23, and disaster information from the in-vehicle communication device 22. The panic factor is estimated based on various information such as the driver's line of sight and facial direction, arm and upper body movements, sound information such as sounds and voices in the vehicle cabin, object detection information around the vehicle, and disaster information. The panic factor estimation result in step S200 is used, for example, in the guidance and vehicle control processing in the autonomous driving mode, which will be described later, as well as in the second vehicle control.

[0076] In the subsequent step S210, for example, the abnormality detection unit 36 ​​determines whether the detected abnormal response is excessive tension. For example, the abnormality detection unit 36 ​​determines excessive tension when the driver is operating the pedal device and / or the steering wheel and the operation is appropriate for the driving environment, but this is not limited to this. Whether the operation is appropriate can be determined, for example, by creating in advance setting data for an appropriate range of pedal device and / or steering wheel operation corresponding to the driving environment recognized by the environment recognition unit 29 and determining whether the operation is within the appropriate range. In this case, for example, the setting data may be created in the form of a data table or the like and stored in a recording medium (not shown) of the vehicle control unit 28. The abnormality detection unit 36 ​​may then read the setting data in the determination processing of step S210. Then, for example, if the vehicle control unit 28 makes a positive determination in step S210, the process proceeds to step S220, and if the vehicle control unit 28 makes a negative determination in step S210, the process proceeds to step S230.

[0077] In step S220, for example, the driving assistance unit 37 determines to execute guidance to the driver corresponding to "hypertension" as vehicle control. Then, for example, the device control unit 34 outputs a guidance execution command to the display device 12 and / or the audio device 13 based on the guidance execution request from the control determination unit 35. For example, the audio device 13 executes guidance by voice, such as "please calm down," to calm the driver of the vehicle. In this way, the vehicle control system 1 can alleviate the hypertension state of the driver of the vehicle.

[0078] In step S221, for example, the abnormality detection unit 36 ​​determines whether the amount of operation by the driver is appropriate based on the driving information and operation information of the pedal device, etc., acquired after step S210. Then, for example, if the determination in step S221 is negative, the vehicle control unit 28 proceeds to step S222. On the other hand, for example, if the determination in step S221 is positive, the vehicle control unit 28 skips to the processing in step S222 and ends the first vehicle control processing.

[0079] In step S222, for example, the driving assistance unit 37 determines to correct the driver's operation. Then, in response to the execution request from the driving assistance unit 37, the control execution unit 39 outputs an execution command for speed control and / or steering control to the cruise control ECU 27. For example, if the first abnormal response indicates that the driver's pedal depression amount, steering wheel operation amount, or both are excessive or insufficient, the cruise control ECU 27 executes correction so that the operation amount falls within an appropriate range. That is, in step S222, the vehicle control system 1 reduces the operation amount if the driver's operation amount is excessive, and increases the operation amount if the driver's operation amount is insufficient. Furthermore, in step S222, even if the driver's operation is not smooth, the vehicle control system 1 increases the operation amount so that the driver's operation is appropriate as intended. As a result, the vehicle control system 1 corrects the operation amount of the pedal device, etc., by the overstrained driver to an appropriate value, thereby executing appropriate deceleration, acceleration, or steering, and improving the driving safety of the host vehicle. Then, for example, after executing the process of step S222, the vehicle control unit 28 ends the first vehicle control process.

[0080] In step S230, for example, the abnormality detection unit 36 ​​determines whether the detected abnormal response is abnormal behavior. For example, the abnormality detection unit 36 ​​determines abnormal behavior when the driver is operating the pedal device and / or steering wheel and the vehicle operation is inappropriate for the driving environment, but this is not limited to this. The abnormal behavior determination may be performed, for example, similar to the above-described determination of hypertension, by creating setting data in advance that anticipates operations corresponding to abnormal behavior and then using the acquired operation information and the setting data. Then, for example, after acquiring the operation information, the abnormality detection unit 36 ​​reads the setting data for determination and determines that the first abnormal response is abnormal behavior if the driver's operation corresponds to abnormal behavior. For example, the setting data used in step S230 corresponds to highly dangerous abnormal behavior, such as reverse steering or inappropriate acceleration, in order to prioritize risk avoidance of the vehicle. Then, for example, if the vehicle control unit 28 determines a positive result in step S230, the process proceeds to step S231. If the vehicle control unit 28 determines a negative result in step S230, the process proceeds to step S240.

[0081] In step S231, for example, the driving assistance unit 37 determines to execute guidance corresponding to the estimated panic factor and "abnormal behavior" and requests execution of the guidance from the device control unit 34. Then, the device control unit 34 outputs an execution command for the guidance to the display device 12 and / or the audio device 13, similar to step S220.

[0082] For example, if the abnormal behavior is caused by mistaking the accelerator for the brake, the audio device 13 or the like may issue a notification or warning indicating the erroneous operation, such as "You are misstepping," or guidance guiding the driver to the correct operation, such as "Please release the pedal." Furthermore, if the panic factor is estimated to be a passenger disorder, the audio device 13 or the like may issue guidance such as "Stop the vehicle in a safe place and call 119." Furthermore, if the panic factor is estimated to be a collapsed load, the audio device 13 or the like may issue guidance encouraging the driver to take the correct action, such as "Please concentrate on driving and decide to pick up the luggage later" or "Please have your passenger pick up the luggage." Furthermore, if the panic factor is estimated to be a disaster such as an earthquake, the audio device 13 or the like may issue guidance such as "Please park the vehicle on the left side of the road." In this way, the guidance in step S231 is appropriately changed depending on the type of panic factor and abnormal behavior. This allows the vehicle control system 1 to guide the driver engaging in abnormal behavior to the correct behavior.

[0083] In step S231, for example, the driving assistance unit 37 may determine to provide haptic feedback to the driver to recognize the correct operation corresponding to the situation, in addition to the guidance, and may issue an execution request to the cruise control ECU 27 via the control execution unit 39. For example, if the abnormal behavior is an erroneous pedal depression and the pedal device is a haptic pedal, the cruise control ECU 27 may cause the pedal device to provide haptic feedback, such as increasing the weight of the pedal depressed by the driver or moving it to the opposite side.

[0084] In the following step S232, for example, the abnormality detection unit 36 ​​determines whether the operation by the driver is appropriate based on the driving environment information and operation information of the pedal device, etc. acquired after step S231. Then, for example, the vehicle control unit 28 advances the process to step S233 if the determination in step S232 is affirmative, or advances the process to step S234 if the determination in step S232 is negative.

[0085] In step S233, for example, the abnormality detection unit 36 ​​determines whether the amount of operation by the driver is appropriate based on the driving information and operation information of the pedal device, etc., acquired after step S231. Then, for example, if the determination in step S233 is negative, the vehicle control unit 28 proceeds to step S222 and performs correction processing to make the amount of operation by the driver appropriate, as described above. On the other hand, if the determination in step S233 is positive, the vehicle control unit 28 skips the processing in step S222, allows the driver to continue operating the vehicle as is, and ends the first vehicle control processing. This is because it is estimated that the driver's operation and the amount of control thereof are appropriate based on the guidance.

[0086] In step S234, since the driver's abnormal behavior continues even after the guidance is executed, for example, the driving support unit 37 switches the driving mode from the manual driving mode to the automatic driving mode. This changes the driving subject from the driver to the system, prevents the driver from performing inappropriate operations, and enables a prompt transition to risk avoidance.

[0087] In the following step S235, for example, the driving assistance unit 37 determines to execute deceleration control and steering control and causes the cruise control ECU 27 to stop the vehicle via the control execution unit 39. In this case, for example, the driving assistance unit 37 acquires information on a safe stopping position for the host vehicle based on locator information and road information, and executes deceleration and steering control so as to stop the host vehicle in a safe position. Furthermore, for example, if there is a possibility that the host vehicle will collide with another object, the driving assistance unit 37 outputs an activation signal to the collision suppression device 26 and executes deceleration and steering control to avoid the collision or mitigate the impact. As a result, even if the driver performs inappropriate operations such as reverse steering or accelerating when deceleration is required, the vehicle control system 1 can override the manual operation with automatic operation by the system, stop the host vehicle, and ensure safety.

[0088] In subsequent step S236, for example, the driving assistance unit 37 determines to issue an alert to the exterior of the host vehicle. Then, for example, the driving assistance unit 37 causes the vehicle exterior alert ECU 25 to issue an alert to the exterior of the host vehicle via the device control unit 34. For example, the vehicle exterior alert ECU 25 turns on the hazard lights in the turn signals and, if the vehicle has an exterior speaker or a display that displays images to the exterior, issues an alert to the exterior that the host vehicle is stopped due to an emergency. This allows the vehicle control system 1 to communicate the status of the host vehicle to the exterior when the host vehicle is stopped by autonomous driving, thereby preventing the occurrence of a secondary traffic accident. The vehicle exterior alert in step S236 may be issued not only after the host vehicle has stopped, but also when autonomous driving-based stop control is initiated, and the timing of this execution may be changed as appropriate.

[0089] In the following step S237, for example, the restart support unit 38 determines whether or not the driver can resume driving. For example, based on the driver information acquired after step S236, the restart support unit 38 determines that driving can be resumed if it is estimated from the driver's facial expression, etc. that the panic has subsided, and otherwise determines that driving cannot be resumed. Then, for example, if the vehicle control unit 28 makes a positive determination in step S237, the process proceeds to step S239, and if the vehicle control unit 28 makes a negative determination in step S237, the process proceeds to step S238.

[0090] In step S238, for example, the restart support unit 38 determines to execute the driving restart support and causes the display device 12 and / or the audio device 13 to execute guidance for restarting driving via the device control unit 34. For example, the device control unit 34 causes the display device 12 and / or the audio device 13 to execute guidance and output a message to the driver by voice, such as "Please stay calm" or "Take a deep breath." Furthermore, for example, the device control unit 34 may cause the audio device 13 to play music to calm the driver. Furthermore, in the driving restart support in step S238, similar to step S231, guidance corresponding to the panic factor estimated in step S200 is executed. For example, if the panic factor is estimated to be a medical condition of the passenger, the audio device 13 or the like executes guidance such as "Call 119" or "Take x first aid measures" (x being specific examples of first aid measures). Furthermore, for example, when it is estimated that the panic factor is an earthquake, audio device 13 or the like executes guidance such as "Stop the engine and wait inside the vehicle until the shaking stops. Once the shaking stops, get out of the vehicle with the key in the ignition and leave the doors unlocked, and evacuate to a safe place." Then, for example, vehicle control unit 28 returns the process to step S237 after step S238.

[0091] In step S239, for example, the restart support unit 38 changes the driving mode back from the automatic driving mode to the manual driving mode. After step S239, for example, the vehicle control unit 28 may cause the audio device 13 or the like to notify the driver that the vehicle is now operable and to provide guidance to encourage the driver to resume driving. This allows the vehicle control system 1 to assist the driver of the vehicle in resuming driving as needed, and to return the vehicle to a state in which the driver can resume driving. After executing step S239, for example, the vehicle control unit 28 ends the first vehicle control process.

[0092] In step S240, since the determinations in both steps S210 and S230 are negative and the driver's first abnormal response is estimated to be incapacity to act, i.e., a freezing reaction, for example, the driving assistance unit 37 determines to execute guidance corresponding to the panic factor and "inability to act." Then, for example, the device control unit 34 outputs a guidance execution command to the display device 12 or the like in response to the execution request from the driving assistance unit 37.

[0093] For example, if the panic factor is estimated to be a collision between the vehicle and some object, the audio device 13 etc. executes guidance to encourage the driver or passengers to take the correct action. Furthermore, if the panic factor is estimated to be a collapse of luggage, the audio device 13 etc. executes guidance such as "Pick up the luggage later and concentrate on driving" or "Please have a passenger pick up the luggage." Furthermore, if the panic factor is estimated to be a disaster such as an earthquake, the audio device 13 etc. executes guidance such as "Please park on the left side of the road" or an announcement to encourage passengers to take the correct action. In this way, the vehicle control system 1 can encourage the driver or passengers to take the correct action when the driver is incapacitated.

[0094] In step S240, for example, the vehicle control unit 28 may execute haptic feedback processing to prompt the driver to perform the correct operation, in addition to providing guidance through the audio device 13 or the like. For example, the vehicle control unit 28 may vibrate a part of the pedal device or the steering wheel when it is necessary to operate these devices to prompt the driver to operate them. Furthermore, when there is a possibility that the host vehicle will collide with another object without any operation, the vehicle control unit 28 may activate the collision suppression device 26 to hasten the execution of a warning of the possibility of a collision and control to avoid the collision.

[0095] In the following step S241, for example, the environment recognition unit 29 determines whether the driver is operating the vehicle based on the operation information acquired after step S240. Then, for example, the vehicle control unit 28 proceeds to step S242 if the determination in step S241 is affirmative, or proceeds to step S234 if the determination in step S241 is negative. As a result, if the driver is not operating the vehicle after the guidance in step S240 is executed, the vehicle control system 1 executes the processes from step S234 onward, switches to the autonomous driving mode, and can perform appropriate vehicle control.

[0096] In step S242, for example, the abnormality detection unit 36 ​​determines whether the driver's operation is appropriate based on the driving information and operation information of the pedal device, etc. Then, for example, if the vehicle control unit 28 makes a negative determination in step S242, it proceeds to step S234. As a result, if the driver operates the vehicle after the guidance in step S240 is executed but the operation is inappropriate, the vehicle control system 1 executes the processes from step S234 onwards, switches to autonomous driving mode, and performs appropriate vehicle control. On the other hand, for example, if the vehicle control unit 28 makes a positive determination in step S242, it is presumed that the driver's vehicle operation is appropriate, so it allows the driver to continue operating the vehicle as is and ends the first vehicle control process.

[0097] The above is an example of a control flow in the first vehicle control. When the vehicle control system 1 detects an immediate response by the first vehicle control, it executes at least one of speed control, steering control, guidance to the occupants, notification to those around the vehicle, correction of the driver's operation, and assistance in resuming driving. As a result, when the driver makes an abnormal response immediately after the occurrence of some panic factor, the vehicle control system 1 can execute not only deceleration control but also appropriate vehicle control in response to the abnormal response.

[0098] The first vehicle control is not limited to the example shown in FIG. 5 . For example, while the example in which the type of abnormal response is determined in the order of hypertension and abnormal behavior has been described, the first vehicle control may reverse this order, or may further include a determination of whether the vehicle is unable to act. Furthermore, the first vehicle control may issue a warning that the amount of operation or operation is inappropriate, and may also notify the details of subsequent control, when executing the operation correction in step S222 or the automatic control to stop the vehicle in step S235. In this way, the first vehicle control may appropriately change the content and order of processes, or add other processes, within the scope of feasibility.

[0099] [Second Vehicle Control] Next, an example of the second vehicle control in step S40 after AE is detected will be described with reference to FIG. 6, but the processing common to the first vehicle control will be briefly described.

[0100] If an AE is detected in step S30, for example, the vehicle control unit 28 proceeds to step S410. In step S410, for example, the abnormality detection unit 36 ​​determines whether the second abnormal response is excessive tension using a method similar to that used in step S210 of the first vehicle control. If the determination in step S410 is affirmative, the vehicle control unit 28 proceeds to step S420, and if the determination in step S410 is negative, the vehicle control unit 28 proceeds to step S430.

[0101] In step S420, for example, the vehicle control unit 28 executes the process of providing guidance to the driver corresponding to "hypertension" in the same manner as in step S220 of the first vehicle control. At this time, for example, the vehicle control unit 28 may cause the audio device 13 or the like to play music that calms the driver in addition to the guidance. As a result, the vehicle control system 1 causes the audio device 13 or the like to execute guidance by voice or the like to calm the driver of the vehicle, such as "please calm down," thereby alleviating the hypertension state of the driver.

[0102] Thereafter, for example, the vehicle control unit 28 proceeds to step S421, which corresponds to step S221 in the first vehicle control, and if the determination in step S421 is negative, i.e., if the amount of operation by the driver is inappropriate, proceeds to step S422, which corresponds to step S222. On the other hand, if the determination in step S421 is positive, for example, the vehicle control unit 28 skips the processing in step S422, allows the driver to continue driving, and ends the processing of the second vehicle control.

[0103] In step S422, for example, the vehicle control unit 28 corrects the amount of operation of the pedal device or the like by the driver, similarly to step S222, to suppress excessive operation amount or increase insufficient operation amount.

[0104] In step S430, for example, the abnormality detection unit 36 ​​determines whether the AE is behaving abnormally using a method similar to that of step S230 in the first vehicle control. Possible abnormal behaviors of the AE include, for example, abnormal behaviors in immediate response, as well as erratic driving of the vehicle, leaving malfunctioning in on-board devices such as windshield wipers or blinkers unattended, and a decrease in the frequency with which the driver checks behind. Therefore, for example, the setting data used to determine whether the AE is behaving abnormally corresponds to abnormal behaviors with a relatively low risk, such as those described above, in addition to abnormal behaviors in immediate response. If the vehicle control unit 28 determines that the AE is behaving abnormally in step S430, the process proceeds to step S431. If the vehicle control unit 28 determines that the AE is behaving abnormally in step S430, the process proceeds to step S440.

[0105] In step S431, for example, the vehicle control unit 28 executes a process of providing guidance to the driver corresponding to the panic factor and "abnormal behavior" estimated in step S220. For example, the audio device 13 or the like executes guidance by voice or the like to encourage correct operation, such as "Please turn off the wipers" if the wipers are malfunctioning, or "Your eyes are too focused on the front" if the driver is checking behind you less frequently. Furthermore, for example, the audio device 13 or the like may execute an announcement not only to the driver but also to passengers to encourage the driver to operate the vehicle correctly.

[0106] The processing of steps S432 to S439 corresponds to the processing of steps S232 to S239 in the first vehicle control. For example, if the vehicle control unit 28 determines in step S432 that the driver's operation is appropriate, the processing proceeds to step S433; otherwise, the processing proceeds to step S434. Note that, for example, in a case where it is estimated that the abnormal behavior will not stop the malfunctioning in-vehicle device, if the malfunctioning in-vehicle device remains operating after the guidance, the vehicle control unit 28 makes a negative determination in step S432 and proceeds to step S434.

[0107] Then, for example, if the vehicle control unit 28 determines in step S433 that the amount of driver operation is appropriate, it allows the driver to continue driving and ends the second vehicle control process. On the other hand, if the determination in step S433 is negative, for example, the vehicle control unit 28 proceeds to step S422 and executes a process to correct the driver's operation. As a result, the vehicle control system 1 can correct the amount of steering by the driver when the vehicle is staggering due to abnormal AE behavior, for example, and allow the vehicle to drive appropriately.

[0108] If the determination in step S432 is negative, for example, the vehicle control unit 28 sequentially executes steps S434 to S437, i.e., switching to autonomous driving mode, stopping the vehicle through automatic control, providing an external alarm, and determining whether driving can be resumed. If an in-vehicle device, such as windshield wipers, is operating in a situation where it is not necessary to operate it, for example, the vehicle control unit 28 executes control to stop the in-vehicle device after step S434. This allows the vehicle control system 1 to stop the in-vehicle device if the driver does not turn it off even after the guidance in step S431. Furthermore, for example, if the vehicle control unit 28 determines that driving cannot be resumed in step S437, the process proceeds to step S438 and executes driving resumption assistance similar to the first vehicle control. Then, for example, the vehicle control unit 28 returns the process to step S437. Thereafter, for example, if the determination in step S437 is affirmative, the vehicle control unit 28 proceeds to step S439, sets the driving mode to the manual driving mode, and ends the second vehicle control process.

[0109] The processing of steps S440 to S442 corresponds to the processing of steps S240 to S242 in the first vehicle control. In step S440, for example, the vehicle control unit 28 executes guidance processing corresponding to "impossible to act." As a result, the audio device 13 or the like executes guidance by voice or the like urging the driver to operate the vehicle or reminding the driver of the operation that should be performed, or executes an announcement to passengers urging the driver to take the correct action. After step S440, for example, the vehicle control unit 28 executes processing to determine whether or not the vehicle is being operated in step S441, and if the determination in step S441 is affirmative, executes processing to determine whether or not the operation is appropriate in step S442. Then, for example, if the determination in either step S441 or S442 is negative, the vehicle control unit 28 proceeds to step S434, and if the determination in step S442 is affirmative, the vehicle control unit 28 causes the driver to continue operating the vehicle and ends the processing of the second vehicle control.

[0110] The above is an example of a control flow in the second vehicle control. When the vehicle control system 1 detects an AE through the second vehicle control, it executes at least one of speed control, steering control, guidance to the occupants, notification to those around the vehicle, correction of the driver's operation, turning off the in-vehicle devices, and assistance in resuming driving. As a result, when the driver exhibits an abnormal response after any panic factor has disappeared, the vehicle control system 1 can execute deceleration control as well as appropriate vehicle control in response to the abnormal response.

[0111] In the vehicle control system 1 according to the embodiment, the driver recognition unit 32 recognizes the driver's state of the vehicle using the driver monitor 18, and the control decision unit 35 detects an abnormal response of the driver based on the driver information from the driver recognition unit 32 and decides to execute vehicle control in response to the abnormal response. The vehicle control system 1 also includes at least one execution unit that executes the vehicle control decided by the control decision unit 35. The vehicle control executed upon detection of an abnormal response is at least one of speed control, steering control, guidance to the occupants, notification to those around the vehicle, correction of the driver's operation, and travel resumption assistance. This enables the vehicle control system 1 to execute appropriate vehicle control in response to the driver's abnormal response. Furthermore, the vehicle control system 1 has the following features.

[0112] (1) The control decision unit 35 has an abnormality detection unit 36 ​​that detects a first abnormal response from the driver immediately after a panic factor occurs, and decides to execute a first vehicle control corresponding to the first abnormal response. This allows the vehicle control system 1 to grasp the abnormal response from the driver immediately after some panic factor occurs, i.e., the immediate response, and execute appropriate vehicle control in accordance with the immediate response.

[0113] (2) The abnormality detection unit 36 ​​detects a second abnormal response after the first vehicle control is executed. Then, the control decision unit 35 decides to execute a second vehicle control corresponding to the second abnormal response. This allows the vehicle control system 1 to grasp the abnormal response of the driver after some panic factor has disappeared, i.e., the after-effect, and to execute appropriate vehicle control in accordance with the after-effect.

[0114] (3) The abnormality detection unit 36 ​​determines whether the driver can resume driving after the host vehicle has stopped due to the execution of deceleration control. If the abnormality detection unit 36 ​​determines that the driver cannot resume driving, the control decision unit 35 decides to execute driving resumption assistance. This allows the vehicle control system 1 to execute driving resumption assistance if it is difficult to resume driving after the host vehicle has stopped due to deceleration control, and to return the host vehicle to a state where it can be driven appropriately.

[0115] (Other Embodiments) While the present disclosure has been described with reference to examples, it is understood that the present disclosure is not limited to those examples or structures. The present disclosure also encompasses various modifications and modifications within the scope of equivalents. In addition, various combinations and forms, as well as other combinations and forms including only one element, more than one, or less than one, are also within the scope and spirit of the present disclosure.

[0116] Each processing unit in the above embodiment is, for example, hardware for arithmetic processing coupled to a RAM, and includes at least one arithmetic core, such as a CPU and a GPU. GPU is an abbreviation for Graphics Processing Unit. Each processing unit may further include an FPGA, an NPU, and an IP core with other dedicated functions. FPGA and NPU are abbreviations for Field-Programmable Gate Array and Neural Network Processing Unit, respectively. Each processing unit may be individually mounted on a printed circuit board, or may be mounted on an ASIC, FPGA, or the like. ASIC is an abbreviation for Application Specific Integrated Circuit. In this way, the hardware configuration of each processing unit can be changed as appropriate.

[0117] The form of the recording medium (non-transitory tangible storage medium) on which various programs, including the vehicle control program corresponding to the vehicle control in the above embodiment, are recorded may be changed as appropriate. Furthermore, the recording medium is not limited to being provided on a circuit board, but may be provided in the form of a memory card or the like, inserted into a slot, and electrically connected to a control circuit such as the vehicle control unit 28. Furthermore, the storage medium may be an optical disk, hard disk drive, solid state drive, or the like, from which the program is copied or distributed to the vehicle control unit 28, etc.

[0118] Furthermore, the vehicle control system 1 according to the embodiment may be configured to include, for example, the various on-board sensors and known sensors other than the on-board devices described above, and may execute control to activate other on-board devices as necessary when an abnormal response by the driver is detected. For example, the vehicle control system 1 may be configured to include a fragrance device, and may execute control to activate the fragrance device and provide a scent to the vehicle cabin that calms the driver when an abnormal response by the driver is detected. In this way, the configuration and the content of the vehicle control executed by the vehicle control system 1 may be changed as appropriate.

[0119] The vehicle control unit 28 and the methods described herein may be implemented by a special-purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the vehicle control unit 28 and the methods described herein may be implemented by a special-purpose computer configured with a processor configured with one or more dedicated hardware logic circuits. Alternatively, the vehicle control unit 28 and the methods described herein may be implemented by one or more special-purpose computers configured with a processor configured with one or more hardware logic circuits in combination with a processor and memory programmed to perform one or more functions. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0120] It goes without saying that in each of the above embodiments, the elements constituting the embodiments are not necessarily essential unless they are specifically stated as essential or are clearly considered essential in principle. Furthermore, in each of the above embodiments, when numerical values ​​such as the number, values, amounts, and ranges of the components of the embodiments are mentioned, they are not limited to the specific numbers unless they are specifically stated as essential or are clearly limited to a specific number in principle. Furthermore, in each of the above embodiments, when the shapes, positional relationships, etc. of the components are mentioned, they are not limited to the shapes, positional relationships, etc., unless they are specifically stated or are clearly limited to a specific shape, positional relationship, etc. in principle.

Claims

1. A vehicle control device comprising: a driver recognition unit (32) that recognizes the state of a driver of the vehicle based on information from a driver monitor (18) mounted on the vehicle; a control decision unit (35) that detects an abnormal response of the driver based on the driver information from the driver recognition unit and decides to execute vehicle control in accordance with the abnormal response; and an execution unit (34, 39) that executes the vehicle control decided by the control decision unit, wherein the vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of operations by the driver, and assistance in resuming driving.

2. A vehicle control device as described in claim 1, wherein the control decision unit has an abnormality detection unit (36) that detects a first abnormal response from the driver immediately after a panic factor occurs, and decides to execute a first vehicle control corresponding to the first abnormal response.

3. A vehicle control device as described in claim 2, wherein the abnormality detection unit detects a second abnormal response after the first vehicle control is executed, and the control decision unit decides to execute a second vehicle control corresponding to the second abnormal response.

4. A vehicle control device as described in claim 2 or 3, wherein the abnormality detection unit determines whether the driver can resume driving after the vehicle has stopped due to the execution of deceleration control, and the control decision unit decides to execute driving resumption assistance if the abnormality detection unit determines that the driver cannot resume driving.

5. A vehicle control method that can be used in a vehicle, comprising: grasping the state of a driver of the vehicle from information from a driver monitor (18) mounted on the vehicle; detecting an abnormal response of the driver based on the grasped driver information; deciding to execute vehicle control in accordance with the abnormal response; and executing the decided vehicle control, wherein the vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of operations by the driver, and assistance in resuming driving.

6. A vehicle control program used to control the driving state of a vehicle or on-board equipment (22, 24-27), the vehicle control program causing at least one execution unit (28) to execute processes including: a process of grasping the state of the driver of the vehicle from information from a driver monitor (18) mounted on the vehicle; a process of detecting an abnormal response of the driver based on the grasped driver information; a process of deciding to execute vehicle control in accordance with the abnormal response; and a process of executing the decided vehicle control, wherein the vehicle control is at least one of speed control, steering control, guidance to occupants, notification to those around the vehicle, correction of operation by the driver, and driving resumption support.

Images