Driver assistance control device, driver assistance method, and computer program

Abstract

It accurately determines whether the driver is in an abnormal state. [Solution] The driver assistance control device 6 of the vehicle 100 is configured to perform driver assistance to address driver abnormalities in response to a determination that the driver of the vehicle 100 is in an abnormal state that makes it difficult to continue driving the vehicle 100, and to determine whether the driver is in an abnormal state based on the driver's response to a predetermined operation or action instruction given to the driver.

Description

Technical Field

[0001] The present invention relates to a driving support control device, a driving support method, and a computer program.

Background Art

[0002] In Patent Document 1, as a conventional vehicle driving control device, it is configured to determine whether a driver is in an abnormal state, and when the driver is in an abnormal state, perform deceleration stop control to decelerate the host vehicle and keep the host vehicle in a stopped state.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] If it is not accurately determined whether the driver is in an abnormal state, there is a risk that deceleration stop control will be performed regardless of whether the driver is in a normal state.

[0005] The present invention has been made paying attention to such problems, and an object thereof is to accurately determine whether a driver is in an abnormal state.

Means for Solving the Problems

[0006] In order to solve the above problems, a driving support control device for a vehicle according to an aspect of the present invention performs driving support for coping with a driver's abnormality in response to a determination that the driver of the vehicle is in an abnormal state in which it is difficult to continue driving the vehicle, and determines whether the driver is in an abnormal state based on a response result of the driver to the instruction when a predetermined operation or movement instruction is given to the driver.

[0007] Furthermore, a vehicle driving assistance method according to one aspect of the present invention provides driving assistance to address a driver abnormality in response to a determination that the vehicle driver is in an abnormal state that makes it difficult to continue driving the vehicle, and determines whether the driver is in an abnormal state based on the driver's response to an instruction given to perform a predetermined operation or action.

[0008] Furthermore, a computer program according to one aspect of the present invention performs driving assistance to address driver abnormalities in response to a determination that the vehicle driver is in an abnormal state that makes it difficult to continue driving the vehicle, and causes the computer to execute a process that determines whether the driver is in an abnormal state based on the driver's response to a predetermined operation or action instruction given to the driver. [Effects of the Invention]

[0009] According to these embodiments of the present invention, it is possible to accurately determine whether a driver is in an abnormal state. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of a vehicle according to one embodiment of the present invention. [Figure 2] This flowchart illustrates the details of the driver abnormal state detection process according to one embodiment of the present invention. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described in detail below with reference to the drawings. In the following description, similar components will be given the same reference numerals.

[0012] Figure 1 is a schematic diagram of a vehicle 100 according to one embodiment of the present invention.

[0013] Vehicle 100 includes a peripheral sensor 1, a vehicle sensor 2, a driver sensor 3, an HMI (Human Machine Interface) 4, an actuator 5, and a control device 6. The peripheral sensor 1, vehicle sensor 2, driver sensor 3, HMI 4, actuator 5, and control device 6 are each connected to communicate via an in-vehicle network 9 that conforms to standards such as a controller area network.

[0014] The surrounding sensor 1 is a sensor for generating surrounding data that represents the conditions around the vehicle 100. In this embodiment, the vehicle 100 is equipped with one or more external cameras 11 as the surrounding sensor 1 for taking pictures of the area around the vehicle 100.

[0015] The external camera 11 captures images of the area around the vehicle 100 at a predetermined frame rate (for example, 10 Hz to 40 Hz) and generates an ambient image showing the area around the vehicle 100. Each time the external camera 11 generates an ambient image, it transmits the generated ambient image to the control device 6 as ambient data.

[0016] In addition to the external camera 11, the vehicle 100 may also be equipped with a distance measuring sensor as a surrounding sensor 1 to measure the distance to targets and features in its vicinity. Examples of distance measuring sensors include LiDAR (Light Detection and Ranging), which measures distance based on the reflected light emitted by radar, and millimeter-wave radar sensors, which measure distance based on the reflected waves emitted by radio waves.

[0017] Vehicle sensor 2 is a sensor for acquiring vehicle data representing the state of vehicle 100. Vehicle 100 according to this embodiment includes, as vehicle sensor 2, a speed sensor 21 that acquires speed data indicating the vehicle's speed, a positioning sensor 22 that acquires current position data indicating the vehicle's current position such as latitude and longitude, a steering sensor 23 that acquires steering data related to steering operations such as steering grip, steering torque, and steering angle, an accelerator sensor 24 that acquires accelerator data related to accelerator operations such as accelerator pedal operation amount, and a brake sensor 25 that acquires brake data related to brake operations such as brake pedal operation input. However, vehicle sensor 2 is not limited to these sensors. Each data acquired by each sensor 21 to 25 is transmitted to the control device 6 as vehicle data.

[0018] The driver sensor 3 is a sensor for generating driver data that represents the driver's state. In this embodiment, the vehicle 100 includes a driver monitor camera 31 as the driver sensor 3 for capturing the driver's appearance, including the driver's face. The driver monitor camera 31 captures the driver's appearance at a predetermined frame rate (for example, 10 Hz to 40 Hz) and generates an appearance image showing the driver's appearance. Each time the driver monitor camera 31 generates an appearance image of the driver, it transmits the generated appearance image to the control device 6 as driver data.

[0019] HMI4 is a user interface for exchanging information between the vehicle 100 and its occupants. HMI4 includes output devices 41 for notifying the vehicle occupants through their bodily senses (e.g., sight, hearing, and touch), and input devices 42 for the vehicle occupants to perform input and response operations. Output devices 41 include, for example, displays (e.g., meter displays, center displays, head-up displays, etc.) and speakers. Input devices 42 include, for example, touch panels, microphones, physical buttons, dials, and other physical controls.

[0020] The HMI 4 notifies the vehicle occupants of information corresponding to the output signal received from the control device 6 via the output device 41, and transmits the data input by the vehicle occupants via the input device 42 to the control device 6.

[0021] The HMI 4 may be pre-mounted in the vehicle 100, or may be a terminal such as a smartphone owned by the vehicle occupants (driver and passengers). In the latter case, for example, information exchange may be performed by communicating wirelessly at short range between the vehicle 100 and the vehicle occupant's terminal, or communication may be performed between the vehicle occupant's terminal and an external server (not shown), and information exchange may be performed indirectly via the server.

[0022] The actuator 5 is a device used for the driving control of the vehicle 100. The vehicle 100 according to the present embodiment includes, as the actuator 5, an acceleration actuator 51 that performs acceleration control of the vehicle 100 (for example, at least one of an engine and a motor), a brake actuator 52 that performs brake control of the vehicle 100 (for example, a hydraulic actuator), and a steering actuator 53 that performs steering control of the vehicle 100 (for example, a steering motor).

[0023] The control device 6 is an ECU (Electronic Control Unit) including a communication unit 61, a storage unit 62, and a processing unit 63.

[0024] The communication unit 61 includes an interface circuit for connecting the control device 6 to the in-vehicle network 9. The communication unit 61 supplies various data received from the outside to the processing unit 63. The communication unit 61 also outputs various signals output from the processing unit 63 to the outside.

[0025] The storage unit 62 has a storage medium such as an HDD (Hard Disk Drive), an SSD (Solid Disk Drive), or a semiconductor memory, and stores various computer programs and data used in the processing by the processing unit 63.

[0026] The processing unit 63 has one or more CPUs (Central Processing Units) and their peripheral circuits, and executes various computer programs stored in the memory unit 62. The processing unit 63 is, for example, a processor. The processing unit 63 may further have other arithmetic circuits such as a logical operation unit, a numerical operation unit, or a graphics processing unit. By executing processing according to the computer program, the processing unit 63 functions as an abnormal state determination unit 71, a recognition unit 72, and a driving support unit 73, and operates as a functional unit (module) that realizes a predetermined function. In the following description, when describing the processing with each functional unit 81 to 83 as the subject, it means that the processing unit 63 is executing the program that realizes each functional unit 81 to 83.

[0027] The following describes the specific processes performed by the control device 6. Specifically, it describes the contents of each functional unit 81 to 83, which are realized by the processing unit 63 executing processes according to the computer program.

[0028] The abnormal condition determination unit 71 determines whether the driver is in an abnormal condition (hereinafter simply referred to as "abnormal condition") that makes it difficult to continue driving, for example, due to a sudden change in physical condition.

[0029] The recognition unit 72 recognizes objects and features around the vehicle 100. For example, the recognition unit 72 sequentially inputs the surrounding image received from the external camera 11 into a classifier to recognize objects such as other vehicles, motorcycles, and pedestrians, as well as features such as curbs, fences, and other similar structures (hereinafter referred to as "boundary structures") and road markings (e.g., lane markings defining driving lanes) within the surrounding image. The classifier can be, for example, a convolutional neural network (CNN) having multiple convolutional layers connected in series from the input side to the output side. The recognition unit 72 also calculates the distance from the vehicle 100 to the objects and features and calculates the position of the objects and features by using, for example, the standard size of the objects and features stored in the memory unit 62 for each type of object and feature, and the size of the objects and features recognized in the surrounding image. Note that the method of recognizing objects and features is not limited to this method, and various known methods may be used for recognition.

[0030] The driver assistance unit 73 controls the actuators 5 based on the targets and features recognized by the recognition unit 72, and provides driver assistance that includes vehicle driving control of the vehicle 100. In this embodiment, the driver assistance unit 73 can provide driver assistance that includes vehicle driving control of the vehicle 100 at a driving control level of Level 3 as defined by the Society of Automotive Engineers (SAE), that is, at a driving control level that does not require the driver to operate each actuator 51 to 53 or monitor the surroundings. The driver assistance unit 73 can also provide driver assistance that includes vehicle driving control of the vehicle 100 at a driving control level in which the driver is involved in driving the vehicle 100, for example, at a driving control level of Level 1 or Level 2 as defined by the SAE.

[0031] The driver assistance unit 73, as one of the driver assistance functions involving vehicle 100 driving control, implements driver abnormality response driving assistance when it determines that the driver is in an abnormal state. Specifically, when the driver assistance unit 73 determines that the driver is in an abnormal state, it first implements notification control, such as a control warning (alarm), to the driver via the HMI 4. Then, after a predetermined time has elapsed since the notification began, it implements deceleration stop control to decelerate the vehicle 100 and keep the vehicle 100 in a stopped state. In other words, driver abnormality response driving assistance includes notification control and deceleration stop control, which includes deceleration control and stop-hold control. In addition, during driver abnormality response driving assistance, the driver assistance unit 73 takes measures to inform surrounding vehicles of the driver abnormality of vehicle 100 (for example, by turning on the hazard lights).

[0032] The method by which vehicle 100 is decelerated and stopped is not particularly limited, as long as the risk of contact with road users outside the vehicle during deceleration and stopping control is taken into consideration. For example, if the vehicle can recognize the lane markings of its own driving lane, it can decelerate and maintain a stopped state along the lane markings of its own driving lane. If the vehicle cannot recognize the lane markings of its own driving lane, it can decelerate and maintain a stopped state along the trajectory of the vehicle in front. If neither the lane markings of its own driving lane nor the vehicle in front can be recognized, it can decelerate and maintain a stopped state while driving straight. In addition, for example, considering the safety of driver rescue and passenger disembarkation, the vehicle may change lanes and move to a lane on the road side or to the shoulder.

[0033] Here, if the accuracy of the abnormal state determination unit 71's determination of whether the driver is in an abnormal state is low, there is a risk that driver abnormality response driving support will be implemented even though the driver is in a normal state. Therefore, the abnormal state determination unit 71 in this embodiment is configured to issue an operation or action instruction to the driver and determine whether the driver is in an abnormal state based on the driver's response to the instruction.

[0034] For example, the abnormal state determination unit 71 can determine that the driver is in an abnormal state if it issues a predetermined operation instruction to the driver via the output device 41 (for example, one or both of the displays and / or speakers), and the driver is unable to perform a response operation corresponding to the operation instruction via the input device 42 within a predetermined time. Examples of predetermined operation instructions include instructions to operate a designated button switch on a touch panel or instructions to operate a designated physical operation.

[0035] Furthermore, for example, the abnormal state determination unit 71 can issue a predetermined operation instruction to the driver via the output device 41 (for example, one or both of the displays and / or speakers), and if the driver fails to perform a response operation to the operation instruction within a predetermined time, it can determine that the driver is in an abnormal state. Examples of predetermined operation instructions include instructions to make the driver perform a specified gesture or pose. The determination of whether or not the driver has performed the specified gesture or pose is not particularly limited, but for example, it can be determined from the appearance of the driver based on the image from the driver monitor camera 31.

[0036] Furthermore, instructions for operation or action to the driver may be given periodically, or only when the driver is in a predetermined abnormal state.

[0037] An abnormal condition is a state in which the driver can be considered to be in an abnormal state. Examples of abnormal conditions include the driver having their eyes closed, the driver having poor posture, and, unless a hands-free driving assistance system is in place, the driver not operating the steering wheel. Poor posture of the driver refers to a state in which the driver is slumped over, looking down, leaning backward, or has their head or upper body tilted or fallen to the side due to muscle relaxation caused by loss of consciousness, etc., or a state in which the driver is arching their back due to rigidity caused by epilepsy, etc.

[0038] Whether the driver has their eyes closed, their posture is poor, or the steering wheel is not being operated can be determined, for example, from the driver's appearance based on the image from the driver monitoring camera 31. Furthermore, whether the steering wheel is not being operated can be determined, for example, based on data acquired by the steering sensor 23, or based on that data and the image from the driver monitoring camera 31.

[0039] Figure 2 is a flowchart illustrating the details of the driver abnormality state determination process performed by the driver support unit 73 and, consequently, the control device 6.

[0040] In step S1, the control device 6 determines whether it is time to issue an instruction to the driver for an operation or action. If the control device 6 periodically issues instructions to the driver for an operation or action, it determines that it is time to issue an instruction to the driver for an operation or action if a predetermined amount of time has elapsed since the previous instruction. Also, if the control device 6 issues instructions to the driver for an operation or action while the driver is in an abnormal state, it determines that it is time to issue an instruction to the driver for an operation or action if the driver is in an abnormal state.

[0041] If it is time for the control device 6 to issue an instruction to the driver for an operation or action, the control device 6 proceeds to the process in step S2. On the other hand, if it is not time for the control device 6 to issue an instruction to the driver for an operation or action, the control device 6 terminates the current process.

[0042] In step S2, the control device 6 issues a predetermined operation instruction or action instruction to the driver via the output device 41 (for example, one or both of the displays and / or speakers).

[0043] In step S3, the control device 6 determines whether it has detected a response operation to an operation instruction if it has issued one, or whether it has detected a response operation to an operation instruction if it has issued one. If the control device 6 has issued an operation instruction to the driver via the output device 41, such as an instruction to operate a specified button switch on the touch panel or an instruction to operate a specified physical operation, it can detect the driver's response operation to that operation instruction via the input device 42. Also, if the control device 6 has instructed the driver via the output device 41 to take a specified gesture or pose as an operation instruction, it can detect whether the specified gesture or pose is being taken based on the image from the driver monitor camera 31.

[0044] If the control device 6 has issued an operation instruction, and has detected a response operation to that instruction, it can determine that the driver is in a normal state and terminate the current process. If it has not detected a response operation to that instruction, it proceeds to the process in step S4.

[0045] In step S4, the control device 6 determines that the driver is in an abnormal state.

[0046] The control device 6 (driving support control device) of the vehicle 100 according to this embodiment, as described above, is configured to provide driving support to address driver abnormalities in response to a determination that the driver of the vehicle 100 is in an abnormal state that makes it difficult to continue driving the vehicle 100, and to determine whether the driver is in an abnormal state based on the driver's response to a predetermined operation or action instruction given to the driver. A predetermined operation instruction is, for example, an instruction to operate a designated button switch on a touch panel, or an instruction to operate designated physical controls including physical buttons and dials, and a predetermined action instruction is, for example, an instruction to have the driver perform a designated gesture or pause.

[0047] As described above, according to this embodiment, a predetermined operation or action instruction is given to the driver, and the driver's response to the instruction is used to determine whether the driver is in an abnormal state. Therefore, it is possible to determine with high accuracy whether the driver is in an abnormal state.

[0048] Although embodiments of the present invention have been described above, these embodiments only represent a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

[0049] For example, in the above embodiment, the abnormal state determination unit 71 may be configured to ask the driver multiple questions with different content and determine whether the driver is in an abnormal state based on the driver's answers to the questions. In this case, the abnormal state determination unit 71 may be configured to determine that the driver is in an abnormal state when the driver fails to answer the questions multiple times in a row. Furthermore, the questions and their answers may be registered in advance by the driver.

[0050] Furthermore, in the above embodiment, for example, the computer program executed in the control device 6 may be provided in the form of a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium, or it may be provided as a computer program product. [Explanation of Symbols]

[0051] 6. Control Device (Driving Assistance Control Device) 100 vehicles

Claims

1. A vehicle driver assistance control device, In response to a determination that the driver of the vehicle is in an abnormal state that makes it difficult to continue driving the vehicle, the system provides driving assistance to address the driver's abnormality. The system is configured to determine whether the driver is in an abnormal state based on the driver's response to a predetermined operation or action instruction given to the driver. Driver assistance control system.

2. The aforementioned instruction for a predetermined operation is an instruction to operate a designated button switch on a touch panel, or an instruction to operate designated physical controls, including physical buttons and dials. The driver assistance control device according to claim 1.

3. The aforementioned instruction for a predetermined action is an instruction to cause the driver to perform a specified gesture or pause. The driver assistance control device according to claim 1.

4. A method of assisting the driving of a vehicle, In response to a determination that the driver of the vehicle is in an abnormal state that makes it difficult to continue driving the vehicle, the system provides driving assistance to address the driver's abnormality. The system issues a predetermined operation or action instruction to the driver and determines whether the driver is in an abnormal state based on the driver's response to the instruction. Driving assistance methods.

5. In response to a determination that the vehicle's driver is in an abnormal state that makes it difficult to continue driving the vehicle, the system provides driving assistance to address the driver's abnormality. The system issues a predetermined operation or action instruction to the driver and determines whether the driver is in an abnormal state based on the driver's response to the instruction. A computer program that instructs a computer to perform a process.

Images