Vehicle control device, vehicle control method, and vehicle control computer program

Abstract

To provide a vehicle control device capable of securing safety of a vehicle when abnormality of a driver is detected.SOLUTION: The vehicle control device includes a determination unit configured to determine a subject of an operation based on an in-vehicle sensor signal from an in-vehicle sensor configured to detect a situation inside a vehicle cabin of the vehicle when an operation for cancelling execution of an emergency stop mode is performed during execution of an emergency stop mode in which an abnormality of a driver of the vehicle is detected, and a control unit configured to continue control of the vehicle according to the emergency stop mode when the subject of the operation is an occupant of the vehicle other than the driver, and to cancel execution of the emergency stop mode when the subject of the operation is the driver.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a computer program for vehicle control that control the running of a vehicle when an abnormality occurs in the driver. [Background technology]

[0002] Research has been conducted into a technology that allows passengers in the front or rear seats to operate an emergency evacuation device that provides emergency evacuation assistance by stopping the vehicle in a safe position without the driver's operation (see Patent Document 1). This emergency evacuation device allows the vehicle to be moved by operating an operating means provided in the vehicle or a display means that allows screen input operation after emergency evacuation assistance has started or after the vehicle has stopped due to emergency evacuation assistance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-19301 Summary of the Invention [Problem to be solved by the invention]

[0004] When a vehicle is being controlled to make an emergency stop due to the detection of a driver abnormality, if the emergency stop control is interrupted by an occupant other than the driver operating the vehicle, there is a risk that the safety of the vehicle cannot be ensured.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a vehicle control device that can ensure the safety of the vehicle when an abnormality in the driver is detected. [Means for solving the problem]

[0006] According to one embodiment, there is provided a vehicle control device that includes: a determination unit that, when an operation to cancel the emergency stop mode is performed while an emergency stop mode for bringing the vehicle to an emergency stop due to detection of an abnormality in the driver of the vehicle is in progress, determines who is performing the operation based on an interior sensor signal from an interior sensor that detects a situation inside the vehicle cabin; and a control unit that continues control of the vehicle in accordance with the emergency stop mode when the operation is performed by a vehicle occupant other than the driver, and cancels the emergency stop mode when the operation is performed by the driver.

[0007] In one embodiment, when the subject of the operation is a vehicle occupant other than the driver, the control unit cancels a control instruction to the vehicle corresponding to an operation to cancel the execution of the emergency stop mode.

[0008] In one embodiment, the vehicle control device further includes a driver state determination unit that determines whether the driver's state has returned to a normal state while the emergency stop mode is being executed, and if the driver's state has returned to a normal state when an operation to cancel the execution of the emergency stop mode is performed, the control unit cancels the execution of the emergency stop mode even if the subject of the operation is a vehicle occupant other than the driver.

[0009] According to another embodiment, there is provided a vehicle control method, which includes, when an operation to cancel the emergency stop mode is performed while an emergency stop mode for bringing the vehicle to an emergency stop due to detection of an abnormality in the driver of the vehicle is in operation, determining who performed the operation based on an interior sensor signal from an interior sensor for detecting a situation inside the vehicle, and continuing control of the vehicle in accordance with the emergency stop mode if the operation is performed by a vehicle occupant other than the driver, and canceling the emergency stop mode if the operation is performed by the driver.

[0010] According to yet another embodiment, there is provided a vehicle control computer program, the vehicle control computer program including instructions for causing a processor mounted on the vehicle to execute the following: when an operation to cancel the emergency stop mode is performed while an emergency stop mode for bringing the vehicle to an emergency stop due to detection of an abnormality in the driver of the vehicle is in progress, the program determines who performed the operation based on an interior sensor signal from an interior sensor for detecting a situation inside the vehicle, and if the operation is performed by a vehicle occupant other than the driver, continues control of the vehicle in accordance with the emergency stop mode; [Effects of the Invention]

[0011] The vehicle control device according to the present disclosure has the effect of ensuring the safety of the vehicle when an abnormality in the driver is detected. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic configuration diagram of a vehicle control system in which a vehicle control device is implemented. [Figure 2] 1 is a hardware configuration diagram of an electronic control device that is one embodiment of a vehicle control device. [Figure 3] FIG. 2 is a functional block diagram of a processor of an electronic control unit related to vehicle control processing. [Figure 4] 10A and 10B are diagrams illustrating the relationship between the subject of operation and whether the emergency stop mode can be continued when a release operation is performed while the emergency stop mode is being executed. [Figure 5] 4 is an operational flowchart of a vehicle control process. DETAILED DESCRIPTION OF THE INVENTION

[0013] A vehicle control device, a vehicle control method executed on the vehicle control device, and a vehicle control computer program will be described below with reference to the drawings. The vehicle control device has the function of a so-called driver abnormality response system (Emergency Driving Stop System, EDSS). When an abnormality that makes it difficult for the driver to continue driving the vehicle is detected, the vehicle control device controls the vehicle according to an emergency stop mode that automatically stops the vehicle. When an operation to cancel the emergency stop mode is performed while the emergency stop mode is being executed, the vehicle control device determines who performed the operation. If the operation is performed by a vehicle occupant other than the driver, the vehicle control device continues to control the vehicle according to the emergency stop mode. In this way, the vehicle control device prevents the emergency stop mode from being canceled while the driver is in an abnormal state.

[0014] FIG. 1 is a schematic diagram of a vehicle control system in which a vehicle control device is implemented. In this embodiment, the vehicle control system 1 is mounted on a vehicle 10 and controls the vehicle 10. The vehicle control system 1 includes an exterior camera 2, a driver monitor camera 3, a notification device 4, and an electronic control unit (ECU) 5, which is an example of a vehicle control device. The exterior camera 2, the driver monitor camera 3, the notification device 4, and the ECU 5 are communicatively connected to each other. The vehicle 10 may be provided with a distance measurement sensor (not shown), such as a LiDAR or radar, that measures the distance from the vehicle 10 to objects present around the vehicle 10. The vehicle 10 may also be provided with a positioning device (not shown), such as a GPS receiver, that measures the position of the vehicle 10 using a satellite positioning system. The vehicle 10 may also be provided with a wireless communication terminal (not shown) that wirelessly communicates with other devices.

[0015] The exterior camera 2 is an example of an exterior imaging unit, and is attached to the vehicle 10 so as to face a predetermined area around the vehicle 10, such as the area ahead of the vehicle 10. Note that the vehicle 10 may be provided with a plurality of exterior cameras with different imaging directions or focal lengths. The exterior camera 2 captures an image of a predetermined area at each predetermined imaging cycle to generate an image of the predetermined area (hereinafter referred to as an exterior image), and outputs the generated exterior image to the ECU 5.

[0016] The driver monitor camera 3 is an example of an in-vehicle sensor for detecting the situation inside the vehicle cabin. The driver monitor camera 3 is mounted near the top edge of the windshield or near the instrument panel, facing the driver, so that the head of the driver seated in the driver's seat of the vehicle 10 and the hands operating a device that accepts an operation to interrupt the emergency stop mode are included in the imaging target area. The driver monitor camera 3 may have a light source such as an infrared LED. The driver monitor camera 3 then captures an image of the imaging target area at predetermined imaging intervals to generate an image (hereinafter referred to as a driver image) showing the driver and other elements, and outputs the generated driver image to the ECU 5.

[0017] The notification device 4 is provided in the passenger compartment of the vehicle 10 and is a device that provides predetermined notifications to the occupants of the vehicle 10 by light, sound, vibration, text display, or image display. To achieve this, the notification device 4 has at least one of, for example, a speaker, a light source, a vibrator, or a display device. When the notification device 4 receives a notification signal representing a predetermined notification to the occupants of the vehicle 10 from the ECU 5, it notifies the occupants by sound from the speaker, light emission or flashing of the light source, vibration of the vibrator, or display of a notification message or icon on the display device.

[0018] The ECU 5 operates as an EDSS, that is, when the ECU 5 detects an abnormality that prevents the driver from continuing to drive the vehicle 10, the ECU 5 controls the vehicle 10 to bring the vehicle 10 to an emergency stop.

[0019] Fig. 2 is a hardware configuration diagram of the ECU 5. As shown in Fig. 2, the ECU 5 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may be configured as separate circuits, or may be configured integrally as a single integrated circuit.

[0020] The communication interface 21 has an interface circuit for connecting the ECU 5 to other devices inside the vehicle. The communication interface 21 passes the outside image received from the outside camera 2 and the driver image received from the driver monitor camera 3 to the processor 23. The communication interface 21 also outputs a notification signal received from the processor 23 to the notification device 4. Furthermore, the communication interface 21 outputs a control signal for controlling the vehicle 10.

[0021] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores various data used in the vehicle control process executed by the processor 23 of the ECU 5. For example, the memory 22 stores parameters of the exterior camera 2, such as the installation position, shooting direction, and focal length. The memory 22 also stores various parameters used to detect driver abnormalities from the driver image. The memory 22 also temporarily stores the exterior image received from the exterior camera 2 and the driver image received from the driver monitor camera 3.

[0022] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The processor 23 executes vehicle control processing for the vehicle 10.

[0023] 3 is a functional block diagram of the processor 23 related to vehicle control processing. The processor 23 has an abnormality detection unit 31, a determination unit 32, a vehicle control unit 33, and a driver state determination unit 34. Each of these units in the processor 23 is, for example, a functional module realized by a computer program running on the processor 23. Alternatively, each of these units in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23.

[0024] The abnormality detection unit 31 detects an abnormality when an abnormality occurs that prevents the driver from continuing to drive the vehicle 10. Note that, for the sake of convenience, hereinafter, when an abnormality occurs that prevents the driver from continuing to drive the vehicle 10, it may be simply referred to as an abnormality occurring in the driver.

[0025] For example, the abnormality detection unit 31 determines the driver's drowsiness level at predetermined intervals based on a series of driver images obtained over a recent fixed period. If the driver's drowsiness level is at a level that prevents the driver from monitoring the area ahead of the vehicle 10, the abnormality detection unit 31 determines that an abnormality has occurred in the driver. To this end, the abnormality detection unit 31 detects the driver's line of sight, the degree of eye opening (hereinafter referred to as "eye opening degree"), and the degree of mouth opening from each of the series of driver images obtained over the recent fixed period. The abnormality detection unit 31 then determines the driver's drowsiness level based on the detected line of sight, eye opening degree, and drowsiness level.

[0026] The anomaly detection unit 31 detects the area in the driver image where the driver's face appears (hereinafter referred to as the face area) by inputting the driver image into a classifier that has been trained in advance to detect the driver's face from the image. Such a classifier is configured as a deep neural network (DNN) with a convolutional neural network (CNN)-type architecture, such as the Single Shot Multibox Detector (SSD) or Faster R-CNN. Alternatively, such a classifier may be configured as a DNN with an attention mechanism, such as a Vision Transformer. Alternatively, such a classifier may be configured as a classifier based on other machine learning techniques, such as the AdaBoost classifier. Such a classifier is trained in advance using a large number of training images depicting human faces according to a predetermined learning method, such as backpropagation. The anomaly detection unit 31 detects the driver's eyes and mouth from the face area in the driver image. In this case, the anomaly detection unit 31 detects edge pixels in the face area by applying an edge detection filter, such as a Sobel filter. The abnormality detection unit 31 then detects lines in which edge pixels are continuous in a substantially horizontal direction, and for each of the left and right eyes, detects two such lines aligned vertically within the range in the face region where the eye is expected to be located as the upper and lower eyelids of that eye. Similarly, the abnormality detection unit 31 detects the area surrounded by two such lines aligned vertically within the range in the face region where the mouth is expected to be located as the driver's mouth. Note that the abnormality detection unit 31 may also detect the upper and lower eyelids of each of the driver's left and right eyes from the driver image using other methods for detecting the upper and lower eyelids from an image. Similarly, the abnormality detection unit 31 may also detect the driver's mouth from the driver image using other methods for detecting the mouth from an image.

[0027] The abnormality detection unit 31 estimates the degree of eye opening of the driver based on the distance between the upper and lower eyelids of each of the left and right eyes in each driver image. For example, the abnormality detection unit 31 may determine the eye opening degree by averaging the distance between the upper and lower eyelids of each of the left and right eyes. The abnormality detection unit 31 may also estimate the eye opening degree using other methods that calculate the eye opening degree from the upper and lower eyelids in the image. The abnormality detection unit 31 then calculates the time from a maximum value to the next maximum value as the duration of one blink of the driver based on the time-series change in the eye opening degree in each of the series of driver images. The abnormality detection unit 31 then counts the number of blinks during a recent fixed period and calculates the average time between blinks as the blink cycle. Furthermore, the abnormality detection unit 31 may, for example, calculate the ratio of the vertical length to the horizontal length of the mouth in each driver image during a recent fixed period and calculate the average value as the degree of mouth opening of the driver. The abnormality detection unit 31 may calculate the degree to which the driver's mouth is open according to another method for calculating the degree to which the mouth is open from the region on the image where the mouth is represented.

[0028] Furthermore, the abnormality detection unit 31 detects the driver's gaze direction from each driver image. For example, the abnormality detection unit 31 detects the corneal reflection image of the light source and the center of gravity of the pupil (hereinafter simply referred to as the pupil center of gravity) from an area surrounded by the upper eyelid and the lower eyelid (hereinafter referred to as the eye area) for at least one of the driver's left and right eyes displayed on the driver image. The corneal reflection image of the light source is also called a Purkinje image. In this case, the abnormality detection unit 31 detects the Purkinje image by, for example, template matching between a Purkinje image template and the eye area. Similarly, the abnormality detection unit 31 may detect the pupil by template matching between a pupil template and the eye area, and determine the center of gravity of the area in which the detected pupil is displayed as the pupil center of gravity. The abnormality detection unit 31 may also detect the Purkinje image and the pupil center of gravity using another method for detecting the Purkinje image and the pupil center of gravity from the eye area. The abnormality detection unit 31 then calculates the direction and distance from the Purkinje image to the pupil center, and detects the driver's gaze direction by referring to a table that shows the relationship between the direction and distance and the driver's gaze direction. Such a table may be stored in advance in the memory 22. The abnormality detection unit 31 then calculates the amount of movement in the gaze direction for each pair of two consecutive driver images captured within a recent fixed period, and calculates the average amount of movement by dividing it by the interval between acquisitions of the driver images to determine the movement speed in the gaze direction.

[0029] The abnormality detection unit 31 determines the drowsiness level of the driver based on at least one of the period and frequency of blinking, the degree of mouth opening, and the speed of movement in the direction of gaze, and determines that an abnormality has occurred in the driver if the drowsiness level is at a level that makes it impossible to monitor the area ahead of the vehicle 10. For example, the abnormality detection unit 31 determines that an abnormality has occurred in the driver if the number of blinks in the most recent fixed period is equal to or greater than a predetermined number, the blinking period is longer than a predetermined time threshold, and the degree of mouth opening is greater than a predetermined opening degree.

[0030] The abnormality detection unit 31 may determine whether an abnormality has occurred in the driver based on other indicators representing the driver's state. For example, the abnormality detection unit 31 may determine that an abnormality has occurred in the driver if the driver's face or eyes cannot be detected from the driver image continuously for a predetermined period of time. Alternatively, the abnormality detection unit 31 may determine that an abnormality has occurred in the driver if the driver's eyes are closed continuously for a predetermined period of time. In this case, the abnormality detection unit 31 may determine that the driver's eyes are closed if the degree of eye opening is equal to or less than an eye closure determination threshold corresponding to closed eyes. Furthermore, if a microphone (not shown) is provided in the vehicle cabin, the abnormality detection unit 31 may detect a specific abnormal sound (e.g., snoring) emitted by the driver from an audio signal representing sound within the vehicle cabin generated by the microphone. The abnormality detection unit 31 may then determine that an abnormality has occurred in the driver if the specific abnormal sound emitted by the driver is detected. The abnormality detection unit 31 may detect the abnormal sound emitted by the driver using any method for detecting a specific abnormal sound from an audio signal. A microphone is another example of an in-vehicle sensor, and an audio signal generated by a microphone is another example of an in-vehicle sensor signal.

[0031] When the abnormality detection unit 31 determines that an abnormality has occurred in the driver, it instructs the determination unit 32 and the vehicle control unit 33 to activate the EDSS function, i.e., to apply the emergency stop mode. On the other hand, when the abnormality detection unit 31 determines that an abnormality has not occurred in the driver, it is not necessary to activate the EDSS function.

[0032] When an operation to cancel the execution of the emergency stop mode (hereinafter, sometimes simply referred to as a cancel operation) is performed while the emergency stop mode is being executed, the determination unit 32 determines the entity that performed the operation. In this embodiment, the period during which the emergency stop mode is being executed includes a phase in which a notice is given to start execution of control to bring the vehicle 10 to an emergency stop phase in which rescue is requested from outside the vehicle after the vehicle 10 has stopped.

[0033] Furthermore, the release operations that are the subject of determination of the operator include operations that can be performed by an occupant of the vehicle 10 other than the driver, such as at least one of a steering operation, a hazard light-off operation, a shift lever operation, an EPB release operation, a parking brake operation, an IG-OFF operation, and a door opening operation. If a switch for releasing the emergency stop mode is provided in a position (for example, an instrument panel) that can be operated by an occupant other than the driver, the operation of that switch is also included in the subjects of determination of the operator. On the other hand, operations that are difficult for an occupant other than the driver to perform, such as the operation of the brake pedal or accelerator pedal, do not need to be the subject of determination of the operator.

[0034] When the device that is the target of the cancellation operation notifies the determination unit 32 that a cancellation operation has been performed, the determination unit 32 inputs a plurality of driver images obtained during a predetermined period (e.g., one to several seconds) including the timing when the cancellation operation was performed, in order of generation of each driver image, to a classifier that has been trained in advance to determine the operating entity. The classifier then outputs a determination result of the operating entity. Such a classifier is configured as a DNN with a recursive structure, such as a recurrent neural network (RNN) or LSTM.

[0035] Alternatively, the classifier may be pre-trained to detect the occupant's body part, such as the occupant's hand or finger, used for the release operation from the driver image. Such a classifier may be configured as a CNN or a DNN with an attention mechanism. Alternatively, such a classifier may be configured as a classifier based on a machine learning algorithm other than a DNN, such as an adaBoost classifier. In this case, the determination unit 32 identifies a driver image generated at or around the time of the release operation as the driver image at the time of the release operation. The determination unit 32 then inputs the driver image at the time of the release operation into the classifier to detect each occupant depicted in the driver image. The determination unit 32 then determines that the occupant closest to the device on which the release operation was performed is the occupant who performed the release operation. Note that device position information indicating the position of each device on the driver image that is the target of the release operation may be pre-stored in the memory 22. The determination unit 32 may then identify the position of the device on the driver image where the release operation was performed by referring to the device that output the notification that the release operation was performed and the device position information. For example, if the device on which the deactivation operation was performed is a hazard lamp, the occupant closest to the position of the hazard lamp on the driver image at the time the deactivation operation was performed is determined to be the occupant who performed the deactivation operation.

[0036] Alternatively, the determination unit 32 may determine the operator of the release operation based on an in-vehicle sensor signal from an in-vehicle sensor other than the driver monitor camera 3. For example, if each seat in the vehicle is provided with a weight sensor or a touch sensor, the sensor signal from the sensor may be used to determine the operator. In this case, the weight sensor or touch sensor of each seat is another example of an in-vehicle sensor. The determination unit 32 may determine that the operator of the release operation is an occupant other than the driver when a sensor signal from a weight sensor provided in the passenger seat or a rear seat indicates that a weight equal to or greater than a predetermined detection threshold immediately before the release operation is performed and that a weight less than the detection threshold is present at the time the release operation is performed. Similarly, the determination unit 32 may determine that the operator of the release operation is an occupant other than the driver when a sensor signal from a touch sensor provided in the passenger seat or a rear seat indicates that something is touching the seat immediately before the release operation is performed and that nothing is touching the seat at the time the release operation is performed. This is because the device for performing the release operation is provided around the driver, and it is expected that when an occupant other than the driver performs the release operation, the occupant will leave the seat. On the other hand, if there is no change in the weight indicated by the sensor signal from any of the weight sensors provided in any of the seats before and after the release operation, or if there is no change in the presence or absence of contact indicated by the sensor signal from any of the touch sensors provided in any of the seats, the determination unit 32 determines that the person performing the release operation is the driver.

[0037] The determination unit 32 outputs the determination result regarding the operator of the executed release operation to the vehicle control unit 33.

[0038] When notified that a driver abnormality has been detected, the vehicle control unit 33 controls the vehicle 10 in accordance with the emergency stop mode. That is, when notified that a driver abnormality has been detected, the vehicle control unit 33 controls the vehicle 10 to decelerate to a predetermined slow speed, and then stops the vehicle 10 at a target stopping position.

[0039] First, in the control notice phase, the vehicle control unit 33 notifies those around the vehicle 10 that the vehicle 10 will make an emergency stop. To do so, the vehicle control unit 33 turns on the hazard lights. The vehicle control unit 33 also notifies the occupants of the vehicle 10 via the notification device 4 that the emergency stop mode will be executed.

[0040] After the control notice phase lasting a predetermined period (e.g., several seconds), the vehicle control unit 33 performs a driving intervention phase in which the vehicle 10 is decelerated and the vehicle 10 is stopped at a target stopping position. For example, the vehicle control unit 33 sets the target stopping position on the shoulder of the road section on which the vehicle 10 is traveling. Alternatively, the vehicle control unit 33 may refer to map information and the current position of the vehicle 10 determined by a positioning device (not shown) to determine whether there is a turnout space on the road section on which the vehicle 10 is traveling within a predetermined distance from the current position of the vehicle 10. If such a turnout space is found, the vehicle control unit 33 may set the turnout space as the target stopping position.

[0041] The vehicle control unit 33 generates a planned driving route (trajectory) of the vehicle 10 from the current position of the vehicle 10 to the target stopping position. The planned driving route is expressed, for example, as a set of target positions of the vehicle 10 at each time when the vehicle 10 travels along the planned driving route. The vehicle control unit 33 then controls each unit of the vehicle 10 so that the vehicle 10 travels along the planned driving route.

[0042] The vehicle control unit 33 decelerates the vehicle 10 to a slow speed and controls the vehicle 10 to move toward the target stopping position at that slow speed. The vehicle control unit 33 also starts decelerating the vehicle 10 and sounds the horn. Furthermore, the vehicle control unit 33 detects objects and lane markings that may be obstacles to the vehicle 10's travel, such as other vehicles, around the vehicle 10, from each of a series of time-series exterior images acquired by the exterior camera 2. The vehicle control unit 33 detects such objects and lane markings by inputting the exterior images to a classifier that has been trained in advance to detect such objects and lane markings. The classifier is configured as, for example, a CNN or a DNN with an attention mechanism. Alternatively, if the vehicle 10 has a distance measurement sensor such as a LiDAR sensor, the vehicle control unit 33 may detect objects around the vehicle 10 based on distance measurement signals obtained by the distance measurement sensor.

[0043] The vehicle control unit 33 generates a planned driving route so that the vehicle 10 does not collide with objects present around the vehicle 10. To this end, the vehicle control unit 33 tracks objects present around the vehicle 10 detected from a time-series series of exterior images, etc., and estimates a predicted trajectory of each object up to a predetermined time ahead from the trajectory obtained from the tracking results. In this case, the vehicle control unit 33 tracks the objects by applying a predetermined tracking process such as Byte Track to the series of exterior images.

[0044] The vehicle control unit 33 converts the coordinates of each tracked object in the image into coordinates on the bird's-eye view image (bird's-eye view coordinates) by performing a viewpoint conversion process using parameters such as the mounting position of the exterior camera 2 on the vehicle 10. The vehicle control unit 33 estimates the position of the detected object at the time each image is captured based on the position of the vehicle 10 measured by a positioning device and the traveling direction of the vehicle 10 measured by a direction sensor (not shown), the estimated distance to the detected object, and the direction from the vehicle 10 to the object. It is assumed that the position of the bottom edge of the object area depicting the detected object represents the position where the object is in contact with the road surface. Therefore, the vehicle control unit 33 can calculate the estimated distance to the detected object based on the position of the bottom edge of the object area on the exterior image and parameters of the exterior camera 2, such as the shooting direction and installation height. Alternatively, the vehicle control unit 33 may use the distance measured by a ranging sensor in the direction corresponding to the object area depicting the detected object as the estimated distance to the detected object. The vehicle control unit 33 can estimate the trajectory of each object being tracked by arranging the estimated positions in chronological order for the object. The vehicle control unit 33 can estimate the predicted trajectory of the object up to a predetermined time ahead by performing prediction processing using a Kalman filter, a particle filter, or the like based on the traveling trajectory of the object being tracked during the most recent predetermined period.

[0045] The vehicle control unit 33 generates a planned driving trajectory based on the predicted trajectory of each object being tracked so that the predicted value of the distance between each object being tracked and the vehicle 10 up to a predetermined time ahead will be equal to or greater than a predetermined distance, and if necessary, lane changes will be completed to the target stopping position.

[0046] Once the planned travel route is set, the vehicle control unit 33 controls each unit of the vehicle 10 so that the vehicle 10 travels along the planned travel route. For example, the vehicle control unit 33 calculates the deceleration of the vehicle 10 according to the planned travel route and the current vehicle speed of the vehicle 10 measured by a vehicle speed sensor (not shown), and controls the power train and brake devices to achieve that deceleration.

[0047] Furthermore, when the vehicle 10 changes its course so that the vehicle 10 travels along a planned travel route, the vehicle control unit 33 calculates a steering angle of the vehicle 10 according to the planned travel route. Then, the vehicle control unit 33 outputs a control signal corresponding to the steering angle to an actuator (not shown) that controls the steering wheels of the vehicle 10. When the vehicle 10 changes lanes, the vehicle control unit 33 turns on a turn signal in the direction indicating the destination lane.

[0048] When the vehicle 10 stops, the vehicle control unit 33 unlocks the doors and continues to sound the horn as a rescue support phase. The vehicle control unit 33 may also notify the driver that an abnormality has occurred via a wireless communication terminal (not shown) mounted on the vehicle 10.

[0049] Furthermore, when a cancellation operation is performed while the emergency stop mode is being executed, the vehicle control unit 33 refers to the determination result of the determination unit 32 as to the operator of the cancellation operation. If the operator is the driver, the vehicle control unit 33 cancels the execution of the emergency stop mode. When canceling the execution of the emergency stop mode, the vehicle control unit 33 notifies the occupants of the vehicle 10 via the notification device 4 that the execution of the emergency stop mode has been canceled. Thereafter, the vehicle control unit 33 controls the traveling of the vehicle 10 in accordance with the driving operation by the driver.

[0050] On the other hand, if the person performing the cancellation operation is an occupant other than the driver, the vehicle control unit 33 continues to control the vehicle 10 in accordance with the emergency stop mode. This prevents the emergency stop mode from being canceled even when the driver is unable to drive the vehicle 10. The vehicle control unit 33 also generates a notification signal indicating that the cancellation operation has been canceled, i.e., indicating that control of the vehicle 10 in accordance with the emergency stop mode will continue, and outputs the notification signal to the notification device 4. This causes the vehicle control unit 33 to notify the occupant of the vehicle 10 via the notification device 4 that the cancellation operation has been canceled.

[0051] If the subject of the release operation is an occupant other than the driver, the vehicle control unit 33 may cancel a control instruction to the vehicle 10 corresponding to the release operation. For example, if the release operation is an operation to release the EPB, the vehicle control unit 33 continues to control the vehicle 10 in accordance with the emergency stop mode and does not release the EPB. Also, if the release operation is an operation to turn off the hazard lights, the vehicle control unit 33 continues to control the vehicle 10 in accordance with the emergency stop mode and does not turn off the hazard lights. In this way, by canceling the control instruction to the vehicle 10 corresponding to the release operation, the vehicle control unit 33 can prevent the vehicle behavior from becoming unstable due to an operation by an occupant other than the driver.

[0052] When the cancel operation is performed, if the driver state determination unit 34 determines that the driver's state has returned to a normal state in which the vehicle 10 can be driven, the vehicle control unit 33 may cancel the execution of the emergency stop mode even if the cancel operation is performed by an occupant other than the driver. In this case, even if the cancel operation is performed by an occupant other than the driver, the driver is in a normal state, so the driver can take over driving control of the vehicle 10 from the vehicle control unit 33. Therefore, the vehicle control unit 33 does not need to make an unnecessary emergency stop of the vehicle 10.

[0053] 4(a) and 4(b) are explanatory diagrams each showing the relationship between the subject of operation and whether the emergency stop mode can be continued when a release operation is performed during execution of the emergency stop mode.

[0054] In the example shown in FIG. 4(a), driver 400 performs a cancel operation by touching switch 410 for canceling the execution of the emergency stop mode. Therefore, the execution of the emergency stop mode is canceled, and control of vehicle 10 is transferred from ECU 5 to driver 400. On the other hand, in the example shown in FIG. 4(b), passenger 401 sitting in the front passenger seat performs a cancel operation by touching switch 410 for canceling the execution of the emergency stop mode. Therefore, control of vehicle 10 according to the emergency stop mode continues. Therefore, it is possible to prevent control of vehicle 10 from being transferred to the driver while the driver is having difficulty driving vehicle 10 normally.

[0055] The driver state determination unit 34 determines whether the driver's state has returned to a normal state in which the driver is able to drive the vehicle 10 while control of the vehicle 10 is being executed in accordance with the emergency stop mode. To this end, the driver state determination unit 34 executes the same processing as the abnormality detection unit 31 while control of the vehicle 10 is being executed in accordance with the emergency stop mode. If none of the conditions for determining that the driver's state is abnormal are satisfied, the driver state determination unit 34 determines that the driver's state has returned to a normal state.

[0056] The driver state determination unit 34 may determine whether the driver's state has returned to a normal state based on whether or not the driver responds when an occupant other than the driver takes some action toward the driver. For example, the driver state determination unit 34 may determine whether the driver responds to a voice uttered by an occupant other than the driver by analyzing audio collected by one or more microphones (not shown) provided in the vehicle 10. If the driver responds, the driver state determination unit 34 determines that the driver has returned to a normal state. If the driver does not respond, the driver state determination unit 34 determines that the driver has not returned to a normal state. In this example, the driver state determination unit 34 distinguishes between a voice uttered by an occupant other than the driver and a voice uttered by the driver by applying a predetermined speaker identification method to an audio signal representing the collected audio. If the driver utters a voice within a predetermined period in response to a voice uttered by an occupant other than the driver, the driver state determination unit 34 determines that the driver has responded, i.e., that the driver has returned to a normal state. On the other hand, if the driver does not make a sound even after a predetermined period of time has elapsed, the driver state determination unit 34 determines that the driver is not responding, i.e., that the driver has not returned to a normal state. Alternatively, the driver state determination unit 34 may determine whether the driver is responding by inputting multiple driver images in the order in which they were generated into a classifier that has been trained in advance to identify whether the driver is responding to actions by passengers other than the driver. If the classifier outputs a classification result indicating that the driver is responding, the driver state determination unit 34 determines that the driver has returned to a normal state. On the other hand, if the classifier does not output a classification result indicating that the driver is responding, the driver state determination unit 34 determines that the driver has not returned to a normal state. Such a classifier is configured as a DNN with a recursive structure, such as an RNN or LSTM.

[0057] When the driver state determination unit 34 determines that the driver's state has returned to a normal state, it notifies the vehicle control unit 33 of the determination result.

[0058] FIG. 5 is an operational flowchart of the vehicle control process executed by the processor 23.

[0059] The abnormality detection unit 31 determines whether or not an abnormality has been detected in the driver (step S101). If an abnormality in the driver has not been detected (step S101-No), the processor 23 repeats the process of step S101.

[0060] On the other hand, if a driver abnormality is detected (step S101-Yes), vehicle control unit 33 starts controlling vehicle 10 in accordance with the emergency stop mode (step S102). Thereafter, determination unit 32 determines whether a cancellation operation has been performed (step S103). If a cancellation operation has not been performed (step S103-No), processor 23 repeats the process of step S103.

[0061] On the other hand, if a cancellation operation has been performed (step S103-Yes), the determination unit 32 determines whether the person performing the cancellation operation is the driver (step S104). If the person performing the operation is the driver (step S104-Yes), the vehicle control unit 33 cancels the execution of the emergency stop mode (step S105). Thereafter, the processor 23 repeats the processes from step S101 onwards.

[0062] On the other hand, if the subject of operation is an occupant other than the driver (step S104-No), the vehicle control unit 33 determines whether or not it has received a determination result from the driver state determination unit 34 that the driver's state has returned to a normal state (step S106). If the driver's state has returned to a normal state (step S106-Yes), the processor 23 repeats the processing from step S105 onwards. On the other hand, if the driver's state has not returned to a normal state (step S106-No), the vehicle control unit 33 continues to control the vehicle 10 in accordance with the emergency stop mode (step S107). Thereafter, the processor 23 repeats the processing from step S103 onwards until the processing in accordance with the emergency stop mode is completed.

[0063] As described above, when an operation to cancel the emergency stop mode is performed while the emergency stop mode is being executed, the vehicle control device determines who performed the operation, and if the operation is performed by a vehicle occupant other than the driver, continues to control the vehicle in accordance with the emergency stop mode. This prevents the emergency stop mode from being canceled while the driver is in an abnormal state. As a result, the vehicle control device can ensure the safety of the vehicle when a driver abnormality is detected.

[0064] According to a modified example, if the canceling operation is performed by a passenger other than the driver, the vehicle control unit 33 may continue to control the vehicle 10 in accordance with the emergency stop mode regardless of the state of the driver. In this case, the processing by the driver state determination unit 34 may be omitted.

[0065] A computer program for realizing the vehicle control process according to the above embodiment or modification may be provided in a form recorded on a computer-readable portable recording medium.

[0066] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0067] 1 Vehicle control system, 10 Vehicle, 2 Exterior camera, 3 Driver monitor camera, 4 Notification device, 5 Electronic control unit (ECU, vehicle control unit), 21 Communication interface, 22 Memory, 23 Processor, 31 Abnormality detection unit, 32 Determination unit, 33 Vehicle control unit, 34 Driver state determination unit

Claims

1. a determination unit that, when an operation to cancel the emergency stop mode is performed while an emergency stop mode is being executed to make an emergency stop of the vehicle due to detection of an abnormality in the driver of the vehicle, determines the entity of the operation based on an interior sensor signal from an interior sensor that detects a situation inside the vehicle; a control unit that continues control of the vehicle in accordance with the emergency stop mode when the subject is an occupant of the vehicle other than the driver, and cancels execution of the emergency stop mode when the subject is the driver; A vehicle control device having the above.

2. The vehicle control device according to claim 1 , wherein the control unit cancels a control instruction to the vehicle corresponding to the operation when the subject is an occupant of the vehicle other than the driver.

3. a driver state determination unit that determines whether the driver's state has returned to a normal state while the emergency stop mode is being executed, 3. The vehicle control device according to claim 1, wherein the control unit cancels the execution of the emergency stop mode when the driver's state has returned to a normal state when the operation is performed, even if the subject is an occupant of the vehicle other than the driver.

4. When an operation to cancel the emergency stop mode is performed during execution of an emergency stop mode in which the vehicle is brought to an emergency stop due to detection of an abnormality in the driver of the vehicle, the subject of the operation is determined based on an interior sensor signal from an interior sensor for detecting a situation inside the vehicle; If the subject is an occupant of the vehicle other than the driver, continuing control of the vehicle in accordance with the emergency stop mode; If the subject is the driver, cancel the execution of the emergency stop mode. A vehicle control method comprising:

5. When an operation to cancel the emergency stop mode is performed during execution of an emergency stop mode in which the vehicle is brought to an emergency stop due to detection of an abnormality in the driver of the vehicle, the subject of the operation is determined based on an interior sensor signal from an interior sensor for detecting a situation inside the vehicle; If the subject is an occupant of the vehicle other than the driver, continuing control of the vehicle in accordance with the emergency stop mode; If the subject is the driver, cancel the execution of the emergency stop mode. A computer program for vehicle control that causes a processor mounted on the vehicle to execute the above.

Images