Vehicle control device

Abstract

This vehicle control device comprises a driver abnormality determining unit that determines whether or not there is an abnormality in the driver, a driver abnormality control unit that determines whether or not it is necessary to automatically stop the vehicle when a predetermined time has elapsed after it has been determined that there is an abnormality in the driver, a braking control unit that executes braking control of the vehicle and stops the vehicle when it is determined that it is necessary to automatically stop the vehicle, a passenger state recognition unit that recognizes the state of a passenger, and a passenger action determination unit that determines whether or not the state of the passenger is an action based on the abnormality in the driver on the basis of the recognition result from the passenger state recognition unit, wherein, when the recognition result from the passenger state recognition unit is determined to be an action based on the abnormality in the driver, it is determined that it is necessary to automatically stop the vehicle without waiting for the predetermined time to elapse. This makes it possible to confirm the occurrence of an abnormality in the driver more quickly without relying on the operation of a device by the passenger, and to shorten the time until the automatic stop begins, thereby reducing anxiety of the passenger when an abnormality occurs in the driver.

Description

Vehicle control device

[0001] The present invention relates to a vehicle control device.

[0002] There is known a technique for stopping a vehicle in motion for the purpose of suppressing a subsequent accident when an abnormality such as a driver losing consciousness is detected.

[0003] For example, Patent Document 1 discloses a vehicle control device including an abnormality detection unit that detects an abnormality of a driver during vehicle travel, a notification control unit that executes notification control for notifying the detected abnormality, and an automatic stop control unit that starts stop control of the vehicle when a predetermined time has elapsed since the notification control unit executed the notification control and there is no cancellation operation of the notification control. The automatic stop control unit starts stop control of the vehicle without waiting for the elapse of the predetermined time when a stop request operation by a passenger is detected before the elapse of the predetermined time.

[0004] Japanese Unexamined Patent Application Publication No. 2022 - 42665

[0005] In the above prior art, when a stop request operation is performed by a passenger even before the elapse of a predetermined time from the detection of the driver's abnormality, stop control for automatically stopping the vehicle is started. Therefore, it is possible to reduce the anxiety of the passengers during the coasting period from the detection of the driver's abnormality until the automatic stop starts. However, for a passenger to execute automatic stop of the vehicle, it is necessary to operate an operator related to an electric parking brake (EPB) as a stop request operation, and it is assumed that the passenger recognizes the existence of a means for automatically stopping the vehicle and correctly understands the operation method of the stop request operation. In addition, since the occurrence of the driver's abnormality is assumed to be sudden, it is considered difficult to perform the stop request operation promptly and accurately without mistake at the time of the abnormality occurrence.

[0006] The present invention has been made in view of the above, and an object thereof is to provide a vehicle control device capable of reducing the anxiety of passengers at the time of the occurrence of a driver's abnormality by more quickly determining the occurrence of the driver's abnormality regardless of the operation of the device by the passengers and shortening the time until the start of automatic stop.

[0007] The present invention includes multiple means for solving the above problems, but to give one example, a driver state recognition unit that recognizes the state of the driver based on the detection results of an occupant detection sensor that detects the driver and passengers of a vehicle; a driver abnormality determination unit that determines whether or not there is an abnormality in the driver based on the recognition results of the driver state recognition unit; a driver abnormality control unit that determines whether or not the vehicle needs to be automatically stopped after a predetermined time has elapsed following the determination that there is an abnormality in the driver by the driver abnormality determination unit; and a warning when the driver abnormality determination unit determines that there is an abnormality in the driver, and when the driver abnormality control unit determines that the vehicle needs to be automatically stopped. The system includes a warning unit that issues a warning in such a situation, a braking control unit that executes braking control of the vehicle and stops the vehicle when the driver abnormality control unit determines that it is necessary to automatically stop the vehicle, a passenger state recognition unit that recognizes the state of the passenger based on the detection result of the passenger detection sensor, and a passenger action determination unit that determines whether the state of the passenger is due to an action caused by a driver abnormality based on the recognition result of the passenger state recognition unit, and if the driver abnormality control unit determines that the recognition result of the passenger state recognition unit is due to an action caused by a driver abnormality, it will determine that it is necessary to automatically stop the vehicle without waiting for the predetermined time to elapse.

[0008] According to the present invention, by determining the occurrence of a driver malfunction more quickly without relying on the passenger's operation of the device, and shortening the time until the automatic stop is initiated, the anxiety of passengers when a driver malfunction occurs can be reduced.

[0009] This diagram shows the vehicle control device along with its related components. This diagram shows a use case for automatic stopping due to driver abnormality detection. This diagram shows a use case for automatic stopping due to a passenger's stop command action. This diagram shows a use case for automatic stopping due to a passenger's action judgment. This flowchart shows the processing details of automatic stopping based on a passenger's stop command action. This flowchart shows the processing details of automatic stopping based on the result of a passenger's action judgment.

[0010] Hereinafter, one embodiment of the present invention will be described with reference to the drawings.

[0011] Figure 1 is a diagram showing the vehicle control device according to this embodiment, along with its related components.

[0012] In Figure 1, the control device 1 (vehicle control device) is a control unit mounted on the vehicle 100 to be controlled, and has the following functional units: occupant state control unit 2, alarm control unit 3, driving control unit 4, and driving support control unit 5.

[0013] The occupant state control unit 2 includes the following functional units: occupant detection unit 2a, driver state recognition unit 2b, driver abnormality determination unit 2c, passenger state recognition unit 2d, passenger action determination unit 2e, and driver abnormality control unit 2f.

[0014] The occupant detection unit 2a uses the functions of an Occupant Monitoring System (OMS), such as an in-vehicle imaging sensor that takes pictures of the interior of the vehicle 100, to detect whether the vehicle 100 has an occupant, including the driver and other passengers. The occupant detection unit 2a also performs facial recognition of the occupants (driver and passengers), and if the occupant is registered, it authenticates them as an occupant sitting in a seat.

[0015] The driver state recognition unit 2b detects and recognizes the characteristics and state of the driver using the detection values ​​(detection results) from the occupant state sensor 11. The occupant state sensor 11 has functions such as an image sensor that obtains images to detect and recognize the state of the driver in the driver's seat and each passenger in the passenger seat and rear seats, a seat sensor that detects passengers by detecting the load on each seat, an in-vehicle microphone that acquires voice information of the driver and passengers, a pulse sensor to detect the driver's pulse, and a body temperature sensor to detect the driver's body temperature, and these functions are comprehensively described. The driver state recognition unit 2b uses, for example, the functions of a driver monitoring system (DMS) using in-vehicle image sensors, and recognizes information about the characteristics and state of body parts such as the position of the driver's eyes, nose, and mouth, skeletal structure, degree of opening and closing of the eyes and mouth, face orientation, and gaze direction, based on the detection results from the occupant state sensor 11. Note that the driver monitoring function of an occupant monitoring system (OMS) may be configured to recognize similar information.

[0016] The driver abnormality determination unit 2c uses the driver's characteristics and state, which are the recognition results of the driver state recognition unit 2b, to determine whether or not there is a driver abnormality while the vehicle 100 is in motion. Here, a driver abnormality refers to a state in which there is a high probability that the driver cannot perform safe driving operations, such as when the driver has lost consciousness due to sudden illness, fainting, etc., or when the driver is not able to concentrate on driving due to falling asleep or being distracted.

[0017] The passenger state recognition unit 2d detects and recognizes the characteristics and state of passengers using the values ​​(detection results) detected by the occupant state sensor 11. The passenger state recognition unit 2d uses the functions of an occupant monitoring system (OMS) that uses, for example, an in-vehicle imaging sensor, and recognizes information about the characteristics and state of body parts of passengers other than the driver, such as the position of the eyes, nose, and mouth, the degree of opening and closing of the eyes and mouth, the position of the head, the orientation of the face, the direction of gaze, and the skeletal structure, based on the detection results from the occupant state sensor 11.

[0018] The passenger action determination unit 2e uses the passenger's characteristics and state recognized by the passenger state recognition unit 2d to determine the passenger's actions and posture. For example, if the passenger's arm movement is a predetermined stop request action, the passenger action determination unit 2e determines that the passenger is performing a stop request action. Here, a stop request action performed by a passenger is a predetermined arm movement of the passenger that is intended to signal (request) the passenger to stop (hereinafter sometimes referred to as a stop signal action), such as raising a hand or waving a hand. In addition, if the passenger speaks to the driver or places a hand on the driver's shoulder, the unit determines that the passenger is showing concern for the driver. Furthermore, the unit may also determine from the passenger's facial expression whether or not the passenger has noticed an abnormality with the driver.

[0019] The driver abnormality control unit 2f executes a stop control to automatically stop the moving vehicle 100 based on the occupant information detected by the occupant detection unit 2a, the driver abnormality determination result of the driver abnormality determination unit 2c, and the passenger status information determined by the passenger action determination unit 2e. The driver abnormality control unit 2f controls each control unit necessary for stopping the vehicle 100.

[0020] The alarm control unit 3 performs warning control to issue warnings from the driver abnormality warning controlled by the driver abnormality control unit 2f until the vehicle is stopped. In warning control, the alarm control unit 3 outputs an audio message and an alarm sound to the audio device 12. The audio device 12 is a device that includes an audio signal processing unit and a speaker. The alarm control unit 3 also outputs a display message and a flashing instruction to the display device 13. The display device 13 is, for example, an MFD (Multi-Function Display) or a device for displaying information provided in the meter panel.

[0021] The driving control unit 4 has the functional units of a braking control unit 4a and a steering control unit 4b.

[0022] The braking control unit 4a controls the braking device 21 based on detection signals from predetermined sensors, operation input information from the driver, and control signals from various functional units (driver abnormality control unit 2f, driving support control unit 5, etc.). Specifically, the braking control unit 4a performs brake control for stopping in response to control signals from the driver abnormality control unit 2f, etc. If the passenger action determination unit 2e determines that a passenger is performing a stop request action (stop instruction action), the driver abnormality control unit 2f outputs a control signal to the braking control unit 4a to perform stopping control that results in a higher deceleration of the vehicle 100 than when there is no passenger stop request action (stop instruction action). The braking control unit 4a also performs automatic brake control for collision avoidance and stopping in response to control signals from the driving support control unit 5, etc.

[0023] The steering control unit 4b controls the steering device 22 based on detection signals from predetermined sensors and driver input information. The steering control unit 4b also performs automatic steering control for lane keeping in response to control signals from the driver assistance control unit 5.

[0024] The driver assistance control unit 5 controls various control units necessary for controlling actions such as braking and steering, based on information about objects detected and recognized by, for example, an image sensor acting as a surrounding monitoring sensor 14. Specifically, the driver assistance control unit 5 uses the detected values ​​from the surrounding monitoring sensor 14 to perform various driving-related controls, such as braking support control for collision avoidance and steering support control for lane keeping. Here, the surrounding monitoring sensor 14 comprehensively refers to an image sensor that obtains captured images for detecting and recognizing objects outside the vehicle, such as preceding vehicles, pedestrians, and lanes, and a GNSS (Global Navigation Satellite System) sensor that determines position based on latitude, longitude, and altitude. Note that the sensor used for object detection is not limited to an image sensor; other sensors such as millimeter-wave radar can also be used.

[0025] In this embodiment, the case in which each function of the vehicle control device 1 is implemented by a single control device (control unit) has been described as an example, but the invention is not limited to this, and some of the functions may be executed by different control devices, or all of the functions may be executed by other control devices.

[0026] The specific operations of the vehicle control device of this embodiment, configured as described above, from driver abnormality detection to vehicle stopping, will now be explained. In the following, use cases for automatic stopping corresponding to driver abnormality detection, use cases for automatic stopping due to a passenger's stop instruction action (action of requesting to stop), and use cases for automatic stopping due to passenger's action judgment (judgment of driver abnormality from the passenger's actions) will be explained as examples.

[0027] Figure 2 shows a use case for automatic shutdown due to driver abnormality detection.

[0028] As shown in Figure 2, while the vehicle 100 is in motion, the driver state recognition unit 2b first recognizes the driver's state based on the information detected from the occupant state sensor 11. If the driver abnormality determination unit 2c recognizes from the information recognized by the driver state recognition unit 2b that the driver's state is abnormal, and the detection of the driver abnormality continues for a predetermined time, it confirms that the driver is abnormal.

[0029] Here, the time when the driver abnormality is detected is defined as time T1, the predetermined time until the driver abnormality is confirmed as Tn1, and the time when the driver abnormality is confirmed as T2.

[0030] When the driver abnormality control unit 2f confirms a driver abnormality at time T2, it initiates warning control, such as outputting a warning sound from the alarm control unit 3, thereby notifying the vehicle that a driver abnormality has been detected.

[0031] Taking into account the possibility of the driver recovering from an abnormality after it has been confirmed, a waiting period is provided between the confirmation of the driver abnormality and the start of the automatic stopping of the vehicle 100, and the automatic stopping is initiated after a predetermined period of time has elapsed.

[0032] If the driver recovers from the abnormal condition within a predetermined time, the driver can cancel the warning notification. Here, canceling the warning notification refers to a specific operation to cancel the warning notification, such as operating a designated control element provided inside the vehicle. Note that the cancellation operation is not limited to the operation of a control element and may be achieved by operating the brakes.

[0033] Once a driver malfunction is confirmed, a warning notification will begin at time T2, and after a predetermined time has elapsed, an automatic shutdown notice will be issued.

[0034] Here, the predetermined time from the start of the warning notification T2 to the start of the automatic stop notification is defined as Tn2, and the time when the automatic stop notification starts is defined as time T3.

[0035] If the driver recovers from the abnormal state within a predetermined time period Tn2 after a driver malfunction warning is issued, the driver can cancel the warning. The predetermined time period Tn2 can be set arbitrarily.

[0036] If a predetermined time Tn3 elapses after the automatic stop warning is issued but no input information for a release operation is detected, the driver abnormality control unit 2f executes an automatic stop of the moving vehicle. At this time, the alarm control unit 3 issues a warning inside the vehicle that the braking control for automatic stopping has been initiated.

[0037] The automatic stop execution notification will be issued from time Tn3 until the vehicle comes to a complete stop.

[0038] Through the above actions, the vehicle 100 can be safely stopped even if the driver becomes abnormal while the vehicle is in motion.

[0039] Figure 3 shows a use case for automatic stopping triggered by a passenger's stop command.

[0040] In the use case shown in Figure 2, a predetermined time interval Tn2 and Tn3 is provided between the start of the warning notification request for a response by the alarm control unit 3, the start of the warning notification for the impending stop, and the execution of the automatic stop by the driver abnormality control unit 2f, taking into consideration the driver's recovery from the abnormality. During this time, the passenger will wait in the moving vehicle while the driver is not operating normally, feeling anxious and frightened, for the predetermined time interval Tn2 and Tn3 until the automatic stop is initiated. The passenger may also realize that the driver's abnormality will not be resolved immediately.

[0041] Therefore, as shown in Figure 3, the driver abnormality detection unit 2c determines that there is a driver abnormality, the passenger action detection unit 2e determines the passenger's action (stop instruction action), and the driver abnormality control unit 2f starts executing automatic stop before the predetermined time Tn2 has elapsed.

[0042] In Figure 3, first, if the occupant detection unit 2a has detected a passenger, the driver abnormality control unit 2f confirms a driver abnormality at time T2, and along with a warning from the alarm control unit 3, notifies the passenger of an action that signals the start of stopping the vehicle.

[0043] The passenger state recognition unit 2d recognizes the passenger's state based on the information detected from the occupant state sensor 11, and the passenger action determination unit 2e determines from the information recognized by the passenger state recognition unit 2d that the passenger's action is a signal to stop the vehicle (stop instruction action).

[0044] The driver abnormality control unit 2f, without waiting for the predetermined time Tn2 to elapse, executes an automatic stop of the moving vehicle based on the passenger action determination unit 2e's determination of the passenger's action to stop the vehicle. At this time, the warning control unit 3 broadcasts a warning inside the vehicle that the automatic stopping braking control has been initiated.

[0045] In addition to determining the state of the driver, by adding an operation of requesting the passenger to stop based on the determination of the passenger's expectation of the driver's recovery, the reliability of the determination of driver abnormality is increased, and it becomes possible to determine the start of automatic stop without waiting for the elapse of a predetermined time Tn2. By starting the automatic stop without waiting for the elapse of the predetermined time Tn2, the vehicle can be stopped at an earlier time than when it is started after the elapse of the predetermined time Tn2 (see FIG. 2). As a result, the passenger can obtain a sense of security at an earlier stage. Also, even if the passenger has no driving experience of a vehicle or is a child and does not know the method of stopping the vehicle, the passenger can start the automatic stop by issuing an instruction for automatic stop (automatic parking) of the vehicle by himself / herself, and can obtain a sense of security at an earlier stage.

[0046] FIG. 4 is a diagram showing a use case of automatic stop based on the determination of the passenger's operation.

[0047] The use case shown in FIG. 4 shows a case where information on the state indicating whether the passenger has recognized the presence of driver abnormality is acquired, and even when there is no stop instruction operation by the passenger, the driver abnormality is determined based on the detection information of driver abnormality and the information on the state of the passenger, and the vehicle is stopped.

[0048] In FIG. 4, first, when the driver abnormality determination unit 2c recognizes the driver's abnormality, and the passenger operation determination unit 2e determines from the information recognized by the passenger state recognition unit 2d that the passenger has recognized the driver's abnormality, for example, by talking to the passenger driver or putting a hand on the shoulder, etc., the driver abnormality time control unit 2f executes automatic stop of the running vehicle without waiting for the elapse of the predetermined time Tn1.

[0049] At this time, the warning control unit 3 notifies the inside of the vehicle of a warning that the braking control of the automatic stop has been started.

[0050] In addition to determining the state of the driver, by adding a determination that the driver is abnormal from the passenger's operation, the reliability of the determination of driver abnormality is increased, and it becomes possible to determine the start of automatic stop of the vehicle without waiting for the elapse of the predetermined time Tn1.

[0051] By automatically stopping the vehicle without waiting for the predetermined time Tn1 to elapse, the vehicle can be stopped earlier than when automatic stopping is performed after the predetermined time Tn1 and Tn2 have elapsed (see Figure 3). This allows passengers to feel safe at an earlier stage. Furthermore, even if a passenger has no driving experience or is a child and does not know how to stop the vehicle, braking control can be started from the moment the passenger notices an abnormality in the driver (the moment it is determined that the passenger has recognized an abnormality in the driver), allowing them to feel safe at an earlier stage.

[0052] The following describes the specific processing steps from driver abnormality detection to vehicle stopping in the vehicle control device of this embodiment, which is configured as described above. The control device 1 performs the following processing while the vehicle is running with driver assistance control such as lane keeping assist control or constant speed driving / inter-vehicle distance control. The control device 1 may also perform the following processing during manual driving when the driver is operating the vehicle themselves without driver assistance control.

[0053] First, we will explain the process of automatic stopping based on a stop command issued by a passenger (see Figure 3).

[0054] Figure 5 is a flowchart showing the process of automatic stopping based on a stop command issued by a passenger.

[0055] In Figure 5, first, the driver state recognition unit 2b of the control device 1 recognizes the characteristics and state of the driver using the values ​​detected by the occupant state sensors 11, such as the in-vehicle imaging sensor, pulse sensor, and body temperature sensor (step S100).

[0056] For example, by processing the acquired image information through image analysis, the system recognizes the driver's state based on external characteristics such as the driver's posture, degree of eye opening and closing, and duration of eye closure. It also recognizes the driver's biometric information such as pulse rate and body temperature.

[0057] Next, the driver abnormality determination unit 2c determines whether or not a driver abnormality has been detected based on the recognition result of the driver state recognition unit 2b (step S110).

[0058] The driver abnormality determination unit 2c determines that the driver is abnormal based on information about the driver's state recognized by the driver state recognition unit 2b, such as when the driver's posture deviates from a normal driving posture, such as when the driver is leaning over the steering wheel, when the driver's eyes remain closed, or when the driver's pulse rate or body temperature exceeds a preset normal range.

[0059] If the result of the determination in step S110 is NO, that is, if no driver abnormality is detected, the processes of steps S100 and S110 are repeated until a driver abnormality is detected and the result of the determination in step S110 becomes YES.

[0060] Furthermore, if the result of the determination in step S110 is YES, then it is determined whether a predetermined time Tn1 has elapsed since the confirmation of the driver abnormality (step S120).

[0061] The driver abnormality detection unit 2c measures the duration of driver abnormality conditions such as poor posture, closed eyes, pulse rate, and body temperature, and confirms a driver abnormality if the driver abnormality condition persists for a predetermined time Tn1.

[0062] If the result of the determination in step S120 is NO, the process from steps S100 to S120 is repeated until the results of the determinations in steps S110 and S120 become YES.

[0063] Furthermore, if the result of the determination in step S110 is YES, the driver abnormality control unit 2f then determines whether or not there are any occupants other than the driver based on the occupant detection information from the occupant detection unit 2a (step S130).

[0064] If the result of the determination in step S140 is NO, the driver abnormality control unit 2f notifies the alarm control unit 3 of a warning that it has determined that there is a driver abnormality (step S140).

[0065] As a result, for example, an audio message and alarm sound will be emitted inside the vehicle, a warning message will be displayed on the MFD, and an indicator in the meter will flash.

[0066] Furthermore, the driver abnormality control unit 2f may, as a warning of driver abnormality, prompt the driver to regain awareness by, for example, notifying them through tactile means such as steering wheel vibration or by notifying them through gradual deceleration.

[0067] Following the processing in step S140, the driver abnormality control unit 2f determines whether a predetermined time Tn2 (response waiting time) has elapsed since the driver abnormality warning notification was started without detecting an operation to cancel the driver abnormality warning notification (step S150).

[0068] If the result of the determination in step S150 is NO, the processes in steps S140 and S150 are repeated until the result of the determination becomes YES. During the execution of steps S140 and S150, driver assistance control continues to be performed on the moving vehicle. If the moving vehicle is being operated manually, the driver abnormality control unit 2f may activate the driver assistance control function when it determines that there is a driver abnormality (if the result in step S110 is YES). Also, if the driver attempts to cancel the driver abnormality warning notification during the execution of steps S140 and S150, the driver abnormality control unit 2f terminates the driver abnormality warning notification. Cancellation operations include, for example, the operation of a specific control element or intentional steering operations by the driver. In the case of cancellation due to steering operations, the driver abnormality control unit 2f determines that it is an intentional steering operation by the driver based on, for example, the amount of change in the steering angle obtained from the steering angle sensor.

[0069] Furthermore, if the result of the determination in step S150 is YES, the driver abnormality control unit 2f then notifies the alarm control unit 3 of the start of automatic shutdown (step S160).

[0070] This allows for, for example, the system to provide voice messages, sound alarms, and display warning messages on the MFD inside the vehicle, while also providing warnings to following vehicles and pedestrians outside the vehicle, such as flashing hazard lights and sounding the horn.

[0071] Following the processing in step S160, the driver abnormality control unit 2f determines whether a predetermined time Tn3 (waiting time for the notification) has elapsed since the automatic stop start notification warning notification was started without detecting an operation to cancel the automatic stop start notification warning notification (step S170).

[0072] If the result of the determination in step S170 is NO, the processes of steps S160 and S170 are repeated until the result of the determination becomes YES. During the execution of steps S160 and S170, driver assistance control continues to be performed on the moving vehicle. Also, if the driver's operation to cancel the automatic stop start warning notification is detected during the execution of steps S160 and S170, the driver abnormality control unit 2f terminates the automatic stop start warning notification. The cancellation operation is, for example, the operation of a specific control element or an intentional steering operation by the driver. In the case of cancellation due to steering operation, the driver abnormality control unit 2f determines that it is an intentional steering operation by the driver based, for example, the amount of change in the steering angle obtained from the steering angle sensor.

[0073] Furthermore, if the result of the determination in step S170 is YES, the driver abnormality control unit 2f instructs the braking control unit 4a to apply the brakes, and the alarm control unit 3 notifies the driver of an automatic stop operation (step S180), and the process ends.

[0074] When the vehicle comes to a stop due to the automatic braking system, it maintains its stationary state. This triggers, for example, voice messages and alarm sounds inside the vehicle, along with a warning message displayed on the MFD (Multi-Function Display). Outside the vehicle, it alerts following vehicles and pedestrians to the automatic stop operation through flashing hazard lights and the sounding of the horn.

[0075] Furthermore, if the result of the determination in step S130 is YES, that is, if it is determined that there is a passenger other than the driver, the driver abnormality control unit 2f then notifies the passenger of the action to signal the passenger to start stopping the vehicle (the content of the stop instruction action) along with a warning from the alarm control unit 3 that an abnormality in the driver has been determined (step S131).

[0076] As a result, for example, an audio message and alarm sound will be emitted inside the vehicle, a warning message will be displayed on the MFD, and an indicator in the meter will flash. For passengers, a simple action such as raising either the left or right arm will be conveyed via an audio message to signal that the passenger should stop the vehicle. In addition, if the occupant detection unit 2a performs facial recognition of the occupant and the occupant is registered, the driver abnormality control unit 2f may add the registered occupant's name to the message indicating that the passenger should stop the vehicle. In this case, if there are multiple passengers, for example, the names of the passengers in the front passenger seat may be added to the message indicating that the passenger should stop the vehicle. Furthermore, if the occupant detection unit 2a has information on the age of the passengers and whether they are adults or children, the names of passengers in the older age group may be added first.

[0077] Furthermore, if the occupant status sensor 11 is equipped with a microphone for detecting sounds inside the vehicle, the passenger's voice may be used as a signal to begin stopping the vehicle. For example, along with a warning from the alarm control unit 3 that it has determined that the driver is abnormal, it notifies the passenger of the passenger's actions that signal the start of stopping the vehicle, as well as an inquiry to start stopping and a request for a voice response from the passenger.

[0078] Furthermore, the driver abnormality control unit 2f may, as a warning of driver abnormality, prompt the driver to regain awareness of the abnormal state by, for example, notifying them through tactile means such as steering wheel vibration or by notifying them through tactile means such as gradual deceleration.

[0079] Next, the passenger state recognition unit 2d recognizes the passenger's state based on the information detected from the passenger state sensor 11, and the passenger action determination unit 2e determines the passenger's action based on the information recognized by the passenger state recognition unit 2d (step S132).

[0080] Next, the passenger state recognition unit 2d determines whether or not it has detected a signal to stop the vehicle (stop instruction action) based on the passenger's actions (step S133).

[0081] If the result of the determination in step S133 is YES, the process in step S180 is performed and the process ends. In addition, if the passenger status recognition unit 2d determines that the passenger's actions are a voice response from the passenger to the inquiry about stopping, the result of the determination in step S133 will also be YES.

[0082] Furthermore, if the determination result in step S133 is NO, that is, if the passenger action determination unit 2e's determination result for the passenger action is not a signal to stop the vehicle (stop instruction action), the driver abnormality control unit 2f then determines whether a predetermined time Tn2 (response waiting time) has elapsed since the notification of the passenger action to stop the vehicle started along with the driver abnormality warning, without detecting an operation to cancel the driver abnormality warning notification (step S134).

[0083] If the result of the determination in step S134 is NO, the process of steps S131 to S134 is repeated until the result of the determination becomes YES. During the execution of steps S131 to S134, driver assistance control continues to be performed on the moving vehicle. Also, if the driver's operation to cancel the automatic stop start warning notification is detected during the execution of steps S131 to S134, the driver abnormality control unit 2f terminates the driver abnormality warning and the notification of the passenger's action to stop the vehicle.

[0084] Furthermore, if the result of the determination in step S134 is YES, the alarm control unit 3 then notifies the passenger of the action to be taken by the passenger (the content of the stop instruction action) that will signal the start of the automatic stop (step S135).

[0085] As a result, for example, an audio message and alarm sound will be emitted inside the vehicle, a warning message will be displayed on the MFD, and an indicator in the meter will flash. For passengers, a simple action such as raising either the left or right arm will be communicated via audio message to signal that the passenger should stop the vehicle. In addition, if the occupant detection unit 2a performs facial recognition of the occupant and the occupant is registered, the driver abnormality control unit 2f may add the registered occupant's name to the message regarding the passenger's signal action to stop the vehicle and notify the occupant. Outside the vehicle, warnings such as flashing hazard lights and horn sounds may be notified to alert following vehicles and pedestrians.

[0086] Furthermore, if the occupant status sensor 11 is equipped with a microphone for detecting sounds inside the vehicle, the passenger's voice may be used as a signal to begin stopping the vehicle. For example, along with a warning from the alarm control unit 3 that it has determined that the driver is abnormal, the system will notify the passenger of the passenger's actions (stop instruction actions) that signal the start of stopping the vehicle, as well as an inquiry to start stopping and a request for a voice response from the passenger.

[0087] Furthermore, the driver abnormality control unit 2f may, as a warning of driver abnormality, prompt the driver to regain awareness of the abnormal state by, for example, notifying them through tactile means such as steering wheel vibration or by notifying them through tactile means such as gradual deceleration.

[0088] Next, the passenger state recognition unit 2d recognizes the passenger's state based on the information detected from the passenger state sensor 11, and the passenger action determination unit 2e determines the passenger's action based on the information recognized by the passenger state recognition unit 2d (step S136).

[0089] Next, the passenger state recognition unit 2d determines whether or not it has detected a signal to stop the vehicle (stop instruction action) based on the passenger's actions (step S137).

[0090] If the result of the determination in step S137 is YES, the process in step S180 is performed and the process ends. In addition, if the passenger status recognition unit 2d determines that the passenger's actions are a voice response from the passenger to the inquiry about stopping, the result of the determination in step S137 will also be YES.

[0091] Furthermore, if the result of the determination in step S137 is NO, that is, if the passenger action determination unit 2e's determination result of the passenger action is not a signal to stop the vehicle (stop instruction action), then the driver abnormality control unit 2f then determines whether a predetermined time Tn3 (waiting time for notification) has elapsed since the notification of the passenger action to stop the vehicle (stop instruction action) started along with the driver abnormality warning, without detecting an operation to cancel the automatic stop start notification warning notification (step S138).

[0092] If the result of the determination in step S138 is NO, the process from S135 to S138 is repeated until the result becomes YES. During the execution of the process from steps S135 to S138, driver assistance control continues to be performed on the moving vehicle. Also, if the driver's operation to cancel the automatic stop start warning notification is detected during the execution of the process from steps S135 to S138, the driver abnormality control unit 2f terminates the automatic stop start warning and the notification of the passenger's action to stop the vehicle. The cancellation operation is, for example, the operation of a specific control element or an intentional steering operation by the driver. In the case of cancellation due to steering operation, the driver abnormality control unit 2f determines that it is an intentional steering operation by the driver based, for example, the amount of change in the steering angle obtained from the steering angle sensor.

[0093] Furthermore, if the result of the determination in step S138 is YES, the process in step S180 is performed and the process is terminated.

[0094] Next, the process of automatic stopping based on the passenger's action judgment (whether or not the passenger perceives a driver abnormality) (see Figure 4) will be explained. In this process, the passenger action judgment unit 2e determines whether or not an automatic stop is necessary even before a driver abnormality is confirmed, based on the passenger's action judgment. The control device 1 performs the following processes while the vehicle is running with driving assistance control such as lane keeping assist control or constant speed driving / inter-vehicle distance control. Note that the control device 1 may also perform the following processes when the driver is manually operating the vehicle without using driving assistance control.

[0095] Figure 6 is a flowchart showing the process of automatic stopping based on the passenger's actions.

[0096] In Figure 6, first, the driver state recognition unit 2b recognizes the characteristics and state of the driver using the values ​​detected by the occupant state sensors 11, such as the in-vehicle imaging sensor, pulse sensor, and body temperature sensor (step S200).

[0097] For example, by processing the acquired image information through image analysis, the system recognizes the driver's state based on external characteristics such as the driver's posture, degree of eye opening and closing, and duration of eye closure. It also recognizes the driver's biometric information such as pulse rate and body temperature.

[0098] Next, the driver abnormality determination unit 2c determines whether or not a driver abnormality has been detected based on the recognition result of the driver state recognition unit 2b (step S210).

[0099] The driver abnormality determination unit 2c determines that the driver is abnormal based on information about the driver's state recognized by the driver state recognition unit 2b, such as when the driver's posture deviates from a normal driving posture, such as when the driver is leaning over the steering wheel, when the driver's eyes remain closed, or when the driver's pulse rate or body temperature exceeds a preset normal range.

[0100] If the result of the determination in step S210 is NO, the process in steps S200 and S210 is repeated until the result of the determination becomes YES.

[0101] Furthermore, if the determination result in step S110 is YES, the passenger state recognition unit 2d then recognizes the passenger's state based on the detection information from the passenger state sensor 11, and the passenger action determination unit 2e determines the passenger's action based on the information recognized by the passenger state recognition unit 2d (step S220).

[0102] Next, the passenger action determination unit 2e determines from the passenger's actions whether or not the passenger is aware of a driver abnormality (step S230).

[0103] The passenger action determination unit 2e determines that the passenger is aware of the driver's abnormality based on information such as the passenger turning their face towards the driver, the passenger looking at the driver, the passenger speaking to the driver, the passenger moving their body closer to the driver, the passenger touching the driver's body with their hand, the passenger trying to correct the driver's poor posture, the passenger showing a surprised expression, the passenger showing a frightened expression, etc. To prevent misjudgments, the determination of whether the passenger is aware of the driver's abnormality may be made not only based on a single action, but also on multiple actions. The determination may also be made based on the passenger's voice. For example, if the unit recognizes that the passenger has uttered words expressing concern for the driver or words related to stopping, it will be determined that the passenger is aware of the driver's abnormality.

[0104] If the result of the determination in step S230 is YES, the driver abnormality control unit 2f instructs the braking control unit 4a to apply the brakes and the alarm control unit 3 notifies a warning that automatic stopping is in progress (step S330), and the process ends.

[0105] When the vehicle comes to a stop due to the automatic braking system, it maintains its stationary state. This triggers, for example, voice messages and alarm sounds inside the vehicle, along with a warning message displayed on the MFD (Multi-Function Display). Outside the vehicle, it alerts following vehicles and pedestrians to the automatic stop operation through flashing hazard lights and the sounding of the horn.

[0106] Furthermore, if the result of the determination in step S230 is NO, that is, if the passenger action determination unit 2e determines that the passenger is not aware of the driver abnormality (this includes cases where the occupant detection unit 2a does not detect any occupants other than the driver), then it is determined whether a predetermined time Tn1 has elapsed since the confirmation of the driver abnormality (step S240).

[0107] If the result of the determination in step S240 is NO, the process from steps S200 to S240 is repeated until the result of the determination becomes YES.

[0108] Furthermore, if the result of the determination in step S240 is YES, the alarm control unit 3 then issues a warning indicating that a driver abnormality has been detected (step S250).

[0109] As a result, for example, an audio message and alarm sound will be emitted inside the vehicle, a warning message will be displayed on the MFD, and an indicator in the meter will flash.

[0110] Furthermore, the driver abnormality control unit 2f may, as a warning of driver abnormality, prompt the driver to regain awareness of the abnormal state by, for example, notifying them through tactile means such as steering wheel vibration or by notifying them through tactile means such as gradual deceleration.

[0111] Next, the passenger state recognition unit 2d recognizes the passenger's state based on the information detected from the passenger state sensor 11, and the passenger action determination unit 2e determines the passenger's action based on the information recognized by the passenger state recognition unit 2d (step S260).

[0112] Next, the passenger action determination unit 2e determines from the passenger's actions whether or not the passenger is aware of a driver abnormality (step S270).

[0113] The passenger action determination unit 2e determines that the passenger is aware of the driver's abnormality based on information such as the passenger turning their face towards the driver, the passenger looking at the driver, the passenger speaking to the driver, the passenger moving their body closer to the driver, the passenger touching the driver's body with their hand, the passenger trying to correct the driver's poor posture, the passenger showing a surprised expression, the passenger showing a frightened expression, etc. To prevent misjudgments, the determination of whether the passenger is aware of the driver's abnormality may be made not only based on a single action, but also on multiple actions. The determination may also be made based on the passenger's voice. For example, if the unit recognizes that the passenger has uttered words expressing concern for the driver or words related to stopping, it will be determined that the passenger is aware of the driver's abnormality.

[0114] If the result of the determination in step S270 is YES, the process in step S330 is performed and the process is terminated.

[0115] Furthermore, if the determination result in step S270 is NO, that is, if the passenger action determination unit 2e determines that the passenger is not aware of the driver abnormality (this includes cases where the occupant detection unit 2a does not detect any occupants other than the driver), the driver abnormality control unit 2f determines whether a predetermined time Tn2 (response waiting time) has elapsed since the driver abnormality warning notification was started without detecting an operation to cancel the driver abnormality warning notification (step S280).

[0116] If the result of the determination in step S280 is NO, the process from steps S250 to S280 is repeated until the result of the determination becomes YES. During the execution of the process from steps S250 to S280, driver assistance control is continuously performed on the moving vehicle. If the moving vehicle is being driven manually, the driver abnormality control unit 2f may activate the driver assistance control function when it determines that there is a driver abnormality (if the result in step S210 is YES). Also, if the driver attempts to cancel the driver abnormality warning notification during the execution of the process from steps S250 to S280, the driver abnormality control unit 2f terminates the driver abnormality warning notification. Cancellation operations include, for example, the operation of a specific control element or intentional steering operations by the driver. In the case of cancellation due to steering operations, the driver abnormality control unit 2f determines that it is an intentional steering operation by the driver based on, for example, the amount of change in the steering angle obtained from the steering angle sensor.

[0117] Furthermore, if the result of the determination in step S280 is YES, the alarm control unit 3 will issue a warning to indicate the start of automatic stopping (step S290).

[0118] This allows for, for example, the system to provide voice messages, sound alarms, and display warning messages on the MFD inside the vehicle, while also providing warnings to following vehicles and pedestrians outside the vehicle, such as flashing hazard lights and sounding the horn.

[0119] Next, the passenger state recognition unit 2d recognizes the passenger's state based on the information detected from the passenger state sensor 11, and the passenger action determination unit 2e determines the passenger's action based on the information recognized by the passenger state recognition unit 2d (step S300).

[0120] Next, the passenger action determination unit 2e determines whether or not the passenger is aware of a driver abnormality based on the passenger's actions (step S310).

[0121] The passenger action determination unit 2e determines whether the passenger is aware of the driver's abnormality based on information such as the passenger turning their face towards the driver, the passenger looking at the driver, the passenger speaking to the driver, the passenger moving their body closer to the driver, the passenger touching the driver's body with their hand, the passenger trying to correct the driver's poor posture, the passenger showing a surprised expression, the passenger showing a frightened expression, etc. The determination may be made not only based on a single action, but also on multiple actions. The determination may also be made based on the passenger's voice. For example, if the unit recognizes that the passenger has uttered words expressing concern for the driver or words related to stopping, it will be determined that the passenger is aware of the driver's abnormality.

[0122] If the result of the determination in step S310 is YES, the process in step S330 is performed and the process is terminated.

[0123] Furthermore, if the result of the determination in step S310 is NO, that is, if the passenger action determination unit 2e determines that the passenger is not aware of the driver abnormality (this includes cases where the occupant detection unit 2a does not detect any occupants other than the driver), the driver abnormality control unit 2f then determines whether a predetermined time Tn3 (waiting time for the notification) has elapsed since the automatic stop start notification warning was started without detecting an operation to cancel the automatic stop start notification warning (step S320).

[0124] If the result of the determination in step S320 is NO, the process in steps S290 to S320 is repeated until the result of the determination becomes YES. During the execution of the process in steps S290 to S320, driver assistance control is continuously performed on the moving vehicle. If the moving vehicle is being operated manually, the driver abnormality control unit 2f may activate the driver assistance control function when it determines that there is a driver abnormality (if the result in step S210 is YES). Also, if the driver cancels the automatic stop warning notification during the execution of the process in steps S290 to S320, the driver abnormality control unit 2f terminates the automatic stop warning notification. The cancellation operation is, for example, the operation of a specific control element or an intentional steering operation by the driver. In the case of cancellation due to steering operation, the driver abnormality control unit 2f determines that it is an intentional steering operation by the driver based on, for example, the amount of change in the steering angle obtained from the steering angle sensor.

[0125] Furthermore, if the result of the determination in step S320 is YES, the process in step S330 is performed and the process is terminated.

[0126] <Note> The present invention is not limited to the embodiments described above, and includes various modifications and combinations that do not depart from the spirit of the invention. Furthermore, the present invention is not limited to having all the configurations described in the embodiments described above, and includes those in which some of the configurations have been omitted. In addition, some or all of the above configurations, functions, etc. may be realized by designing, for example, an integrated circuit. Furthermore, each of the above configurations, functions, etc. may be realized in software by having a processor interpret and execute a program that realizes each function.

[0127] 1...Control device (vehicle control device), 2...Occupant status control unit, 2a...Occupant detection unit, 2b...Driver status recognition unit, 2c...Driver abnormality judgment unit, 2d...Passenger status recognition unit, 2e...Passenger action judgment unit, 2f...Driver abnormality control unit, 3...Warning control unit, 4...Driving control unit, 4a...Braking control unit, 4b...Steering control unit, 5...Driving support control unit, 11...Occupant status sensor, 12...Audio device, 13...Display device, 14...Surroundings monitoring sensor, 21...Braking device, 22...Steering device, 100...Vehicle

Claims

1. A driver state recognition unit that recognizes the state of the driver based on the detection results of an occupant detection sensor that detects the driver and passengers of a vehicle; a driver abnormality determination unit that determines whether or not there is an abnormality in the driver based on the recognition results of the driver state recognition unit; a driver abnormality control unit that determines whether or not the vehicle needs to be automatically stopped after a predetermined time has elapsed following the determination that there is an abnormality in the driver by the driver abnormality determination unit; a warning unit that issues a warning when the driver abnormality determination unit determines that there is an abnormality in the driver, and a warning when the driver abnormality control unit determines that it is necessary to automatically stop the vehicle; a braking control unit that executes braking control of the vehicle and stops the vehicle when the driver abnormality control unit determines that it is necessary to automatically stop the vehicle; a passenger state recognition unit that recognizes the state of the passenger based on the detection results of the occupant detection sensor; and a passenger action determination unit that determines whether or not the passenger's action is due to an abnormality in the driver based on the recognition results of the passenger state recognition unit. The vehicle control device is characterized in that, when the driver abnormality control unit determines that the recognition result of the passenger state recognition unit is an operation based on a driver abnormality, it determines that it is necessary to automatically stop the vehicle without waiting for the predetermined time to elapse.

2. A vehicle control device according to claim 1, wherein the driver abnormality control unit determines, based on the recognition result of the passenger status recognition unit, whether or not the passenger has performed an action to instruct the vehicle to start automatic stopping as an action based on the driver abnormality.

3. A vehicle control device according to claim 1, wherein the driver abnormality control unit determines, based on the recognition result of the passenger state recognition unit, whether the passenger has performed an action that indicates the passenger has recognized that the driver is abnormal, as an action based on the driver's abnormality.

4. A vehicle control device according to claim 2, wherein the warning unit, when the driver abnormality determination unit determines that there is an abnormality in the driver and the occupant detection sensor detects the passenger, issues an instruction action warning prompting the passenger to take action to instruct the automatic stopping of the vehicle, and the driver abnormality control unit, after the instruction action warning, when the passenger action determination unit determines that the passenger is taking action to instruct the automatic stopping of the vehicle, determines that the automatic stopping of the vehicle is necessary without waiting for the predetermined time to elapse.

5. A vehicle control device according to claim 3, wherein the passenger action determination unit determines whether the passenger has recognized that there is an abnormality in the driver based on at least one of the passenger's facial expression, posture, and actions, which are the recognition results of the passenger state recognition unit, and the driver abnormality control unit determines that it is necessary to automatically stop the vehicle without waiting for the predetermined time to elapse, when the driver abnormality determination unit has determined that there is an abnormality in the driver and the passenger action determination unit has determined that the passenger has recognized that there is an abnormality in the driver.

Images