Driver assistance systems

Abstract

The system detects driver abnormalities caused by oral automatisms and activates or deactivates the vehicle's driver abnormality response system. [Solution] The driving assistance system comprises: an image acquisition unit that sequentially acquires multiple images including the face of the driver operating the vehicle; an audio acquisition unit that acquires voices emitted by the driver; a detection unit that extracts multiple feature points from the face portion shown in the multiple images and detects the displacement of the multiple feature points; a determination unit that, when no voice from the driver is detected in synchronization with the displacement of the multiple feature points, uses machine learning to determine whether or not a specific behavior of the driver's mouth that would interfere with the driver's driving operations has occurred; and a notification unit that notifies the determination result of the determination unit.

Description

Technical Field

[0001] The present invention relates to a driving support system that detects driver abnormalities and provides driving support.

Background Art

[0002] As an in-vehicle system mounted on a vehicle, an advanced driving support system (ADAS) that provides driving support for a driver is known. For example, it detects the surrounding environment of the vehicle using a plurality of in-vehicle cameras and sensing devices, and detects the driver's posture, face direction, line-of-sight direction, etc. using a driver monitoring system. When it is detected that there is a possibility that the safe driving may be hindered due to the positional relationship between the vehicle and an object such as another vehicle or a pedestrian, the in-vehicle system issues an alarm to the driver using an in-vehicle monitor or a speaker, and provides various types of information in text or voice.

[0003] Driving operations may become difficult due to driver abnormalities. For example, it may be difficult for the driver to continue safe driving while checking the surrounding environment of the vehicle due to sudden symptoms that the driver himself / herself cannot predict in advance, such as a sudden illness, cerebrovascular disease, heart disease, or unconscious state such as fainting of the driver. In order to cope with a situation where it becomes difficult to continue driving due to a sudden change in the driver's physical condition, a driver abnormality response system (EDSS, Emergency Driving Stop System) has been developed that detects driver abnormalities and automatically stops the vehicle in an emergency. The driver abnormality response system detects driver abnormalities and automatically decelerates the vehicle and automatically parks it in a safe place such as the side of the road. For example, a physical emergency switch for manually activating the driver abnormality response system may be provided near the driver's seat. In this case, by the driver or a passenger operating the emergency switch, the driver abnormality response system can be activated to safely stop the vehicle or evacuate the vehicle to a safe place.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-144818 [Overview of the project] [Problems that the invention aims to solve]

[0005] Patent Document 1 discloses a driver assistance device equipped with an operator that can deactivate automatic stop control, which detects when the driver is incapacitated and slows down and stops the vehicle. The driver assistance device reduces driver confusion regarding the timing of activation and deactivation of automatic stop control, and deactivates the driver abnormality response system according to the operation status of the operator operated by the driver or passenger.

[0006] The basic design document for the automatic driver abnormality detection system formulated by the Ministry of Land, Infrastructure, Transport and Tourism of Japan defines multiple items and indicators for detecting driver abnormalities, including driver state, driving behavior, and vehicle behavior. Driver state includes not only external conditions such as the driver's posture, facial expression, and limb movements, but also internal conditions such as the driver's hemodynamics and brain nerve responses. Driving behavior includes the driver's operation of various controls such as the steering wheel, accelerator pedal, and brake pedal, as well as safe driving actions such as the driver visually checking the surrounding environment. Vehicle behavior refers to the movement of the vehicle, such as swaying and changes in speed. Thresholds are set for various indicators as criteria for judging driver abnormalities. When an indicator reaches a threshold, the system performs a response confirmation actuation to the driver, and if the driver does not respond within a predetermined time, the system detects that a driver abnormality has occurred.

[0007] Driver conditions resulting from cerebrovascular disease, heart and aortic disease, diabetes, and epilepsy include unsteady posture, closed eyes, and loss of consciousness. If these conditions cause the driver to remain inattentive to the road ahead, the risk of driving increases. Driver abnormality response systems detect driver abnormalities by observing changes in the driver's appearance while the vehicle is in motion. However, if the driver experiences a physical rigidity condition such as epilepsy, it is difficult to accurately detect the driver's inability to drive. In particular, with symptoms related to brain dysfunction such as epilepsy, it is conceivable that the driver's body may suddenly become rigid, rendering them unable to drive, or that the driver's attention may suddenly become distracted, making it difficult to perform safe driving maneuvers in response to the surrounding environment.

[0008] Onboard cameras that capture the driver's condition and various sensing devices embedded in the driver's seat can detect changes in the driver's appearance, or driver monitoring systems can detect the driver's facial movements and gaze direction. However, it has been difficult to detect states of distraction or loss of consciousness caused by minor changes in the driver's appearance, such as partial paralysis of the driver's body or face. In particular, in cases of epilepsy, temporary brain dysfunction can cause the driver to unexpectedly lose the ability to operate the vehicle and temporarily lose consciousness. In other words, an epileptic seizure that the driver cannot predict in advance may lead to dangerous driving. Even if an epileptic seizure occurs in a driver, if it is only partial paralysis of the driver's body or face, the changes in the driver's appearance are minor and it is difficult to detect it as an abnormality in the driver.

[0009] As described above, advanced driver assistance systems and driver abnormality response systems using driver monitoring systems have difficulty accurately detecting sudden inability to drive due to the driver's epileptic seizure or other reasons, and performing automatic stopping control to move the vehicle to a safe location.

[0010] If a sudden epileptic seizure that could lead to the driver becoming incapacitated is falsely detected, the driver abnormality response system may activate unexpectedly, causing the vehicle to automatically stop and move to the side of the road against the driver's intention. When a vehicle automatically stops or moves to the side of the road against the intention of a normal driver, it can cause confusion in the surrounding environment, such as other vehicles and pedestrians, and may impair traffic safety. Therefore, it is necessary to exclude false detections of epileptic seizures that could lead to the driver becoming incapacitated and to appropriately set the activation and deactivation timing of the driver abnormality response system.

[0011] The embodiment of the present invention aims to provide a driving assistance system that detects oral automatism as an epileptic symptom with minimal outward changes from a facial image of a driver operating a vehicle, determines that the driver is abnormal due to oral automatism, and activates or deactivates the vehicle's driver abnormality response system. [Means for solving the problem]

[0012] An embodiment of the present invention provides a driving assistance system comprising: an image acquisition unit that sequentially acquires a plurality of images including the face portion of a driver operating a vehicle; an audio acquisition unit that acquires voices emitted by the driver; a detection unit that extracts a plurality of feature points from the face portion shown in the plurality of images and detects the displacement of the plurality of feature points; a determination unit that, when no voice from the driver is detected in synchronization with the displacement of the plurality of feature points, uses a machine learning model that has learned the displacement of the plurality of feature points to determine whether or not a specific behavior of the driver's mouth that interferes with the driver's driving operation has occurred; and a notification unit that notifies the determination result of the determination unit. [Effects of the Invention]

[0013] In an embodiment of the present invention, oral automatism can be detected as an epileptic symptom with minimal visible changes from the facial image of the driver operating the vehicle, and an abnormality in the driver due to oral automatism can be determined, thereby enabling or deactivating the driver abnormality response system installed in the vehicle. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing the configuration of a driver assistance system according to the first embodiment of the present invention. [Figure 2] This is a block diagram showing the configuration of a driver assistance system according to a second embodiment of the present invention. [Figure 3] This flowchart shows the processing of the driver assistance system according to the second embodiment of the present invention. [Modes for carrying out the invention]

[0015] Embodiments of the present invention will be described in detail with reference to the drawings. The driver assistance system according to the embodiment of this application is equipped with a driver abnormality response system function (hereinafter referred to as the "EDSS function"). The conditions for activating the EDSS function are when the driver is unable to drive or is unconscious due to epileptic symptoms. Recently, attention has been focused on "elderly epilepsy," in which epileptic symptoms develop in older age, even though the person did not develop epileptic symptoms at a young age. The driver assistance system according to the embodiment of this application aims to improve the performance of the EDSS function by additionally detecting elderly epileptic symptoms as a condition for activating the EDSS function.

[0016] In elderly individuals, cases of complex partial seizures affecting specific parts of the body or face have been reported as epilepsy. For example, these include short-term loss of consciousness where the person does not respond when spoken to, hand automatisms such as repetitive meaningless hand movements, and oral automatisms such as repetitive chewing motions. The driving assistance system of this embodiment is characterized by detecting oral automatisms and activating, deactivating, or stopping the EDSS function.

[0017] It is possible to capture images of the driver's face using an in-car camera and detect mouth movements. These movements include jaw movement and movements of the orbicularis oris muscle and other facial muscles. Oral automatisms involve both jaw movement and orbicularis oris muscle movement. To detect the combination of jaw and orbicularis oris muscle movement, the entire face image is meshed, and the mesh intersections are captured as feature points. Changes or movements of these feature points are then machine-learned, and an artificial intelligence (AI) system can detect oral automatisms. For example, "LipSync3D," a machine learning model developed by a US company, is known. This model uses a "3D Face Mesh" (hereinafter referred to as "FaceMesh") to mesh face images for machine learning of the subject's mouth movements and speech, synchronizing the subject's video with deepfake mouth movements. FaceMesh is a machine learning model that detects key points (or feature points) of a person's face in real time from a person's face image, outputting 3D key points from a face image of a predetermined resolution. Another known machine learning program is "ViViT (A Video Vision Transformer)." ViViT is a video classification model that can identify the movements of people appearing in videos. While the machine learning model applied to the driver assistance system of this embodiment is not limited to FaceMesh or ViViT, known technologies can be used to detect mouth movements from facial images of people appearing in images and videos.

[0018] The driver assistance system of this embodiment detects oral automatism as an epileptic symptom. Specifically, the driver assistance system uses artificial intelligence (AI) and machine learning to detect specific behaviors that suggest the onset of oral automatism from the driver's facial images and videos captured by the in-vehicle camera. If the specific behavior continues for a certain period of time, the driver assistance system activates the EDSS function and pulls the vehicle over to a safe location and moves it to a safe location. Oral automatism is an unnatural movement around the mouth of the driver's face, and the driver's outward appearance changes are small. For this reason, it is difficult to distinguish oral automatism from other mouth movements even with AI and machine learning, and there is a possibility of false detection of oral automatism. For example, if the system detects that the driver is chewing gum, it is difficult to distinguish this from oral automatism based solely on a video of the driver's face. Therefore, it is necessary to distinguish between oral automatism, which occurs unconsciously by the driver, and the driver's conscious mouth movements.

[0019] The driving assistance system of the present embodiment includes means to eliminate false detections that would otherwise occur, such as detecting the driver's conscious mouth movements as oral automatisms, and thereby to stop (or deactivate) the EDSS function that may be activated in response to a false detection. For example, the driver may set a predetermined time (e.g., 3 minutes) using a timer and stop detecting oral automatisms for the predetermined time specified by the driver. Alternatively, detection of oral automatisms may be stopped at predetermined timings, such as when the driver eats or drinks and moves their mouth, when the driver talks to themselves or hums, when the driver's hands move around their face, when small objects are moved around the driver's face by the driver or a passenger, or when the driver speaks.

[0020] The means for removing false detection of oral automatisms provided in the driving support system is not limited to a timer that sets a predetermined time to stop the detection of oral automatisms. The driving support system may include various operators to cancel the EDSS function that may be activated by false detection of oral automatisms. Examples of operators for canceling the EDSS function include a physical switch operated by a driver or a passenger, a software switch that operates in response to the driver's voice, and a sensing device that detects the driver's gestures or body movements. When the driver utters a statement to stop the detection of abnormalities such as oral automatisms for a predetermined time, the software switch is activated and the EDSS function is canceled. Also, when the driver performs a specific action such as waving a hand in front of the in-vehicle camera, the sensing device detects the specific action and the EDSS function may be canceled. Note that the driver may set an arbitrary time in advance as the cancellation time of the EDSS function.

[0021] [1] First Embodiment Next, the configuration and functions of the driving support system 10 according to the first embodiment of the present invention will be described. FIG. 1 is a block diagram showing the configuration of the driving support system 10. The driving support system 10 is configured by implementing an EDSS function in an in-vehicle system that cooperates with an in-vehicle camera CM, a microphone MC, a speaker SP, and a display monitor DM such as a car navigation system mounted on a vehicle. The in-vehicle system is composed of a processor such as a CPU or a GPU and a memory, and realizes a desired function when the processor executes a predetermined program. Alternatively, the EDSS function may be implemented in a plurality of electronic control units (ECUs, Electronic Control Unit) connected to an in-vehicle network such as CAN (Controller Area Network).

[0022] The driving support system 10 detects the oral automatisms of the driver based on the facial image data of the driver acquired by the image acquisition unit 11 in cooperation with the in-vehicle camera CM and the voice data of the driver acquired by the voice acquisition unit 12 in cooperation with the microphone MC, and determines whether there is an abnormality that causes an obstacle to the driving operation due to the oral automatisms.

[0023] In FIG. 1, the image acquisition unit 11 acquires, as an image input, the facial image (or the upper body image) of the driver captured by the in-vehicle camera CM. When the in-vehicle camera CM operates continuously during the driving of the vehicle, the video of the driver's face is captured. Alternatively, when the in-vehicle camera CM repeats on / off periodically, a plurality of facial images of the driver are continuously captured in time series. The image acquisition unit 11 generates image data based on the image input.

[0024] The detection unit 13 meshes the facial image using software such as FaceMesh and extracts feature points corresponding to the mesh intersections. The feature points of the driver's face change according to the driver's expression, speech, and face orientation. The change in the feature points of the driver's face part refers to the change in the position of the feature points in the mesh and the change in the relative positions and distances of a plurality of feature points. The detection unit 13 generates face feature data indicating a plurality of feature points extracted from the facial image and the displacement of each feature point.

[0025] The voice acquisition unit 12 acquires, as a voice input, the speech of the driver by the microphone MC mounted on the vehicle. In addition to the speech of the driver, the microphone MC picks up a plurality of sounds such as the speech of passengers, noise generated inside the vehicle, and noise from outside the vehicle. Therefore, the directivity of the microphone MC may be enhanced to selectively acquire only the voice emitted from the driver. Also, the voice of the driver may be recorded in advance, and only the voice pattern of the driver or a specific frequency band may be selectively picked up. Alternatively, noise other than the speech of the driver may be removed by noise cancellation. The voice acquisition unit 12 generates and outputs voice data based on the voice of the driver acquired by the microphone MC.

[0026] The determination unit 14 performs the function of the EDSS to determine driver abnormalities caused by the driver's oral automatisms. Specifically, the determination unit 14 detects the driver's oral automatisms based on the driver's facial feature data and voice data, and determines whether or not the driver is abnormal. The determination unit 14 is equipped with a machine learning model ML for determining the driver's oral automatisms. The machine learning model ML is equipped with a video classification model such as ViViT, for example, and infers the presence or absence of oral automatisms based on the displacement of multiple feature points that make up the driver's facial image. As mentioned above, it distinguishes between conscious mouth movements such as the driver making a gesture of chewing gum, and oral automatisms such as involuntary mouth movements due to the driver's epileptic symptoms. Furthermore, if the driver's mouth moves unnaturally even though the voice data does not contain the driver's speech, it is highly likely that the voice data and the driver's mouth movements are not synchronized, and the driver is experiencing oral automatisms. For this reason, the presence or absence of oral automatisms may be inferred based on whether or not the voice data contains the driver's speech. The determination unit 14 determines whether there is an abnormality in the driver associated with the onset of oral automatism, based on the inference results of the machine learning model ML. Even if oral automatism is inferred in the driver, if the symptoms subside in a short time, it is determined that there is no impairment to driving operations. For this reason, if the driver's oral automatism does not subside in a short time, the determination unit 14 may determine that there is an abnormality in the driver associated with oral automatism. The determination unit determines the driver's abnormality based on the inference results of the machine learning model ML and generates driver abnormality data.

[0027] The notification unit 15 outputs pre-recorded audio to the speaker SP based on driver abnormality data from the judgment unit 14. For example, it plays a warning sound indicating a driver abnormality and outputs it from the speaker SP. The notification unit 15 also provides information to the driver or passenger by displaying warning displays and warning messages based on the driver abnormality data on the display monitor DM. As a warning display, a yellow or red warning mark indicating a warning may be displayed on the display monitor DM. As a warning message, a pre-recorded message urging the driver or passenger to pay attention may be displayed on the display monitor DM. Alternatively, as guidance notification to the driver, an explanatory text encouraging an evacuation action such as slowing down the vehicle and moving to a designated location may be displayed on the display monitor DM.

[0028] The determination unit 14 works in conjunction with the vehicle stopping means SV and decelerates and stops the vehicle based on driver abnormality data. The vehicle stopping means SV may be an accelerator ECU or brake ECU mounted on the vehicle. In addition to decelerating and stopping the vehicle, if the vehicle needs to take evasive action, the vehicle stopping means SV may automatically steer the vehicle to move it to a safe location.

[0029] The determination unit 14 is linked to the release means CS, and the EDSS function of the determination unit 14 is deactivated at the driver's initiative. The release means CS is composed of a physical switch such as an emergency switch or a hazard switch. Alternatively, a software switch or a sensing device may be used as the release means CS. Furthermore, a predetermined cancellation time may be set using a timer or the like as the release means CS. In this case, even if the determination unit 14 determines that the driver is abnormal due to a driver automatism, the output of driver abnormality data from the determination unit 14 may be stopped until the cancellation time has elapsed.

[0030] [2] Second embodiment Next, the configuration and functions of the driver assistance system 100 according to the second embodiment of the present invention will be described. Figure 2 is a block diagram showing the configuration of the driver assistance system 100. In Figure 2, the same reference numerals are used for components that are the same as those in Figure 1. That is, the driver assistance system 100 comprises an image acquisition unit 11 that works in conjunction with an in-vehicle camera CM that captures an image of the driver D's face, and a voice acquisition unit 12 that works in conjunction with a microphone MC that picks up the driver D's voice.

[0031] The driver assistance system 100 implements an HMI (Human Machine Interface) 130 and EDSS 140 to realize EDSS functionality in the in-vehicle system. In addition, the driver assistance system 100 is equipped with an oral automatism detection model AM that works in conjunction with EDSS 140 to detect oral automatisms in driver D and determine abnormalities in driver D.

[0032] HMI130 is an in-vehicle HMI that has been modified and adjusted to comply with EDSS140. An in-vehicle HMI is an interface for smooth command, operation, and information exchange between the driver (D) and the in-vehicle system, and includes, for example, the touch panel of a car navigation system, a head-up display (HUD), sensing devices, meters, and controls. The in-vehicle HMI enables gesture recognition of the driver (D), integration with terminal devices such as smartphones, and voice recognition, making it easier to transmit the driver's intentions regarding vehicle control and driving operations to the in-vehicle system.

[0033] The image acquisition unit 11 acquires video footage of driver D's face captured by the in-vehicle camera CM. In order to detect driver D's epileptic symptoms, the image acquisition unit 11 acquires video footage of driver D's face from the high-resolution in-vehicle camera CM and generates driver D's face image data. As driver D's face image data, a video capturing changes in driver D's facial expression, or multiple images capturing changes in driver D's facial expression, may be used.

[0034] The voice acquisition unit 12 picks up the voice of driver D using a microphone MC installed in the vehicle. The microphone MC may be installed in the car navigation system installed in the vehicle. Alternatively, a separate voice input device may be provided and wirelessly communicated with the voice acquisition unit 12 to collect the voice of driver D. The voice acquisition unit 12 outputs the voice data of driver D.

[0035] The HMI 130 facilitates driver D's input to the EDSS 140. In the driver assistance system 100, the HMI 130 is configured, for example, with a touch panel display, and displays menus related to multiple setting inputs. In addition to various menus for setting the vehicle status, the touch panel display may also display menus related to the detection of epileptic seizures.

[0036] The EDSS140 incorporates a driver abnormality detection unit 150 and a vehicle control unit 160. The driver abnormality detection unit 150 receives facial image data and voice data of driver D and determines whether or not there is an abnormality in the driver. The driver assistance system 100 focuses particularly on oral automatism as an epileptic seizure of driver D. For this reason, the driver abnormality detection unit 150 works in conjunction with the epileptic seizure detection unit 170 in addition to the oral automatism detection model AM.

[0037] The epileptic seizure detection unit 170 determines whether or not driver D has had an epileptic seizure. It determines that an epileptic seizure has occurred if driver D continues to exhibit unnatural behavior while driving the vehicle. For example, if the facial image data of driver D reveals a state of physical rigidity, unnatural movements of the limbs or face, or unnatural changes in facial expression, it determines that driver D has had an epileptic seizure. The epileptic seizure detection unit 170 has driver status information related to the driver's unnatural behavior pre-recorded in a memory device, and it reads the driver status from the facial image data of driver D to determine whether or not driver D has exhibited unnatural behavior.

[0038] The epileptic seizure detection unit 170 determines whether or not driver D has had an epileptic seizure, but it cannot necessarily determine whether or not driver D has had a complex partial seizure. Complex partial seizures are accompanied by small external displacements of driver D. Therefore, the driver assistance system 100 determines whether or not driver D has had a complex partial seizure by accurately detecting small external displacements of driver D. In particular, the driver assistance system 100 uses the oral automatism detection model AM to determine whether or not oral automatism is present as an example of a complex partial seizure.

[0039] The oral automatism detection model AM is created in advance at a research and development facility such as a data center outside the vehicle. The oral automatism detection model AM is designed to infer the presence or absence of oral automatism by performing machine learning such as ViViT on facial video data of multiple drivers or individual drivers. The driver abnormality detection unit 150 applies the facial image data and voice data of driver D to the oral automatism detection model AM to determine whether or not driver D is experiencing oral automatism. For example, if a change in the mouth in the facial image data is observed during a time when there is no voice data indicating driver D's speech, or if the voice data of driver D and the change in the mouth shown in the facial image data are not synchronized, the oral automatism detection model AM determines that driver D is experiencing oral automatism. If oral automatism is detected in driver D, the driver abnormality detection unit 150 detects whether or not driver D is abnormal. The signs of oral automatism in driver D do not immediately indicate an abnormality in driver D that impairs driving operations; there is a possibility of misdetecting small displacements of driver D's mouth as oral automatism. Therefore, the driver abnormality detection unit 150 detects a driver abnormality when the likelihood of oral automatism occurring is high. If driver D has set a cancellation time in advance, oral automatism detection is stopped, and the presence or absence of a driver abnormality is detected after the cancellation time has elapsed.

[0040] A known object detection method called YOLO (You Only Look Once) detects the region of an object from various images and identifies the type of object. Recently, it has become possible to extract regions with a high probability of containing an object from an image using deep neural networks, and a confidence score that indicates the distinction of an object from the background has been adopted. In the oral automatism judgment model AM, a confidence score is calculated to distinguish and identify the movement of driver D's mouth from the background. For example, a confidence score consisting of a decimal number in the range of 0 to 1 is calculated as the correlation between specific behaviors related to oral automatism and the movement of driver D's mouth. The closer the confidence score is to "1", the higher the probability that the movement of driver D's mouth is a sign of oral automatism. A predetermined threshold may also be set for the presence or absence of oral automatism. If the confidence score calculated for the movement of driver D's mouth is above the threshold, it may be determined that driver D is experiencing oral automatism.

[0041] In the driver assistance system 100, the presence or absence of an epileptic seizure in driver D is determined by the epileptic seizure determination unit 170, which is linked with the EDSS 140, and the presence or absence of oral automatism is determined by the oral automatism determination model AM. Furthermore, if the mouth of driver D's face makes unnatural movements during a period when the voice data does not include the driver's speech, it may be determined that there is a high probability that driver D is experiencing oral automatism.

[0042] If a cancellation time is set in the driver abnormality detection unit 150, an abnormality in driver D is determined after the cancellation time has elapsed, based on the determination of driver D's oral automatism. When an abnormality in driver D is determined, there is a risk that the driver will become unable to drive, so the vehicle control unit 160 controls the vehicle to steer to a predetermined area or decelerate. The vehicle control unit 160 may be a drive force control ECU mounted on the vehicle, such as an accelerator ECU or a brake ECU. The vehicle control unit 160 may also notify the outside of the vehicle that driver D has had an epileptic seizure and oral automatism has been detected. If the vehicle is equipped with a car navigation system with an EDSS function, a message may be displayed on the display indicating that driver D has early-stage mild elderly epilepsy. For example, a warning message may be displayed on the display indicating that driver D may have difficulty operating the vehicle due to an epileptic seizure, or advice may be given to driver D to seek medical attention.

[0043] Next, the processing of the driver assistance system 100 will be described. Figure 3 is a flowchart of the processing of the driver assistance system 100. The flowchart in Figure 3 consists of steps S10 to S140, where steps S80 to S110 represent oral automatism determination OA. Steps S120 to S140 can be implemented using an existing driver abnormality response system.

[0044] In the flowchart shown in Figure 3, step S10 corresponds to the processing between the in-vehicle camera CM and the image acquisition unit 11, where a facial image of driver D is acquired from the in-vehicle camera CM and facial image data is generated. Step S20 corresponds to the processing of the HMI 130, where the cancellation time t is set by driver D to disable the EDSS function. Step S50 corresponds to the processing between the microphone MC and the voice acquisition unit 12, where the voice of driver D is acquired by the microphone MC.

[0045] After step S20, when the flow moves to step S30, driver D can set a cancellation time t arbitrarily by selecting the EDSS function menu on the touch panel display. The unit of the cancellation time may be set to minutes or seconds. For example, the EDSS function may be stopped by pre-setting a cancellation time t of several minutes or tens of seconds, which is the time it takes for the driver to consciously move their mouth. Alternatively, the EDSS function may be emergency stopped by a switch installed in the vehicle. In this case, the EDSS function may be forcibly stopped for only the preset time of cancellation time t, for example, 3 minutes. Once the cancellation time t is set in the timer and the timer starts operating, the cancellation time t decreases in seconds (t=t-1), and when the cancellation time t becomes zero (t=0), the EDSS function is released (steps S30, S40).

[0046] In step S50, since the audio acquired by the microphone MC contains noise other than the speech of driver D, noise cancellation may be performed to remove the noise and filter out only the speech of driver D (step S60).

[0047] When the cancellation time t ends (step S40, YES), the flow moves to step S70. In step S70, it is determined whether or not driver D has spoken based on the audio data after noise cancellation (step S60). Here, in order to determine whether or not driver D's speech is synchronized with facial image data (facial feature data), it may also be determined whether or not the current time is within the period in which driver D's speech can be detected. If driver D's speech is not detected in step S70 (step S70, YES), the flow proceeds to oral automatism determination OA. If driver D's speech is detected (step S70, NO), it is presumed that driver D's mouth movements were synchronized with the driver's speech, so oral automatism determination OA is not performed.

[0048] In the oral automatism detection OA, the driver assistance system 100 extracts an image of the lower half of driver D's face from the driver D's facial image data and adjusts the image size (step S80). This allows the system to extract facial feature data showing the displacement of feature points of driver D's mouth from the image data acquired by the in-vehicle camera CM. The facial feature data is input into a trained model such as the oral automatism detection model AM (step S90). The trained model calculates a confidence score indicating the likelihood that the movement of driver D's mouth included in the facial feature data is a sign of oral automatism (step S100). The confidence score may be expressed as a decimal number between 0 and 1.

[0049] In the oral automatism detection OA, a threshold for the confidence score required to determine whether driver D's mouth movements constitute oral automatism is pre-set. For example, a support vector machine (SVM) may be set up to perform a mechanical classification of whether or not oral automatism is present by taking probabilistic statistics on the correlation between multiple facial images and oral automatism. Alternatively, a numerical value corresponding to the boundary that can distinguish oral automatism from other mouth movements may be set as the confidence score threshold. In step S110, if the confidence score is above the threshold, the facial feature data is determined to indicate oral automatism.

[0050] If the confidence score is below the threshold and driver D's mouth movements cannot be identified as oral automatism (step S110, NO), the flow in Figure 3 ends. If the confidence score is above the threshold and driver D's oral automatism is detected (step S110, YES), the flow proceeds to step S120. In step S120, the driver assistance system 100 performs an EDSS abnormality determination following the detection of driver D's oral automatism. It is anticipated that driver D may begin speaking due to the time required for the oral automatism determination OA. For example, even if driver D moves their mouth silently and is determined to have oral automatism, driver D may then begin speaking. Taking into account such a time lag in driver D's speech, the driver assistance system 100 performs an EDSS abnormality determination (step S120) if driver D's speech is not detected even after the oral automatism determination OA (step S70, YES). Here, a certain time buffer of a few seconds may be allowed before the EDSS abnormality detection begins. That is, if driver D's oral automatism is detected and driver D's speech is not detected, and if driver D's oral automatism is likely to interfere with driving operations, the driver assistance system 100 will determine that driver D has an abnormality.

[0051] If an abnormality in driver D is detected, the driver assistance system 100 executes EDSS vehicle control, automatically controlling the vehicle's accelerator, brakes, steering wheel, etc., to decelerate the vehicle or take evasive action (step S130).

[0052] If the EDSS function is to be temporarily deactivated depending on the driver's condition, if the driver D's conscious mouth movements are to be excluded from the EDSS abnormality judgment, an emergency button may be provided to eliminate false detection or misjudgment of mouth automatism. The emergency button may also be used in conjunction with the vehicle's hazard switch. If the driver assistance system 100 is equipped with a hazard switch to deactivate the EDSS, when the driver D or a passenger operates the hazard switch, the EDSS abnormality judgment in step S120 and the EDSS vehicle control in step S130 are stopped (step S140).

[0053] As described above, the driver assistance system 100 adds a driver D oral automatism detection OA to the EDSS function equipped in the vehicle, and also provides a means to deactivate the EDSS function (such as a hazard switch). In particular, the conditions for activating the EDSS abnormality detection (step S120) are: (a) determination that driver D is experiencing oral automatism, (b) absence of speech from driver D, and (c) failure of the deactivation means such as a hazard switch. Furthermore, in order to prevent the EDSS function from being activated unexpectedly against driver D's intention, the system includes: (d) prevention of false detection or false determination of driver D's oral automatism, (e) setting of the duration by driver D, and (f) deactivation of the EDSS function by driver D.

[0054] Next, the features and effects of embodiments of the present invention will be described. (1) The driving assistance system (10) comprises an image acquisition unit (11) that successively acquires multiple images including the face of the driver operating the vehicle; an audio acquisition unit (12) that acquires voices emitted by the driver; a detection unit (13) that extracts multiple feature points from the face portion shown in the multiple images and detects the displacement of the multiple feature points; a determination unit (14) that, when no voice from the driver is detected in synchronization with the displacement of the multiple feature points, uses a machine learning model (ML) that has learned the displacement of the multiple feature points to determine whether or not a specific behavior of the driver's mouth that would interfere with the driver's driving operation has occurred; and a notification unit (15) that notifies the determination result of the determination unit (14).

[0055] The image acquisition unit (11) acquires multiple images or videos of the driver captured by the in-vehicle camera (CM) and generates face image data. The detection unit (13) performs meshing on the driver's face image data and extracts multiple feature points corresponding to the mesh intersections. As the driver's facial expression changes, particularly the movement of the mouth, the mesh itself is displaced, and the relative positions and distances of the multiple feature points corresponding to the mesh intersections are also displaced. Therefore, by detecting the displacement of the multiple feature points extracted from the driver's face image, the detection unit (13) can detect changes in the driver's facial expression, particularly the movement of the mouth.

[0056] The determination unit (14) implements the EDSS function described above. That is, when the driver's voice is not detected in synchronization with the displacement of multiple feature points on the driver's face, it means that only the mouth of the driver's face is moving, even though the driver is not speaking. The determination unit (14) uses a machine learning model (ML) such as ViViT to learn the displacement of multiple feature points and determines the specific behavior of the driver's mouth. Subsequently, the determination unit (14) determines whether the displacement of multiple feature points on the driver's face corresponds to a specific behavior of the mouth that interferes with the driver's driving. The specific behavior of the driver's mouth means that the driver is unconsciously moving only their mouth silently, and indicates an unconscious state that the driver temporarily falls into, or a situation in which the driver is distracted by other things, such as thinking about something, and cannot concentrate on driving. When movement of the driver's mouth is detected during a period when there is no driver voice, the determination unit (14) determines that the specific behavior of the driver's mouth has occurred.

[0057] When the determination unit (14) determines that a specific behavior of the driver's mouth has occurred, the notification unit (15) notifies the driver of a warning or alerts the driver because there is a high possibility that it will interfere with the driver's driving operations. In this case, the driver may be notified by playing a predetermined message through the speaker SP. Alternatively, the warning or predetermined message may be displayed on the display monitor DM.

[0058] The determination unit (14) is provided with a release mechanism (CS). The driver can voluntarily deactivate or deactivate the EDSS function by operating the release mechanism (CS), such as a stop button or an emergency switch. Specific movements of the driver's mouth do not necessarily cause interference with driving operations. Therefore, when the driver consciously moves their mouth, such as when chewing gum, the driver may activate the release mechanism (CS) at their own initiative to prevent the driver from misinterpreting specific movements of the driver's mouth as abnormal driving operations. This prevents the determination unit (14) from mistakenly detecting an abnormality that interferes with driving operations due to specific movements of the driver's mouth.

[0059] (2) The machine learning model (ML) has learned in advance the displacement of a predetermined feature point of a face image in relation to the specific behavior of the driver's mouth, and the determination unit (14) may determine whether or not the specific behavior of the driver's mouth has occurred based on the inference result obtained by inputting the displacement of multiple feature points of multiple images of the driver into the machine learning model (ML).

[0060] Multiple images may be obtained in advance from multiple drivers, each facial image may be meshed to extract multiple feature points, and in particular, multiple feature points corresponding to the mesh intersections in the driver's mouth area may be used to create a machine learning model (ML). In this case, the displacement of feature points in the facial image of a given driver will be machine-learned, but the machine learning model (ML) may also be trained on the displacement of feature points in the facial image of a specific driver operating the vehicle. This improves the accuracy of the machine learning model (ML). The machine learning model (ML) infers that a specific behavior of the driver's mouth occurred from the displacement of feature points in the driver's facial image.

[0061] (3) The determination unit (14) may calculate a confidence level indicating the likelihood of the occurrence of a specific behavior of the driver's mouth based on the displacement of multiple feature points of multiple images of the driver obtained by a machine learning model (ML) and the displacement of a predetermined facial image feature point obtained in advance in relation to the specific behavior of the driver's mouth, and determine that a specific behavior of the driver's mouth that interferes with the driver's driving has occurred when the confidence level is above a predetermined threshold.

[0062] A confidence level indicating the likelihood of a specific behavior of the driver's mouth occurring is calculated based on the correlation between the displacement of feature points in an image of the driver's face and the displacement of feature points in a predetermined face image corresponding to the movement of the driver's mouth. For example, the determination unit (14) considers the displacement of the mouth region in an image of a predetermined person who exhibited a specific behavior of the mouth and estimates the probability that the displacement of the mouth region in an image of a driver's face can be estimated to be a specific behavior, and calculates this probability as the confidence level. In other words, the higher the probability that the displacement of the mouth region in an image of a driver can be estimated to be a specific behavior, the higher the confidence level. Alternatively, a threshold for the confidence level may be set as the boundary between whether or not the specific behavior of the driver's mouth causes an abnormality in driving operation. In this case, if the confidence level is above the threshold, the determination unit (14) determines that a specific behavior of the mouth that interferes with the driver's driving operation has occurred. This prevents the driver's specific behavior of the mouth from being immediately judged as an abnormality that interferes with driving operation based on a low confidence level.

[0063] (4) The system may also be equipped with a release mechanism (CS) that excludes other behaviors different from the specific behavior of the driver's mouth from the determination unit (14)'s determination targets. There are cases where a driver, while silent, consciously performs an action that may be suspected of being a specific behavior of the mouth, such as chewing gum. In this case, the driver can voluntarily activate the release mechanism (CS) to prevent the EDSS function from being activated. This prevents the determination unit (14) from determining a specific behavior of the mouth as an abnormality that interferes with driving operations, regardless of the driver's state, awareness, and the environment around the vehicle, simply because the driver consciously moves their mouth.

[0064] (5) The system is equipped with setting means (130) for setting the duration of a specific behavior of the driver's mouth, and the determination unit (14) may determine that a specific behavior of the driver's mouth has occurred when the specific behavior of the driver's mouth continues for a longer period than the duration set in the setting means. The setting means (130) is, for example, a timer that the driver can set at will. The setting means (130) may also be a form of the release means (CS). If the time period in which the driver consciously performs an action that is suspected to be a specific behavior of the mouth is known in advance, the determination unit (14) may set a predetermined duration as a condition for stopping the EDSS function, and during that duration, the abnormality determination associated with the specific behavior of the driver's mouth may be stopped. This makes it possible to identify in advance the time period in which the driver's mouth movements continue and stop the determination of specific behavior of the mouth. Furthermore, by setting a duration, it is possible to prevent false detections in which the driver stops moving their mouth for a short time and this is determined to be a specific behavior of the mouth.

[0065] (6) The vehicle is equipped with a vehicle stopping means (SV) that stops or moves the vehicle according to the determination result of the determination unit (14), and when the specific behavior of the driver's mouth continues for longer than the duration set in the setting means (130), the determination unit (14) may drive the vehicle stopping means (SV). If the specific behavior of the driver's mouth continues for longer than a predetermined duration, the vehicle stopping means (SV) may automatically perform accelerator or brake operations to decelerate the vehicle or perform a vehicle evasive action. If the specific behavior of the driver's mouth continues for longer than the duration, there is a possibility that the driver has lost consciousness or is unable to drive due to epileptic symptoms or the like, so the vehicle stopping means (SV) may activate the EDSS function to decelerate the vehicle and perform an evasive action.

[0066] (7) The specific behavior of the driver's mouth may be a specific symptom exhibited by the driver. The specific symptom may be, in particular, oral automatism among epileptic symptoms. When a driver develops an epileptic symptom, it is expected that the driver may become unable to drive due to the physical restraint of the driver, or the driver may lose consciousness. Unnatural movements of the driver's mouth, which do not show any outward changes, are difficult to distinguish from epileptic symptoms. However, since the driver assistance system (10) can determine whether or not the driver has oral automatism, the EDSS function can be activated before the driver becomes unable to drive, thereby supporting the driver's safe driving.

[0067] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0068] 10,100 Driver Assistance Systems 11 Image acquisition unit 12. Audio acquisition unit 13 Detection unit 14 Judgment section 15 Hochi Department 130 HMI 140 EDSS 150 Driver abnormality detection unit 160 Vehicle Control Unit 170 Epilepsy Seizure Assessment Unit AM Oral Automatism Diagnosis Model CM In-car camera CS release method D Driver DM Display Monitor MC Microphone ML Machine Learning Models SP Speaker SV Vehicle Stopping Method

Claims

1. An image acquisition unit that sequentially acquires multiple images including the face of the driver operating the vehicle, A voice acquisition unit that acquires voices emitted by the aforementioned driver, A detection unit extracts multiple feature points from the face portion shown in the multiple images and detects the displacement of the multiple feature points, When the voice from the driver is not detected in synchronization with the displacement of the plurality of feature points, a determination unit determines whether or not a specific behavior of the driver's mouth that interferes with the driver's driving operations has occurred, using a machine learning model that has learned the displacement of the plurality of feature points. A notification unit that notifies the determination result of the determination unit, A driver assistance system characterized by being equipped with the following.

2. The driving assistance system according to claim 1, characterized in that the machine learning model has learned in advance the displacement of a predetermined feature point of a facial image in relation to the specific behavior of the driver's mouth, and the determination unit determines whether or not the specific behavior of the driver's mouth has occurred according to the inference result obtained by inputting the displacement of the plurality of feature points of the plurality of images of the driver into the machine learning model.

3. The driver assistance system according to claim 1, characterized in that the determination unit calculates a confidence level indicating the likelihood of the occurrence of the specific behavior of the driver's mouth based on the displacement of the plurality of feature points of the plurality of images of the driver obtained by the machine learning model and the displacement of a predetermined face image feature point that has been determined in advance in relation to the specific behavior of the driver's mouth, and determines that the specific behavior of the driver's mouth that interferes with the driver's driving operation has occurred when the confidence level is above a predetermined threshold.

4. The driving support system according to claim 1, further comprising a release means for excluding other behaviors of the driver's mouth that are different from the specific behavior of the mouth from the determination unit's determination target.

5. The system includes setting means for setting the duration for which the specific behavior of the driver's mouth continues, The driving support system according to claim 1, characterized in that the determination unit determines that the specific behavior of the driver's mouth has occurred when the specific behavior of the driver's mouth continues for a longer period than the duration set in the setting means.

6. The system includes a vehicle stopping means for stopping or moving the vehicle according to the determination result of the determination unit, The driving support system according to claim 5, characterized in that when the specific behavior of the driver's mouth continues for a longer period than the duration set in the setting means, the determination unit drives the vehicle stopping means.

7. The driving assistance system according to any one of claims 1 to 6, characterized in that the specific behavior of the driver's mouth is a specific symptom that manifests in the driver.

Images