System and method for detecting microsleep-related behavior of a driver while driving

Abstract

By analyzing the process of drowsy driving in actual driving situations, we can detect microsleep in drivers while driving, which occurs before an accident occurs due to drowsy driving, and provide safety measures based on that detection. [Solution] Based on images of the driver while driving taken by an in-vehicle camera 3, the driver's anti-sleepiness behavior and signs of microsleep behavior are detected, and based on images of the vehicle while driving taken by an exterior camera 5, abnormal vehicle behavior is detected. The increase rate of the detected detection results of the driver's anti-sleepiness behavior, signs of microsleep behavior or abnormal vehicle behavior is measured, and if the increase rate is in a predetermined relationship, it is determined that the driver is at risk of causing a collision accident while driving.

Description

[Technical field]

[0001] The present invention relates to a system and method for detecting microsleep-related behavior that simultaneously evaluates a driver's entire body and a vehicle's driving behavior. [Background technology]

[0002] Truck traffic accidents are a major social issue that cause both human and property damage, with drowsy driving being a key factor. According to the National Police Agency, drowsy driving is treated as a type of distracted driving, which is a violation of the Road Traffic Act, and distracted driving has been ranked first in the number of fatal accidents by legal violation for 11 consecutive years from 2012 to 2022, and many studies have emphasized its seriousness and risk of fatality. Collisions caused by drowsy driving are a global problem that has yet to be resolved.

[0003] Microsleeps, also known as momentary sleepiness, are short, irresistible periods of excessive sleepiness. Microsleeps while driving increase the likelihood of drowsiness-related crashes. Microsleeps caused by excessive sleepiness have been objectively measured using electroencephalography data, sleep-related behaviors, and performance level assessments. Behavioral assessments of microsleeps to date have focused on eye closure, changes in pupil diameter, and performance during driving simulations and psychomotor vigilance tests. Recent studies have also attempted to detect driver drowsiness by using deep learning models to evaluate various behavioral features in video samples.

[0004] However, most of these studies and technologies evaluate the driver or vehicle behavior individually. As collision prevention measures, technologies that detect the driver's eye state, driving assistance technologies (such as automatic cruising functions), and vehicle control technologies (such as lane departure prevention functions) are used, but collisions cannot be completely prevented. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2013-257691 A [Patent Document 2] Patent Publication No. 2022-89223 [Non-patent literature]

[0006] [Non-Patent Document 1] Celecia, A., Figueiredo, K., Vellasco, M., Gonz´alez, R., 2020. A portable fuzzy driver drowsiness estimation system. Sensors (Basel) 20(15), 4093. [Non-Patent Document 2] Cicchino, JB, 2018. Effects of lane departure warning on police-reported crash rates. J. Saf. Res. 66, 61-70. [Non-Patent Document 3] Malla, AM, Davidson, PR, Bones, PJ, Green, R., Jones, RD, 2010. Automated video-based measurement of eye closure for detecting behavioral microsleep. Annu Int Conf IEEE Eng Med Biol Summary of the Invention [Problem to be solved by the invention]

[0007] Previous research into microsleep has focused on evaluating the driver's eye and mouth movements, and no research has been conducted to simultaneously evaluate the whole body and vehicle driving behavior. In addition, physiological evaluation of the driver is difficult in actual driving situations, and evaluations using simulations may not reflect the effects of actual driving situations.

[0008] Therefore, our goal is to analyze the process of drowsy driving in actual driving situations, thereby enabling early detection of microsleep in drivers while driving, which occurs before an accident occurs due to drowsy driving, and to provide safety measures based on this detection. [Means for solving the problem]

[0009] In order to solve the above problems, the inventors have discovered a method for detecting microsleep early or immediately before it occurs by using footage recorded inside and outside the vehicle by an onboard camera to evaluate microsleep-related behavior from three perspectives: anti-sleepiness behavior of the driver while driving up to a collision accident, signs of microsleep behavior, and abnormal vehicle behavior while driving, and have completed the present invention, which is completely different from conventional evaluations that are limited to the driver's eye and mouth movements. That is, the present invention has been made to solve the above-mentioned problems, and embodiments of the present invention may include at least some of the configurations described below.

[0010] [1] An in-vehicle imaging means capable of photographing a driver while driving; A recording means capable of recording the voice of a driver while driving; An outside image capturing means capable of capturing an image of a vehicle while it is being driven; A speed detection means for detecting a speed of a vehicle being driven; A control means 1 connected to the in-vehicle imaging means; A control means 2 connected to the speed detection means and the outside-of-vehicle image capturing means; An analysis means for analyzing the detection results of the control means 1 and / or the control means 2; A determination program for determining microsleep-related behavior; Equipped with The control means 1 Detecting anti-sleepiness behavior of the driver while driving and a symptom of microsleep behavior of the driver while driving based on an image of the driver while driving captured by the in-vehicle imaging means; The control means 2 Detecting the vehicle behavior abnormality based on an image of the vehicle being driven captured by the vehicle exterior image capturing means; The analysis means detects anti-drowsiness behavior of the driver while driving. Number of detections , the driver's microsleep behavioral manifestation while driving Number of detections, and The number of detections of abnormal vehicle behavior 、 Measure the rate of increase over time, When the respective increase rates are in a predetermined relationship, the driver is likely to cause a collision accident. The judgment program A system for detecting microsleep-related behavior in a driver while driving, characterized by determining whether the driver is able to drive or not. [2] An in-vehicle imaging means capable of imaging a driver while driving; A recording means capable of recording the voice of a driver while driving; An outside image capturing means capable of capturing an image of a vehicle while it is being driven; A speed detection means for detecting a speed of a vehicle being driven; A control means 1 connected to the in-vehicle imaging means; A control means 2 connected to the speed detection means and the outside-of-vehicle image capturing means; An analysis means for analyzing the detection results of the control means 1 and / or the control means 2; A determination program for determining microsleep-related behavior; A method for detecting microsleep-related behavior, comprising: The control means 1 Detecting anti-sleepiness behavior of the driver while driving and a symptom of microsleep behavior of the driver while driving based on an image of the driver while driving captured by an in-vehicle imaging means capable of capturing an image of the driver while driving; The control means 2 Detecting the abnormal vehicle behavior based on an image of the vehicle being driven captured by an exterior image capturing means capable of capturing an image of the vehicle being driven; The analysis means Detected anti-drowsiness behavior of the driver while driving Number of detections , the driver's microsleep behavioral manifestation while driving Number of detections, and The number of detections of abnormal vehicle behavior 、 Measure the rate of increase over time, When the respective increase rates are in a predetermined relationship, the driver is likely to cause a collision accident. The judgment program A method for detecting microsleep-related behavior of a driver while driving, comprising: [3] When the predetermined relationship is such that, in the presence of the detection number of microsleep behavioral signs and / or vehicle behavior abnormalities of the driver while driving, the increase rate of the detection number of the anti-sleepiness behavior while driving is higher than the increase rate of the detection number of the microsleep behavioral signs and / or vehicle behavior abnormalities, when the increase rate of the detection number of the anti-sleepiness behavior of the driver while driving is 1.5 times or more, the driver is likely to cause a collision accident on a general road. The judgment program [2] A method for detecting microsleep-related behavior of a driver while driving, characterized by determining whether the driver is experiencing microsleep-related behavior. [4] When the predetermined relationship is such that, in the presence of the detection number of microsleep behavioral signs and / or vehicle behavior abnormalities of the driver while driving, the increase rate of the detection number of the anti-sleepiness behavior while driving is higher than the increase rate of the detection number of the microsleep behavioral signs and / or vehicle behavior abnormalities, when the increase rate of the detection number of the anti-sleepiness behavior of the driver while driving turns negative, the driver is likely to cause a collision accident on a highway. The judgment program [2] A method for detecting microsleep-related behavior of a driver while driving, characterized by determining whether the driver is experiencing microsleep-related behavior. [5] When the predetermined relationship is such that, in the presence of the driver's microsleep behavioral signs and / or the number of detections of vehicle behavior abnormalities while driving, the increase rate of the number of detections of anti-sleepiness behavior is positive and the increase rate of the number of detections of vehicle behavior abnormalities is negative, the driver is likely to cause a collision accident. The judgment program A method for detecting microsleep-related behavior of a driver while driving as described in [2], characterized by comprising a step of determining. [6] When the predetermined relationship is such that, in the presence of the driver's microsleep behavioral signs and / or the number of detections of vehicle behavior abnormalities, the increase rate of the number of detections of anti-drowsiness behavior is positive and the increase rate of the number of detections of vehicle behavior abnormalities is also positive, the driver is likely to cause a collision accident. The judgment program A method for detecting microsleep-related behavior of a driver while driving as described in [2], characterized by comprising a step of determining. [7] When the predetermined relationship is such that, in the presence of the detection number of microsleep behavioral signs and / or abnormal vehicle behavior of the driver while driving, the increase rate of the detection number of anti-sleepiness behavior is negative and the increase rate of the detection number of microsleep behavioral signs is positive, the driver is likely to cause a collision accident. The judgment program A method for detecting microsleep-related behavior of a driver while driving as described in [2], characterized by comprising a step of determining. [8] The judgment program To drivers who are driving on expressways that are judged to have the potential to cause a collision 、 [1] A system for detecting microsleep-related behavior in drivers while driving, characterized by warning of a collision with an obstacle in the left front direction of the vehicle while driving. [9] A system for detecting microsleep-related behavior of a driver while driving as described in [1], further comprising a device for detecting obstacles in the left front direction of the vehicle. Effect of the Invention

[0011] According to the system and method for detecting microsleep-related behavior of a driver while driving according to the present invention, it is possible to easily detect when anti-sleepiness behavior, which is a precursor to a drowsy state caused by frequent microsleep, changes from an increase to a decrease in anti-sleepiness behavior, which is a precursor to a drowsy state caused by frequent microsleep, on general roads or expressways, from the process of changes in driver behavior and vehicle behavior, which are classified as anti-sleepiness behavior, signs of microsleep behavior, and abnormal vehicle behavior. [Brief description of the drawings]

[0012] [Figure 1] Block diagram showing the system [Diagram 2] Imaging the situation leading up to a collision [Diagram 3] Flow diagram for detecting microsleep-related behavior [Figure 4] Flow diagram for detecting microsleep-related behavior on public roads [Diagram 5] Flow diagram for detecting microsleep-related behavior on highways [Figure 6]Types of microsleep-related behaviors by type on public roads [Figure 7] Types of microsleep-related behaviors by type on highways [Figure 8] Graph of time series data of microsleep-related behavior [Figure 9] Conceptual diagram of drowsy driving accidents on expressways and microsleep-related behavior [Figure 10] Flow diagram for detecting microsleep-related behavior five minutes before a collision [Figure 11] Graph showing time series data of microsleep-related behavior from 5 minutes before the collision to the collision DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the drawings.

[0014] <Microsleep-related behavior detection system D> As shown in FIG. 1, a microsleep-related behavior detection system D is provided in a vehicle 10 to detect microsleep-related behavior.

[0015] The microsleep-related behavior detection system D may be, for example, an optional part that is retrofitted to the vehicle 10, or may be pre-installed in the vehicle 10.

[0016] Preferably, the microsleep-related behavior detection system D is an optional part that is retrofitted to the vehicle 10. More preferably, the microsleep-related behavior detection system D is a retrofit drive recorder as an optional part.

[0017] The microsleep-related behavior detection system D includes an in-vehicle camera 3 as an in-vehicle imaging means capable of capturing an image of the driver of the vehicle 10, a control means 1 connected to the in-vehicle camera 3, a control unit 6, and an alarm device 7 as an alarm means connected to the control unit 6 and warning the driver of the vehicle 10 about falling asleep while driving. The alarm device 7 may be a smartphone carried by the driver. The in-vehicle camera 3 is preferably a camera capable of capturing an image of the entire body of the driver.

[0018] The in-vehicle camera 3 is not particularly limited as long as it can capture the entire body of the driver of the vehicle 10. If the entire body can be captured, it is possible to follow every posture of the driver and grasp information such as the driver's overall appearance, posture, and hand movements. Examples of such cameras include well-known small cameras such as stereo cameras, monocular cameras, and optical cameras. A camera attached to a drive recorder may also be used.

[0019] In particular, since it is said that drowsiness can be detected from blinking and eye movements, it is preferable to be able to capture the driver's facial movements, line of sight, and degree of eye opening in addition to the entire body of the driver, and therefore an in-vehicle camera 3' may be provided in addition to the in-vehicle camera 3. Examples of cameras include spectroscopic cameras and infrared cameras, but infrared cameras are preferable because they can respond to changes in light conditions and can provide clear images even when driving at night. There are no particular limitations on the microphone 4 as long as it can record the driver's voice. Examples of the microphone 4 include well-known microphones. The in-vehicle camera 3 and the microphone 4 are installed, for example, on the dashboard of the vehicle 10 and are arranged facing the driver. It is preferable to install them on the dashboard on the passenger seat side. This allows the in-vehicle camera 3 to capture the movements of the entire body of the driver, including the face (see arrow a in FIG. 1). If the in-vehicle camera 3 is installed directly in front of the driver, such as on the steering wheel, the three-dimensional effect and depth are lost, making it difficult to confirm the movements of the entire body of the driver. On the other hand, if the in-vehicle camera 3 is installed on the dashboard on the passenger side, the driver can be photographed diagonally upward, and the photographing range is expanded, so that the whole body image of the driver can be confirmed from a bird's-eye view. Note that the in-vehicle camera 3 and microphone 4 may be installed above the driver's seat of the vehicle 10, as long as the whole body image of the driver can be photographed.

[0020] When an in-vehicle camera 3' is also installed in addition to the in-vehicle camera 3, the in-vehicle camera 3' is preferably placed in a position where it can capture images of the driver's facial movements and the degree of eye opening. The position is not particularly limited as long as it can capture the driver's eye and facial movements, and may be set, for example, on the rearview mirror, the top of the windshield, or the center visor of the instrument panel of the vehicle 10. The in-vehicle camera 3' is intended to check the driver's eye movements and facial expressions, so it may be installed in front of the driver.

[0021] In order to detect the whole body movements of the driver of the vehicle 10, particularly anti-drowsiness behavior and cessation of body movements with higher accuracy, an electromagnetic wave sensor may be used in addition to the in-vehicle camera 3. The electromagnetic wave sensor can detect the body movements of the driver (movement of the head, limbs (arms, legs, hands, knees, elbows)). When using an electromagnetic wave sensor, it is preferable to install it so as to eliminate blind spots of the in-vehicle camera 3, for example, on the upper part of the windshield of the vehicle 10 or on the ceiling above the driver's seat. Whether the in-vehicle camera 3 or the electromagnetic wave sensor is used, both are non-contact systems and do not interfere with the driver's safe driving environment.

[0022] The in-vehicle camera 3 is connected to the control means 1, and the exterior camera 5 and a speed detection means, which will be described later, are connected to the control means 2 via a wired or wireless connection, and the control means 1 and the control means 2 are stored in the control unit 6. This makes it possible to input images of the driver captured by the in-vehicle camera 3 and images of the area in front of the vehicle 10 captured by the exterior camera 5 to the control unit 6. It is preferable that information captured by the in-vehicle camera 3' and the electromagnetic wave sensor is also stored in the control unit 6.

[0023] There are no particular limitations on the type of exterior camera 5, so long as it can capture images of the road ahead in the direction of travel of vehicle 10 and the lanes on the road ahead. For example, a publicly known small camera is used as the exterior camera 5. For example, the exterior camera 5 is disposed above the driver's seat of vehicle 10, facing forward of the windshield. This allows the exterior camera 5 to capture images of the road ahead in the direction of travel of vehicle 10 and the lanes ahead (see arrow b in FIG. 1). The exterior camera 5 may be installed on the dashboard of vehicle 10.

[0024] The control unit 6 is a control unit that executes electrical control of the microsleep-related behavior detection system D, and is composed of a microcomputer equipped with a CPU, ROM, RAM, etc. The control unit 6 has a judgment program P for judging whether the driver is drowsy at the wheel. The judgment program P is not particularly limited and is designed by a known method, and is stored in the control unit 6 together with an analysis means. The method of judging microsleep-related behavior using the judgment program P will be described later.

[0025] The judgment program P may be run taking into account the driving time. For example, in the case of a long-distance truck driver, the accumulated driving time since the start of work or the continuous driving time can be input to predict the occurrence of microsleep-related behavior taking into account the driver's degree of fatigue.

[0026] The control unit 6 may also store AI (artificial intelligence). The AI ​​can analyze the images obtained from the in-vehicle camera 3, microphone 4, and outside camera 5 into three categories: "anti-drowsiness behavior," "signs of microsleep behavior," and "abnormal vehicle behavior," which will be described later, detect microsleep-related behavior, and issue a warning to the driver, instruct the vehicle to make an emergency stop, or forcibly stop the vehicle in a safe area. Although a huge capacity is required to store and analyze the captured images and data in the cloud, it is preferable that the AI ​​installed in the vehicle 10 analyzes the captured images in real time and immediately provides feedback to the driver, since a large-capacity cloud is not required. A hybrid method in which critical real-time processing is performed in the vehicle and large-volume data processing and complex calculations are performed in the cloud may also be used.

[0027] The warning device 7 warns the driver when the driver's microsleep-related behavior is detected. Examples of the warning device 7 include a speaker that warns the driver by voice, a smartphone carried by the driver, and a display that warns the driver by image, and the like, and a speaker is preferable.

[0028] The warning device 7 is connected by wire or wirelessly to the control unit 6. As a result, when the judgment program P of the control unit 6 detects the driver's microsleep-related behavior, the warning device 7 is made operable by the control unit 6.

[0029] The microsleep-related behavior detection system D includes a speed detection means for detecting the speed of the vehicle (not shown). A GPS device is preferable. The GPS device may be a commercially available one. Preferably, the GPS device is built into the drive recorder. For example, the system includes a map storage unit for acquiring arbitrary map information, a GPS sensor for acquiring the position information of the vehicle 10, a storage unit for storing the position information of the vehicle 10, a timer for observing the passage of time, and a calculation unit for calculating the speed of the vehicle 10 from the change in the position information of the vehicle 10 per unit time, and is capable of acquiring the position information of the vehicle 10 and calculating the speed of the vehicle 10 (FIGS. 2(D) and (E)).

[0030] <Indicators of microsleep-related behavior> The indicators used in this invention to detect microsleep-related behavior and determine whether a driver is likely to cause a collision accident are classified into three categories: "anti-sleepiness behavior," "signs of microsleep behavior," and "abnormal vehicle behavior," in order to perform video analysis based on images of the driver and the vehicle while driving.

[0031] <Definition of anti-sleepiness behavior> Anti-sleepiness behaviors were defined as behaviors intended to promote wakefulness by the driver while driving, including "touching," "stretching," "yelling," "yawning," and "blinking."

[0032] <Definition of microsleep behavioral signs> Microsleep behavioral signs were defined as behaviors strongly suggestive of microsleep, including "no body movement," "relaxed antigravity muscles," "eyes more than half-closed," and "eyes closed for more than 1 second." Percentage of eyelid closure over pupil (PERCLOS) (Dinges, 1998; Thropp, 2018; Marquart, 2015; Jackson, 2016) is the percentage of time the eyes are closed per minute, where "eyes closed" refers to a time of 0.5 seconds or more between opening and closing the eyelids. The length of time the eyes are closed corresponds to strong drowsiness, so in this invention, the eye closure time is set to 1 second.

[0033] <Definition of vehicle behavior abnormality> Abnormal vehicle behavior was defined as inappropriate lane crossing, where the vehicle crosses a lane and then returns to the original lane, and speed reduction of 10% or more on a straight road with no vehicle ahead.

[0034] <Detecting anti-drowsiness behaviors of drivers while driving> In the microsleep-related behavior detection system D, as shown in FIG. 1, while driving a vehicle 10, an in-vehicle camera 3 captures the driver's body movements, and a microphone 4 records the driver's voice. Based on the images captured by the in-vehicle camera 3 and the voice recorded by the microphone 4, a control unit 6 can detect the driver's behaviors to promote wakefulness, including "touching," "stretching," "yelling," "yawning," and "blinking" over time. In addition, the control unit 6 can detect the driver's anti-sleepiness behavior while driving, based on the number of times the driver "touches," "stretches," "yells," "yawns," and "blinks" are measured over time. "Touching" refers to the driver touching his or her face, head, or body. This is to resist sleepiness.

[0035] The above-mentioned anti-drowsiness behavior of the driver while driving can be detected by a known method. "Touching", "stretching", and "yawning" are body movements, and an image of the driver's body movement is acquired, the position of each part of the driver's body is recognized in time series, and the comparison reference data set in advance for multiple playback positions in the example video data and the position data of each part of the driver's body recognized in time series corresponding to the multiple playback positions are compared to calculate the degree of agreement (for example, JP 2023-55313 A and the like are references). "Shouting" may be acquired from the voice data of the driver recorded in time series by the microphone 4. "Blinking" may be acquired from the driver's body movement image, eye movement, or facial video (for example, JP 2023-55313 A, JP 2017-205317 A and the like are references). Alternatively, anti-drowsiness behavior may be detected from slight body movements of the driver obtained from an electromagnetic wave sensor (see, for example, Japanese Patent Application Laid-Open No. 7-59757).

[0036] <Detecting microsleep behavioral signs in drivers while driving> In the microsleep-related behavior detection system D, as shown in FIG. 1, an in-vehicle camera 3 captures the driver's body movements while driving a vehicle 10. Then, based on the images captured by the in-vehicle camera 3, the control unit 6 can detect behaviors suggestive of drowsiness, including the driver's "no body movement," "antigravity muscle relaxation," "eyes half-closed or more," and "eyes closed for one second or more" over time. In addition, the control unit 6 can detect microsleep behavioral signs of the driver while driving based on the number of times the driver's "no body movement," "antigravity muscle relaxation," "eyes half-closed or more," and "eyes closed for one second or more" are measured over time. "A state in which antigravity muscles are relaxed" refers to the head being bent forward or backward due to relaxation of the neck muscles.

[0037] The above-mentioned detection of microsleep behavioral signs of a driver while driving can be performed by a known method. The "state of no body movement" and "state of relaxed antigravity muscles" are physical movements, and a method is disclosed in which an in-car camera and a control unit are used to monitor the driver of a car, and the control unit detects the degree of closure of the driver's eyelids and the tilt of the head from the image captured by the in-car camera (for example, JP 2022-89223 A and the like are references). The "state of eyes half-closed or more" and "state of eyes closed for one second or more" may be obtained from an image of the driver's body movement (for example, JP 2023-55313 A, JP 2017-205317 A and the like are references).

[0038] <Detection of abnormal vehicle behavior while driving> In the microsleep-related behavior detection system D, as shown in FIG. 1, an exterior camera 5 captures the behavior of a vehicle while it is being driven. The control unit 6 can detect the behavior of the vehicle, including "inappropriate lane crossing in which the vehicle while being driven crosses lanes and then returns to the original lane" and "speed reduction in which the vehicle while being driven slows down by 10% or more on a straight road with no preceding vehicles" based on the images captured by the exterior camera 5. The control unit 6 can also detect abnormal vehicle behavior while it is being driven based on the number of times that the vehicle while being driven "inappropriate lane crossing in which the vehicle while being driven crosses lanes and then returns to the original lane" and "speed reduction in which the vehicle while being driven slows down by 10% or more on a straight road with no preceding vehicles" are measured over time.

[0039] The above-mentioned detection of abnormal vehicle behavior during driving can be performed by a known method. "Inappropriate lane crossing in which a vehicle being driven crosses lanes and then returns to the original lane" refers to the meandering of a vehicle being driven. Driving environment information of the vehicle being driven, including the lane ahead of the vehicle 10, captured by the exterior camera 5, may be acquired, and meandering state information including the presence or absence of detection of a meandering state of the vehicle being driven may be acquired based on the driving environment information (see, for example, JP 2022-187304 A). "Speed ​​reduction of 10% or more on a straight road without a vehicle ahead" refers to the vehicle behavior abnormality, which may be detected by acquiring driving environment information of a vehicle ahead of the vehicle 10 captured by the exterior camera 5, and using the speed detected by a speed detection means based on the driving environment information (see, for example, JP 2009-73462 A).

[0040] <Method of detecting microsleep-related behavior> The driver's actions and voice, as well as the vehicle's behavior, were counted every second from 60 seconds before the collision, as recorded by a drive recorder installed in a truck involved in a collision, to analyze microsleep-related behavior in collision accidents caused by microsleep. As microsleep-related behaviors, the number of times driver or vehicle behaviors were classified into three categories, "anti-sleep behavior," "signs of microsleep behavior," and "abnormal vehicle behavior," were counted from 60 seconds before the collision until the collision. The specific data acquisition method will be described later, but the results for the 32 collision cases that occurred on general roads are shown in Figure 6, and the results for the 20 collision cases that occurred on expressways are shown in Figure 7. In Figures 6 and 7, "†" indicates a truck equipped with an advanced emergency braking system, "fb" means rapid blinking, "h" means tapping, "l" means fidgeting, "s" means stretching, "t" means touching, "v" means shouting, "y" means yawning, "ab" means no body movement, "c" means eyes closed, "eh" means eyes half-closed, "ilc" means inappropriate lane departure, and "sr" means unnatural deceleration.

[0041] A piecewise regression analysis was performed on the three microsleep-related behaviors 60 seconds before a crash on general roads and expressways shown in Figures 6 and 7. Figure 8 shows the trends. On both general roads and expressways, drivers' anti-sleepiness behaviors decrease from about 40 seconds before the crash. On the other hand, the tendency for behavioral signs of microsleep and abnormal vehicle behavior to occur increases from about 40 seconds before the crash.

[0042] As shown in Figures 6 and 8, in accidents on general roads, anti-sleepiness behavior, microsleep behavior signs, and abnormal vehicle behavior signs gradually increased over time. Microsleep behavior signs began 22 seconds before the collision, and a significant increase was observed immediately before the collision. A significant increase in abnormal vehicle behavior was observed 9 seconds before the collision.

[0043] As shown in Figures 7 and 8, in highway accidents, microsleep behavioral signs showed little change from 60 seconds to 17 seconds before the collision, but increased significantly from 17 seconds before the collision. On the other hand, abnormal vehicle behavior gradually increased from 60 seconds before the collision, and increased significantly from 12 seconds before the collision. Anti-sleepiness behavior also increased until about 40 seconds before the collision, but then gradually decreased.

[0044] As described above, the indicators of microsleep-related behaviors, such as anti-sleepiness behaviors, microsleep behavior symptoms, and abnormal vehicle behavior, change over time in different patterns on general roads and expressways. For this reason, it is desirable to separately determine the detection of microsleep-related behaviors on general roads and expressways.

[0045] Microsleep-related behavior 60 seconds before a collision can be detected by the point in time when the driver's anti-drowsiness behavior and microsleep behavior signs, as well as the rate of increase in abnormal vehicle behavior that appears over time, change (hereinafter referred to as the "change point") and the percentage change in the rate of increase before and after the change point (hereinafter referred to as the "change rate").

[0046] Figure 9 is a conceptual diagram showing the change points and the rate of change in the increase rate (change rate) of anti-sleepiness behavior, microsleep behavior signs, and abnormal vehicle behavior in a drowsy driving accident on a highway. The change points in a drowsy driving accident appear at different times for anti-sleepiness behavior, microsleep behavior signs, and abnormal vehicle behavior, and the change rates are also different. As shown in Figure 9, the change points of anti-sleepiness behavior, microsleep behavior signs, and abnormal vehicle behavior in a drowsy driving accident on a highway from 60 seconds before the actual collision to the collision appear at different times, and the change rates are also different.

[0047] 1. Method for detecting microsleep-related behavior The flow of the method for detecting microsleep-related behavior in the microsleep-related behavior detection system D will be described. The present invention includes the steps of detecting anti-sleepiness behavior and microsleep behavioral signs of the driver while driving based on an image of the driver while driving captured by an in-vehicle imaging means capable of capturing an image of the driver while driving; detecting the vehicle behavior abnormality based on an image of the vehicle being driven captured by an exterior image capturing means capable of capturing an image of the vehicle being driven; Measuring the time-to-time increase rate of the detected anti-sleepiness behavior of the driver while driving, the microsleep behavioral signs of the driver while driving, or the number of detections of the vehicle behavior abnormality, respectively; The method further includes a step of determining that the driver is likely to cause a collision accident when the respective increase rates have a predetermined relationship. Fig. 3 is a flowchart showing an example of a process for determining that the driver is likely to cause a collision accident in the microsleep-related behavior detection system D.

[0048] As shown in FIG. 3, in step (hereinafter, simply referred to as "S") 101, an image of the driver while driving is acquired, for example, image data and voice data input from the in-vehicle camera 3 and microphone 4 (FIG. 1), and in S102, the image and voice of the driver are detected, and in S103, the anti-sleepiness behavior of the driver is detected by the analysis means and the judgment program P stored in the control unit 6, and in S104, the driver's microsleep behavior sign is detected. Simultaneously with S101, in S105, an image of the vehicle while driving input from the outside camera 5 is detected (S106), and the analysis means and the judgment program P stored in the control unit 6 detect abnormal vehicle behavior (S107). Images of the driver while driving and images of the vehicle while driving are repeatedly acquired.

[0049] In S108, the analysis means and judgment program P stored in the control unit 6 measure the increase rate per unit time of the number of detections of anti-drowsiness behavior, microsleep behavior symptoms, and vehicle behavior abnormality, and in S109, if the respective increase rates have a predetermined relationship, it is judged in S110 that the driver may cause a collision accident while driving. The driver may be notified of the judgment result by an alarm device.

[0050] 2. A predetermined relationship for determining whether a driver is likely to cause a collision while driving Figures 6 and 7 show the counts of the occurrences of anti-sleepiness behaviors, microsleep behavioral signs, and abnormal vehicle behaviors on general roads and expressways, by the time of occurrence, from 60 seconds before a collision until the collision. In order to determine the possibility of a driver causing a collision accident from this data, a regression analysis was performed on the data shown in Figures 6 and 7. The results are shown in Tables 1 and 2. Table 1 shows the results of the regression analysis of the data on general roads, and Table 2 shows the results of the regression analysis of the data on expressways. [Table 1] [Table 2]

[0051] The cut off point in Tables 1 and 2 refers to a specific point where the pattern of data changes. For example, if the trend of the data changes over time, it indicates the point of change. The slope in Tables 1 and 2 represents the rate of change in Y for a unit change in X. The slope of a linear function (Y / X) where X is time and Y is the change in the number of occurrences of each event is the rate of increase. If the slope is positive, as X increases, Y also increases, and if the slope is negative, Y decreases. A regression equation is an equation that expresses the relationship between X and Y. Regression equations are not limited to linear functions, but can also be quadratic or exponential functions.

[0052] 3. Judgment that the driver is at risk of causing a collision on a general road In Table 1, the cut off point for anti-sleepiness behavior in sections A and B is -41, with the change point being between 42 and 41 seconds before the collision, and the increase rate of anti-sleepiness behavior differs between sections A and B. For anti-sleepiness behavior, there is a tendency for Y (anti-sleepiness behavior) to increase with an increase in X (passage of time) in both sections A and B, but in section B from 41 seconds before the collision to the collision, the increase rate is 1.52, which is more than 1.5 times larger than the 0.85 in section A. Specifically, 1.52 / 0.85=1.78, which is more than 1.5 times the increase rate of the number of detected anti-sleepiness behaviors. In Table 1, the cut-off point for behavioral signs of microsleep in sections A and B is -22, with the change point being between 23 and 22 seconds before the collision. Section B from 22 seconds before the collision to the collision has a larger increase rate of 0.87 than section A (0.19), but at 42 seconds before the collision the increase rate is still 0.19. For abnormal vehicle behavior, the cut off point for sections A and B is -9, and the change point is between 10 and 9 seconds before the collision. The increase rate from 9 seconds before the collision to the collision increases significantly from 0.41 to 8.39, but at 42 seconds before the collision the increase rate is 0.41.

[0053] As described above, when focusing on the anti-sleepiness behavior whose change point appears between 42 seconds and 41 seconds before the collision, when the predetermined relationship is such that, in the presence of the detection results of the microsleep behavior signs and / or vehicle behavior abnormalities of the driver while driving, the increase rate of the detection results of the anti-sleepiness behavior of the driver while driving is higher than the increase rate of the detection results of the microsleep behavior signs and / or vehicle behavior abnormalities, when the change in the increase rate of the detection results of the anti-sleepiness behavior of the driver while driving is 1.5 times or more, it can be determined that the driver may cause a collision accident on a general road. In other words, the present invention includes a step of determining that the driver may cause a collision accident on a general road when the change in the increase rate of the detection number of the anti-sleepiness behavior of the driver while driving is 1.5 times or more in the presence of the detection number of the microsleep behavior signs and / or vehicle behavior abnormalities of the driver while driving, the increase rate of the detection number of the anti-sleepiness behavior of the driver while driving is higher than the increase rate of the detection number of the microsleep behavior signs and / or vehicle behavior abnormalities of the driver while driving.

[0054] 4. It is determined that the driver is at risk of causing a collision on the highway. The results of the regression analysis of the expressway data are shown in Table 2. In Table 2, the data patterns are divided into C and D sections over time. For anti-sleepiness behavior in Table 2, the cut off point for sections C and D is -38, with the change point being between 39 and 38 seconds before the collision, and the increase rate of anti-sleepiness behavior differs between sections C and D. The increase rate of anti-sleepiness behavior is 1.32 in section C, indicating a tendency for anti-sleepiness behavior to increase over time, but in section D, from 38 seconds before the collision to the collision, it is -0.66, indicating a decrease. For the behavioral signs of microsleep in Table 2, the cut off point for sections C and D is -17, with the change point being between 18 and 17 seconds before the collision. Section D, from 17 seconds before the collision to the collision, had a larger increase rate of 3.91 than section C, which had an increase rate of 0.11, but at 38 seconds before the collision the increase rate was still 0.11. For abnormal vehicle behavior, the cut off point for sections C and D is -12, with the change point being between 13 and 12 seconds before the collision. At 12 seconds before the collision, the increase rate increases significantly from 0.18 to 4.2, but at 38 seconds before the collision, the increase rate is 0.18.

[0055] As described above, when focusing on the anti-sleepiness behavior whose change point appears between 39 seconds and 38 seconds before the collision, when the predetermined relationship is such that the increase rate of the detection result of the anti-sleepiness behavior while driving is higher than the increase rate of the detection result of the microsleep behavior sign and / or vehicle behavior abnormality of the driver while driving, when the increase rate of the detection result of the anti-sleepiness behavior of the driver while driving turns negative, it can be determined that the driver may cause a collision accident on the expressway. In other words, the present invention includes a step of determining that the driver may cause a collision accident on the expressway when the increase rate of the detection number of the anti-sleepiness behavior while driving is higher than the increase rate of the detection number of the microsleep behavior sign and / or vehicle behavior abnormality of the driver while driving, when the increase rate of the detection number of the anti-sleepiness behavior while driving turns negative, when the increase rate of the detection number of the anti-sleepiness behavior while driving is higher than the increase rate of the detection number of the microsleep behavior sign and / or vehicle behavior abnormality of the driver while driving, when the detection number of the anti-sleepiness behavior while driving is higher than the increase rate of the detection number of the microsleep behavior sign and / or vehicle behavior abnormality of the driver while driving.

[0056] 5. Prescribed Relationships When Focusing on Microsleep Behavioral Signs (1) On general roads Focusing on microsleep behavior signs, which show a change point between 23 and 22 seconds before a collision, when the specified relationship is such that, in the presence of detection results of anti-sleepiness behavior and / or abnormal vehicle behavior of the driver while driving, the increase rate of the detection results of microsleep behavior signs is higher than the increase rate of the detection results of anti-sleepiness behavior while driving, when the change in the increase rate of microsleep behavior signs of the driver while driving increases by more than three times, it can be determined that there is a possibility that the driver will cause a collision accident on a public road.

[0057] (2) On expressways Focusing on microsleep behavior signs, which show a change point between 18 seconds and 17 seconds before a collision, it can be determined that the driver is likely to cause a collision accident on a highway when the specified relationship is such that, in the presence of detection results of anti-sleepiness behavior of the driver while driving and / or abnormal vehicle behavior, the change in the rate of increase in the detection results of microsleep behavior signs of the driver while driving increases by 20 times or more after the increase rate of the detection results of the anti-sleepiness behavior of the driver while driving turns negative.

[0058] 6. Predetermined relationship when focusing on vehicle behavior abnormality (1) On general roads Focusing on vehicle behavior abnormalities that show a change point between 10 seconds and 9 seconds before a collision, when the specified relationship indicates that the change in the rate of increase in the detection results of the vehicle behavior abnormalities while driving increases by 10 times or more in the presence of detection results of anti-drowsiness behavior and / or microsleep behavioral signs of the driver while driving, it can be determined that the driver is likely to cause a collision accident on a public road.

[0059] (2) On expressways Focusing on vehicle behavior abnormalities, it can be determined that the driver is likely to cause a collision accident on a highway when the specified relationship is such that, in the presence of detection results of anti-sleepiness behavior and / or microsleep behavioral signs of the driver while driving, the change in the rate of increase in the detection results of the vehicle behavior abnormalities while driving increases by 10 times or more after the increase rate of the detection results of the driver's anti-sleepiness behavior while driving turns negative.

[0060] <Microsleep-related behavior detection system that warns of collisions with obstacles in the left front direction> A characteristic of collisions caused by drowsiness at the wheel is that they are often side collisions (see "Collision site" in Table 4). In particular, 55.0% of collisions on expressways are collisions on the left side of the vehicle. It is desirable that the microsleep-related behavior detection system D warns a driver who is driving on an expressway that is determined to have the possibility of causing a collision of the vehicle being driven with an obstacle in the left front direction of the vehicle. It is also desirable to further include a device that detects obstacles in the left front direction of the vehicle. It is desirable that the obstacle detection device includes, but is not limited to, an irradiation device that irradiates laser light at a predetermined interval, an optical sensor, a laser light judgment unit, and an alarm unit. Furthermore, it is desirable to incorporate the microsleep-related behavior detection system D or a method for detecting microsleep-related behavior into an Advanced emergency braking system (AEBS) operating mechanism to improve the timing of AEBS operation and warning.

[0061] <System and method for detecting microsleep-related behavior from 5 minutes before a collision> In order to prevent future accidents, especially collisions triggered by microsleep behavioral signs, it is effective to detect microsleep-related behaviors at an earlier stage of a collision. "Earlier" means several minutes before a collision, and an example would be 5 minutes before a collision, which is a time when microsleep-related behaviors can be detected and the vehicle can be stopped in a safe area with ample time after detection. In particular, from April 2024, the maximum speed on expressways for medium and large trucks with a total weight of 8 tons or more will be reduced from the current 80 km / h to 90 km / h, so it is important to identify the possibility of a collision earlier in order to avoid accidents.

[0062] A specific detection method will be explained using the flow diagram in Figure 10. Detection results of "anti-sleepiness behavior," "signs of microsleep behavior," and "abnormal vehicle behavior" are obtained from images captured inside and outside the vehicle, information on the driver's body movements acquired by an electromagnetic wave sensor, and the driver's voice, and the hourly increase rate is calculated. The detection method and the method of acquiring the detection results can be the same as those described above.

[0063] As shown in the flow diagram in Figure 10, When the number of detections of anti-sleepiness behavior increases (i.e., the increase rate is positive) and the number of detections of abnormal vehicle behavior decreases (i.e., the increase rate is negative) in the presence of microsleep behavioral signs of a driver while driving and / or the number of detections of abnormal vehicle behavior, it is determined that there is a possibility of the appearance of microsleep behavioral signs. Since it is determined that there is a possibility of a collision accident triggered by the microsleep behavioral signs, the driver is alerted via the alarm device 7 and encouraged to take a break. When the increase rate in the number of detections of anti-drowsiness behavior is positive and the increase rate in the number of detections of abnormal vehicle behavior is negative, the driver is in a state where he or she is resisting drowsiness and attempting to correct the abnormal vehicle behavior, so it can be determined that the anti-drowsiness behavior is effectively contributing to suppressing the vehicle behavior, and a warning is sufficient.

[0064] on the other hand, As shown in the flow diagram in Figure 10, When the number of detections of anti-sleepiness behavior increases (i.e., when the rate of increase is positive) and the number of detections of vehicle behavior abnormalities also increases (i.e., when the rate of increase is also positive) in the presence of microsleep behavioral signs of a driver while driving and / or the number of detections of abnormal vehicle behavior, and this state continues for 3 seconds or more, it is determined that there is a possibility that microsleep behavioral signs will appear. Since it is determined that there is a possibility that a collision accident will occur due to the microsleep behavioral signs, a warning is given to the driver via the alarm device 7, and the driver is prompted to stop the vehicle. When the increase rate of the number of detections of anti-sleepiness behavior and the increase rate of the number of detections of abnormal vehicle behavior are both positive, it can be determined that the anti-sleepiness behavior is not effectively contributing to suppressing the vehicle behavior because the vehicle is actually behaving abnormally despite the driver's attempts to resist drowsiness, and a "warning" that is stronger than a caution is issued. Because the driver is exhibiting anti-sleepiness behavior and is seen to be attempting to resist drowsiness, a warning is issued to encourage the driver to stop the vehicle himself.

[0065] As shown in the flow diagram in Figure 10,When the number of detections of anti-sleepiness behavior decreases (i.e., when the increase rate is negative) and the number of detections of microsleep behavior signs increases (i.e., when the increase rate is positive) in the presence of microsleep behavior signs of a driver and / or abnormal vehicle behavior while driving, it is determined that there is a possibility of a collision accident triggered by microsleep behavior signs. When the number of anti-sleepiness behaviors detected is decreasing but the number of microsleep behavior signs is increasing, this indicates that the driver is unable to resist sleepiness and is unable to drive safely, so the vehicle must be stopped immediately. Therefore, when the above-mentioned state occurs, it is preferable to use a driver abnormality response system in which the vehicle determines the road shape, automatically moves the vehicle to a safe zone, and makes an emergency stop. EXAMPLES

[0066] The present invention will be described in detail below by way of examples, but the present invention is not limited to these examples in any way.

[0067] <Collision data> We evaluated the characteristics of microsleep-related behaviors in professional truck drivers involved in truck collision accidents. Of 14,695 people (day shift: 11,613, night shift: 3,082) working for one transportation company, 194 people experienced truck collision accidents caused by drowsiness between April 2016 and March 2021. Of these, 52 cases in which dashcam footage from 60 seconds before the collision was available were studied. The driver's age and working hours were confirmed using reports made at the time of the collision. Obesity level was confirmed using data from the most recent health examination at the time of the collision, and the accident type and collision site were confirmed using dashcam footage. The type of shift work was fixed as day shift and night shift, and there was no rotation within each driver. Sleep-disordered breathing screening was performed using multiple devices (Pulsox-300i, Konica Minolta, Tokyo, Japan; Pul-Sleep LS-140, Fukuda Denshi, Tokyo, Japan; Pulse Watch PMP-200Gplusx; Smart Watch PMP-300EX, Philips, Best, Netherlands; ApnoMonitor7; ApnoMonitor mini, Chest Co, Tokyo, Japan; Somny, NGK, Japan; NGK Spark Plug Co., Aichi, Japan). Sleep-disordered breathing was defined as a respiratory event index of 5 events / h or more. Day shifts were defined as working hours from 8:00 am to 7:59 pm, and night shifts were defined as working hours from 8:00 pm to 7:59 am.

[0068] <Recording and analysis of drive recorder footage> Two drive recorders were installed at the front of the truck (Fig. 2). The external drive recorder was installed at the top center of the windshield to evaluate the vehicle's behavior, and the internal drive recorder was installed on the dashboard to evaluate the driver's behavior. Drive recorder footage of truck collisions recorded on a 16-GB secure digital (SD) memory card was available for 85 of 194 cases, either inside or outside the vehicle (DR-9100, DRT-7100, HORIBA, Kyoto, Japan). Of the 85 cases, 33 cases were excluded because they did not have 60 seconds of footage before the collision, and the remaining 52 cases with footage of 60 seconds or more were analyzed. Of the 52 cases, 32 and 20 were collision accidents that occurred on general roads and expressways, respectively. In Japan, general roads are usually 1-2 lanes, have a speed limit of 30-50 km / h, and start / stop and right / left turns are frequent, so drivers need to pay attention to pedestrians, obstacles, and other vehicles. However, expressways are toll roads, usually with 2-3 lanes, a speed limit of 80-100km / h, and no traffic lights or right or left turns. Therefore, the driving conditions are obviously different between general roads and expressways, and the degree of attention required and the behavior of the vehicle are different. Therefore, in this invention, general roads and expressways were evaluated separately. The recorded data was analyzed using DR Player EX software (HORIBA). The interior and exterior video footage captured 60 seconds before the truck collision was visually reviewed. The screens displaying the driver and vehicle behavior could be enlarged and reduced, and playback could be paused. The driver's behavior, vehicle behavior, and driving speed were simultaneously visually analyzed on a second-by-second basis from a total of 3,120 seconds of footage obtained from two dashcams. In this method, the interior and exterior dashcam footage was analyzed from three perspectives: the driver's anti-sleepiness behavior, behavioral signs of microsleep, and abnormal vehicle behavior. This made it possible to simultaneously and intensively evaluate the three perspectives during driving.

[0069] <Definition of microsleep-related behavior> Behaviors related to microsleep were classified into three categories: "anti-sleep behavior," "microsleep behavioral signs," and "abnormal vehicle behavior." The indicators of microsleep-related behavior used in the present invention were defined for the purpose of performing the video analysis of the present invention, since such classification has not been done so far. The driver's anti-sleep behavior was defined as behaviors to promote wakefulness, such as touching, stretching, shouting, yawning, and blinking. Behavioral signs of microsleep were defined as behaviors suggesting sleepiness, such as no body movement, relaxation of antigravity muscles, eyes half-closed or closed for more than one second. Abnormal vehicle behavior was defined as inappropriate lane departure (recovering after turning the steering wheel after going over a straight or curved road) and unnatural speed reduction (10% or more on a straight road with no vehicles ahead, then reducing speed). One of the indices used to detect drowsiness while driving is the percentage of eyelids closed over pupils over time (PERCLOS) (Dinges, 1998; Thropp, 2018; Marquart, 2015; Jackson, 2016), which has been shown to be highly correlated with subjective drowsiness and psychomotor alertness tests, and is particularly useful when drowsiness is strong. PERCLOS is the percentage of time the eyes are closed per minute, with "eyes closed" referring to a time of 0.5 seconds or more between opening and closing the eyelids. As the length of time the eyes are closed corresponds to strong drowsiness, the eye closure time was set to 1 second in this study.

[0070] <Data Analysis> The data obtained are shown in Figures 6 and 7. The data obtained were statistically analyzed. All statistical analyses were performed using STATA version 15.1 (Stata Corp, College Station, TX, USA). Continuous variables were expressed as mean ± standard deviation (SD) or median (interquartile range [IQR]), and categorical variables were expressed as numbers or percentages. The Shapiro-Wilk test was first performed to evaluate the normality of the data. Student's t-test and Mann-Whitney U-test were used as appropriate to compare continuous variables such as basic characteristics and behavioral characteristics of traffic accidents on general roads and highways. For dichotomous data, the chi-square test was used. The Clopper-Pearson confidence interval method was used to estimate the 95% confidence interval for dichotomous data. A split regression analysis was performed to estimate the breakpoint and regression of each behavior before the collision. The breakpoint of each behavior was defined as the point where the trend of regression to the scatter plot estimated using the split regression model changed. The Clopper-Pearson confidence interval was used to evaluate the accident rate. All comparisons were two-sided, and p < 0.05 was considered statistically significant.

[0071] <Driver and crash characteristics> Characteristics of 52 professional truck drivers involved in crashes caused by drowsiness while driving are shown in Table 4. There were no significant differences in age and obesity between the 32 drivers who experienced crashes on general roads and the 20 drivers who experienced crashes on expressways. Drivers who experienced crashes on expressways were more likely to experience sleep-disordered breathing than drivers who experienced crashes on general roads (p=0.032). All collisions were analyzed in detail using dashcam footage. Twenty-four collisions occurred during the day shift and 28 during the night shift, with a significant difference between general roads and expressways (p<0.001). The types of collisions were rear-end collisions with the vehicle ahead in 25 cases and side collisions in 27 cases. The type and location of the collisions were significantly different between general roads and expressways (0.011 and 0.009, respectively). On general roads, rear-end collisions accounted for 62.5% of all collisions, with the most common location being the front of the vehicle (62.5%). On expressways, side collisions accounted for 75% of collisions, with the left side accounting for 55% of collisions.

[0072] <Real-time dashcam footage of rear-end and side-impact collisions> An example of a collision is shown below. In Figure 6 case 13, the driver's body motionless state ("ab") was observed for the first time from 52 seconds to 42 seconds before the collision, and repeated inappropriate line crossing ("ilc") was observed before and after the body motionless state. Then, 21 seconds before the collision, there was a 4-second extension, and 13 seconds before the collision, there was a second body motionless state ("ab"). Just 5 seconds after the driver's neck was bent backward to hit the back of the head against the headrest due to neck muscle relaxation, the driver rear-ended a stopped vehicle in front without sufficient deceleration. Also, in Figure 6 case 1, rapid blinking ("fb") ​​as an anti-sleepiness behavior was observed from 60 seconds to 52 seconds before the collision. Then, from 38 seconds before the collision, the absence of body motion ("ab"), slow blinking ("eh"), and eye closure ("c") were observed as microsleep behavioral signs. In addition, inappropriate lane departure ("ILC") occurred 30 seconds before the impact, and unnatural deceleration ("SR") was observed 16 seconds before the impact. After repeating these abnormal behaviors, the driver was observed to have his eyes closed ("C") from 4 seconds before the impact until 2 seconds before the impact. He opened his eyes 1 second before the impact, but it was too late, as the vehicle veered off the road and crashed into the bushes on the left side.

[0073] <Classification of microsleep-related behaviors> As shown in Table 3, microsleep-related behaviors were classified into general roads (Figure 6) and expressways (Figure 7) based on the analysis of the entire 3,120 seconds of drive recorder footage in second units. The frequency of occurrence of each microsleep-related behavior indicates the percentage of drivers in which the behavior occurred among the 52 drivers. In terms of drowsiness prevention behaviors, the most frequent item was "touching" (touching the face, head, or body), which was found in 46.2% of all drivers. Next, in terms of microsleep behavioral signs, the most common item was "cessation of body movement," which was found in 75.0% of all drivers. Finally, in terms of vehicle behavior abnormalities, the item "unnatural lane departure" was observed in the highest percentage of all drivers (78.8%). Note that the frequency of occurrence includes overlapping behaviors. [Table 3]

[0074] <Characteristics of the collision> As shown in Table 4, 24 collision accidents occurred during the day shift and 28 during the night shift, with a significant difference being observed between general roads and expressways (p < 0.001). The types of collisions were rear-end collisions with the vehicle in front in 25 cases and side collisions in 27 cases. The type and location of the collision was significantly different between general roads and expressways (p = 0.011 and 0.009, respectively). On general roads, rear-end collisions accounted for approximately 63% of all accidents, with the most common location of collision being the front of the vehicle (approximately 63%). For collisions on expressways, side collisions accounted for approximately 75% of collisions, with the left side accounting for approximately 55% of collisions. This strongly suggests the need to take measures against side collisions in collision accidents. [Table 4]

[0075] <Trends in microsleep-related behavior> A piecewise regression analysis was performed on the data on microsleep-related behaviors on general roads and expressways. Figure 8 shows the trend of how the data on microsleep-related behaviors changed over time, and the results of the regression analysis are shown in Tables 1 and 2. On both general roads and expressways, drivers' anti-sleepiness behaviors (in Figures 6 and 7, rapid blinking "fb", tapping "h", shaking "l", stretching "s", touching "t", yelling "v", and yawning "y") generally decreased from 40 seconds before the collision, and anti-sleepiness behaviors were observed in only 1 out of 32 cases (3.1%) on general roads and 3 out of 20 cases (15.0%) on expressways 1 second before the collision. In addition, the tendency for microsleep behavioral signs (cessation of body movement "ab", closed eyes "c", half-closed eyes "eh" in Figures 6 and 7) and abnormal vehicle behavior (inappropriate lane departure "ilc", unnatural deceleration "sr" in Figures 6 and 7) to occur was different from anti-sleepiness behavior. In collisions on general roads, microsleep behavioral signs and abnormal vehicle behavior gradually increased from 60 seconds before the collision. Microsleep behavioral signs began 22 seconds before the collision and were observed in 28 of 32 cases (87.5%) 1 second before the collision, showing a significant increase (p=0.006). Abnormal vehicle behavior showed a significant increase from 9 seconds before the collision and was observed in 25 of 32 cases (78.1%) 1 second before the collision (p<0.001). In highway accidents, microsleep behavioral signs showed little change from 60 seconds to 17 seconds before the collision, but increased significantly sharply 17 seconds before the collision (p=0.021). Also, they were observed 1 second before the collision in 15 of 20 cases (75.0%). Meanwhile, abnormal vehicle behavior gradually increased from 60 seconds before the collision, and increased significantly sharply 12 seconds before the collision (p=0.045). Abnormal vehicle behavior was observed in 16 of 20 cases (80.0%) 1 second before the collision. Repost [Table 1] Repost [Table 2]

[0076] <Segment analysis of general roads> Table 1 (A) and (B) are the results of regression analysis of the number of occurrences of anti-sleepiness behavior, microsleep behavioral signs, and abnormal vehicle behavior per hour, which are microsleep-related behaviors from 60 seconds before the collision on a general road shown in Figure 5 to the collision. Anti-sleepiness behavior, microsleep behavioral signs, and abnormal vehicle behavior were evaluated by dividing microsleep-related behaviors on general roads into Section A and Section B. The cut off point in the table is a specific point in time at which the data pattern changes, and the data pattern is divided into Section A and Section B in chronological order.

[0077] 1. Anti-sleepiness behavior Table 1 (A) shows that the pattern of data over time changed before and after the point 41 seconds before the collision. The cut off point is -41, and the change point is between 42 and 41 seconds before the collision. The slope of section A from 60 seconds before the collision to 42 seconds before the collision is 0.85, and anti-sleepy behaviors also increase over time. In section B from 41 seconds before the collision to the collision, the slope is 1.52, and the number of anti-sleepy behaviors also increases over time. The rate of increase in anti-sleepy behaviors is greater in section B, with an increase rate of about 1.8 times that of section A.

[0078] 2. Microsleep Behavioral Signs Table 1 (A) shows that the pattern of data over time changed before and after the point 22 seconds before the collision. The cut off point is -22, and the slope of section A from 60 seconds before the collision to 23 seconds before the collision is 0.19, so the increase in microsleep behavior signs over time is slower than that of anti-sleepiness behavior. In section B from 22 seconds before the collision to the collision, the slope is 0.87, and although the increase in the number of microsleep behavior signs over time is similarly slow, the change in the rate of increase is about 4.6 times.

[0079] 3. Abnormal vehicle behavior Table 1 (A) shows that the data pattern over time changed before and after the point 9 seconds before the collision. The cut off point is -9, and the change point is between 10 seconds and 9 seconds before the collision. The slope of section A from 60 seconds before the collision to 10 seconds before the collision is 0.41, which is smaller than anti-drowsiness behavior, and no significant increase in vehicle behavior abnormalities appears until just before the collision. However, in section B from 9 seconds before the collision to the collision, the slope is 8.39, and the number of vehicle behavior abnormalities also increases suddenly over time from 9 seconds before the collision to after the collision. The increase rate of vehicle behavior abnormalities in section B is about 20 times higher than in section A.

[0080] <Expressway section analysis> (C) and (D) in Table 2 are the results of a regression analysis of the number of occurrences of anti-sleepiness behavior, microsleep behavioral signs, and abnormal vehicle behavior per hour, which are microsleep-related behaviors from 60 seconds before the collision on the expressway shown in Figure 7 to the collision. Anti-sleepiness behavior, microsleep behavioral signs, and abnormal vehicle behavior were evaluated by dividing microsleep-related behaviors on the expressway into sections C and D. The cut off point in the table is a specific point in time at which the data pattern changes, and the data pattern was divided into sections C and D in chronological order.

[0081] 1. Anti-sleepiness behavior In the anti-sleepiness behavior of Table 2 (C), the pattern of data over time changes around the time 38 seconds before the collision. The cut off point is -38, and the change point is between 39 seconds and 38 seconds before the collision. The slope of section C from 60 seconds before the collision to 39 seconds before the collision is 1.32, and the anti-sleepiness behavior also increases relatively significantly over time. In section D from 38 seconds before the collision to the collision, the slope is -0.66, and the number of anti-sleepiness behaviors decreases over time. The increase rate of anti-sleepiness behaviors is significantly different between section C and section D. This shows that the anti-sleepiness behaviors, which are the precursor to a state of drowsiness due to frequent microsleeps, increase over time and decrease, making it impossible for drivers to resist drowsiness, and indicating that the risk of a collision is imminent. Trucks are heavy vehicles, and it takes time to brake and stop on the highway, so it is significant to be able to determine drowsy driving at this point.

[0082] 2. Microsleep Behavioral Signs Table 2 (C) shows that the pattern of data over time changed before and after the point 17 seconds before the collision. The cut off point is -17, and the change point is between 18 and 17 seconds before the collision. The slope of section C from 60 seconds before the collision to 18 seconds before the collision is 0.11, so the increase in microsleep behavior signs over time is gradual. In section D from 17 seconds before the collision to the collision, the slope is 3.91, and the increase in the number of microsleep behavior signs over time is greater than that in section C, with the change in the rate of increase being about 35 times that of section C.

[0083] 3. Abnormal vehicle behavior Table 2 (C) shows that the abnormal vehicle behavior changes over time before and after the point 12 seconds before the collision. The cut off point is -12, with the change point being between 13 and 12 seconds before the collision. The slope of section A from 60 seconds before the collision to 13 seconds before the collision is 0.18, so the increase in abnormal vehicle behavior over time is gradual. However, in section D from 12 seconds before the collision to the collision, the slope is 4.2, and the number of abnormal vehicle behavior also increases significantly over time. The rate of increase in abnormal vehicle behavior is greater in section D, with an increase rate about 23 times that of section C.

[0084] <Analysis of time series data on microsleep-related behavior from 5 minutes before to the collision> Time series data on microsleep-related behaviors is shown in Figure 11. Data was collected under the same conditions as those used in Figures 6 to 8, and the number of "anti-sleepiness behaviors," "signs of microsleep behaviors," and "abnormal vehicle behavior" detected in the five minutes prior to the collision was measured for 30 newly obtained collision accident cases.

[0085] The horizontal axis is every 5 seconds, and the vertical axis is the number of times that "anti-sleepiness behavior," "signs of microsleep behavior," and "abnormal vehicle behavior" were detected (for example, a scale of 100 seconds represents the average number of times the above were detected from 98 seconds to 102 seconds).

[0086] According to the flow diagram in Fig. 10, Fig. 11 shows the parts where attention should be called, a warning, or an emergency vehicle stop should be issued depending on the rate of increase in the number of detections of "anti-sleepiness behavior," "signs of microsleep behavior," and "abnormal vehicle behavior."

[0087] When checking the time series data from 5 minutes before the collision, it can be seen that the number of detections of anti-drowsiness behavior increases around 290 seconds before the collision, around 275 seconds before the collision, and around 115 to 90 seconds before the collision, and the number of detections of abnormal vehicle behavior decreases. This state indicates that the driver is resisting drowsiness despite being attacked by it, but it can be determined that the anti-drowsiness behavior is effectively contributing to suppressing the vehicle behavior, so a warning is issued. Note that the vehicle may be stopped in this state.

[0088] Around 210 seconds before the collision and between 70 and 40 seconds before the collision, the number of detections of anti-drowsiness behavior and abnormal vehicle behavior increases, and this state continues for more than 3 seconds. Since it can be determined that anti-drowsiness behavior is not effectively contributing to suppressing the vehicle behavior, a "warning" that is stronger than a warning is issued. The vehicle may be stopped in this state.

[0089] As shown in Fig. 11, if the driver stops the vehicle at the Caution 1 and 2 and Warning 1 and 2, there is enough time before the collision and the driver has time to slow down and stop the vehicle. It is assumed that the collision accident can be easily avoided. On the other hand, from about 8 seconds before the collision, the number of anti-sleepiness behaviors detected decreased and the number of microsleep behavior signs detected increased. This state indicates that the driver is unable to resist drowsiness and is unable to drive safely, so the vehicle must be brought to an emergency stop.

[0090] <Microsleep-related behavior detection system that warns of collisions with obstacles> There were 13 collisions out of 52 (6 on ordinary roads, 7 on expressways, † in Figures 6 and 7) caused by drowsy driving by trucks equipped with an advanced emergency braking system (AEBS), of which 8 were rear-end collisions, and only 3 (37.5%) involved rear-end collisions in which the AEBS was activated (3 of the 5 cases in which the AEBS was not activated were offset collisions, and the remaining 2 were of unknown cause, and all did not have an alarm sound). In the 3 rear-end collisions in which the AEBS was activated, the time from the alarm sounding to the collision was 0.7 to 2.4 seconds, and the distance traveled during this period was 6.6 to 24 m. As such, the current AEBS does not activate in all rear-end collisions, and even if it does activate, the timing of issuing the alarm is too late, and the braking distance is insufficient to prevent rear-end collisions. For this reason, it is necessary to improve the sensitivity of the sensor to avoid rear-end collisions and increase the activation rate. By linking it with the early determination of drowsy driving by the microsleep-related behavior detection system D, the timing of activating the AEBS can be advanced, increasing the probability of preventing rear-end collisions. [Industrial Applicability]

[0091] According to the system and method for detecting microsleep-related behavior of a driver while driving according to the present invention, it is possible to reliably detect microsleep-related behavior, which is the precursor to a drowsy state and which has different patterns on general roads and expressways, and to accurately prevent collision accidents caused by drowsy driving. [Explanation of symbols]

[0092] 3. In-car camera 4. Microphone 5. Exterior camera 6. Control unit 10. Vehicle

Claims

1. An in-vehicle imaging means capable of imaging a driver while driving; A recording means capable of recording the voice of a driver while driving; An outside image capturing means capable of capturing an image of a vehicle while it is being driven; A speed detection means for detecting a speed of a vehicle being driven; A control means 1 connected to the in-vehicle imaging means; A control means 2 connected to the speed detection means and the outside-of-vehicle image capturing means; an analysis means for analyzing the detection results of the control means 1 and / or the control means 2; A determination program for determining microsleep-related behavior, The control means 1 Detecting anti-sleepiness behavior of the driver while driving and a symptom of microsleep behavior of the driver while driving based on an image of the driver while driving captured by the in-vehicle imaging means; The control means 2 Detecting the vehicle behavior abnormality based on an image of the vehicle being driven captured by the vehicle exterior image capturing means; The analysis means measures an increase rate over time of the number of detections of anti-drowsiness behavior of the driver while driving, the number of detections of microsleep behavioral signs of the driver while driving, and the number of detections of the vehicle behavior abnormality, A system for detecting microsleep-related behavior of a driver while driving, characterized in that the judgment program determines that the driver is likely to cause a collision accident when the respective increase rates are in a predetermined relationship.

2. An in-vehicle imaging means capable of imaging a driver while driving; A recording means capable of recording the voice of a driver while driving; An outside image capturing means capable of capturing an image of a vehicle while it is being driven; A speed detection means for detecting a speed of a vehicle being driven; A control means 1 connected to the in-vehicle imaging means; A control means 2 connected to the speed detection means and the outside-of-vehicle image capturing means; an analysis means for analyzing the detection results of the control means 1 and / or the control means 2; A determination program for determining microsleep-related behavior; A method for detecting microsleep-related behavior, comprising: The control means 1 detects anti-sleepiness behavior of the driver while driving and a microsleep behavior sign of the driver while driving based on an image of the driver while driving captured by an in-vehicle imaging means capable of capturing an image of the driver while driving, The control means 2 detects the vehicle behavior abnormality based on an image of the vehicle being driven that is captured by an outside-vehicle image capturing means capable of capturing an image of the vehicle being driven, The analysis means measures the hourly increase rate of the number of detections of the anti-sleepiness behavior of the driver while driving, the number of detections of the microsleep behavioral signs of the driver while driving, and the number of detections of the vehicle behavior abnormality, A method for detecting microsleep-related behavior of a driver while driving, characterized in that the judgment program determines that there is a possibility that the driver may cause a collision accident when the respective increase rates are in a predetermined relationship.

3. The method for detecting microsleep-related behavior of a driver while driving, as described in claim 2, characterized in that the specified relationship is such that when the increase rate of the detection rate of anti-sleepiness behavior of the driver while driving is higher than the increase rate of the detection rate of microsleep behavior signs and / or abnormal vehicle behavior of the driver while driving in the presence of the detection rate of microsleep behavior signs and / or abnormal vehicle behavior of the driver while driving, when the increase rate of the detection rate of anti-sleepiness behavior of the driver while driving is 1.5 times or more, the judgment program judges that there is a possibility that the driver will cause a collision accident on a public road.

4. The method for detecting microsleep-related behavior of a driver while driving, as described in claim 2, characterized in that the specified relationship is such that, when the increase rate of the detection number of microsleep behavior signs and / or vehicle behavior abnormalities of the driver while driving is higher than the increase rate of the detection number of microsleep behavior signs and / or vehicle behavior abnormalities of the driver while driving, when the increase rate of the detection number of anti-sleepiness behavior of the driver while driving turns negative, the judgment program judges that there is a possibility that the driver will cause a collision accident on a highway.

5. The method for detecting microsleep-related behavior of a driver while driving as described in claim 2, characterized in that the predetermined relationship includes a step in which the judgment program judges that the driver is likely to cause a collision accident when the increase rate in the number of detections of anti-drowsiness behavior is positive and the increase rate in the number of detections of vehicle behavior abnormalities is negative in the presence of microsleep behavioral signs and / or the number of detections of vehicle behavior abnormalities of the driver while driving.

6. The method for detecting microsleep-related behavior of a driver while driving as described in claim 2, characterized in that the judgment program determines that the driver is likely to cause a collision accident when the specified relationship indicates that, in the presence of microsleep behavioral signs of the driver while driving and / or the number of detections of vehicle behavior abnormalities, the increase rate of the number of detections of anti-drowsiness behavior is positive and the increase rate of the number of detections of vehicle behavior abnormalities is also positive for more than 3 seconds.

7. The method for detecting microsleep-related behavior of a driver while driving as described in claim 2, characterized in that the predetermined relationship includes a step in which the judgment program judges that the driver is likely to cause a collision accident when the increase rate of the detection number of anti-drowsiness behavior detection signs is negative and the increase rate of the detection number of microsleep behavior detection signs is positive in the presence of microsleep behavioral signs and / or abnormal vehicle behavior of the driver while driving.

8. A system for detecting microsleep-related behavior of a driver while driving as described in claim 1, characterized in that the system warns a driver while driving on a highway that has been determined by the judgment program to have the possibility of causing a collision accident of a collision with an obstacle to the left front direction of the vehicle.

9. The system for detecting microsleep-related behavior of a driver while driving as claimed in claim 1, further comprising a device for detecting an obstacle in the left front direction of the vehicle.

Images