Detection method, detection program, and on-vehicle device

The in-vehicle device calculates the aspect ratio of a driver's eyes to select appropriate indicators for drowsiness detection, improving accuracy and reducing false positives.

JP2024157959APending Publication Date: 2024-11-08DENSO TEN LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023072663
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-26
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Conventional eye detection methods struggle with false positives due to variations in eye shape and the difficulty in accurately determining eyebrow positions, leading to ineffective drowsiness detection.

Method used

An in-vehicle device uses image recognition to calculate the aspect ratio of a driver's eyes, selecting indicators like eye closure rate and duration based on this ratio to accurately detect drowsiness, thereby reducing false positives.

Benefits of technology

The method effectively suppresses false drowsiness detections by adapting to individual eye shapes and sizes, enhancing detection accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157959000001_ABST
    Figure 2024157959000001_ABST
Patent Text Reader

Abstract

To suppress erroneous detection of dozing.SOLUTION: An on-vehicle device includes a controller for executing processing to detect dozing of a driver of a vehicle. The controller identifies an aspect ratio of an eye of the driver based on an image photographed by a camera arranged in the vehicle. The controller selects one or more indexes from the plurality of indexes related to opening / closing of the eye for detecting the dozing of the driver based on the aspect ratio. The controller detects the dozing of the driver based on the image photographed by the camera with the use of the selected indexes.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for detecting whether a driver is drowsy. [Background technology]

[0002] Conventionally, there are known techniques for detecting the state or behavior of a person's eyes based on images. However, in such techniques, the detection accuracy may decrease due to differences in the shape of each person's eyes.

[0003] For example, if the subject has small eyes, the change in the image between when the eyes are open and when they are closed will be smaller than when the eyes are not small, and the accuracy of detecting actions such as blinking may decrease.

[0004] In response to this, Patent Document 1 describes a method for detecting blinks using not only changes in the shape of the eye itself but also changes in the positional relationship between the eyebrows and the eyes, in order to prevent a decrease in detection accuracy due to narrow eyes. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2000-163564 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the conventional technology has a problem in that it may not be possible to prevent false detection of drowsiness.

[0007] In the technology described in Patent Document 1, in order to improve the detection accuracy, it is necessary to obtain information on the position of not only the eyes but also the eyebrows by image recognition. However, depending on the function of the image recognition engine or the performance of the device that executes the image recognition, it may be difficult to obtain the position of the eyebrows. When focusing only on the change in the shape of the eyes, it is still difficult to prevent the occurrence of false detections.

[0008] Furthermore, the technology described in Patent Document 1 is intended to detect blinking, and is not necessarily applicable to detection of drowsiness.

[0009] The present invention has been made in view of the above, and has an object to prevent erroneous detection of drowsiness at the wheel. [Means for solving the problem]

[0010] A detection method according to one aspect of an embodiment is a detection method executed by an in-vehicle device having a controller that performs processing to detect whether a driver of a vehicle is dozing off, and includes determining an aspect ratio of the driver's eyes based on an image captured by a camera equipped in the vehicle, selecting one or more indicators from a plurality of indicators for detecting whether the driver is dozing off based on the aspect ratio, and using the selected indicators to detect whether the driver is dozing off based on the image captured by the camera. Effect of the Invention

[0011] According to one aspect of the embodiment, false detection of drowsiness can be suppressed. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an in-vehicle device according to an embodiment. [Diagram 2] FIG. 2 is a flowchart showing a flow of processing performed by the in-vehicle device according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating a method for determining the aspect ratio of the eye. [Figure 4] FIG. 4 is a diagram illustrating a method for determining the aspect ratio of the eye. [Diagram 5] FIG. 5 is a diagram for explaining a method for calculating the degree of fineness of the eye. [Figure 6] FIG. 6 is a diagram illustrating erroneous detection. [Figure 7]FIG. 7 is a diagram showing an example of detection processing according to the degree of narrowness of the eyes. [Figure 8] FIG. 8 is a diagram illustrating the clearing of the buffer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A detection method, a detection program, and an in-vehicle device according to an embodiment will be described in detail below with reference to the accompanying drawings. The present invention is not limited to the embodiment.

[0014] The configuration of the in-vehicle device will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the in-vehicle device according to the embodiment.

[0015] The in-vehicle device 10 functions as a DMS (driver monitoring system). The DMS is a system that detects whether the driver is looking away from the vehicle, falling asleep, etc., based on an image captured inside the vehicle cabin. The in-vehicle device 10 may be a drive recorder equipped with a function as a DMS.

[0016] In the embodiment, a method for detecting whether the driver is dozing off by the in-vehicle device 10 will be described. The in-vehicle device 10 detects the driver's eyes from an image captured inside the vehicle cabin using an image recognition engine. Then, the in-vehicle device 10 detects whether the driver is dozing off based on whether the driver's eyes are open or closed.

[0017] As shown in FIG. 1, the in-vehicle device 10 includes a camera 11, an output unit 12, a communication unit 13, a controller 14, a memory 15, and an interface 16.

[0018] The camera 11 is installed in the vehicle cabin facing the driver's seat of the vehicle. When the driver is seated in the driver's seat, the camera 11 can capture an image of the driver's face.

[0019] The output unit 12 outputs data. The output unit 12 is an output device such as a display and a speaker. For example, the output unit 12 outputs an alarm sound when dozing is detected.

[0020] The communication unit 13 is a device for communicating data with other devices, and is, for example, an in-vehicle communication module capable of wireless communication.

[0021] The controller 14 reads and executes a program stored in the memory 15. The controller 14 is a microcomputer, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), a SoC (System on a Chip), or the like.

[0022] The controller 14 may be a single processor, a multi-processor configuration, or a multi-core configuration having multiple cores in a single chip connected to a single socket.

[0023] The memory 15 is a storage medium such as a flash memory, etc. The memory 15 functions as a read only memory (ROM) or a random access memory (RAM), but the following description will be mainly given assuming that the memory 15 is a RAM.

[0024] The interface 16 is an interface for inputting and outputting data between the devices included in the in-vehicle device 10 .

[0025] The controller 14 performs a process for detecting whether the driver is dozing off. The process for detecting whether the driver is dozing off will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of the process performed by the in-vehicle device according to the embodiment.

[0026] 2, after the ACC of the vehicle is turned on (step S101), the controller 14 causes the camera 11 to start capturing an image of the driver's face and starts judging whether the eyes are open or closed (step S102). After step S102, the camera 11 continues capturing the face image. The controller 14 controls the start of capturing images, etc., by transmitting a signal to the camera 11 via the interface 16.

[0027] Controller 14 creates a moving image by having camera 11 capture still images continuously or intermittently at regular time intervals (e.g., 0.05 seconds). Each frame that constitutes the moving image captured by camera 11 is called a face image frame.

[0028] The controller 14 uses an image recognition engine to determine whether the driver's eyes are open or closed for each frame. The image recognition engine is a program that uses, for example, a trained neural network to recognize the positions of specific objects and boundaries between objects in an image.

[0029] The controller 14 stores the face image frames in the memory 15 in association with the image capture time and the result of the open / close determination (for example, a binary value of Open or Close).

[0030] Next, the in-vehicle device 10 calculates the degree of narrowness of the eyes from the frame of the face image (step S103). At this time, the controller 14 identifies the aspect ratio of the driver's eyes based on the image captured by the camera 11 provided in the vehicle. Then, the controller 14 calculates the narrowness of the eyes based on the aspect ratio.

[0031] A method for determining the aspect ratio of the eyes will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining a method for determining the aspect ratio of the eyes. Note that Fig. 3 shows the eyes of user U1 in an open state.

[0032] First, the controller 14 identifies the positions of the feature points using an image recognition engine. Here, the image recognition engine recognizes two points on the boundary between the upper eyelid and the eyeball, one point on the outer corner of the eye, one point on the inner corner of the eye, and two points on the boundary between the lower eyelid and the eyeball as feature points.

[0033] 3, the controller 14 calculates the distance h1 between a line passing through two points on the boundary between the upper eyelid and the eyeball and a line passing through two points on the boundary between the lower eyelid and the eyeball. The controller 14 also calculates the distance w1 between a point on the outer corner of the eye and a point on the inner corner of the eye. The controller 14 then calculates the aspect ratio of the eyes of the user U1 as h1 / w1.

[0034] In Fig. 3, an example in which the driver is a user U1 is described. In contrast, Fig. 4 shows an example in which the driver is a user U2. Fig. 4 is a diagram for explaining a method for determining the aspect ratio of the eyes. Note that Fig. 4 shows the eyes of the user U2 in an open state.

[0035] 4, the controller 14 calculates the distance h2 between a line passing through two points on the boundary between the upper eyelid and the eyeball and a line passing through two points on the boundary between the lower eyelid and the eyeball. The controller 14 also calculates the distance w2 between a point on the outer corner of the eye and a point on the inner corner of the eye. The controller 14 then calculates the aspect ratio of the eyes of the user U1 as h2 / w2.

[0036] As shown in Figures 3 and 4, the aspect ratios of users U1 and U2 are significantly different. The aspect ratio also changes depending on the opening and closing of the eyes. Therefore, the controller 14 calculates the degree of narrowness of the driver's eyes based on the distribution of the aspect ratios of the driver's eyes, which are determined based on each of a plurality of images captured by the camera 11 at different times.

[0037] Fig. 5 is a diagram for explaining a method for calculating the degree of thinness of the eyes. The controller 14 sorts a plurality of aspect ratios in ascending order. Then, as shown in Fig. 5, the controller 14 plots the aspect ratios, numbered in the sorted order, on a coordinate plane. The horizontal axis of the coordinate plane corresponds to the number. The vertical axis of the coordinate plane corresponds to the aspect ratio.

[0038] The coordinate plane in Figure 5 plots aspect ratios at various times, such as the aspect ratio when the eyes are closed, the aspect ratio when the eyes are naturally open, the aspect ratio when the eyes are strongly open, the aspect ratio when the eyes are halfway closed, and the aspect ratio when the eyes are halfway open.

[0039] As shown in Fig. 5, the distribution of aspect ratios differs for each user. Assume that user U1 has narrower eyes than user U2. For example, the aspect ratio of the eyes of user U1 varies less than the aspect ratio of the eyes of user U2.

[0040] Here, the controller 14 calculates the reciprocal of the slope of a line approximating the aspect ratio of each user as the degree of thinness of the eyes. In other words, the greater the slope of the line, the smaller the degree of thinness of the eyes. Therefore, the degree of thinness of the eyes of user U1 is greater than the degree of thinness of the eyes of user U2.

[0041] In this way, by using multiple images instead of just one image, the controller 14 can more accurately calculate the degree of thinness by taking into account changes in the shape of the eye.

[0042] Although the method for calculating the degree of narrowing of the eyes has been explained using two different drivers (user U1 and user U2) as an example, in reality, the controller 14 calculates the degree of narrowing of the eyes for one driver who is driving the vehicle.

[0043] Returning to Fig. 2, the controller 14 judges whether the degree of narrowing of the driver's eyes is equal to or greater than a threshold (step S104). After that, the controller 14 selects one or more indicators from among a plurality of indicators related to the opening and closing of the eyes for detecting the driver's drowsiness, based on the degree of narrowing of the eyes calculated from the aspect ratio. It is assumed that the threshold has been set in advance.

[0044] The multiple indexes include an eye-closing rate and a duration. The eye-closing rate is an example of a first index. The duration is an example of a second index. The controller 14 calculates the eye-closing rate and the duration based on the result of the open / close determination.

[0045] The eye closure rate is the percentage of time that the eyes are closed. The duration is the time that the eyes are closed. The larger the eye closure rate and duration, the easier it is to detect drowsiness. This is because a driver who is about to fall asleep tends to close his or her eyes more frequently and keep them closed for a longer period of time.

[0046] Here, when the degree of narrowing of the driver's eyes is large, there is a tendency for false detection to occur in the detection using the eye closure rate. That is, this is because when the degree of narrowing of the driver's eyes is large, it is likely that the eyes are closed even when they are actually open.

[0047] The false detection will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the false detection. The controller 14 stores the results of determining whether the driver's eyes are closed or open for each of a plurality of images captured by the camera 11 at different times in a buffer in the memory in association with each time.

[0048] That is, the controller 14 stores data as shown in Fig. 6 as a buffer in the memory 15 as a result of the open / closed determination. Fig. 6 shows an example of the result of the open / closed determination when it is assumed that the user U1 and the user U2 close their eyes for the same period and open their eyes for the same period. Note that the buffer contains the results of the open / closed determination for the past 30 seconds, for example.

[0049] "Open" and "Close" are judgment results. In addition, the judgment results are associated with time.

[0050] The degree of narrowing of the eyes of the user U2 is small (less than the threshold value). The controller 14 correctly determines the user U2 as “Closed” during the period when the eyes are closed (the period from t1 to t2 and the period from t3 to t4), and correctly determines the user U2 as “Open” during the period when the eyes are open (the period from t2 to t3).

[0051] On the other hand, the degree of narrowing of the eyes of user U1 is large (above the threshold). The controller 14 correctly determines that the user U1 is “Closed” during the periods when the eyes are closed (periods from t1 to t2 and from t3 to t4). On the other hand, the controller 14 erroneously determines that the user U1 is “Closed” partially during the periods when the eyes are open (periods from t2 to t3).

[0052] The controller 14 calculates the eye closure rate during the period from t1 to t4, that is, within the buffer, as (number of eyes determined to be Closed) / ((number of eyes determined to be Open)+(number of eyes determined to be Closed)).

[0053] 6, the correct eye closure rate for both user U1 and user U2 is 6 / 16. The controller 14 calculates the eye closure rate for user U2 to be 6 / 16. On the other hand, the controller 14 calculates the eye closure rate for user U1 to be 12 / 16. The higher the eye closure rate, the easier it is to detect dozing, and therefore the higher the possibility of false detection for user U1.

[0054] A false positive is a false positive that is detected as a person dozing when in fact the person is not dozing. A failure to detect a person dozing when in fact the person is dozing is called a decrease in detection performance or a missed detection.

[0055] Therefore, the controller 14 determines whether or not to use the eye closure ratio among the indices for detecting dozing in step S104 of Fig. 2. That is, the controller 14 selects one or more indices based on the degree of thinness from among the eye closure ratio, which is the ratio of the time the eyes are closed, and the duration, which is the time the eyes are kept closed.

[0056] The duration is the number of consecutive times in the buffer where the determination result is "Close."

[0057] 7 is a diagram showing an example of a detection process according to the degree of narrowing of the eyes. The detection process of drowsiness that can be executed by the controller 14 includes two detection processes: a detection process using the eye closure rate and a detection process using the duration. The controller 14 may execute either one of the two detection processes or may execute both of them.

[0058] 7, when the degree of narrowing of the eyes is small (less than the threshold), the controller 14 operates both the detection process using the eye closure rate and the detection process using the duration. On the other hand, when the degree of narrowing of the eyes is large (above the threshold), the controller 14 stops the detection process using the eye closure rate and operates the detection process using the duration. In this way, the controller 14 prevents the occurrence of false detection when the degree of narrowing of the eyes is large (above the threshold).

[0059] The controller 14 realizes control of such detection processing by the processing from step S104 onward in FIG.

[0060] If the degree of thinness is equal to or greater than the threshold (Yes at step S104), the controller 14 selects the duration and proceeds to step S 105. In this case, the controller 14 selects the duration without selecting the eye closure rate.

[0061] In step S105, the controller 14 stops the dozing detection based on the eye closure rate (step S105). Also, the controller 14 clears (deletes) the buffer without using it (step S106).

[0062] FIG. 8 is a diagram for explaining the clearing of the buffer. As shown in FIG. 8, the controller 14 may have many erroneous judgment results remaining in the buffer. In such a state, when the driver is changed, before the judgment results of the new driver are stored in the buffer, drowsiness may be detected based on the judgment results of the previous driver, resulting in an erroneous detection. The controller 14 clears the buffer to avoid the occurrence of an erroneous detection when the driver is changed.

[0063] On the other hand, if the degree of thinness is less than the threshold (step S104, No), the controller 14 selects the eye closure rate and the duration, and proceeds to step S107. In this case, the controller 14 selects both the eye closure rate and the duration. The controller 14 also uses the buffer without clearing it (step S108).

[0064] The controller 14 executes detection of drowsiness by the ongoing process (step S109). That is, the controller 14 detects the driver's drowsiness based on the image captured by the camera 11 using the selected index.

[0065] Thereafter, if ACC is not OFF (step S110, No), the controller 14 proceeds to step S111. If ACC is OFF (step S110, Yes), the controller 14 ends the process.

[0066] If the driver of the vehicle has not changed (step S111, No), the controller 14 returns to step S109 and repeats the process. On the other hand, if the driver of the vehicle has changed (step S111, Yes), the controller 14 returns to step S103 and calculates the degree of narrowing of the eyes again.

[0067] As described above, the controller 14 identifies the aspect ratio of the driver's eyes based on the image captured by the camera 11 provided in the vehicle. The controller 14 selects one or more indices from among a plurality of indices related to eye opening and closing for detecting the driver's drowsiness based on the aspect ratio. The controller 14 uses the selected indices to detect the driver's drowsiness based on the image captured by the camera 11.

[0068] In this way, the in-vehicle device 10 can prevent erroneous detection of drowsiness at the wheel even when the driver's eyes are quite narrow, by changing the index used depending on the aspect ratio of the driver's eyes.

[0069] Furthermore, when the degree of squinting of the driver's eyes is small, the in-vehicle device 10 can prevent the detection performance from deteriorating by using both the eye closure rate and the duration.

[0070] In order to prevent false positives, it is possible to increase the threshold value of the eye closure rate when detecting dozing, but in that case, even if false positives are reduced, the detection performance will decrease.

[0071] Further advantages and modifications may readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof. [Explanation of symbols]

[0072] 10 Onboard equipment 11 Camera 12 Output section 13. Communications Department 14 Controller 15 Memory 16 Interface

Claims

1. A detection method executed by an in-vehicle device having a controller that performs processing for detecting drowsiness of a driver of a vehicle, comprising: Identifying an aspect ratio of the driver's eyes based on an image captured by a camera installed in the vehicle; selecting one or more indicators from a plurality of indicators related to eye opening and closing for detecting drowsiness of the driver based on the aspect ratio; Using the selected indicator, it is detected that the driver is dozing off based on the image captured by the camera. Detection method.

2. The controller: Calculating the degree of narrowness of the driver's eyes based on a distribution of aspect ratios of the driver's eyes identified based on each of a plurality of images captured by the camera at different times; Selecting one or more indices from among the plurality of indices based on the degree of thinness. The detection method according to claim 1 .

3. The controller: Selecting one or more indicators based on the degree of narrowing from a first indicator that is a percentage of time that the eyes are closed and a second indicator that is a duration that the eyes are closed. The detection method according to claim 2.

4. The controller: When the degree of thinness is equal to or greater than a threshold, the second index is selected, and when the degree of thinness is less than the threshold, the first index and the second index are selected. The detection method according to claim 3.

5. The controller: determining whether the driver's eyes are open or closed for each of a plurality of images captured by the camera at different times, and storing the results in a memory as a buffer in association with each of the times; If the degree of thinness is equal to or greater than a threshold, clear the buffer. The detection method according to claim 4.

6. A controller that performs processing to detect whether a vehicle driver is dozing off, Identifying an aspect ratio of the driver's eyes based on an image captured by a camera installed in the vehicle; selecting one or more indicators from a plurality of indicators related to eye opening and closing for detecting drowsiness of the driver based on the aspect ratio; Using the selected indicator, it is detected that the driver is dozing off based on the image captured by the camera. A detection program that executes the process.

7. A controller for performing processing for detecting drowsiness of a driver of a vehicle, The controller: Identifying an aspect ratio of the driver's eyes based on an image captured by a camera installed in the vehicle; selecting one or more indicators from a plurality of indicators related to eye opening and closing for detecting drowsiness of the driver based on the aspect ratio; Using the selected indicator, it is detected that the driver is dozing off based on the image captured by the camera. In-vehicle device.

Citation Information

Patent Citations

  • Eye tracking device and wink detector

    JP2000163564A