Driver distraction recognition system

Abstract

This invention provides a driver distraction detection system, comprising: a camera to capture images of the driver's eyes and obtain information about the driver's gaze direction; an infrared array sensor to detect the driver's eye position and perform eye tracking to obtain information about the driver's eye movement; a vehicle computer equipped with a heart rate calculation module, which, by combining gaze direction information and eye movement information, generates an eye movement range and uses an algorithm and the heart rate calculation module to determine whether the eye movement range exceeds a warning area and the driver's mental state; a first transmission module; and a second transmission module; wherein the first transmission module is connected to the infrared array sensor and the vehicle computer; and the second transmission module is connected to the vehicle computer and the camera.

Description

[Technical Field]

[0001] This invention relates to an identification system and method, and more particularly to a driver distraction identification system and method. [Previous Technology]

[0002] Intelligent transportation and autonomous driving technologies are booming. Autonomous driving is a continuously evolving technology that relies on the coordinated operation of three major systems: perception, decision-making, and control. Coupled with recent advancements in artificial intelligence technologies, these systems are being trained to significantly accelerate the development of self-driving cars. Furthermore, their practical applications are expanding from highways to more complex urban roads. Ensuring vehicle safety, driving comfort, and reducing traffic accidents are considered major goals of modern society. However, poor driver performance is a major cause of traffic accidents. Fatigue and distraction can cause drivers to lose control of the road.

[0003] Currently, there are methods that can detect a user's facial expressions and even track the position of the user's pupils to determine the user's current emotional state. Methods that can track a user's gaze use infrared light and cameras to track the direction of the user's gaze and perform eye movement data analysis. Based on changes in the user's gaze position, the user's current mental state can be determined, such as drowsiness, distraction, or anxiety.

[0004] In addition, systems for assessing a driver's mental state have emerged to improve driving safety. These systems typically require detecting the driver's pupil position and are often combined with various physiological monitoring devices before they can be used. However, they also have many areas for improvement. For example, human pupils are quite small and very similar in color to the iris, causing errors when cameras track pupil position, and sometimes even failing to detect them. Furthermore, wearing various physiological monitoring devices is inconvenient and restricts the driver's movement, leading to decreased judgment and longer reaction time while driving, ultimately increasing the risk of accidents.

[0005] Therefore, how to provide a driver distraction detection system and method that can solve the problems of insufficient accuracy of current driving safety systems and the restriction of driver's actions caused by physiological monitoring devices remains an important issue to be solved. [Summary of the Invention]

[0006] In view of the above problems, the present invention provides a driver distraction detection system, which includes a camera to capture the driver's eyes and obtain the driver's gaze direction information; an infrared array sensor to detect the driver's eye position, perform eye tracking, and obtain the driver's eye movement information; an in-vehicle computer to combine the gaze direction information and the eye movement information to generate an eye movement range, and then use an algorithm and a heart rate calculation module to determine whether the eye movement range exceeds the warning area and the driver's mental state; a first transmission module; and a second transmission module; wherein the first transmission module is connected to the infrared array sensor and the in-vehicle computer; wherein the second transmission module is connected to the in-vehicle computer and the camera.

[0007] In one embodiment of a driver distraction detection system of the present invention, the camera is equipped with embedded vision technology.

[0008] In one embodiment of a driver distraction detection system of the present invention, the infrared array sensor transmits data internally through a serial transceiver or a system on a chip (SoC).

[0009] In one embodiment of a driver distraction detection system of the present invention, the first transmission module includes a first Ethernet network, a first Camera Serial Interface (CSI), or a first Universal Serial Bus (USB).

[0010] In one embodiment of a driver distraction detection system of the present invention, the second transmission module includes a second Ethernet network, a second Camera Serial Interface (CSI), or a second Universal Serial Bus (USB).

[0011] In one embodiment of a driver distraction recognition system of the present invention, the heart rate calculation module includes a gaze direction algorithm, a heart rate extraction algorithm, and a heart rate change calculation method.

[0012] The present invention further provides a driver distraction detection method, comprising the following steps: (a) turning on a camera and an infrared array sensor, wherein the camera continuously captures the driver's eyes while the driver is driving and generates the driver's gaze direction information; (b) the infrared array sensor continuously tracks the driver's gaze movement position while the driver is driving and generates the driver's eye movement information; (c) the vehicle computer receives the eye movement information through a first transmission module and the gaze direction information through a second transmission module, then combines the eye movement information and the gaze direction information to generate an eye movement range, and executes an algorithm to determine whether the eye movement range exceeds the warning area. On the other hand, the heart rate calculation module combines the eye movement range to determine whether the driver's mental state is normal; (d) if so, the in-vehicle sound system continuously emits a warning sound to remind the driver to pay attention to the vehicle condition, and continues to repeat steps (a) to (c); (e) If not, continue repeating steps (a) to (c) until the driver stops driving and turns off the camera and infrared array sensor.

[0013] In one embodiment of a driver distraction detection method of the present invention, the camera has embedded vision technology.

[0014] In one embodiment of a driver distraction detection method of the present invention, the infrared array sensor transmits data internally through a serial transceiver or a system on a chip (SoC).

[0015] In one embodiment of a driver distraction detection method of the present invention, the first transmission module includes a first Ethernet network, a first Camera Serial Interface (CSI), or a first Universal Serial Bus (USB).

[0016] In one embodiment of a driver distraction detection method of the present invention, the second transmission module includes a second Ethernet network, a second Camera Serial Interface (CSI), or a second Universal Serial Bus (USB).

[0017] In one embodiment of a driver distraction recognition method of the present invention, the heart rate calculation module includes a gaze direction algorithm, a heart rate extraction algorithm, and a heart rate change calculation method.

Implementation Method

[0019] To enable the esteemed examiner to understand the structural features, content, advantages, and effects of this invention, this invention will be described in detail with reference to the accompanying drawings and in the form of embodiments. The following description of specific examples of the drawings, elements, and arrangements used is for illustrative purposes only and to assist in the description. The content mentioned is not intended to limit this invention, and it is foreseeable that the shapes in the drawings will differ due to manufacturing processes and / or tolerances. Therefore, the scope of the invention in actual implementation should not be limited. The following will describe the illustrative embodiments in detail with reference to the drawings.

[0020] In this invention, unless otherwise explicitly specified and limited, the terms "set", "fixed", "connected", etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, a direct connection, or an indirect connection through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0021] In the description of this work, “first,” “second,” and “third” are used for descriptive purposes only and are not intended to indicate relative importance.

[0022] Furthermore, spatial relative terms, such as "below", "under", "lower than", "above", "front", "back", "left" and "right", are used to easily describe the relationship between the elements or features shown in the diagram; in addition, spatial relative terms include not only the directions depicted in the diagram, but also the different directions in which the elements are used or operated.

[0023] Please refer to Figure 1 first, which is a schematic diagram of the driver distraction recognition system of the present invention. It includes an infrared array sensor 1, a first transmission module 2, a vehicle computer 3, a second transmission module 4, and a camera 5. The first transmission module 2 is connected to the infrared array sensor 1 and the vehicle computer 3. The second transmission module 4 is connected to the vehicle computer 3 and the camera 5. The camera 5 and the infrared array sensor 1 are both mounted on the rearview mirror of the car and face the driver to detect the driver's face. The camera 5 is used to capture the driver's eyes and generate gaze direction information 35. The infrared array sensor 1 is used to detect the position of the driver's eyeballs and track the movement position and route of the driver's eyeballs, and then generate eye movement information 34. The vehicle computer 3 combines the gaze direction information 35 and the eye movement information 34 to generate an eye movement range, then executes an algorithm, and then uses the heart rate calculation module 36 to combine the eye movement range to determine whether the eye movement range exceeds the warning area and the driver's mental state.

[0024] The infrared array sensor 1 transmits internal information through the first serial transmitter 11 or the first single-chip system 12 (SoC).

[0025] The camera 5 transmits internal information through the second serial transmitter 51 or the second single-chip system 52.

[0026] The infrared array sensor 1 transmits data with the vehicle computer 3 through the first Ethernet 21, the first camera serial interface 22 or the first serial bus 23 in the first transmission module 2.

[0027] On the other hand, the vehicle computer 3 transmits data to the camera 5 through the second Ethernet 41, the second camera serial interface 42, or the second universal serial bus 43 in the second transmission module 4.

[0028] In one embodiment of the present invention, a FLIR Blackfly S series camera 5 (model: BFS-U3-51S5C-C) is used, paired with a specific sensor 53 (model: Sony IMX250 CMOS), which gives it high performance and low latency characteristics, supports thermal imaging, and also has embedded vision technology 54, which has a resolution of five million pixels and can capture high-quality images at a rate of 75 frames per second.

[0029] In another embodiment of the present invention, a Basler Ace 2 series camera 5 (model: Basler Ace 2 a2A1920-160ucBAS) is used, and it is also equipped with a specific sensor 53 (model: Sony IMX250 CMOS), which can capture images at a rate of 160 frames per second, has advanced image processing functions, and supports automatic exposure and gain control, and can accurately track the driver's eye position in real time.

[0030] In one embodiment of the present invention, the infrared array sensor 1 used is a Melexis MLX90640. This infrared array sensor 1 has 32*24 pixels, a field of view of 110°*75°, and uses an internal integrated circuit (I²C) as an interface, thus possessing both high sensitivity and low power consumption. In addition, its measurable temperature range is -40°C to 85°C, making it suitable for measuring the temperature distribution and range of a driver.

[0031] In another embodiment of the present invention, the infrared array sensor 1 used is a FLIR Lepton 3.5. This infrared array sensor 1 has 160*120 pixels, high resolution and good thermal imaging capability, and also has a field of view of 57°*42°. It uses a Serial Peripheral Interface (SPI) as the interface, and its measurable temperature range is -10°C to 80°C, which makes it have high resolution characteristics and can track the driver's temperature changes and movement position more in real time.

[0032] In one embodiment of the present invention, the vehicle computer 3 series NVIDIA Jetson AGX Orin used has an artificial intelligence computing capability of 200 trillion operations per second, and is equipped with a specific central processing unit 31 (8-core ARM Cortex-A78AE v8.2 64-bit CPU), a specific graphics processing unit (2048-core NVIDIA Ampere Architecture GPU), and a specific memory 33 (32 GB LPDDR5), which enables it to process a large amount of visual data in a very short time.

[0033] In another embodiment of the present invention, the vehicle computer 3 series Qualcomm Snapdragon Ride used has an artificial intelligence computing capability of 30 trillion operations per second, and is equipped with a specific central processing unit 31 (8-core Kryo 585 CPU), a specific graphics processing unit (Adreno 650 GPU) and a specific memory 33 (16 GB LPDDR5), which can support multi-sensor fusion and complex driving assistance functions.

[0034] Please refer to Figures 1 to 3 simultaneously. In this invention, a camera 5 installed on the rearview mirror continuously captures images of the driver's eyes while the driver is driving. The camera 5 can analyze the captured images using embedded vision technology 54, calculate the driver's gaze direction based on the driver's eye position, and generate the driver's gaze direction information 35. On the other hand, the infrared array sensor 1 can track the driver's temperature changes and movement position in real time, thereby generating eye movement information 34. In addition, the infrared array sensor 1 uses the internal integrated circuit (I²C) of the system on a chip (SoC) or the serial peripheral interface (SPI) of the serial transmitter to transmit data. Next, the vehicle computer 3 combines the gaze direction information 35 from the camera 5 and the eye movement information 34 from the infrared array sensor 1 to generate the eye movement range. The vehicle computer 3 communicates with the camera 5 and the infrared array sensor 1 via Ethernet, Camera Serial Interface (CSI), or Universal Serial Bus (USB) for data transmission. Figure 2 is a schematic diagram of the warning area of ​​the present invention. In one embodiment of the present invention, a first warning area 6A, a second warning area 6B, and a third warning area 6C are provided, located in the left rearview mirror, the center rearview mirror, and the right rearview mirror, respectively. The present invention does not particularly limit the number and location of the warning areas.

[0035] The vehicle computer 3 then uses an algorithm to determine whether the eye movement range exceeds the warning area. Simultaneously, the vehicle computer 3 also controls the operation of other devices within the vehicle, such as the reversing camera, in-car internet access, audio-visual entertainment, fault detection, and mobile communication. On the other hand, the heart rate calculation module 36 within the vehicle computer 3, combined with the eye movement range, determines the driver's mental state. If the final calculation result from the vehicle computer 3 indicates that the driver's mental state is poor, a warning sound will be played through the in-car audio system to remind the driver to pay attention to road and vehicle conditions until the driver's mental state returns to normal or the driving stops.

[0036] Please refer to Figure 3, which is a schematic diagram of the mental state judgment system of the present invention. The vehicle computer 3 further includes a heart rate calculation module 36. First, the gaze direction information 35 and eye movement information 34 are transmitted to the central processing unit 31 and the image processing unit 32. The central processing unit 31 and the image processing unit 32 combine the gaze direction information 35 and the eye movement information 34 to form the eye movement range. The heart rate calculation module 36 includes a gaze direction algorithm 361, a heart rate extraction algorithm 362, and a heart rate change calculation algorithm 363. Then, a mental state judgment result 37 is generated to determine whether the driver's mental state is normal, dazed, or fatigued. If the final judgment result is abnormal, a warning sound will be played through the in-vehicle audio system to remind the driver to pay attention to road and vehicle conditions until the driver's mental state returns to normal or the driving stops.

[0037] Please refer to Figures 2 and 3 simultaneously. In one embodiment of the present invention, the driver's mental state is determined by the angle between the eye movement range and the first warning area 6A, the second warning area 6B, or the third warning area 6C, wherein the angle is calculated using the following formula (1). Formula (1)

[0038] Referring to Figures 1 and 4, the present invention also provides a driver distraction detection method, which includes the following steps:

[0039] Step S110: After the driver sits in the car, the camera 5 and the infrared array sensor 1 are turned on. During the driving process, the camera 5 will continuously detect the driver's eyes, analyze the images captured during the photography process, and calculate the driver's gaze direction based on the driver's eye position to generate the driver's gaze direction information 35.

[0040] Step S120: On the other hand, during driving, the infrared array sensor 1 will track the driver's temperature changes and movement position in real time, and will also continuously track the driver's eye movement position, thereby generating driver's eye movement information 34.

[0041] Step S130: The vehicle computer 3 receives eye movement information 34 through the first transmission module 2 and gaze direction information 35 through the second transmission module 4. It then transmits the eye movement information 34 and gaze direction information 35 to the processing unit within the vehicle computer 3, including the central processing unit 31 and the image processing unit 32. The gaze direction information 35 and eye movement information 34 are combined to generate an eye movement range, and an algorithm is used to determine whether the eye movement range exceeds the warning area. Meanwhile, the heart rate calculation module 36 in the vehicle computer 3 combines the eye movement range to determine the driver's mental state.

[0042] Step S140: Determine if the driver's mental state is abnormal. Abnormal conditions include fatigue or distraction. After calculation, if the eye movement range exceeds the warning area, or the heart rate calculation result is abnormal, it indicates that the driver is unable to concentrate on driving.

[0043] Step S150: If, after step S140, the calculation result of the vehicle computer 3 indicates that the driver's mental state is abnormal, a warning sound will be played through the in-vehicle sound system to remind the driver to pay attention to the vehicle and road conditions, and steps S110 to S140 will continue to be repeated.

[0044] Step S160: If the calculation result of the vehicle computer 3 after step S140 is that the driver's mental state is normal, then continue to repeat steps S110 to S140 until the driver stops driving and turns off the camera 5 and the infrared array sensor 1.

[0045] The gaze direction information 35 obtained in step S110 is obtained by analyzing the image captured by the camera 5 through the embedded vision technology 54 of the camera 5.

[0046] In one embodiment of the present invention, the infrared array sensor 1 transmits data through the internal integrated circuit (I²C) of the system on a chip (SoC) or the serial peripheral interface (SPI) of the serial transceiver.

[0047] The first transmission module 2 includes a first Ethernet network 21, a first camera serial interface (CSI) 22, or a first universal serial bus (USB), which receives and transmits eye movement information 34.

[0048] The second transmission module 4 includes a second Ethernet network 41, a second camera serial interface 42 (CSI), or a second universal serial bus 43 (USB), which receives and transmits line-of-sight information 35.

[0049] In view of this, the driver distraction recognition system and method provided by the present invention combines multiple sensors for tracking, and the hardware device can also process large amounts of data in real time, which significantly improves the reaction speed and accuracy. Furthermore, data fusion is performed through artificial intelligence algorithms, thereby enhancing the accuracy and immediacy of the judgment of driving status. Thus, it solves the problems of insufficient accuracy of previous mental state judgment systems and inconvenience in measuring physiological information, and further improves driving safety.

[0050] In summary, the embodiments are only used to illustrate the content of the present invention and are not intended to limit the scope of the present invention. Therefore, any modifications or changes that can be easily conceived by those skilled in the art within the scope of the present invention can be covered within the patent scope of the present invention. [Simplified Explanation of the Diagram]

[0018] Figure 1 is a schematic diagram of the driver distraction recognition system of the present invention. Figure 2 is a schematic diagram of the warning area of ​​the present invention. Figure 3 is a schematic diagram of the mental state judgment system of the present invention. Figure 4 is a flowchart of the driver distraction recognition method of the present invention.

Claims

1. A driver distraction detection system, comprising: a camera that captures the eyes of a driver to obtain gaze direction information of the driver; an infrared array sensor that detects the position of the driver's eyeballs, performs eye tracking, and obtains eye movement information of the driver; a vehicle computer equipped with a heart rate calculation module that, by combining the gaze direction information and the eye movement information, generates an eye movement range, and uses an algorithm and the heart rate calculation module to determine whether the eye movement range exceeds a warning area and the driver's mental state, wherein the heart rate calculation module uses a gaze direction algorithm, a heart rate extraction algorithm, and a heart rate change calculation method to generate a mental state judgment result; a first transmission module; and a second transmission module; wherein the first transmission module is connected to the infrared array sensor and the vehicle computer; wherein the second transmission module is connected to the vehicle computer and the camera; The warning area includes a first warning area with the left rearview mirror, a second warning area with the middle rearview mirror, and a third warning area with the right rearview mirror; and the driver's mental state is judged by the angle between the eye movement range and the first warning area, the second warning area, or the third warning area, wherein the angle is calculated by the following formula (1).

2. The driver distraction detection system as described in claim 1, wherein the camera utilizes an embedded vision technology.

3. The driver distraction detection system as described in claim 1, wherein data transmission within the infrared array sensor is performed via a serial transceiver or a system on a chip (SoC).

4. The driver distraction detection system as described in claim 1, wherein the first transmission module includes a first Ethernet network, a first Camera Serial Interface (CSI), or a first Universal Serial Bus (USB).

5. The driver distraction detection system as described in claim 1, wherein the second transmission module includes a second Ethernet network, a second Camera Serial Interface (CSI), or a second Universal Serial Bus (USB).

Images