Fatigue driving detection method based on facial recognition analysis

Abstract

The invention relates to a fatigue driving detection method based on facial recognition analysis, and the method comprises the following steps: S1, video collection: during the driving of a vehicle, a camera collects the facial video of a driver at regular time, and after the continuous driving of the driver reaches a set time length, the camera automatically starts video collection to increase the monitoring of a fatigue state; s2, data extraction: extracting heart rate monitoring data, blood oxygen monitoring data, pulse wave detection data and respiratory rate detection data of a driver; s3, performing data analysis and fatigue judgment according to the data obtained in the step S1 and the step S2; and S4, outputting a result and warning. The method has the advantages that fatigue driving detection is realized through a non-contact facial recognition analysis method, and fatigue driving is judged through cooperation of multiple physiological indexes, so that the analysis method is more accurate and reliable.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving detection, and in particular to a fatigue driving detection method based on facial recognition analysis. Background Art

[0002] Fatigue driving refers to the physical and mental fatigue of the driver after long-term continuous driving, which leads to decreased reaction ability, lack of concentration, weakened judgment and weakened control of the vehicle, thereby increasing the risk of traffic accidents. Fatigue driving is one of the main causes of traffic accidents, posing a serious threat to people's lives and property safety. Therefore, the development of effective fatigue driving detection technology and solutions is of great significance to prevent traffic accidents.

[0003] With the continuous advancement of science and technology, it has become possible to detect physiological states using non-contact methods. At present, with the development of computer technology and artificial intelligence, fatigue driving detection technology based on machine vision and deep learning has gradually become a hot topic of research. This technology uses ordinary cameras to collect driver's facial information. Through the analysis and processing of the RPPG model, the RPPG model uses the camera to capture the periodic color changes caused by the blood flow pulsation caused by the heartbeat in the skin area, and then restores the corresponding BVP signal and measures physiological indicators. The principle of this technology is based on the three-layer structure of human skin, namely the epidermis, dermis and subcutaneous tissue layer, in which the dermis and subcutaneous tissue layers are rich in capillaries. The blood flow caused by the beating of the heart causes the blood flow of the capillaries and the hemoglobin content therein to change periodically, which in turn causes the skin to absorb light in a periodic manner. Although this change is very weak, it can still be captured by the camera. Based on this principle, the RPPG technology extracts the BVP signal and measures physiological indicators related to the cardiac cycle by analyzing the periodic color changes caused by the heartbeat in the skin area.

[0004] As people pay more and more attention to driving safety, automatic identification of driver fatigue has become a key research direction of automobile passive safety. However, current fatigue driving detection is based on artificially set rules, such as timed reminders to the driver and remote monitoring of the driver's driving behavior. However, the above methods are not only highly subjective, but also affect driving comfort and safety. There is an urgent need for a fatigue driving detection method that can accurately detect the driver's fatigue state based on the driver's physiological state. Summary of the invention

[0005] In view of the above problems, an object of the present invention is to provide a fatigue driving detection method based on facial recognition analysis to overcome the deficiencies of the above-mentioned prior art.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for detecting fatigue driving based on facial recognition analysis comprises the following steps:

[0008] Step S1: Video acquisition. During the driving of the vehicle, the camera regularly acquires the driver's facial video. When the driver has been driving continuously for a set time, the camera automatically starts video acquisition to increase the monitoring of fatigue status. The regular acquisition is once every 30 minutes. The facial video is acquired for ten seconds using the in-car camera at a frame rate of 30 frames per second. The set duration is continuous driving for more than 2 hours.

[0009] Step S2: data extraction, extracting the driver's heart rate monitoring data, blood oxygen monitoring data, pulse wave detection data, and respiratory rate detection data;

[0010] Step S3: Obtain data according to step S1 and step S2 to perform data analysis and fatigue determination.

[0011] Step S31: The normal heart rate range is 60-100 beats / minute. In a fatigued state, the heart rate will increase, and in a fatigued state, the heart rate will exceed 100 beats / minute or be lower than 60 beats / minute;

[0012] Step S32: Normal blood oxygen saturation is between 95% and 100%. When you are tired, the blood oxygen saturation will decrease. When it is lower than 95%, it indicates that you are in a fatigue state.

[0013] Step S33: The normal pulse range is 60-100 beats / minute, and when in fatigue state, it will exceed 100 beats / minute or be lower than 60 beats / minute;

[0014] Step S34: The blinking frequency is about 15-20 times per minute. When the blinking frequency is less than 15 times, it indicates that the eye is in a fatigue state.

[0015] Step S35: When one or more of the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 exceeds the above normal range, it is determined that there is a fatigue driving situation; wherein the fatigue driving situation is divided into a first-level fatigue index to a fourth-level fatigue index, wherein the first level is mild fatigue and the fourth level is severe fatigue; the first-level fatigue index refers to one of the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 exceeding the normal range, and the fourth-level fatigue index refers to the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 all exceeding the normal range;

[0016] Step S4: Result output and warning,

[0017] When the determination in step S3 is fatigue driving, the vehicle alarm system immediately warns the driver through sound, vibration or screen display to remind him to take a rest. At the same time, the data in step S3 is recorded and reported to the cloud data center for subsequent analysis and supervision.

[0018] As a preferred embodiment of the present invention, step S1 includes the following steps:

[0019] Step S1.1 When capturing video, the following conditions should be ensured:

[0020] (1) The lighting in the car is sufficient and uniform: avoid excessive or dark light that affects the clear capture of facial features;

[0021] (2) The driver’s face is not obstructed: Ensure that the driver does not wear a mask, hat, or other items that obstruct the face in a large area;

[0022] (3) The camera should be in a suitable position and angle: it should be installed in a position that can clearly capture the driver's entire face from the front without being interfered with by the vehicle interior or other objects.

[0023] As a preferred embodiment of the present invention, step S2 also includes the following steps:

[0024] Step S201: heart rate monitoring, according to the optical characteristics of light penetration and reflection of the skin, when light hits the skin surface, part of the light will be reflected back, while the other part will penetrate the skin and be absorbed by the blood. Since the blood volume changes periodically with the beating of the heart, the absorption of light also changes periodically. The camera of step S1 is used to obtain an image of the skin area, wherein the image includes the skin area with rich blood vessels, such as the forehead and cheeks;

[0025] Step S202: signal extraction, analyzing and processing the image acquired in step S201, extracting signals related to blood volume changes, including extracting changes in pixel values ​​in the image, specifically calculating changes in the mean and variance of the red, green, and blue color channels as original signals;

[0026] Step S204: signal processing, eliminating noise and interference from the original signal extracted in step S202;

[0027] Step S205: Physiological parameter calculation: based on the processed original signal as a pulse signal, the heart rate physiological parameters are obtained by calculating the frequency and period characteristics of the pulse signal.

[0028] As a preferred embodiment of the present invention, step S2 also includes the following steps:

[0029] Step S211: Blood oxygen monitoring data, using the camera in step S1 to capture light reflected or transmitted through the skin, to monitor the driver's blood oxygen status in real time and non-invasively, where the relationship between blood oxygen and skin tone is: by analyzing the intensity changes of light reflected or transmitted by the skin, and the absorption ratio of light of different wavelengths, the relative content of oxygenated hemoglobin and reduced hemoglobin in the blood can be inferred.

[0030] As a preferred embodiment of the present invention, step S2 also includes the following steps:

[0031] Step 221: Obtain a video image of the driver through the camera of step S1, focus on the chest area, use image analysis technology to detect the slight ups and downs of the chest, and then derive the breathing frequency.

[0032] The advantages and positive effects of the present invention are: fatigue driving detection is achieved through a non-contact facial recognition analysis method, and fatigue driving is determined collaboratively through multiple physiological indicators, making the analysis method more accurate and reliable. DETAILED DESCRIPTION

[0033] This embodiment provides a method for detecting fatigue driving based on facial recognition analysis, comprising the following steps:

[0034] Step S1: Video acquisition. During the driving of the vehicle, the camera regularly acquires the driver's facial video. When the driver has been driving continuously for a set time, the camera automatically starts video acquisition to increase the monitoring of fatigue status. The regular acquisition is once every 30 minutes. The facial video is acquired for ten seconds using the in-car camera at a frame rate of 30 frames per second. The set duration is continuous driving for more than 2 hours.

[0035] Step S2: data extraction, extracting the driver's heart rate monitoring data, blood oxygen monitoring data, pulse wave detection data, and respiratory rate detection data;

[0036] Step S3: Obtain data according to step S1 and step S2 to perform data analysis and fatigue determination.

[0037] Step S31: The normal heart rate range is 60-100 beats / minute. In a fatigued state, the heart rate will increase, and in a fatigued state, the heart rate will exceed 100 beats / minute or be lower than 60 beats / minute;

[0038] Step S32: Normal blood oxygen saturation is between 95% and 100%. When you are tired, the blood oxygen saturation will decrease. When it is lower than 95%, it indicates that you are in a fatigue state.

[0039] Step S33: The normal pulse range is 60-100 beats / minute, and when in fatigue state, it will exceed 100 beats / minute or be lower than 60 beats / minute;

[0040] Step S34: The blinking frequency is about 15-20 times per minute. When the blinking frequency is less than 15 times, it indicates fatigue, and the pupil will shrink when fatigued;

[0041] Step S35: When one or more of the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 exceeds the above normal range, it is determined that there is a fatigue driving situation; wherein the fatigue driving situation is divided into a first-level fatigue index to a fourth-level fatigue index, wherein the first level is mild fatigue and the fourth level is severe fatigue; the first-level fatigue index refers to one of the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 exceeding the normal range, and the fourth-level fatigue index refers to the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 all exceeding the normal range;

[0042] Step S4: Result output and warning,

[0043] When the determination in step S3 is fatigue driving, the vehicle alarm system immediately warns the driver through sound, vibration or screen display to remind him to take a rest. At the same time, the data in step S3 is recorded and reported to the cloud data center for subsequent analysis and supervision.

[0044] Furthermore, in this embodiment, step S1 includes the following steps:

[0045] Step S1.1 When capturing video, the following conditions should be ensured:

[0046] (1) The lighting in the car is sufficient and uniform: avoid excessive or dark light that affects the clear capture of facial features;

[0047] (2) The driver’s face is not obstructed: Ensure that the driver does not wear a mask, hat, or other items that obstruct the face in a large area;

[0048] (3) The camera is in the right position and angle: it should be installed in a position where the driver's entire face can be clearly captured from the front, without interference from the vehicle interior or other objects. By collecting video in the above situations and conditions, the driver's fatigue state can be more effectively monitored and driving safety can be improved.

[0049] Furthermore, in this embodiment, step S2 also includes the following steps:

[0050] Step S201: heart rate monitoring, according to the optical characteristics of light penetration and reflection of the skin, when light hits the skin surface, part of the light will be reflected back, while the other part will penetrate the skin and be absorbed by the blood. Since the blood volume changes periodically with the beating of the heart, the absorption of light also changes periodically. The camera of step S1 is used to obtain an image of the skin area, wherein the image includes the skin area with rich blood vessels, such as the forehead and cheeks;

[0051] Step S202: signal extraction, analyzing and processing the image acquired in step S201, extracting signals related to blood volume changes, including extracting changes in pixel values ​​in the image, specifically calculating changes in the mean and variance of the red, green, and blue color channels as original signals;

[0052] Step S204: signal processing, eliminating noise and interference from the original signal extracted in step S202;

[0053] Step S205: Physiological parameter calculation, based on the processed original signal as the pulse signal, the heart rate physiological parameters are obtained by calculating the frequency and period characteristics of the pulse signal. The RPPG model (existing model) is used as the basis for extracting the heart rate physiological parameters. The principle is to perform time series analysis on the intensity value of each pixel in the selected ROI (color channel separation) on the image color channel. The time series is an observation sequence arranged in chronological order. It is a continuous rise or continuous decline change presented by the time series over a long period of time. Therefore, the periodic change signal caused by blood flow can be extracted; wherein, the RPPG model belongs to the existing technology model. The RPPG model calculates the heart rate value according to the frequency corresponding to the main peak in the spectrum. This heart rate detection technology based on image processing has the advantages of non-contact, convenience and ease of use, but it is also easily affected by factors such as ambient light and motion interference, and corresponding measures need to be taken to improve the accuracy and stability of detection. Analyze the pulsation of the driver's neck blood vessels. The contrast and brightness adjustment values ​​are based on the quality and characteristics of the original image. The contrast will be adjusted to a specific ratio (such as every 15 minutes to 20 times), and the brightness will be increased or decreased by a certain value to highlight the characteristics of the blood vessels. The specific threshold value of the threshold segmentation is determined by experiment or experience to determine a specific grayscale value or color value range to accurately separate the blood vessels from the background. For example, pixels with a grayscale value between 80-150 are identified as blood vessels. The number of monitoring points will select several (such as 3-5) key monitoring points. The specific number depends on the morphology of the blood vessels and the accuracy requirements of the analysis. The displacement change measurement value is in the continuous frame. The displacement of the monitoring point will be measured in pixels. The monitoring point moves 5 pixels in a certain period of time. The final calculated heart rate value: in units of heart beats per minute, 60 beats / minute-100 beats / minute, etc.

[0054] Furthermore, in this embodiment, step S2 also includes the following steps:

[0055] Step S211: Blood oxygen monitoring data, using the camera in step S1 to capture the light reflected or transmitted through the skin, real-time and non-invasive monitoring of the driver's blood oxygen status, providing an important physiological indicator for judging the degree of fatigue. By understanding the changes in blood oxygen content, it can assist in judging whether the driver is in a state of fatigue or physical discomfort, thereby improving driving safety. The relationship between blood oxygen and skin tone is: by analyzing the intensity changes of light reflected or transmitted by the skin, and the absorption ratio of light of different wavelengths, the relative content of oxygenated hemoglobin and reduced hemoglobin in the blood is inferred. Among them, the flow of blood in the microvessels under the facial skin will cause slight changes in skin color, which can be detected by video image analysis technology. For example, the principle of photoplethysmography (RPPG) is adopted to extract color change information from each pixel or area of ​​interest (such as cheek) of the video image, and a signal similar to a pulse wave is obtained after signal processing. The pulse wave frequency range of a normal adult is generally 60-100 times / minute. In a resting state, some characteristic parameters of the pulse wave also have a certain range, for example, the pulse wave conduction velocity is 5-10 meters / second. However, these values ​​will change when the driver is fatigued driving. When the driver is fatigued, the heart rate (reflected in the pulse wave frequency) will change, such as a decrease in heart rate (lower than 60 beats / minute, because the drowsiness caused by fatigue leads to a low activity state of body function) or irregular changes in heart rate; at the same time, the amplitude and shape of the pulse wave also change, such as the amplitude becomes smaller, the waveform becomes duller, etc., and a sound alarm is issued, such as the voice prompt "Fatigue driving, please pay attention to rest", the sound can be relatively sharp or continuous to attract the driver's attention.

[0056] Furthermore, in this embodiment, step S2 also includes the following steps:

[0057] Step 221: Obtain the video image of the driver through the camera of step S1, focusing on the chest area (although facial recognition mainly focuses on the face, the upper part of the chest can also be partially covered within the field of view). Use image analysis technology (flow tracking method) to detect the slight ups and downs of the chest, and then derive the breathing rate. The optical flow following method can analyze the movement direction and speed of the pixels in the image. The ups and downs of the chest with breathing will cause the vertical movement of the pixels in the chest area. The breathing rate is estimated by calculating the frequency of this movement. The breathing rate of a normal adult in a quiet state is generally 12-20 times / minute. In the case of fatigue driving, the breathing rate will be abnormal. For example, when overly tired, the breathing rate will be lower than 12 times / minute, which is due to the slow breathing caused by the decline of physical function and the inhibition of the nervous system; or the breathing rate will be higher than 20 times / minute, because of the physical stress response caused by fatigue or potential health problems (such as anxiety caused by fatigue affecting breathing). Set a special warning light on the dashboard of the vehicle, for example, when an abnormal breathing rate is detected, a yellow or red warning light will light up. At the same time, relevant prompt information is displayed on the car's display screen, such as "Abnormal breathing rate, fatigue driving state, please take a rest."

[0058] Normal heart rate range in this embodiment: The heart rate of a normal adult in a resting state is generally 60-100 beats / minute. When the heart rate is continuously lower than 60 beats / minute (excluding physiological bradycardia in special groups such as athletes) or continuously higher than 100 beats / minute, it is related to physical fatigue or stress state related to fatigue driving. For example, when the body is extremely tired due to long-term driving, an abnormal heart rate will occur. For example, a continuous increase in heart rate is a stress response produced by the body to cope with fatigue.

[0059] Normal range of blood oxygen saturation: Normal blood oxygen saturation (SpO 2 ) is generally between 95% and 100%. If the blood oxygen saturation is lower than 95%, it indicates that the body is in a state of hypoxia. In the case of fatigue driving, the driver's blood oxygen saturation decreases due to fatigue-related factors such as shallow breathing and poor blood circulation. For example, if you maintain a posture while driving for a long time, your breathing function will be affected, which will in turn reduce the blood oxygen saturation.

[0060] Normal pulse range: The normal pulse frequency is basically consistent with the heart rate, which is 60-100 times / minute. An abnormal pulse (such as too fast, too slow or irregular rhythm) reflects the driver's physical fatigue or potential health problems. Similar to the heart rate, when the pulse continues to deviate from the normal range, it can be used as a basis for judging fatigue driving.

[0061] Taking all indicators into consideration, if these indicators are used to judge fatigue driving, there is such a judgment logic: when one or more indicators of heart rate, blood oxygen, and pulse exceed the above normal range, it can be judged that there is fatigue driving. However, it should be noted that these physiological indicators are affected by many factors, such as individual health status (certain diseases cause abnormal indicators), environmental factors (such as high altitude areas affect blood oxygen saturation), etc. Therefore, this judgment method should be combined with other fatigue driving judgment methods (such as driving behavior analysis, etc.) to comprehensively judge the fatigue driving state. , it is judged as fatigue driving.

[0062] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for detecting fatigue driving based on facial recognition analysis, characterized in that: The following steps are involved: Step S1: Video acquisition. During the driving of the vehicle, the camera regularly acquires the driver's facial video. When the driver has been driving continuously for a set time, the camera automatically starts video acquisition to increase the monitoring of fatigue status. The regular acquisition is once every 30 minutes. The facial video is acquired for ten seconds using the in-car camera at a frame rate of 30 frames per second. The set duration is continuous driving for more than 2 hours. Step S2: data extraction, extracting the driver's heart rate monitoring data, blood oxygen monitoring data, pulse wave detection data, and respiratory rate detection data; Step S3: Obtain data according to step S1 and step S2 to perform data analysis and fatigue determination. Step S31: The normal heart rate range is 60-100 beats / minute. In a fatigued state, the heart rate will increase, and in a fatigued state, the heart rate will exceed 100 beats / minute or be lower than 60 beats / minute; Step S32: Normal blood oxygen saturation is between 95% and 100%. When you are tired, the blood oxygen saturation will decrease. When it is lower than 95%, it indicates that you are in a fatigue state. Step S33: The normal pulse range is 60-100 beats / minute, and when in fatigue state, it will exceed 100 beats / minute or be lower than 60 beats / minute; Step S34: The blinking frequency is about 15-20 times per minute. When the blinking frequency is less than 15 times, it indicates that the eye is in a fatigue state. Step S35: When one or more of the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 exceeds the above normal range, it is determined that there is a fatigue driving situation; wherein the fatigue driving situation is divided into a first-level fatigue index to a fourth-level fatigue index, wherein the first level is mild fatigue and the fourth level is severe fatigue; the first-level fatigue index refers to one of the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 exceeding the normal range, and the fourth-level fatigue index refers to the heart rate, blood oxygen, pulse, and blinking frequency in steps S31 to S34 all exceeding the normal range; Step S4: Result output and warning, When the determination in step S3 is fatigue driving, the vehicle alarm system immediately warns the driver through sound, vibration or screen display to remind him to take a rest. At the same time, the data in step S3 is recorded and reported to the cloud data center for subsequent analysis and supervision.

2. A method for detecting fatigue driving based on facial recognition analysis according to claim 1, characterized in that: Step S1 includes the following steps: Step S1.1 When capturing video, the following conditions should be ensured: (1) The lighting in the car is sufficient and uniform: avoid excessive or dark light that affects the clear capture of facial features; (2) The driver’s face is not obstructed: Ensure that the driver does not wear a mask, hat, or other items that obstruct the face in a large area; (3) The camera should be in a suitable position and angle: it should be installed in a position that can clearly capture the driver's entire face from the front without being interfered with by the vehicle interior or other objects.

3. The method for detecting fatigue driving based on facial recognition analysis according to claim 1, characterized in that: Step S2 also includes the following steps: Step S201: heart rate monitoring, according to the optical characteristics of light penetration and reflection of the skin, when light hits the skin surface, part of the light will be reflected back, while the other part will penetrate the skin and be absorbed by the blood. Since the blood volume changes periodically with the beating of the heart, the absorption of light also changes periodically. The camera of step S1 is used to obtain an image of the skin area, wherein the image includes the skin area with rich blood vessels, such as the forehead and cheeks; Step S202: signal extraction, analyzing and processing the image acquired in step S201, extracting signals related to blood volume changes, including extracting changes in pixel values ​​in the image, specifically calculating changes in the mean and variance of the red, green, and blue color channels as original signals; Step S204: signal processing, eliminating noise and interference from the original signal extracted in step S202; Step S205: Physiological parameter calculation: based on the processed original signal as a pulse signal, the heart rate physiological parameters are obtained by calculating the frequency and period characteristics of the pulse signal.

4. The method for detecting fatigue driving based on facial recognition analysis according to claim 1, characterized in that: Step S2 also includes the following steps: Step S211: Blood oxygen monitoring data, using the camera in step S1 to capture light reflected or transmitted through the skin, to monitor the driver's blood oxygen status in real time and non-invasively, where the relationship between blood oxygen and skin tone is: by analyzing the intensity changes of light reflected or transmitted by the skin, and the absorption ratio of light of different wavelengths, the relative content of oxygenated hemoglobin and reduced hemoglobin in the blood can be inferred.

5. The method for detecting fatigue driving based on facial recognition analysis according to claim 1, characterized in that: Step S2 also includes the following steps: Step 221: Obtain a video image of the driver through the camera of step S1, focus on the chest area, use image analysis technology to detect the slight ups and downs of the chest, and then derive the breathing frequency.