A fatigue monitoring system based on computer vision and sensors

Abstract

The present invention discloses a fatigue monitoring system based on computer vision and sensors, which includes a driver face capture module, a driver body capture module, a driver body state monitoring module, a core main control, a response module, a data sharing module, and an exception handling module. The present invention can be easily deployed on an automobile without directly affecting the operation of the automobile, without affecting the safe driving of the automobile, and is easy to install. After testing, the sensitivity of this method is more than 4 times higher than that of the method of detecting fatigue and dangerous driving through the running state of the automobile. The fatigue detection method of the present invention uses a plurality of data including the internal state and the external state of the driver's body for fusion analysis, which greatly reduces the situation of fatigue misjudgment while ensuring the fatigue monitoring accuracy. Compared with the fatigue detection method that only uses computer vision or only wears sensors, the misjudgment rate is reduced by more than 40%.

Description

Technical Field

[0001] The present invention relates to a fatigue monitoring system based on computer vision and sensors, and belongs to the technical field of monitoring systems. Background Art

[0002] In current existing driver fatigue monitoring systems, most of the monitoring methods are relatively single.

[0003] Patent Publication No. CN103310590A discloses a driver fatigue analysis and warning system and method. The system includes a camera unit, a fatigue analysis unit, and an alarm device. The camera unit is used to obtain real-time video information of the driver. The fatigue analysis unit is used to receive the video information obtained by the camera unit to determine whether the driver is fatigued. The fatigue analysis unit includes one or more of a face capture detector, a face pose recognition device, a living body recognition device, and a face attribute analysis device. The alarm device is used to receive the alarm signal sent by the fatigue analysis unit and issue an alarm message. The present invention can analyze the fatigue degree of the driver, has a high enough recognition accuracy, is well compatible with the existing monitoring network to achieve timely warning, and has stable operation, is easy to upgrade and maintain, and has low cost. The present invention can detect the driver's fatigue driving and give a timely warning, and can effectively avoid traffic accidents caused by fatigue driving.

[0004] Patent Publication No. CN207594736U discloses a fatigue driving detection and reminder system based on vehicle networking, including: an intelligent vehicle terminal, which includes an interface module connected to the data interface of the vehicle, a collection module connected to the interface module, and a communication module connected to the collection module. The collection module is used to obtain driving data; an electroencephalogram detection module, which is connected to the collection module and is used to detect the fatigue degree data of the driver; a remote server, which is connected to the communication module; and a mobile terminal, which is connected to the remote server. The utility model collects driving data and driver fatigue degree data through the collection module respectively. The communication module sends the data to the remote server, and the remote server analyzes the data and transmits the analysis result to the mobile terminal to remind the driver to correct the driving behavior, solving the problem that the existing technology is not objective and comprehensive enough in monitoring the driver's driving behavior.

[0005] As described above, in a part of the two prior arts, only the driver's state is detected through computer vision, and in a part, only the vehicle running state is detected to judge the driver's fatigue degree. The confidence levels of the results detected by these two methods are both relatively low. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a fatigue monitoring system based on computer vision and sensors to solve the problems raised in the above background art.

[0007] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0008] A fatigue monitoring system based on computer vision and sensors, including a driver face capture module, a driver body capture module, a driver body state monitoring module, a core main control, a response module, a data sharing module, and an exception handling module;

[0009] The driver face capture module, the driver body capture module, and the driver body state monitoring module transmit the collected data to the core main control, and the core main control processes and analyzes the obtained driver face, driver body state, and body state data;

[0010] The response module makes corresponding response actions according to the fatigue detection result calculated by the core main control;

[0011] The data sharing module transmits the data in the vehicle to the cloud for real-time recording;

[0012] The exception handling module gives a reminder and self-repairs when an exception occurs during the operation of the system.

[0013] As a further improvement of the present invention, the driver face capture module is used to capture the facial features of the driver, and by detecting key points on the driver's face, the movement behavior of the driver's facial organs is obtained, so as to evaluate the fatigue degree.

[0014] As a further improvement of the present invention, the driver body capture module is used to collect the sitting posture of the driver, and the fatigue degree of the driver is judged by the change of the driver's sitting posture.

[0015] As a further improvement of the present invention, the driver body state monitoring module collects heart rate information through a bracelet and converts it into an electrical signal.

[0016] As a further improvement of the present invention, the core main control is a GPU computing device with jetson nano as the main body, which is responsible for data operation.

[0017] jetson nano is a high-performance AI edge terminal, and GPU refers to a graphics processing unit.

[0018] As a further improvement of the present invention, the response module includes a bracelet and an alarm device; the alarm device is a flash light, a sound, and an awakening spray installed in the cockpit.

[0019] As a further improvement of the present invention, the driver face capture module uses a neural network method to train the face data through a target detection neural network, and uses the trained face target detection model to locate the face.

[0020] As a further improvement of the present invention, the driver face capture module and the driver body capture module use an infrared camera and a high-definition camera installed at the upper end of the vehicle front windshield to collect the driver's head image;

[0021] The infrared camera and the high-definition camera are connected to the core main control through wireless connection or wired connection.

[0022] As a further improvement of the present invention, the bracelet is worn on the driver's wrist, and the bracelet is connected to the main control through Bluetooth to transmit data.

[0023] The beneficial effects produced by adopting the above technical solutions are as follows:

[0024] The present invention can be easily deployed on the vehicle without directly affecting the operation of the vehicle, can not affect the safe driving of the vehicle, and is easy to install at the same time. The whole set of systems only needs to install a small camera in front of the driver's seat and place the main control device in the idle position in the vehicle. At the same time, the bracelet worn by the driver is small and comfortable to wear. After testing, the sensitivity of this method is more than 4 times higher than that of the method of detecting fatigue and dangerous driving through the vehicle running state, and the interval time from the driver's fatigue degree to the dangerous value to the warning is less than 2s. It can ensure the safe driving of the driver in real time while not affecting the driver's normal driving of the vehicle.

[0025] The fatigue detection method of the present invention uses a number of data including the driver's internal state and external state for fusion analysis, which greatly reduces the situation of fatigue misjudgment while ensuring the accuracy of fatigue monitoring. Compared with the fatigue detection methods that only use computer vision or only use wearable sensors, the misjudgment rate is reduced by more than 40%. Description of the Drawings

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 It is a schematic diagram of the working process of the present invention. Detailed Embodiments

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and in no way limits the present application and its application or use. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0029] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments of the present application. As used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0030] Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and values set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn in actual proportional relationships. Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods, and devices should be regarded as part of the authorized specification. In all the examples shown and discussed here, any specific value should be construed as merely exemplary and not as a limitation.

[0031] Therefore, other examples of the exemplary embodiments may have different values. It should be noted that: like reference numerals and letters denote like items in the following drawings, and thus, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0032] In the description of the present application, it should be understood that the orientation or positional relationships indicated by orientation words such as "front, back, up, down, left, right", "horizontal, vertical, perpendicular, horizontal", and "top, bottom" are generally based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description. Without contrary description, these orientation words do not indicate and imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and thus cannot be construed as limiting the protection scope of the present application; the orientation words "inside, outside" refer to the inside and outside relative to the outline of each component itself.

[0033] Embodiment 1

[0034] This embodiment provides a driver fatigue monitoring system, including:

[0035] The driver face capture module is used to capture the facial features of the driver, and by detecting key points on the driver's face, obtain the movement behavior of the driver's facial organs, so as to evaluate the fatigue level.

[0036] The driver body capture module is used to collect the driver's sitting posture and judge the driver's fatigue level through changes in the driver's sitting posture.

[0037] The driver body status monitoring module: The regular beating of the heart will cause changes in blood light transmittance. Collect this change information and convert it into an electrical signal, which corresponds to the heart rate information. At one end of the bracelet in contact with the skin, there is a generator and a photosensitive receiver. When the generator irradiates the skin surface with a light beam of a certain wavelength, the light beam will return to the photosensitive receiver through transmission or reflection. During this process, due to the absorption and attenuation of the light beam by the wrist skin muscle tissue and blood, the intensity of the light detected by the detector will decrease. In the actual scenario, when the driver is fatigued while driving, obvious changes such as a decrease in heart rate will occur in the driver's internal physiological state. The core main control processes and analyzes the data transmitted from the driver's face, the driver's body status, and the body status monitoring bracelet. The main control needs to have GPU computing power;

[0038] The response module: The response module mainly includes alarm devices (including flashlights, speakers, and wake-up sprays) on the bracelet and in the vehicle. The response module will make corresponding response reactions according to the fatigue detection results calculated by the core main control, such as Figure 1 As shown, when the fatigue level does not exceed 60, the response system does not make a reaction. When the fatigue level is greater than 60 and less than 80, the response system will give a voice reminder, reminding the driver that they are currently fatigued while driving and asking them to rest in time. When the fatigue level is greater than or equal to 80, at this time the fatigue level has reached a relatively dangerous level and an accident may occur at any time. At this time, the response system will increase the frequency and intensity of the voice reminder, and at the same time give the driver a boost by releasing wake-up sprays and other means.

[0039] The data sharing module: The data in the vehicle (including the driver's physiological state) will be transmitted to the cloud in real time for recording, which is convenient for subsequent data analysis. At the same time, the cloud can know the running state of the currently fatigued driving vehicle in order to give emergency reminders.

[0040] The exception handling module: When the system runs abnormally, corresponding reminders and self-repair operations are performed.

[0041] Embodiment 2

[0042] This embodiment provides a face acquisition module for a driver fatigue monitoring system. The face acquisition module mainly uses a neural network method to train face data through a target detection neural network and uses the trained face target detection model to locate the face. After obtaining the face information, the system will first classify the face according to the fatigue level. Similarly, this step is also carried out using a neural network method. First, the neural network is used to train facial pictures with different fatigue levels to obtain a classifier model capable of discretely locating the fatigue level of the face. This model is used to discretely locate the fatigue level of the driver to obtain the fatigue level vision_1. Then, the system will perform key point detection on the acquired facial features. Through key point detection, the positions of the driver's eye organs can be obtained. Then, by analyzing the changes in the blinking frequency and amplitude of the driver's eyes over a time line, the analysis method is to collect the blinking frequency and opening / closing degree within 10 minutes, take the data within every 60s as a data point, and observe the change trend of the blinking frequency and opening / closing degree on this time line. If there is a continuous decreasing trend in the data, the degree of data reduction is used as the fatigue level vision_2. Substitute vision_1 and vision_2 into the non-linear fatigue level solving equation to solve the final computer vision fatigue level fatigue_vision. Then, monitor the heart rate data measured on the monitoring bracelet, take the change curve of the heart rate within 10 minutes, and average the data every 20s as valid data points. Then, compare with the heart rate value in the driver's normal state, and use the degree of weakening of the heart rate as the obtained fatigue level fatigue_sensor. Perform non-linear calculation on fatigue_vision and fatigue_sensor to obtain the final fatigue level fatigue. Here, the non-linear calculation means that under normal circumstances, fatigue = fatigue_vision * 0.6 + fatigue_sensor * 0.6. When data anomalies occur (when one party's data jumps out of the normal range), then fatigue = (the still normal data). If both are abnormal, fatigue still takes the value calculated last time and does not change. In this way, the confidence level of the fatigue level obtained by combining the data of computer vision and the monitoring bracelet is higher, and the fatigue misjudgment situation that occurs in ordinary detection methods is greatly reduced.

[0043] Embodiment III

[0044] As Figure 1As shown in the figure, this embodiment provides the following specific implementation steps for a driver fatigue monitoring system: First, perform hardware installation. Install the infrared camera and the ordinary high-definition camera at the upper end of the vehicle's front windshield. The specific position should not affect the driver's normal driving of the vehicle and both cameras can obtain the driver's head image normally. The connection method between the camera and the host can be freely selected according to the space inside the vehicle. The IP camera is connected wirelessly and the USB camera is connected to the core main control by wire. The core main control is a GPU computing power device with Jetson Nano as the main body, responsible for data operation. It can be installed at an idle position inside the vehicle. Then, wear the heart rate monitoring bracelet on the driver's wrist, and the bracelet transmits data to the main control through Bluetooth connection. All relevant hardware installations are completed here.

[0045] When the system starts to operate, first, the real-time image obtained by the ordinary high-definition camera in front of the driver is transmitted to the core main control. The program in the main control first calculates the average value of the V component (brightness) of the image in the HSV color space. When the average value of the V component of the pixel point is less than 40, the system will judge that the current environment is a dark environment. At this time, the system will select to receive the image data of the infrared camera as Img_1 until the image in the ordinary high-definition camera is not in a dark environment, and then the system will re-select the image data of the high-definition camera as Img_1.

[0046] The real-time image data of the driver will be transmitted to the core main control through the camera, and the main control will process the driver's image data Img_1 according to the pre-set program. The system will first run the target detection program, and use the target detection algorithm (YOLOv4) for face positioning and classification (the model file is the model pre-trained with driver fatigue images). At this time, the system can obtain the diagonal point coordinates of the smallest external rectangle of the driver's face in the image, the upper left corner coordinates (X1, Y1) and the lower right corner coordinates (X2, Y2). At the same time, the system can perform a discrete classification of the fatigue degree of the face, divided into four grades: normal, slightly fatigued, fatigued, and severely fatigued, corresponding to scores of 0, 30, 60, and 100 respectively. Assign the score to weight_1.

[0047] The face coordinates obtained in step c are used to segment the coordinate part of the image Img_2. The Img_2 image is the face image. Key point detection is performed on Img_2, also using deep learning. Using the PFLD algorithm, the coordinate points around the eyes can be obtained in Img_2. By judging the opening and closing degree of the eyes represented by these coordinate points, the blinking frequency of the driver can be calculated. The specific calculation method is to calculate the maximum and minimum points of the Y-axis coordinate values of the coordinate points around the human eyes in the image coordinate system, and calculate the difference D_value (pixel difference) between the two. When the driver opens his eyes, D_value reaches the maximum value. When the driver closes his eyes, D_value reaches the minimum value. Each process of D_value from the maximum to the minimum is a blinking process. The number of blinks within 10s is the blinking frequency eye_frequency, and the value of D_value is the opening and closing degree of the eyes eye_range. The eye_frequency and eye_range within 60s are linearly fitted with respect to time t to obtain the slope k_fre of the eye_frequency data and the slope k_ran of the eye_range. At the same time, calculate the average values (absolute values) mean_fre and mean_ran of eye_frequency and eye_range, and calculate the differences D_fre and D_ran between them and the normal blinking frequency common_fre and opening and closing amplitude common_ran of the driver. Then the fatigue calculation formula at this time is weight_2 = -c * (k_fre * D_fre + k_ran * D_ran), where c is the balance weight factor, usually taken as 4.

[0048] Fuse the fatigue factors obtained by the visual method to obtain the final visual processing result fatigue_v. The solution formula is fatigue_v = weight_1 + weight_2.

[0049] The bracelet worn by the driver can obtain the driver's heart rate data in real time. The bracelet takes 1 minute as the time interval, collects the heart rate heart_rate and uploads the data to the core main control in real time in the format of 8-bit integer numbers through Bluetooth. The core main control also linearly fits the heart_rate data within 10 minutes to obtain the slope k_hr and the mean value mean_hr, and then gets the fatigue measurement processing result of the bracelet heart rate data fatigue_s = -10 * k_hr * (75 - mean_hr)

[0050] The final fatigue fatigue has a solution formula: fatigue = (fatigue_v + fatigue_s) / 2. Perform a threshold judgment on the fatigue calculated in g, and the control factor control has a solution formula:

[0051] Perform corresponding actions according to the obtained control value as follows:

[0052]

[0053] The loop of the steps between b and j completes the real-time monitoring of the driver's fatigue state.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; it is obvious for those skilled in the art to combine multiple technical solutions of the present invention. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fatigue monitoring system based on computer vision and sensors, characterized in that: It includes a driver face capture module, a driver body capture module, a driver body state monitoring module, a core main control, a response module, a data sharing module, and an exception handling module; The driver face capture module, the driver body capture module, and the driver body state monitoring module transmit the collected data to the core main control, and the core main control processes and analyzes the obtained driver face, driver body state, and body state data; The response module makes corresponding response reactions according to the fatigue detection results calculated by the core main control; The data sharing module transmits the data in the vehicle to the cloud for real-time recording; The exception handling module gives reminders and self-repairs when the system runs abnormally; The driver face capture module uses a neural network method to train face data through a target detection neural network and uses the trained face target detection model for face positioning; The driver face capture module and the driver body capture module use an infrared camera and a high-definition camera installed at the upper end of the vehicle front windshield to collect the driver's head images; The infrared camera and the high-definition camera are connected to the core main control through wireless connection or wired connection; The driver's real-time image data will be transmitted to the core main control through the camera. The main control will process the driver's image data Img_1 according to a predetermined program. The system will first run the target detection program and use the target detection algorithm YOLOv4 for face positioning and classification. The model file of the target detection algorithm is a model pre-trained with driver fatigue images. At this time, the system can obtain the diagonal point coordinates of the smallest external rectangle of the driver's face in the image, the upper left corner coordinates (X1, Y1) and the lower right corner coordinates (X2, Y2). At the same time, a discrete classification of the fatigue degree of the face can be carried out, which is divided into four grades: normal, slightly fatigued, fatigued, and severely fatigued, corresponding to scores 0, 30, 60, and 100 respectively, and the score is assigned to weight_1; The face coordinates obtained through the above steps are used to segment the coordinate part of the image Img_2. The Img_2 image is the face image. Key point detection is performed on Img_2, also using the deep learning method. The PFLD algorithm can obtain the coordinate points around the eyes in Img_2. By judging the opening and closing degree of the eyes represented by these coordinate points, the blinking frequency of the driver can be calculated. The specific calculation method is to calculate the maximum and minimum points of the Y-axis coordinate values of the coordinate points around the human eyes in the image coordinate system, and calculate the difference between the two, which is the pixel difference D_value. When the driver opens his eyes, D_value reaches the maximum value. When the driver closes his eyes, D_value reaches the minimum value. Each process of D_value from the maximum to the minimum is a blinking process. The number of blinks within 10s is the blinking frequency eye_frequency, and the value of D_value is the opening and closing degree of the eyes eye_range. The eye_frequency and eye_range within 60s are linearly fitted with respect to time t to obtain the slope k_fre of the eye_frequency data and the slope k_ran of the eye_range. At the same time, the average values mean_fre and mean_ran of the eye_frequency and eye_range are calculated. The average values are also absolute values. Calculate the differences D_fre and D_ran between them and the normal blinking frequency common_fre and the opening and closing amplitude common_ran of the driver in the normal state. Then the fatigue calculation formula at this time is weight_2 = -c*(k_fre*D_fre + k_ran*D_ran), where c is the balance weight factor, taking 4. The fatigue factors obtained by the visual method are fused to obtain the final visual processing result fatigue_v. The solution formula is fatigue_v = weight_1 + weight_2.

2. The fatigue monitoring system based on computer vision and sensors according to claim 1, wherein: The driver face capture module is used to capture the facial features of the driver, and by performing key point detection on the driver's face, obtain the movement behavior of the driver's facial organs, so as to evaluate the fatigue degree.

3. The fatigue monitoring system based on computer vision and sensors according to claim 1, characterized in that: The driver body capture module is used to collect the driver's sitting posture and judge the driver's fatigue degree through the change of the driver's sitting posture.

4. A fatigue monitoring system based on computer vision and sensors according to claim 1, characterized in that: The driver body state monitoring module collects heart rate information through a bracelet and converts it into an electrical signal.

5. A fatigue monitoring system based on computer vision and sensors according to claim 1, characterized in that: The core main control is a gpu computing power device with jetsonnano as the main body, responsible for data operation.

6. The fatigue monitoring system based on computer vision and sensors according to claim 1, characterized in that: The response module includes a bracelet and an alarm device; the alarm device is a flash light, a sound and a wake-up spray installed in the cockpit.

7. A fatigue monitoring system based on computer vision and sensors according to claim 4, characterized in that: The bracelet is worn on the driver's wrist, and the bracelet is connected to the main control through Bluetooth to transmit data.

Images