Fatigue early warning system and method based on steering wheel

Abstract

The invention relates to a fatigue early warning system and method based on a steering wheel, and the method comprises the following steps: carrying out the real-time fatigue detection of a driver based on a heart rate detection module disposed on the steering wheel, obtaining a plurality of face images of the driver in a preset time period, inputting the obtained images into an eye opening and closing state evaluation model, and obtaining a fatigue detection result, and reminding a target object according to the fatigue detection result. According to the invention, an eye opening and closing state evaluation model is combined with heart rate detection, so that a comprehensive fatigue early warning system can be provided for a driver. Through real-time monitoring and accurate algorithm analysis, early warning can be given out in time when a driver is in a fatigue state, accuracy of clearance height detection and driving safety are ensured, and accidents are prevented.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue warning systems, and in particular to a fatigue warning system based on a steering wheel and a method thereof. Background Art

[0002] Large-cargo transport fleets travel long distances and take long driving time, so they usually choose to avoid peak hours and usually need to travel at night or early morning when there are fewer vehicles to avoid peak congestion. Especially for the benchmark vehicles of special large-cargo transport fleets, it is necessary to observe while driving to confirm the clearance height. If the driver is tired and does not carefully observe the distance between the benchmark and the bottom of the bridge or tunnel, or does not stop several times to check the distance between the benchmark and the bottom of the bridge or tunnel, it may result in failure to detect the actual height of the bottom of the bridge or tunnel that can pass, resulting in the inability of subsequent fleet transport vehicles to pass, which will affect traffic and the quality of large-cargo transportation.

[0003] Therefore, there is an urgent need to provide a steering wheel-based fatigue warning system and method thereof to solve the problems in the background technology. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, a fatigue warning system based on a steering wheel and a method thereof are provided.

[0005] The technical solution adopted by the present invention to solve the technical problem is a fatigue warning method based on a steering wheel, comprising the following steps: S100: Performing real-time fatigue detection on the driver based on the heart rate detection module provided on the steering wheel, and acquiring a plurality of facial images of the driver within a preset time period, S200: Input the acquired image into the eye opening and closing state evaluation model to obtain fatigue detection results. S300: Prompt the target object according to the fatigue detection result.

[0006] Furthermore, S2 includes the following steps: S201: Identify facial contours and extract important key feature points of the face; S202: determining the position of the driver's face relative to the central control screen, determining the orientation of the face and the angle of the face relative to the central control screen, so as to determine whether the driver is monitoring the measurement data displayed on the central control screen; S203: inputting the eye image into the eye opening and closing state evaluation model, and determining the eye opening and closing state in the image by matching the image set in the model where the head is at the same position; S204: Obtain fatigue detection results based on the eye opening and closing conditions.

[0007] Furthermore, in S201, the acquired driver's facial image is used to recognize the face through MediaPipe, and important key feature points of the face, including eyes, nose, and mouth, are extracted from the detected facial contour.

[0008] Furthermore, an invalid image set is determined, and the eye opening and closing conditions in the image confirm the closed eyes image set and the open eyes image set. Each image in the set is recorded once. The closed eyes image set has N images, which is recorded as N times of closed eyes. The open eyes image set has M images, which is recorded as M times of open eyes. The invalid image set has L images, which is recorded as L times. That is, the number of invalid pictures is L times, the number of valid pictures is N+M times, and the total number is L+N+M times, where L, M, and N are natural numbers.

[0009] Furthermore, in S204, the number of invalid photos is divided by the total number of times to calculate the proportion of invalid photos in the total number of times, so as to determine the time when the central control screen is not monitored. If the proportion is greater than the first threshold, it is determined that the driver has not performed the monitoring task and there is a possibility of being in a fatigued state and fatigue driving. If the proportion is less than or equal to the first threshold, it is determined that the driver is performing the monitoring task.

[0010] Furthermore, in S204, the proportion of the number of eye closing times in the number of valid images is calculated based on the number of eye closing times and the number of eye opening times. If the proportion is greater than a second threshold, it is determined that the driver is in a fatigue state.

[0011] Furthermore, in S204, if the proportion is greater than the third threshold, it is determined that the driver is in a serious fatigue state, and the vehicle fatigue driving information will be sent directly to the terminal device of the team leader to prompt the driver to take a rotation and re-check the clearance height situation. At the same time, an alarm will be sounded in the vehicle and a warning light will flash outside the vehicle to alert subsequent vehicles in the fleet.

[0012] Furthermore, in S203, the rotation vector and the translation vector are calculated by the solvePnP algorithm using the detected 3D model points, thereby determining the angle between the object coordinate system and the camera coordinate system, and the translation vector determines the position of the head in the camera coordinate system.

[0013] Furthermore, in S203, the eye region image is input into the trained eye opening and closing state evaluation model, and the opening and closing state of the eyes is determined by comparing the states of the eyes in the image.

[0014] The present invention also provides a fatigue warning system based on a steering wheel, including an image acquisition module for collecting driver facial information; a clearance height detection module, including an infrared ranging sensor arranged at the bottom of the benchmark, for measuring the height from the benchmark vehicle to the bottom of the bridge or the bottom of the tunnel; a heart rate detection module for detecting the driver's blood oxygen and heart rate; a display module for displaying the clearance height result, and an alarm module for emitting a sound signal according to a control signal for warning. If the proportion is greater than the first threshold, it is determined that the driver has not implemented the monitoring task and there is a possibility of being in a fatigued state and fatigue driving. If the proportion is less than or equal to the first threshold, it is determined that the driver is implementing the monitoring task.

[0015] The present invention is reasonably designed and has the following advantages: The present invention combines the eye opening and closing state evaluation model with heart rate detection to provide a comprehensive fatigue warning system for the driver. Through real-time monitoring and accurate algorithm analysis, it can issue a warning in time when the driver is in a fatigue state, ensure the accuracy of clearance height detection and driving safety, and prevent accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a method flow chart of a fatigue warning method based on a steering wheel in an embodiment of the present invention.

[0018] Figure 2 It is a specific flow chart of step S200 in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0020] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form also includes the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this description, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0021] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The steering wheel-based fatigue warning system includes an image acquisition module for collecting the driver's facial information; a clearance height detection module, including an infrared ranging sensor arranged at the bottom of the benchmark, for measuring the height from the benchmark vehicle to the bottom of the bridge or the bottom of the tunnel; a heart rate detection module for detecting the driver's blood oxygen and heart rate; a display module for displaying the clearance height result, and an alarm module for emitting a sound signal for warning according to a control signal.

[0023] It should be noted that the benchmark of the benchmark vehicle is set on the roof, and the infrared distance sensor is located at the bottom of the benchmark. The actual height of the bottom of the bridge and the tunnel can be determined by the benchmark and the infrared distance sensor, so as to confirm the maximum clearance height of the bridge, and then confirm whether there are obstacles at the bottom of the bridge or the bottom of the tunnel, and whether the passage of the fleet will be restricted. The benchmark is an adjustable benchmark, and its height represents the actual height of the highest large item in the fleet. According to the height measurement from the benchmark vehicle to the bottom of the bridge detected by the infrared distance sensor, the clearance height of the bridge and the tunnel can be detected to ensure that the vehicle will not collide with the bridge when passing. The bottom of the bridge and the tunnel can be dynamically monitored in real time, and the height data of the bottom of the bridge and the tunnel can be continuously obtained to help detect abnormal situations such as approaching or exceeding the benchmark height, so as to avoid exceeding the highest vehicle height of the fleet. The infrared distance sensor is not affected by ambient light, which improves the detection accuracy and working stability of the clearance height detection module, and is friendly to the bottom of the bridge and the tunnel environment. However, the rough and uneven surface of the bridge bottom and tunnel will affect the reflection and diffusion of infrared rays, resulting in the accuracy of the detection results. Therefore, when the control module determines that the clearance height is abnormal, the driver needs to stop the car to check the on-site situation and manually determine whether there is an actual obstruction or the detection data is inaccurate due to the uneven bottom of the bridge or tunnel.

[0024] The display module includes a central control screen located on the right side of the steering wheel, which is used to display the clearance height result. In this embodiment, the clearance height is the sum of the detection height detected by the infrared ranging sensor and the vehicle height. It should be noted that the maximum height of the cargo-carrying vehicle is the sum of the preset benchmark height and the vehicle height. If the detection height minus the benchmark height, combined with the measurement error coefficient is greater than or equal to the preset threshold, that is, all fleet vehicles can pass safely. If it is less than or equal to the threshold, the ECU vehicle electronic control unit will send an alarm signal to the alarm module and the central control screen display module. Generally, when the benchmark vehicle performs a clearance height detection, the road section will be restricted and closed, the vehicle speed should be less than 5km / h, preferably less than 3km / h, and when the vehicle speed exceeds 5km / h, the ECU vehicle electronic control unit will also send an alarm signal to the alarm module and the central control screen display module.

[0025] The heart rate detection module is arranged on the steering wheel, and includes a first blood oxygen heart rate detection unit and a second blood oxygen heart rate detection unit, and the first blood oxygen detection unit and the second blood oxygen detection unit are respectively located at opposite ends of the steering wheel grip part. The blood oxygen detection unit is used to detect the driver's heart rate, and with the help of the sensor part in contact with the skin, the driver's heart rate and blood oxygen level data are identified, and the data is transmitted to the ECU vehicle control unit. When the blood oxygen level is lower than the fourth threshold or the heart rate is lower than the fifth threshold, it is determined that the driver is likely to be in a fatigue state. The ECU vehicle control unit will send a warning signal to the central control display screen, and the display screen pops up to display a fatigue driving reminder. At the same time, the steering wheel and the horn in the car will issue an in-car alarm, and a voice prompt "Please keep driving focused" will be issued. When the blood oxygen value and the heart rate value are not detected, the driver is not driving the car, and the ECU vehicle control unit sends a standby signal to the infrared ranging sensor and the image acquisition module.

[0026] The heart rate detection module determines the driver's state through vital signs information. When the blood oxygen detection unit does not detect vital signs information and the image detection unit does not detect a face image, it means that the driver is stopping the car to check the on-site situation. At this time, the ECU vehicle control unit does not send an alarm or prompt information. It should be noted that when the infrared ranging sensor detects abnormal data and the control module determines it as an abnormal result, the driver needs to stop the car to check the on-site situation and manually determine whether there is an actual obstruction or the detection data is inaccurate due to the uneven bottom of the bridge or tunnel. During this period, the driver did not drive, that is, the driver did not perform the monitoring task. At this time, the ECU vehicle control unit does not send an alarm or prompt information, and there is no need to press the detection button again to turn off the detection function of the image acquisition module.

[0027] The present invention also provides a fatigue warning method based on a steering wheel, comprising the following steps: S100: Performing real-time fatigue detection on the driver based on the heart rate detection module provided on the steering wheel, and acquiring a plurality of facial images of the driver within a preset time period, S200: Input the acquired image into the eye opening and closing state evaluation model to obtain fatigue detection results. S300: Prompt the target object according to the fatigue detection result.

[0028] S200 includes the following steps: S201: Identify facial contours and extract important key feature points of the face; S202: determining the position of the driver's face relative to the central control screen, determining the orientation of the face and the angle of the face relative to the central control screen, so as to determine whether the driver is monitoring the measurement data displayed on the central control screen; S203: inputting the eye image into the eye opening and closing state evaluation model, and determining the eye opening and closing state in the image by matching the image set in the model where the head is at the same position; S204: Obtain fatigue detection results based on the eye opening and closing conditions.

[0029] Collect driver's facial images at preset intervals, obtain multiple driver's facial images within the preset time period, recognize faces from the images through MediaPipe, and extract important key facial features from the detected facial contours, including eyes, nose, and mouth. If MediaPipe cannot recognize a face in the image or the face is incomplete, the detected image will be directly classified as an invalid image set. The target object is the driver or the subsequent fleet vehicle.

[0030] The solvePnP algorithm is used to determine the position of the driver's face relative to the central control screen, the orientation of the face and the angle relative to the central control screen, thereby determining whether the driver is monitoring the measurement data displayed on the central control screen. At the same time, the ECU vehicle electronic control unit confirms the closed-eye image set and the open-eye image set based on the above data.

[0031] In one embodiment, a three-dimensional model of facial key points is created based on important facial key point features extracted from the detected facial contour, and the three-dimensional coordinates of key features such as eyes, nose and mouth are defined; the detected three-dimensional model points are used to calculate the rotation vector and translation vector through the solvePnP algorithm to determine the angle between the object coordinate system and the camera coordinate system, and the translation vector determines the position of the head in the camera coordinate system.

[0032] The eye region image is input into the trained eye opening and closing state evaluation model, and the opening and closing state of the eyes is determined by comparing the state of the eyes in the image. The eye image is input into the eye opening and closing state evaluation model, and the image set with the head in the same position is matched. The input image is compared with the open and closed eye samples marked in the image set, and a binary classifier is used to determine whether the eyes in each frame are open or closed.

[0033] It should be noted that the image set includes facial images at various angles and positions relative to the image acquisition module, and each image has a label indicating the opening and closing state of the eyes, such as "open" or "closed".

[0034] In the eye opening and closing state evaluation model, the input eye image is first preprocessed to optimize image quality and ensure image consistency. The input eye image is denoised, scaled, and normalized. Next, the input image is matched with a set of labeled images that contain open and closed eyes samples under different lighting and angles through image registration technology. The purpose of matching is to ensure that the head position is the same in the image set for reliable comparison. After the matching is completed, the model compares the input image and the labeled samples to extract key features such as shape, edge, and texture information, which are automatically extracted by deep learning models such as convolutional neural networks. These features are then input into a pre-trained binary classifier to determine whether the eyes in the image are open or closed. Finally, the ECU vehicle electronic control unit divides multiple facial images into a closed eye image set and an open eye image set according to the "open eyes" state and the "closed eyes" state.

[0035] Combining the aforementioned invalid picture set, closed-eye image set and open-eye image set, each image in the set is recorded as once, the closed-eye image set has N images, which is recorded as N times of closed eyes, the open-eye image set has M images, which is recorded as M times of open eyes, and the invalid image set has L images, which is recorded as L times of invalidity, that is, the number of invalid pictures is L times, the number of valid pictures is N+M times, and the total number is L+N+M times, where L, M, and N are natural numbers.

[0036] After identifying the number of times the driver closes his eyes, opens his eyes, and has invalid pictures within a preset time period, the number of invalid pictures can be divided by the total number of times to determine the proportion of the number of invalid pictures in the total number of times. It is understandable that the driver cannot face the central control screen in real time while driving, that is, the driver's facial image cannot be collected, that is, the driver's facial contour and facial feature point information cannot be identified, that is, the number of invalid pictures mentioned above. In order to determine whether the driver carefully checks the clearance height under the bridge during slow driving, it is necessary to divide the number of invalid pictures by the total number of times to calculate the proportion of the number of invalid pictures in the total number of times, thereby determining the time when the central control screen is not monitored.

[0037] If the proportion is greater than the first threshold, it is determined that the driver has not implemented the monitoring task and is in a fatigued state and there is a possibility of fatigue driving. If the proportion is less than or equal to the first threshold, it is determined that the driver is implementing the monitoring task, and then proceed to the next step. If it is determined that the driver has not implemented the monitoring task, the ECU vehicle control unit will send a reminder to implement the monitoring task to the central control display. It is understandable that the monitoring task reminder can be sent to the driver through a pop-up prompt on the display screen to warn the driver to implement the monitoring task, or it can be sent to the driver through a voice prompt to monitor the clearance height of the bottom of the bridge on the display screen. Perform another test within the next preset time period. If the monitoring task is still not implemented, the vehicle fatigue driving information is sent to the team leader's terminal device to prompt the driver to take turns to re-test the clearance height. At the same time, an alarm is sounded in the car, and a warning light flashes outside the car to prompt the subsequent fleet vehicles.

[0038] In order to determine whether the driver is in a fatigued state during the driving detection period, it is necessary to calculate the proportion of eye closing times in the total number of times based on the number of times the eyes are closed and the number of times the eyes are opened, and further determine the length of time the eyes are closed.

[0039] After identifying the number of times the driver closes and opens his eyes during a preset time period, the proportion of the number of times the eyes are closed in the number of valid pictures can be used to determine whether the driver is fatigued during the driving detection period. It is understandable that in order to determine whether the driver is in a fatigued state during the driving detection period, it is necessary to calculate the proportion of the number of times the eyes are closed in the total number of times based on the number of times the eyes are closed and the number of times the eyes are opened, and further determine the length of time the eyes are closed.

[0040] If the proportion is greater than the second threshold, it is determined that the driver is in a fatigued state. In this embodiment, when determining whether the driver is performing a monitoring task, the proportion of the number of eye closures in the number of valid images can be used. If the proportion is less than the second threshold, it can be determined that the driver is in a healthy working state. It is understandable that the second threshold can be set according to the actual situation when the driver is fatigued. If the proportion is greater than or equal to the second threshold, it can be determined that the driver is in a fatigued state during the driving detection period. The ECU vehicle control unit will send a fatigue driving reminder to the central control display screen. At the same time, the steering wheel and the horn in the car will sound an in-car alarm, and a voice prompt "Please stay focused on driving" will be issued.

[0041] If the proportion is greater than the third threshold, it is determined that the driver is in a serious fatigue state, and the vehicle fatigue driving information will be sent directly to the team leader's terminal device to prompt the driver to take a rotation and re-check the clearance height. At the same time, an alarm will be sounded in the car and a warning light will flash outside the car to alert subsequent vehicles in the fleet.

[0042] The present invention combines the eye opening and closing state evaluation model with heart rate detection to provide a comprehensive fatigue warning system for the driver. Through real-time monitoring and accurate algorithm analysis, it can issue a warning in time when the driver is in a fatigue state, ensure the accuracy of clearance height detection and driving safety, and prevent accidents.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fatigue warning method based on a steering wheel, characterized in that: The following steps are involved: S100: Performing real-time fatigue detection on the driver based on the heart rate detection module provided on the steering wheel, and acquiring a plurality of facial images of the driver within a preset time period, S200: Input the acquired image into the eye opening and closing state evaluation model to obtain fatigue detection results. S300: Prompt the target object according to the fatigue detection result.

2. The method for early warning of fatigue based on a steering wheel according to claim 1, characterized in that: S200 includes the following steps: S201: Identify facial contours and extract important key feature points of the face; S202: determining the position of the driver's face relative to the central control screen, determining the orientation of the face and the angle of the face relative to the central control screen, thereby determining whether the driver is monitoring the measurement data displayed on the central control screen; S203: inputting the eye image into the eye opening and closing state evaluation model, and determining the eye opening and closing state in the image by matching the image set in the model where the head is at the same position; S204: Obtain fatigue detection results based on the eye opening and closing conditions.

3. The fatigue warning method based on the steering wheel according to claim 2 is characterized in that: In S201, the acquired driver's facial image is used to recognize the face through MediaPipe, and important key feature points of the face, including eyes, nose, and mouth, are extracted from the detected facial contour.

4. The method for early warning of fatigue based on a steering wheel according to claim 2, characterized in that: In S204, an invalid image set is determined, and the eye opening and closing conditions in the image confirm the closed eyes image set and the open eyes image set. Each image in the set is recorded once. The closed eyes image set has N images, which is recorded as N times of closed eyes. The open eyes image set has M images, which is recorded as M times of open eyes. The invalid image set has L images, which is recorded as L times. That is, the number of invalid pictures is L times, the number of valid pictures is N+M times, and the total number is L+N+M times, where L, M, and N are natural numbers.

5. The method for early warning of fatigue based on the steering wheel according to claim 4, characterized in that: In S204, the number of invalid photos is divided by the total number of times to calculate the proportion of invalid photos in the total number of times, so as to determine the time when the central control screen is not monitored. If the proportion is greater than the first threshold, it is determined that the driver has not performed the monitoring task and there is a possibility of being in a fatigued state and fatigue driving. If the proportion is less than or equal to the first threshold, it is determined that the driver is performing the monitoring task.

6. The method for early warning of fatigue based on the steering wheel according to claim 5, characterized in that: In S204, the proportion of the number of eye closing times in the number of valid images is calculated based on the number of eye closing times and the number of eye opening times. If the proportion is greater than a second threshold, it is determined that the driver is in a fatigue state.

7. The method for early warning of fatigue based on a steering wheel according to claim 6, characterized in that: In S204, if the proportion is greater than the third threshold, it is determined that the driver is in a serious fatigue state, and the vehicle fatigue driving information will be sent directly to the terminal device of the team leader to prompt the driver to take a rotation and re-check the clearance height situation. At the same time, an alarm will be sounded in the vehicle and a warning light will flash outside the vehicle to alert subsequent vehicles in the fleet.

8. The method for early warning of fatigue based on a steering wheel according to claim 1, characterized in that: In S203, the detected 3D model points are used to calculate the rotation vector and the translation vector through the solvePnP algorithm, so as to determine the angle between the object coordinate system and the camera coordinate system, and the translation vector determines the position of the head in the camera coordinate system.

9. The method for early warning of fatigue based on a steering wheel according to claim 1, characterized in that: In S203, the eye region image is input into the trained eye opening and closing state evaluation model, and the opening and closing state of the eyes is determined by comparing the states of the eyes in the image.

10. Steering wheel-based fatigue warning system, characterized by: It includes an image acquisition module for collecting the driver's facial information; a clearance height detection module, including an infrared distance sensor arranged at the bottom of the benchmark, for measuring the height from the benchmark vehicle to the bottom of the bridge or the bottom of the tunnel; Heart rate detection module, used to detect the driver's blood oxygen and heart rate; The display module is used to display the clearance height result, and the alarm module is used to emit a sound signal for warning according to the control signal. If the proportion is greater than the first threshold, it is determined that the driver has not performed the monitoring task and is in a fatigue state and there is a possibility of fatigue driving. If the proportion is less than or equal to the first threshold, it is determined that the driver is performing the monitoring task.

Images