Driving fatigue detection system based on abnormal actions of driver

Abstract

The invention discloses a driving fatigue detection system based on abnormal actions of a driver, and relates to the technical field of driving fatigue detection, the system comprises a video acquisition unit, a video preprocessing unit, a feature extraction unit, a fatigue detection algorithm unit and a fusion detection unit; the video preprocessing unit is used for preprocessing each hand and head action image to obtain preprocessed hand and head action images; the feature extraction unit is used for carrying out key feature extraction on the preprocessed hand and head action images to obtain hand and head action features, and the fatigue detection algorithm unit is used for carrying out fatigue detection according to all the hand and head action features to obtain an action recognition result; and the fusion detection unit determines a final fatigue detection result of the driver according to the action recognition result. The driving fatigue detection precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of driving fatigue detection, and in particular to a driving fatigue detection system based on abnormal driver behavior. Background Art

[0002] Driver fatigue driving is one of the important causes of traffic accidents. However, the existing driver fatigue detection methods have low fatigue state detection accuracy, so there is an urgent need for a high-precision driver fatigue detection system. Summary of the invention

[0003] The purpose of this application is to provide a driving fatigue detection system based on abnormal driver actions, which can improve the accuracy of driving fatigue detection.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] The present application provides a driving fatigue detection system based on abnormal driver actions, comprising a video acquisition unit, a video preprocessing unit, a feature extraction unit, a fatigue detection algorithm unit and a fusion detection unit;

[0006] Wherein, the video acquisition unit is used to: acquire the driver's action video in real time; the action video includes a plurality of action images;

[0007] The video preprocessing unit is used to: preprocess each action image to obtain a preprocessed action image;

[0008] The feature extraction unit is used to extract key features from the preprocessed action image to obtain hand and head action features; the hand and head action features include hand position, head key information, hand posture and hand movement trajectory;

[0009] The fatigue detection algorithm unit is used to: perform fatigue detection according to all hand and head motion features to obtain motion recognition results; the motion recognition results include the real-time motion type of the driver in the motion video and the number of times each real-time motion type is completed;

[0010] The fusion detection unit is used to determine the final fatigue detection result of the driver according to the action recognition result.

[0011] Optionally, the feature extraction unit is further used for:

[0012] Using the trained hand detection model, perform hand detection on the preprocessed action images to obtain a hand position recognition result corresponding to each preprocessed action image;

[0013] Using the trained head detection model, performing head detection on the preprocessed action images to obtain a head position recognition result corresponding to each preprocessed action image;

[0014] Using the trained posture recognition model, posture recognition is performed on the hand position recognition results and the head position recognition results respectively to obtain the posture information in each of the preprocessed action images; the hand position recognition results, the head position recognition results and the posture information constitute the hand and head action features.

[0015] Optionally, the fatigue detection algorithm unit is further used to: utilize a trained classification model to perform fatigue detection according to all hand and head motion features to obtain a motion recognition result.

[0016] Optionally, the fusion detection unit is further used to: use a fusion algorithm to determine the final fatigue detection result of the driver according to the action recognition result.

[0017] Optionally, the fusion algorithm is a weighted average algorithm, a Bayesian network or a decision tree algorithm.

[0018] Optionally, the trained hand detection model and the trained head detection model are models obtained by training a convolutional neural network, a YOLO target detection model or a Haar feature cascade classifier.

[0019] Optionally, the trained posture recognition model is a model obtained by training an OpenPose model or a MediaPipe model.

[0020] Optionally, the trained classification model is a model trained using a support vector machine algorithm or a random forest algorithm.

[0021] Optionally, the preprocessing includes grayscale processing, noise filtering and image enhancement processing.

[0022] Optionally, the driving fatigue detection system based on abnormal driver behavior further includes an alarm unit; the alarm unit is used to send a warning signal through a fatigue warning device when the final fatigue detection result of the driver is a fatigue state.

[0023] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0024] The present application provides a driving fatigue detection system based on abnormal driver movements. A feature extraction unit extracts key features from preprocessed motion images to obtain hand and head motion features. A fatigue detection algorithm unit is used to: perform fatigue detection based on all hand and head motion features to obtain motion recognition results; a fusion detection unit determines the final fatigue detection result of the driver based on the motion recognition results. The present application determines the final fatigue detection result by fusing the hand and head motion recognition results, taking both the hand and face levels into consideration, thereby solving the problem of low detection accuracy of existing driving fatigue detection methods and achieving accurate detection of driving fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of the structure of a driving fatigue detection system based on abnormal driver behavior provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] In an exemplary embodiment, Figure 1 As shown, a driving fatigue detection system based on abnormal driver actions is provided, including a video acquisition unit, a video preprocessing unit, a feature extraction unit, a fatigue detection algorithm unit and a fusion detection unit.

[0030] Wherein, the video acquisition unit is used to: acquire the driver's action video in real time; the action video includes a plurality of action images.

[0031] The video preprocessing unit is used to preprocess each action image to obtain a preprocessed action image.

[0032] The feature extraction unit is used to extract key features from the preprocessed action image to obtain hand and head action features; the hand and head action features include hand position, head key information, hand posture and hand movement trajectory.

[0033] The fatigue detection algorithm unit is used to: perform fatigue detection according to all hand and head movement features to obtain a movement recognition result; the movement recognition result includes the real-time movement type of the driver in the movement video and the number of movement completions of each real-time movement type.

[0034] The fusion detection unit is used to determine the final fatigue detection result of the driver according to the action recognition result.

[0035] The action video is acquired in real time by the front camera. The front camera is installed in the car and can acquire the action video data of the driver's head, hands and arms in real time and transmit it to the video acquisition unit. The video acquisition unit is used to acquire the action video transmitted by the front camera and transmit it to the video preprocessing unit.

[0036] The preprocessing in the video preprocessing unit includes grayscale processing, noise filtering and image enhancement processing. Among them, grayscale processing: converting the collected color image into a grayscale image can reduce the computational complexity; noise filtering: using Gaussian filtering or median filtering to remove noise in the image can improve the image quality; image enhancement processing: using contrast enhancement or histogram equalization methods can improve the clarity and contrast of the image.

[0037] The feature extraction unit is also used for:

[0038] Using the trained hand detection model, perform hand detection on the preprocessed action images to obtain a hand position recognition result corresponding to each preprocessed action image;

[0039] Using the trained head detection model, performing head detection on the preprocessed action images to obtain a head position recognition result corresponding to each preprocessed action image;

[0040] Using the trained posture recognition model, posture recognition is performed on the hand position recognition results and the head position recognition results respectively to obtain the posture information in each of the preprocessed action images; the hand position recognition results, the head position recognition results and the posture information constitute the hand and head action features.

[0041] The feature extraction unit extracts hand and head motion features from the preprocessed image, including hand position, head position information, hand posture and hand motion trajectory, through key point detection-key point connection-posture representation-posture estimation. Head key information includes the motion trajectory of head key points, and head key points include left eye and right eye; hand posture information includes hand posture and the motion trajectory of hand key points. Hand key points include finger key points and palm key points.

[0042] The trained hand detection model and the trained head detection model are models obtained by training a convolutional neural network or a Haar feature cascade classifier, etc. The trained posture recognition model is a model obtained by training an OpenPose model or a MediaPipe model, etc.

[0043] Action recognition: Detect action types based on hand position and posture changes, such as stretching arms, yawning, rubbing eyes, scratching head, slow movements, and abnormal departure from the steering wheel. The slow movement of actions is measured by the average distance per unit time of hand feature points (i.e., hand key points), and the driver's slow movement is detected by setting a threshold. Action recognition models: Hourglass network structure, OpenPose, ResNet, deep residual network; hand and hand posture recognition detection models: HandPose, MediaPipe Hands.

[0044] Key point connection: The key points are connected to form the bone structure or posture representation of the hand. Then the connection from the palm center to the finger joints is completed based on the predefined hand anatomical structure.

[0045] Posture representation: Convert the connected hand key points into a specific posture representation. This involves calculating the angles between joints, the direction of the palm, or the extension of the fingers. This includes Euler angles, quaternions, or rotation matrices. The simplified Euler angle sequence is roll-pitch-yaw (roll-rotates around the X axis, pitch-rotates around the Y axis, yaw-rotates around the Z axis) or yaw-pitch-roll (yaw-rotates around the Z axis, pitch-rotates around the Y axis, roll-rotates around the X axis).

[0046] Assuming the Euler angles are (φ, θ, ψ), the corresponding rotation matrix R can be expressed as: R = R z (ψ)·R y (θ)·R x (φ).

[0047] The steps for hand and head gesture and action recognition are as follows:

[0048] Step 1: Hand and facial keypoint detection

[0049] The trained hand detection model and face detection model are used to detect the key point positions of the hand and face respectively. The key point positions of the hand are the hand position recognition results, and the key point positions of the face are the head position recognition results. The key points detected are as follows: facial key points: eyes, nose, mouth, etc.; hand key points: finger joints and palm center, etc.

[0050] Step 2: Calculate key point positions

[0051] Get the key point position:

[0052] Eye Center: (E x ,E y ), where E x is the horizontal coordinate of the eye center position, E y is the horizontal coordinate of the eye center position; it can be calculated from the coordinates of the two eyes.

[0053] Finger tip (assuming it is the index finger tip): (F x ,F y ), where F x is the horizontal coordinate of the index finger tip position, F y is the horizontal coordinate of the index finger tip position.

[0054] Palm center: (P x ,P y ), where P x is the horizontal coordinate of the palm center position, P y is the horizontal coordinate of the palm center position.

[0055] Step 3: Calculate distance and angle

[0056] To recognize hand movements, the following can be calculated:

[0057] Step 3.1: Distance between finger tip and eye center: Detect whether the finger tip is close to the eye. The distance calculation formula is as follows:

[0058] D=(F x -P x ) 2 +(F y -P y ) 2 ,If the distance D between the finger tip and the eye center is less than the set threshold T, the finger is considered to be close to the eye.

[0059] Step 3.2: Distance between the center of the palm and the fingertips: Determine if the hand is rubbing the eyes, or doing something else other than simply being close together.

[0060] D hand =(F x -P x )2 +(F y -P y ) 2 , when the distance D between the center of the palm and the fingertips hand When it changes within a certain range and the distance D between the fingertip and the center of the eye continues to remain less than the set threshold T, the action can be further judged.

[0061] Step 4: Time Series Analysis

[0062] By analyzing the action images after continuous frame preprocessing, it is possible to observe the relationship between the key points of the finger and the eye changing over time. If the detected finger tip remains near the eye for a certain period of time, the action can be determined.

[0063]

[0064] Among them, N is the judgment result corresponding to the action image after continuous frame preprocessing, D i is the distance between the fingertip and the eye center in the action image after preprocessing the i-th frame; δ() is the indicator function, when the distance D between the fingertip and the eye center in the action image after preprocessing the i-th frame i When the threshold T is set, δ() is 1, otherwise it is 0.

[0065] Step 5: Posture and Action Recognition

[0066] Combining the above distance and time series analysis, construct the feature vector of the hand movement (such as the distance from the fingertip to the center of the eye, the distance between the palm center and the fingertip, the number of time series frames, etc.), and then use a machine learning classification algorithm (such as support vector machine, random forest or neural network) for action recognition.

[0067] The fatigue detection algorithm unit is also used to: use the trained classification model to perform fatigue detection according to all hand and head movement features to obtain movement recognition results. The trained classification model is a model trained using a support vector machine algorithm or a random forest algorithm.

[0068] The fusion detection unit is also used to determine the final fatigue detection result of the driver according to the action recognition result by using a fusion algorithm, wherein the fusion algorithm is a weighted average algorithm, a Bayesian network or a decision tree algorithm.

[0069] The fusion detection unit combines the hand motion detection results with the eye and head motion detection results to determine the final fatigue detection result to improve the overall accuracy of fatigue detection. Head motions include: blinking, yawning, nodding, etc.

[0070] Combined with the results of eye and head movement monitoring, fusion algorithms such as weighted average, Bayesian network or decision tree are used to improve the accuracy of fatigue detection. For example, when the weighted average algorithm is used to determine the final fatigue detection result of the driver, a fatigue value is set to zero initially, a threshold is set according to each action type, and whether the action is completed is determined based on the predicted probability of the real-time action type output by the trained classifier and the probability threshold corresponding to the action type. Different weights are assigned to each action type. For each real-time action type, the weighted average of the fatigue value is calculated based on the number of completed actions of the real-time action type multiplied by the weight of the real-time action type, and the final fatigue detection result is determined based on the weighted average of the fatigue value and the set fatigue detection threshold. When the weighted average of the fatigue value is greater than the set fatigue detection threshold, the final fatigue detection result of the driver is a fatigue state, and when the weighted average of the fatigue value is not greater than the set fatigue detection threshold, the final fatigue detection result of the driver is a non-fatigue state.

[0071] In one example of the present application, the driving fatigue detection system based on abnormal driver behavior further includes an alarm unit; the alarm unit is used to: when the final fatigue detection result of the driver is a fatigue state, send out an alarm signal through a fatigue warning device. When the driver's fatigue state is detected, the fatigue warning device sends out an audible, optical or vibration alarm to remind the driver.

[0072] The front camera in the car collects the video data of the driver's hand and arm movements in real time and transmits it to the video acquisition module. The video acquisition module transmits the video data to the image preprocessing unit for graying, noise filtering and image enhancement. The preprocessed image data is transmitted to the feature extraction unit to extract the hand position, posture and motion trajectory. The feature extraction unit transmits the extracted hand and arm movement features to the fatigue detection algorithm unit for analysis to detect the driver's fatigue state. When it is detected that the driver is in a fatigue state, the fatigue detection algorithm unit transmits the hand movement detection result to the fusion detection unit for fusion analysis with the eye and head movement detection results. The movement recognition result output by the fatigue detection algorithm unit is a movement recognition result including the hand and head. The fusion detection unit generates the final fatigue detection result based on the fusion algorithm, triggers the alarm system, and issues a sound, light or vibration alarm to remind the driver.

[0073] In addition, the existing driving fatigue detection methods mainly focus on monitoring the driver's eye and head movements, but ignore the changes in hand and arm movements. Abnormal hand and arm movements, such as rubbing eyes, scratching head, slow movements and abnormal separation from the steering wheel, are obvious characteristics of driver fatigue. Therefore, the fatigue detection method based on hand and arm movements has important application value. By monitoring and analyzing the driver's hand and arm movements, the driver's fatigue state can be detected and warned in time to improve driving safety. In addition, the system of the present invention can be combined with existing driving fatigue detection methods, such as eye and head movement monitoring, to improve the accuracy and reliability of fatigue detection. Specifically: The present application integrates the facial movement detection results and the hand movement detection results to determine the final fatigue detection results, taking into account the two levels of hand and face, solves the problem of low detection accuracy of the existing driving fatigue detection methods, and realizes accurate detection of driving fatigue. In addition, the method based on video monitoring of the present application does not require physical contact sensors and does not interfere with the driver's behavior; it can capture and analyze the driver's hand movements in real time and issue fatigue alarms in time; it combines a variety of image processing and machine learning algorithms to accurately identify fatigue behavior and reduce false alarms; it is suitable for various driving environments and different types of vehicles, and has good versatility.

[0074] Some code examples:

[0075]

[0076]

[0077]

[0078] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A driving fatigue detection system based on abnormal driver behavior, characterized in that: The driving fatigue detection system based on abnormal driver behavior includes a video acquisition unit, a video preprocessing unit, a feature extraction unit, a fatigue detection algorithm unit and a fusion detection unit; Wherein, the video acquisition unit is used to: acquire the driver's action video in real time; the action video includes a plurality of action images; The video preprocessing unit is used to: preprocess each action image to obtain a preprocessed action image; The feature extraction unit is used to extract key features from the preprocessed action image to obtain hand and head action features; the hand and head action features include hand position, head key information, hand posture and hand movement trajectory; The fatigue detection algorithm unit is used to: perform fatigue detection according to all hand and head motion features to obtain motion recognition results; the motion recognition results include the real-time motion type of the driver in the motion video and the number of times each real-time motion type is completed; The fusion detection unit is used to determine the final fatigue detection result of the driver according to the action recognition result.

2. The driving fatigue detection system based on abnormal driver behavior according to claim 1 is characterized in that: The feature extraction unit is also used for: Using the trained hand detection model, perform hand detection on the preprocessed action images to obtain a hand position recognition result corresponding to each preprocessed action image; Using the trained head detection model, performing head detection on the preprocessed action images to obtain a head position recognition result corresponding to each preprocessed action image; Using the trained posture recognition model, respectively perform posture recognition on the hand position recognition results and the head position recognition results to obtain posture information in each of the preprocessed action images; The hand position recognition results, head position recognition results and posture information constitute the hand and head movement features.

3. The driving fatigue detection system based on abnormal driver behavior according to claim 1 is characterized in that: The fatigue detection algorithm unit is also used for: Using the trained classification model, fatigue detection is performed based on all hand and head movement features to obtain the action recognition results.

4. The driving fatigue detection system based on abnormal driver behavior according to claim 1, characterized in that: The fusion detection unit is also used for: The final fatigue detection result of the driver is determined according to the action recognition result by utilizing a fusion algorithm.

5. The driving fatigue detection system based on abnormal driver behavior according to claim 4 is characterized in that: The fusion algorithm is a weighted average algorithm, a Bayesian network or a decision tree algorithm.

6. The driving fatigue detection system based on abnormal driver behavior according to claim 2, characterized in that: The trained hand detection model and the trained head detection model are models obtained by training a convolutional neural network, a YOLO target detection model or a Haar feature cascade classifier.

7. The driving fatigue detection system based on abnormal driver behavior according to claim 2, characterized in that: The trained posture recognition model is a model obtained by training an OpenPose model or a MediaPipe model.

8. The driving fatigue detection system based on abnormal driver behavior according to claim 3 is characterized in that: The trained classification model is a model trained using a support vector machine algorithm or a random forest algorithm.

9. The driving fatigue detection system based on abnormal driver behavior according to claim 1, characterized in that: The preprocessing includes grayscale processing, noise filtering and image enhancement processing.

10. The driving fatigue detection system based on abnormal driver behavior according to claim 1, characterized in that: The driving fatigue detection system based on abnormal driver behavior also includes an alarm unit; the alarm unit is used to send a warning signal through a fatigue warning device when the final fatigue detection result of the driver is a fatigue state.

Images