A method for detecting abnormal driver behavior based on deep learning

Abstract

A method for detecting abnormal driver behavior based on deep learning includes the following steps: Step 1: Using the YOLOv5 algorithm to detect distracted driving behavior on a target driver, an analysis result is obtained to determine whether the driver has engaged in various types of distracted driving behavior; Step 2: Using a combination of the Perclos and Dlib algorithms to detect fatigue driving on the target driver, an analysis result is obtained to determine whether the driver has engaged in fatigue driving behavior; Step 3: Providing an early warning based on the abnormal driver behavior identified and analyzed in Steps 1 and 2. This method can detect whether a driver has engaged in abnormal driving behavior and provide a corresponding early warning for any abnormal driver behavior.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and specifically relates to a method for detecting abnormal driver behavior based on deep learning. Background Art

[0002] With the increasing awareness of road traffic safety and the rapid development of science and technology, the field of abnormal driver behavior detection has attracted widespread attention and in-depth research from experts and scholars at home and abroad, and has achieved significant research results. Currently, research in abnormal driver behavior detection technology focuses on three main areas: recognition methods based on physiological characteristics, recognition methods based on behavioral characteristics, and recognition methods based on computer vision.

[0003] Computer vision-based detection and recognition methods primarily utilize computer image processing and pattern recognition technologies to extract and analyze driver-related aspects such as body movements, head movements, and facial movements. By identifying and classifying the extracted image information, the driver's driving behavior and mental state can be discerned. Compared to the other two methods, this method offers the advantages of a simpler method for obtaining driver information, minimal impact on driving, and non-contact operation. These advantages have made it the most valuable and promising method for detecting driver behavior.

[0004] In the past, most of the research focused on the detection of driver fatigue driving. The main method was to detect the driver's facial features, especially the movement of the eyes and mouth on the face to detect the driver's fatigue level. The rest of the detection of other abnormal driving behaviors of the driver during driving is achieved by detecting the driver's body movements or abnormal objects. The present invention proposes a method for real-time monitoring of the driver's overall behavior. This method of overall recognition and detection breaks through the limitations of previous methods for detecting abnormal driving behaviors of drivers, and can achieve a more objective and comprehensive analysis of the driver's driving status. Summary of the Invention

[0005] In order to overcome the above technical problems, the present invention provides a method for detecting abnormal driver behavior based on deep learning, which can detect whether the driver has abnormal driving behavior and issue corresponding warnings for the driver's abnormal behavior.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for detecting abnormal driver behavior based on deep learning, comprising the following steps:

[0008] Step 1: Use the YOLOv5 algorithm to detect distracted driving behaviors of the target driver and obtain the analysis results of whether the driver has various distracted driving behaviors;

[0009] Step 2: Perform fatigue driving detection on the target driver by combining Perclos and Dlib algorithms to obtain an analysis result of whether the driver has fatigue driving behavior;

[0010] Step 3: Issue an early warning based on the judgment results of the driver's abnormal behavior obtained by the identification and analysis in steps 1 and 2.

[0011] Furthermore, the specific steps included in step 1 are as follows:

[0012] (1) Create a dataset of abnormal driver behavior. Use the LabelImg tool to label the target images to be detected, so that they can be put into the model for training. A total of 1,436 images in 6 categories are created as a dataset for driver distraction behavior detection.

[0013] (2) The YOLOv5 algorithm model is used for model training. The YOLOv5 algorithm uses GIoULoss as the loss function of Boundingbox. GIOU is a loss function for bounding box prediction derived from IOU. GIOULoss can be expressed by formula (1):

[0014]

[0015] Among them, A is the positive sample set, B is the negative sample set, IOU is the intersection-over-union ratio of positive and negative samples, and c is the minimum closure area;

[0016] (3) The dataset is divided into a training set and a test set, and the YOLOv5 convolutional neural network is trained in a ratio of 8:2. The training is stopped when the loss function of the validation set reaches the minimum, and the optimal weight parameters of the model under the dataset are obtained;

[0017] (4) Use the YOLOv5 network loaded with the optimal weight parameters to detect the images in the test set. By detecting whether there are special objects, the corresponding distracting behavior can be determined.

[0018] Furthermore, in step 2, the Dlib algorithm and the Perclos algorithm are used to perform fatigue driving detection. Dlib calculates the Euclidean distance of the face to locate the eyes and mouth of the target face, and then the Perclos algorithm is used to identify the state of the target's eyes and mouth to determine whether the target is tired. Specifically, the following steps are included:

[0019] (1) Dlib is used for face location and recognition specifically as follows:

[0020] 1) Load the face recognition model and calculate all the feature vectors of faces in the known image;

[0021] 2) Load the face recognition model and calculate all the feature vectors corresponding to the faces in the image to be recognized;

[0022] 3) Calculate the Euclidean distance between facial feature vectors. The Euclidean distance is calculated as shown in formula (2).

[0023]

[0024] (2) Perclos refers to the percentage of a specific period of time occupied by the closing time of the eyes; during the process of closing and opening the eyes of a human body, the curve of the change of the degree of eye opening over time can be used to obtain the duration of a certain degree of opening and closing of the target object to be measured during the experiment, so that the Perclos value can be calculated based on the data reflected by it. The Perclos value can be calculated by formula (3);

[0025]

[0026] Where t1 represents the time from the target subject's eyes being fully open to 20% of the time from closing, t2 represents the time from the eyes being fully open to 80% of the time from closing, t3 represents the time from the eyes being fully open to 20% of the time from the next opening, and t4 represents the time from the eyes being fully open to 80% of the time from the next opening.

[0027] The fatigue detection process involves capturing image elements and sending the captured facial data to the monitoring device's control terminal, which contains an algorithm for detecting driver fatigue. The control terminal uses the captured facial data to identify and determine fatigue status.

[0028] Furthermore, the principle process for identifying the state of the eyes in a face is to determine whether the area of the target face's eyelids covering the eyeball exceeds a preset ratio. If the coverage area exceeds the preset ratio, the face is judged to be tired, otherwise it is normal. The recognition and judgment of the mouth state are consistent with the eyes.

[0029] Furthermore, in step 3, different warnings need to be given for the driver's distracted behavior and fatigued driving behavior. Through a window interface of Pyside2, in this interface, the user can choose to use the camera to perform real-time behavior detection on the driver and to detect the recorded video. Regardless of real-time monitoring of the camera or video detection of the surveillance video, if the driver exhibits the above-mentioned distracted driving or fatigued driving behavior, corresponding warnings will be given for the driver's different abnormal driving behaviors in this window interface.

[0030] Beneficial effects of the present invention.

[0031] The deep learning-based driver behavior detection method proposed in this invention utilizes the YOLOv5 algorithm described in step 1 to achieve real-time detection and early warning of driver distraction. Furthermore, the combination of the Perclos and Dlib algorithms described in step 2 enables real-time detection and early warning of driver fatigue. This method can provide early warnings of potential driver abnormalities, thereby helping to reduce the probability of traffic accidents caused by improper driving behavior and ensuring traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Network model diagram of the YOLOv5 algorithm.

[0033] Figure 2 Perclos algorithm change curve.

[0034] Figure 3 Mobile phone detection for abnormal driver behavior detection.

[0035] Figure 4 Driver abnormal behavior detection: drinking water detection.

[0036] Figure 5 Mask detection for abnormal driver behavior detection.

[0037] Figure 6 Fatigue detection for abnormal driver behavior detection.

[0038] Figure 7 Driver abnormal behavior detection - line of sight detection. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings and attached tables in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0040] like Figure 1-Figure 7 As shown: A method for detecting abnormal driver behavior based on deep learning, comprising the following steps;

[0041] Step 1: Use the YOLOv5 algorithm to detect distracted driving behaviors of the target driver and obtain the analysis results of whether the driver has various distracted driving behaviors;

[0042] Step 2: Perform fatigue driving detection on the target driver by combining Perclos and Dlib algorithms to obtain an analysis result of whether the driver has fatigue driving behavior;

[0043] Step 3: Issue an early warning based on the judgment results of the driver's abnormal behavior obtained by the identification and analysis in steps 1 and 2.

[0044] The specific steps of step 1 are as follows:

[0045] (1) Create a dataset of abnormal driver behavior. Use the LabelImg tool to label the target images to be detected, so that they can be put into the model for training. A total of 1,436 images in 6 categories are created as a dataset for driver distraction behavior detection.

[0046] (2) The YOLOv5 algorithm model is used for model training. The YOLOv5 algorithm uses GIoULoss as the loss function of Boundingbox. GIOU is a loss function for bounding box prediction derived from IOU. GIOULoss can be expressed by formula (1):

[0047]

[0048] Among them, A is the positive sample set, B is the negative sample set, IOU is the intersection-over-union ratio of positive and negative samples, and c is the minimum closure area;

[0049] (3) The dataset is divided into a training set and a test set, and the YOLOv5 convolutional neural network is trained in a ratio of 8:2. The training is stopped when the loss function of the validation set reaches the minimum, and the optimal weight parameters of the model under the dataset are obtained;

[0050] (4) Use the YOLOv5 network loaded with the optimal weight parameters to detect the images in the test set. By detecting whether there are special objects, the corresponding distracting behavior can be determined.

[0051] In step 2, the Dlib algorithm and the Perclos algorithm are used to detect fatigue driving. Dlib calculates the Euclidean distance of the face to locate the eyes and mouth of the target face, and then uses the Perclos algorithm to identify the state of the target's eyes and mouth to determine whether the target is tired. Specifically, the following steps are performed:

[0052] (1) Dlib is used for face location and recognition specifically as follows:

[0053] 1) Load the face recognition model and calculate all the feature vectors of faces in the known image;

[0054] 2) Load the face recognition model and calculate all the feature vectors corresponding to the faces in the image to be recognized;

[0055] 3). Calculate the Euclidean distance between facial feature vectors. The Euclidean distance is calculated as shown in formula (2).

[0056]

[0057] (2) Perclos refers to the percentage of a specific period of time occupied by the closing time of the eyes; during the process of closing and opening the eyes of a human body, the curve of the change of the degree of eye opening over time can be used to obtain the duration of a certain degree of opening and closing of the target object to be measured during the experiment, so that the Perclos value can be calculated based on the data reflected by it. The Perclos value can be calculated by formula (3);

[0058]

[0059] Where t1 represents the time from the target subject's eyes being fully open to 20% of the time from closing, t2 represents the time from the eyes being fully open to 80% of the time from closing, t3 represents the time from the eyes being fully open to 20% of the time from the next opening, and t4 represents the time from the eyes being fully open to 80% of the time from the next opening.

[0060] The fatigue detection process involves capturing image elements and sending the captured facial data to the monitoring device's control terminal, which contains an algorithm for detecting driver fatigue. The control terminal uses the captured facial data to identify and determine fatigue status.

[0061] The principle process for identifying the state of the eyes in a face is to determine whether the area of the target face's eyelids covering the eyeball exceeds a preset ratio. If the coverage area exceeds the preset ratio, the face is judged to be tired, otherwise it is normal. The mouth state recognition is consistent with the eye state.

[0062] In step 3, different warnings need to be given for the driver's distracted behavior and fatigued driving behavior. Through a window interface of Pyside2, in this interface, the user can choose to use the camera to perform real-time behavior detection on the driver and to detect the recorded video. Regardless of the real-time monitoring of the camera or the video detection of the surveillance video, if the driver exhibits the above-mentioned distracted driving or fatigued driving behavior, corresponding warnings will be given for the driver's different abnormal driving behaviors in this window interface.

[0063] Example:

[0064] like Figure 3 As shown in the figure, by calling and comparing the model trained by the YOLOv5 algorithm, if a mobile phone is detected in the target, the corresponding label of the mobile phone is fed back. If the main function receives the fed-back mobile phone label, it is determined that the driver is driving illegally using a mobile phone. Otherwise, the driver is considered to be driving normally.

[0065] like Figure 4 As shown in the figure, by calling the model trained by the YOLOv5 algorithm, if a water cup is detected in the target, the corresponding label of the water cup is fed back. If the main function receives the fed-back water cup label, it determines that the driver has violated the driving regulations by drinking water. Otherwise, the driver is considered to be driving normally.

[0066] like Figure 5 As shown in the figure, by calling the model trained by the YOLOv5 algorithm, if a mask is detected in the target, the corresponding mask label is fed back. If the main function receives the fed-back mask label, it is determined that the driver is wearing a mask. Otherwise, it is determined that the driver is not wearing a mask as required and is engaging in irregular driving behavior.

[0067] like Figure 6 As shown in the figure, the Dlib algorithm is used to calculate the Euclidean distance of the face to locate the position of the eyes and mouth on the target face. After the positioning is completed, the Perclos algorithm is used to determine whether the driver's eyes and mouth are closed or open and the number of times the driver's eyes and mouth are closed is counted. Then, the formula (4) is used to determine whether the driver is fatigued driving behavior.

[0068] perclos=(Rolleye / Roll)+(Rollmouth / Roll)*0.2 (4)

[0069] Among them: Rolleye is the number of times the target face is detected blinking, Rollmouth is the number of times the target face is detected opening its mouth, and Roll starts from zero and increases by one after each frame.

[0070] like Figure 7 As shown in the figure, by calling and comparing the model trained with the YOLOv5 algorithm, if the system successfully recognizes the driver's face, it will feedback the face label. If the main program receives the face label, it will determine that the driver has no vision deviation. Otherwise, it will be determined that the driver has distracted driving behavior such as vision deviation.

Claims

1. A method for detecting abnormal driver behavior based on deep learning, characterized in that Detecting abnormal driving behavior of a driver, including the following steps: Step 1: Use the YOLOv5 algorithm to detect distracted driving behaviors of the target driver and obtain the analysis results of whether the driver has various distracted driving behaviors; Step 2: Perform fatigue driving detection on the target driver by combining Perclos and Dlib algorithms to obtain an analysis result of whether the driver has fatigue driving behavior; Step 3: issuing an early warning based on the abnormal driver behavior judgment results obtained from the identification and analysis in Steps 1 and 2; The specific steps of step 1 are as follows: (1) Create a dataset of abnormal driver behavior. Use the LabelImg tool to label the target images to be detected, so that they can be put into the model for training. A total of 1,436 images in 6 categories are created as a dataset for driver distraction behavior detection. (2) The YOLOv5 algorithm model is used for model training. The YOLOv5 algorithm uses GIoULoss as the loss function of Boundingbox. GIOU is a loss function for bounding box prediction derived from IOU. GIOULoss can be expressed by formula (1): Among them, A is the positive sample set, B is the negative sample set, IOU is the intersection-over-union ratio of positive and negative samples, and c is the minimum closure area; (3) The dataset is divided into a training set and a test set, and the YOLOv5 convolutional neural network is trained in a ratio of 8:

2. The training is stopped when the loss function of the validation set reaches the minimum, and the optimal weight parameters of the model under the dataset are obtained; (4) Using the YOLOv5 network loaded with the optimal weight parameters to detect the images in the test set, the corresponding distracting behavior can be determined by detecting whether there are special objects; In step 2, the Dlib algorithm and the Perclos algorithm are used to detect fatigue driving. Dlib calculates the Euclidean distance of the face to locate the eyes and mouth of the target face, and then uses the Perclos algorithm to identify the state of the target's eyes and mouth to determine whether the target is tired. Specifically, the following steps are performed: (1) Dlib is used for face location and recognition specifically as follows: 1) Load the face recognition model and calculate all the feature vectors of faces in the known image; 2) Load the face recognition model and calculate all the feature vectors corresponding to the faces in the image to be recognized; 3) Calculate the Euclidean distance between facial feature vectors. The Euclidean distance is calculated as shown in formula (2). (2) Perclos refers to the percentage of a specific period of time occupied by the closing time of the eyes; during the process of closing and opening the eyes of a human body, the curve of the change of the degree of eye opening over time can be used to obtain the duration of a certain degree of opening and closing of the target object to be measured during the experiment, so that the Perclos value can be calculated based on the data reflected by it. The Perclos value can be calculated by formula (3); Where t1 represents the time from the target subject's eyes being fully open to 20% of the time from closing, t2 represents the time from the eyes being fully open to 80% of the time from closing, t3 represents the time from the eyes being fully open to 20% of the time from the next opening, and t4 represents the time from the eyes being fully open to 80% of the time from the next opening. The fatigue detection process is as follows: by collecting elements from the image, the collected target face data is sent to the control terminal of the monitoring device. The control terminal contains an algorithm for detecting driver fatigue, and the control terminal uses the collected facial data to identify the fatigue state. The Dlib algorithm is used to calculate the Euclidean distance of the face to locate the position of the eyes and mouth on the target face. After the positioning is completed, the Perclos algorithm is used to determine whether the driver's eyes and mouth are closed or open and the number of times the driver's eyes and mouth are closed is counted; then the formula (4) is used to determine whether the driver is fatigued driving behavior perclos=(Rolleye / Roll)+(Rollmouth / Roll)*0.2 (4) Among them: Rolleye is the number of times the target face is detected blinking, Rollmouth is the number of times the target face is detected opening its mouth, and Roll starts from zero and increases by one after each frame.

2. A method for detecting abnormal driver behavior based on deep learning as claimed in claim 1, characterized in that To detect abnormal driving behavior of the driver, the principle process of recognizing and judging the state of the eyes on the face is to determine whether the area covered by the eyelids of the target face over the eyeball exceeds a preset ratio. If the coverage area exceeds the preset ratio, it is judged as tired, otherwise it is normal. The recognition and judgment of the mouth state are consistent with the eyes.

3. A method for detecting abnormal driver behavior based on deep learning as claimed in claim 1, characterized in that Detect the driver's abnormal driving behavior. In step 3, different warnings need to be given for the driver's distracted behavior and the driver's fatigued driving behavior. Through a window interface of Pyside2, in this interface, the user can choose to use the camera to perform real-time behavior detection on the driver and to detect the recorded video. Whether it is real-time monitoring of the camera or video detection of the surveillance video, if the driver exhibits the so-called distracted driving or fatigued driving behavior, in this window interface, corresponding warnings will be given for the driver's different abnormal driving behaviors.

Images