Fatigue driving detection method of neural network model based on deep learning

Through a deep learning-based neural network model, combined with feature pyramid network and RT-DETR module, the YOLOV8 model is optimized, which solves the problems of slow detection speed and low accuracy in the existing technology, and realizes efficient and stable fatigue driving detection. It is applicable In the on-board terminal.

CN119027919APending Publication Date: 2024-11-26LIAONING TECHNICAL UNIVERSITY

Patent Information

Application Number
CN202410957858.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing fatigue detection system has slow detection speed and low accuracy of output results, resulting in poor real-time performance when deploying on-board terminals.

Method used

Adopting a neural network model based on deep learning, using the YOLOV8 model for fatigue driving detection, fusing feature maps of different scales through feature pyramid network and path aggregation network, and introducing AIFI module in RT-DETR to replace SPPF module to optimize backbone network , reduce the number of convolutional channels, adopt shared parameter heads and improved detection head structure, reducing the calculation amount and model complexity.

Benefits of technology

It improves the accuracy and detection efficiency of face feature detection, reduces the hardware resource requirements, and enables efficient fatigue driving detection on low-end terminals, achieving better real-time and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027919A_ABST
    Figure CN119027919A_ABST
Patent Text Reader

Abstract

The invention relates to a fatigue driving detection method of a neural network model based on deep learning, and the method comprises the steps: collecting a picture with a face image of a driver; inputting the image into a calculation model, wherein the calculation model is built based on a YOLOV8 model; the calculation model carries out the preprocessing of an image, and comprises the steps: firstly, generating a unified image size 640 * 640; secondly, performing color channel conversion on the image to convert an image format into an RGB (Red, Green and Blue) format; performing normalization processing on the image, and zooming image pixels from a range of [0, 255] to [0, 1]; inputting the preprocessed image into a calculation model, and extracting basic features of the image; fusing the feature maps of different scales by using a feature pyramid network and a path aggregation network to obtain three feature maps; and processing the three feature maps, generating a detection result, and obtaining the category of each candidate frame and the position of the bounding box. According to the invention, fatigue driving detection is realized by using the neural network model, and the detection method carries out lightweight transformation on an existing operation model, so that the needed hardware resources are less, and therefore, when the method is applied to a low-configuration terminal, better detection efficiency can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of fatigue driving detection, and specifically relates to a fatigue driving detection method based on a neural network model of deep learning. Background Art

[0002] Current fatigue driving detection methods are mainly divided into active detection methods and passive detection methods. Active detection methods mainly include oral questions and questionnaires. Their advantages are simplicity and low cost, but they can only be used for a detection behavior before the driver drives. Because fatigue driving mainly occurs in a series of distracting behaviors generated during driving, it cannot effectively and in real time warn and deal with fatigue driving. Passive detection methods mainly include detection based on physiological characteristics and detection based on driver's facial features. Detection based on physiological characteristics mainly relies on electroencephalography (ECG), electrooculogram (EOG), electrocardiogram (ECG), electromyogram (EMG), etc. for judgment. Its advantages are accurate recognition and high precision. However, because it requires complex external equipment, it will affect the driver's driving habits and psychological state. In addition, the equipment is numerous and the convenience is poor, so it is not universal. The facial feature recognition method only requires a camera to collect the driver's facial information, and will not affect the driver's driving behavior. It also has the advantages of low cost, high recognition rate, and the ability to issue warnings in a timely and effective manner. It is more advantageous for fatigue driving detection at this stage and is currently a major research hotspot.

[0003] Some advanced driver assistance systems (ADAS) today already incorporate monitoring of abnormal driving behavior and provide some form of warning, however, the widespread implementation of these systems is still very challenging as the detection of abnormal behavior requires accurate and real-time identification. To monitor one or more measures of driver behavior via a vehicle ad hoc network (VANET), Al Sultan et al. extended VANET by integrating a real-time probabilistic model based on a dynamic Bayesian network (DBN) to infer the driver's dynamic behavior in combination with contextual information related to the driver, vehicle, and environment to detect four behavioral indicators, namely drunkenness, fatigue, aggressiveness, and normality. Due to the large number of parameters and considering that VANET operates as a mobile ad hoc network (MANET), the transmission of contextual information to the real-time probabilistic model via dedicated short-range communication (DSRC) brings considerable computational requirements. Therefore, reducing the number of parameters is crucial to achieve a computationally efficient network.

[0004] The existing technology for detecting fatigue driving focuses on the high-precision detection of facial features. However, in most previous studies, in order to improve the detection accuracy of the network model, the limitations of the computing resources of the application terminal are often ignored. The data of various factors such as the driver's gender, race, clothing, etc. have a lot of data noise, the calculation amount is huge, the detection speed is slow, and the accuracy of the output results is low. Therefore, the current detection system has relatively poor real-time performance, which to a certain extent restricts the deployment of fatigue driving detection systems on vehicle-mounted terminals. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the existing fatigue detection system has a slow detection speed and a low accuracy rate of output results.

[0006] The present invention solves the above-mentioned problem by adopting the following technical solution: A fatigue driving detection method based on a deep learning neural network model, comprising: S1: Collect pictures with driver’s facial images; S2: Input the image into the computational model, which is built based on the YOLOV8 model; S3: The computational model preprocesses the image. First, a uniform image size of 640*640 is generated. Then, the image is converted to RGB format by color channel conversion. The image is normalized to scale the image pixels from the range of [0, 255] to [0, 1]. S4: input the preprocessed image into the calculation model to extract the basic features of the image; S5: Use feature pyramid network and path aggregation network to fuse feature maps of different scales to obtain three feature maps; S6: Process the three feature maps to generate detection results and obtain the category and bounding box position of each candidate box.

[0007] Compared with the prior art, the present invention adopting the above structure has the following beneficial effects: The present invention utilizes a neural network model to implement fatigue driving detection, improves the accuracy of multi-feature detection of facial features, and efficiently and stably extracts facial feature points; the present invention performs a lightweight transformation on the existing computing model so that it requires fewer hardware resources, thereby achieving better detection efficiency when applied to low-configuration terminals.

[0008] As a preferred embodiment, a further technical solution of the above structure is: Preferably, the collected images include multiple types of images of drivers of different races, those wearing glasses and those not wearing glasses, those in normal and fatigue driving states, and those of male and female drivers.

[0009] Preferably, image preprocessing is performed at the input end of the computing model, specifically by using mosaic data enhancement, adaptive anchor box calculation and adaptive image scaling methods to find the long side of the image, scaling the image to 640 based on the long side, scaling the short side in the same proportion as the long side scaling ratio, and then supplementing the short side with a gray edge to 640, thereby converting the original image size into a network input size of 640*640.

[0010] Preferably, feature extraction is performed in the backbone network of the computing model. The backbone network of this computing model is transformed by introducing the AIFI module in RT-DETR to replace the SPPF module of YOLOV8. This backbone network consists of five convolution modules, four C2f modules and one SPPF module; the main structure of the convolution module is a 2D convolution, a BatchNorm2d and a SiLU activation function.

[0011] Preferably, the feature fusion is performed in the bottleneck network part of the computing model, and the data processing process is as follows: the outputs P3, P4, and P5 of the feature extraction network are input into the PAN-FPN network structure to fuse the feature maps of multiple scales, P5 is fused with P4 after upsampling to obtain F1, F1 is fused with P3 after a C2f layer and an upsampling to obtain T1, T1 is fused with F1 after a convolution layer to obtain F2, F2 is fused with a C2f layer to obtain T2, T2 is fused with P5 after a convolution layer to obtain F3, F3 is fused with a C2f layer to obtain T3, and finally three feature maps T1, T2, and T3 are obtained.

[0012] Preferably, the result output is completed at the prediction end of the computing model. At the prediction end of the computing model, a detection head is decomposed into two parts, each of which is composed of two 3×3 convolutions and one 1×1 convolution. The convolution layer performs a convolution operation on the input data by means of a sliding window, and the two parts use the same convolution kernel for feature extraction; a CIOU loss function is set in the regression branch of the detection head, and the loss function is LMPDIoU.

[0013] The present invention improves the original SPPF module of YOLOV8, and uses the AIFI (attention-based intra-scale feature interaction) module in RT-DETR to enhance the feature interaction capabilities within and between scales, and reduce the attention operations between features of multiple scales, and optimize the speed and computational consumption of the backbone network in detecting targets. AIFI mainly uses the self-attention mechanism to focus on the image features in the image, thereby improving the high-level performance of the model in object detection and recognition, while reducing unnecessary computational consumption.

[0014] The present invention proposes a method for sharing parameter heads, which merges two identical convolution modules into one module and improves it, and uses the same convolution kernel to perform feature extraction, so as to reduce the number of convolutions, reduce the number of convolution channels that were previously excessive, reduce the complexity of the model and the risk of overfitting, so as to achieve the purpose of lightweight model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Fig. 1 is the backbone network structure of the computing model of the present invention; Figure 2 It is the bottleneck network (neck) structure of the present invention; Figure 3 It is the detection part in the prediction end (head) of the present invention; Figure 4 It is the improved detection head of the present invention; Figure 5 It is the test result of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described below in conjunction with embodiments, the purpose of which is only to better understand the content of the present invention. Therefore, the examples given do not limit the protection scope of the present invention.

[0017] The present invention mainly aims at improving the lightweight of the system, and achieves the purpose of lightweight by compressing the model size and computing power, so that it can be better applied to various vehicle-mounted systems. The present invention provides a fatigue driving detection method based on a deep learning neural network model, including: S1: Collect pictures with driver’s facial images; S2: Input the image into the computational model, which is built based on the YOLOV8 model; S3: The computational model preprocesses the image. First, a uniform image size of 640*640 is generated. Then, the image is converted to RGB format by color channel conversion. The image is normalized to scale the image pixels from the range of [0, 255] to [0, 1]. S4: input the preprocessed image into the calculation model to extract the basic features of the image; S5: Use feature pyramid network and path aggregation network to fuse feature maps of different scales to obtain three feature maps; S6: Process the three feature maps to generate detection results and obtain the category and bounding box position of each candidate box.

[0018] The data set of the present invention has a total of 11,561 images, including the public data set YAWDD and the domestic open source data set fdd-dataset, including male and female drivers of different races, wearing glasses and not wearing glasses, and normal and fatigue driving states. Since the YAWDD data set is a video data set, one picture is extracted every 10 frames, and the extracted images are labeled, wherein 1011 opened_mouth bounding box labels, 8855 opened_eyes bounding box labels, 10501 closed_mouth bounding box labels, and 1969 closed_eyes bounding box labels are labeled, and the data set is divided into a training set, a test set, and a validation set according to a ratio of 7:2:1.

[0019] The YOLO series of algorithms have attracted much attention due to their high efficiency and accuracy. In order to achieve the goal of efficient and lightweight detection, the present invention selects YOLOv8n as the basic model for improvement. The following is a detailed description of the adjustments made by the present invention based on YOLOv8n. The entire YOLO network can be divided into four parts: input, backbone, bottleneck, and head.

[0020] At the input end, mosaic data enhancement, adaptive anchor box calculation and adaptive image scaling are used for preprocessing.

[0021] See also Figure 1 The backbone network (Backbone) is composed of five convolution modules, four C2f modules and one SPPF module. The main structure of the convolution module is a 2D convolution, a BatchNorm2d and a SiLU activation function. The function of the entire convolution module is to downsample and enhance the nonlinear representation ability of the model. The SPPF module is improved on the basis of the SPP (Spatial Pyramid Pooling) structure of YOLOv5. It solves the redundancy and efficiency problems of SPP in the calculation process. However, in order to reduce the amount of calculation of the algorithm, expand the receptive field and enhance the feature extraction ability of the network, this paper introduces the AIFI module in RT-DETR to optimize and improve the SPPF module.

[0022] RT-DETR is a real-time end-to-end object detector that effectively processes multi-scale features by decoupling internal scale interactions and cross-scale fusion, and proposes IoU-aware query selection, which further improves performance by providing higher quality initial object queries to the decoder. In addition, different decoders are used to flexibly adjust the inference speed without retraining. It has good performance in terms of processing speed and accuracy. Therefore, this paper uses the AIFI (attention-based internal scale feature interaction) module in RT-DETR to improve the original SPPF module to enhance the intra-scale and inter-scale feature interaction capabilities, reduce the attention operations between features of multiple scales, and optimize the speed and computational consumption of the backbone network in detecting targets. AIFI mainly uses the self-attention mechanism to focus on the image features in the image, thereby improving the model's high-level performance in object detection and recognition, while reducing unnecessary computational consumption.

[0023] The bottleneck network (Neck) part plays the main role of feature extraction and fusion in the entire network structure. The Neck part of yolov8 uses PAN-FPN. Figure 2 This is a flow chart of a Neck section. Figure 2 The main steps are to input the outputs P3, P4, and P5 of the feature extraction network (Backbone) into the PAN-FPN network structure to fuse feature maps of multiple scales, and fuse P5 with P4 after upsampling to obtain F1. F1 is fused with P3 after a C2f layer and an upsampling to obtain T1. T1 is fused with F1 after a convolution layer to obtain F2. F2 is fused with a C2f layer to obtain T2. ​​T2 is fused with P5 after a convolution layer to obtain F3. F3 is fused with T3 after a C2f layer, and finally T1, T2, and T3 are obtained.

[0024] After being processed by the backbone network and the bottleneck network, the prediction end processes the three feature maps T1, T2, and T3 to generate the output results of the model. The main part of the Head is to decompose a detection head into two parts and add the CIOU loss function to its detection head regression branch. The network structure can be found in Figure 3 .

[0025] In the original yolov8 network, the computational complexity of the head part accounts for more than 40% of the entire network, mainly due to the 12 3×3 convolutions and 6 1×1 convolutions in the detection head part. Each detection head divides it into two parts, each of which is composed of two 3×3 convolutions and one 1×1 convolution. The convolution layer performs convolution operations on the input data by means of a sliding window. Since the two parts perform the same convolution operations, the present invention proposes a method for sharing parameter heads, merging the two identical convolution modules into one module and improving it, and using the same convolution kernel for feature extraction, thereby reducing the number of convolutions, reducing the number of convolution channels that were previously excessive, reducing the complexity of the model and the risk of overfitting, so as to achieve the purpose of lightweight model. The improved detection head structure can be found in Figure 4 The EMSC (Efficient Multi-Scale Conv) is a multi-scale efficient convolutional network proposed by the present invention. Compared with the 3×3 convolutional network with the same number of channels, the number of parameters and the amount of calculation are greatly reduced. The main idea is to extract convolutional features of the number of channels through grouped convolution. However, because these features are extracted on a single feature channel, the information between each channel is independent. Therefore, a 1×1 convolution is used to fuse and extract the features of each group, and finally a high compression of the model is achieved.

[0026] The loss function mainly consists of three parts: bounding box positioning loss, target confidence loss, and target classification loss. It is a key component of target detection and has an important impact on the prediction performance of the model. The CIoU loss function used in the original Yolov8 model cannot effectively reflect the difference between the true values ​​of width and height and their confidence. There is also an imbalance in the bounding box regression loss between high-quality samples and low-quality samples. Moreover, most existing bounding box regression loss functions cannot be optimized when the predicted box has the same aspect ratio as the actual annotation box but the width and height values ​​are completely different. Therefore, this paper uses a bounding box regression loss function based on MPDIoU called LMPDIoU, which is an IoU loss based on the minimum point distance. It outperforms existing loss functions, has faster convergence speed and more accurate regression results.

[0027] Since the present invention improves fatigue driving recognition based on a lightweight structure, in order to reflect the improvement of the algorithm in high-precision lightweighting, this paper selects the current mainstream lightweight model for comparison, and compares the following algorithms for accuracy and lightweight indicators to reflect the advantages of this algorithm; Method mAP FLOPs / G Params / M Size / MB SDD 0.774 28.4 20.52 105.1 Yolov5n 0.928 4.1 1.77 3.8 Yolov7-Tiny 0.956 13.1 6.02 12.3 Yolov8s 0.977 28.8 11.2 23.7 This algorithm 0.97 6.2 2.83 5.7 From this table, it can be seen that the model size of this algorithm is only 5.7MB, which can be deployed in various vehicle-mounted systems. The parameter volume and computing power are only 2.83M and 6.2GFLoPs, which can better save computing resources in specific applications and has good application for the deployment of fatigue driving systems in vehicle-mounted terminals. The improved algorithm of the modified computing model of the present invention exceeds SDD, yolov7-Tiny and yolov8s in terms of floating-point operations and parameter volume, proving that the improvement of the algorithm in this paper in terms of lightweight is effective, and at the same time exceeds SDD and yolov7-Tiny in terms of accuracy, and also proves its performance in high-precision detection. In comparison with yolov5n, although it is not as good as yolov5n in terms of lightweight indicators, it is 4.2% higher in accuracy. Therefore, it can be proved that this algorithm can ensure high-precision detection while lightweighting the model.

[0028] The output test results can be found in Figure 5 ,It can be seen that this algorithm has a good detection effect on different races, drivers with and without glasses, and can accurately identify the state of the eyes and mouth, and has good judgment and recognition for the analysis of fatigue driving.

[0029] The above description is only a preferred feasible embodiment of the present invention, and does not limit the scope of rights of the present invention. All equivalent changes made using the contents of the present specification and its drawings are included in the scope of rights of the present invention.

Claims

1. A fatigue driving detection method based on a deep learning neural network model, It is characterized in that include, S1: Collect pictures with driver’s facial images; S2: Input the image into the computational model, which is built based on the YOLOV8 model; S3: The computational model preprocesses the image. First, a uniform image size of 640*640 is generated. Then, the image is converted to RGB format by color channel conversion. The image is normalized to scale the image pixels from the range of [0, 255] to [0, 1]. S4: input the preprocessed image into the calculation model to extract the basic features of the image; S5: Use feature pyramid network and path aggregation network to fuse feature maps of different scales to obtain three feature maps; S6: Process the three feature maps to generate detection results and obtain the category and bounding box position of each candidate box.

2. The fatigue driving detection method based on the deep learning neural network model according to claim 1, It is characterized in that The images collected include a variety of images of different races, drivers with and without glasses, normal and fatigued driving conditions, and male and female drivers.

3. The fatigue driving detection method based on the deep learning neural network model according to claim 1, It is characterized in that Image preprocessing is performed at the input end of the computing model. Specifically, mosaic data enhancement, adaptive anchor box calculation and adaptive image scaling methods are used to find the long side of the image. Based on the long side, the image is scaled to 640, the short side is scaled in the same proportion as the long side, and then the short side is supplemented with a gray edge to 640, thereby converting the original image size into a network input size of 640*640.

4. The fatigue driving detection method based on the deep learning neural network model according to claim 1, It is characterized in that The feature extraction is carried out in the backbone network of the computing model. The backbone network of this computing model is transformed by introducing the AIFI module in RT-DETR to replace the SPPF module of YOLOV8. This backbone network consists of five convolution modules, four C2f modules and one SPPF module; the main structure of the convolution module is a 2D convolution, a BatchNorm2d and a SiLU activation function.

5. The fatigue driving detection method based on the deep learning neural network model according to claim 1, It is characterized in that The feature fusion is carried out in the bottleneck network part of the calculation model, and the data processing process is as follows: the outputs P3, P4, and P5 of the feature extraction network are input into the PAN-FPN network structure to fuse the feature maps of multiple scales, P5 is fused with P4 after upsampling to obtain F1, F1 is fused with P3 after a C2f layer and an upsampling to obtain T1, T1 is fused with F1 after a convolution layer to obtain F2, F2 is fused with a C2f layer to obtain T2, T2 is fused with P5 after a convolution layer to obtain F3, F3 is fused with a C2f layer to obtain T3, and finally three feature maps T1, T2, and T3 are obtained.

6. The fatigue driving detection method based on the deep learning neural network model according to claim 1, Features: The result output is completed at the prediction end of the calculation model. The two detection heads at the prediction end of this calculation model are merged into one part, which consists of an EMSC convolution and a 1×1 convolution. The convolution layer performs convolution operations on the input data through a sliding window, and extracts convolution features of the number of channels through the idea of ​​grouped convolution; the MPDIoU loss function is set in the regression branch of the detection head to replace CIoU, which is called LMPDIoU.

Citation Information

Patent Citations

  • Fatigue driving detection method, equipment, storage medium and device

    CN115713751A

  • Fatigue driving detection method in low-light environment

    CN118334630A

Cited By

  • Lightweight vehicle target detection method and system based on RT-DETR

    CN121861623A

  • Lightweight vehicle target detection method and system based on RT-DETR

    CN121861623B