Fatigue driving detection system and method based on convolutional neural network

Abstract

The invention discloses a fatigue driving detection system and method based on a convolutional neural network, and relates to the technical field of automobile safe driving, and the system comprises a data collection module, a data preprocessing module, a feature extraction module, a fatigue detection module and a result output module. Feature extraction and fatigue detection are performed on the image data of the driver by using the convolutional neural network, and features in the image can be automatically learned, so that the detection accuracy is improved; image data of a driver are collected and processed in real time by adopting a computer vision technology, so that the fatigue state of the driver can be quickly and accurately detected, and the detection efficiency is improved; according to the fatigue driving detection system, special sensor equipment is not needed, and fatigue driving detection can be realized only by using a common camera, so that the detection cost is reduced; and the fatigue driving detection system can timely and accurately detect the fatigue state of the driver and remind the driver through sound or a display and the like, so that the driving safety is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile safe driving, and in particular to a fatigue driving detection system and method based on a convolutional neural network. Background Art

[0002] Fatigue driving refers to the phenomenon that after a long period of continuous driving, the driver's physiological and psychological functions become disordered, and the driving skills objectively decline. In this state, the driver's attention, perception, thinking ability, judgment, willpower and decision-making ability will be affected, leading to driving errors or complete loss of driving ability, which may cause traffic accidents. Fatigue driving has become one of the important causes of traffic accidents, seriously threatening the lives of drivers and pedestrians. Traditional fatigue driving detection methods are mainly based on the driver's physiological characteristics, such as heart rate, brain waves, etc., or based on the driver's behavioral characteristics, such as blinking frequency, yawning, etc. However, these methods have some limitations, such as the need for special sensor equipment, or manual observation and analysis of the driver's behavioral characteristics. This method has the problems of long detection and analysis time, low efficiency, and low detection accuracy.

[0003] With the development of computer vision and deep learning technology, fatigue driving detection methods based on convolutional neural networks (CNN) have gradually become a research hotspot. CNN has powerful feature extraction and classification capabilities, and can automatically learn features in images, thereby realizing accurate detection of fatigue driving. To this end, this application proposes a fatigue driving detection system and method based on convolutional neural networks to solve the above problems. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the deficiencies in the prior art, the present invention provides a fatigue driving detection system and method based on a convolutional neural network, which solves the technical problems raised in the above background.

[0006] (II) Technical solution

[0007] To achieve the above purpose, the technical solution adopted by the present invention is: a fatigue driving detection system based on convolutional neural network, including a data acquisition module, a data preprocessing module, a feature extraction module, a fatigue detection module and a result output module;

[0008] The data acquisition module is used to collect image data of the driver;

[0009] The data preprocessing module is used to preprocess the collected image data, including operations such as image cropping, normalization, and grayscale conversion;

[0010] The feature extraction module is used for performing feature extraction on the preprocessed image data by using a convolutional neural network to obtain a feature vector of the image;

[0011] The fatigue detection module is used to detect the fatigue state of the driver using a classifier according to the feature vector of the image to determine whether the driver is in a fatigue state;

[0012] The result output module is used to output the result of fatigue detection to the user.

[0013] Preferably, the convolutional neural network model is VGG-16, and the classifier is a support vector machine (SVM).

[0014] Preferably, the data acquisition module acquires the driver's image data, specifically facial images and eye images.

[0015] Preferably, the detection result outputted by the result output module to the user can be outputted visually or auditorily, and the visual result is displayed to the driver through a display, and the auditory result is prompted to the driver through sound.

[0016] A method for detecting fatigue driving based on a convolutional neural network comprises the following steps:

[0017] Step 1: Collect image data, using a camera to collect the driver's image data, including facial images and eye images;

[0018] Step 2: Preprocess the image data, preprocess the collected image data, including image cropping, normalization, grayscale and other operations;

[0019] Step 3: Extract image features, use convolutional neural network to extract features from preprocessed image data to obtain the feature vector of the image;

[0020] Step 4: Fatigue state detection: Based on the feature vector of the image, a classifier is used to detect the driver's fatigue state to determine whether the driver is in a fatigue state;

[0021] Step 5: Output the test results to the user, such as displaying the test results on a display or prompting the driver through sound.

[0022] Preferably, the image data in step 1 is collected and stored in a local hard disk.

[0023] Preferably, in the image data preprocessing in step 2, the collected image data needs to be cropped to a fixed size, the fixed size of the image cropping is 224×224 pixels, normalization is performed to normalize the pixel values ​​of the image to the range of [0,1], and grayscale is performed to convert the image into a grayscale image.

[0024] Preferably, when extracting image features in step three, a VGG-16 convolutional neural network is used to extract features from the preprocessed image data, a pre-trained weight file is used to initialize the VGG-16 model during training, and then the data set is used to fine-tune the model.

[0025] Preferably, when the fatigue state is detected in step 4, a support vector machine (SVM) classifier is used to detect the driver's fatigue state based on the feature vector of the image, and the SVM classifier is trained using a data set during the training process.

[0026] Preferably, when the result in step five is output, the result of the fatigue detection will be output to the user, including the driver's fatigue state, which is divided into fatigue or alertness, and an assessment of the degree of fatigue, such as mild fatigue, moderate fatigue, severe fatigue, etc., and the detection result will be displayed on a display or the driver will be prompted by sound.

[0027] (III) Beneficial effects

[0028] The beneficial effects of the present invention are:

[0029] The fatigue driving detection system and method based on convolutional neural network can automatically learn the features in the image and improve the detection accuracy by using convolutional neural network to extract features and detect fatigue of the driver's image data; it can quickly and accurately detect the driver's fatigue state and improve the detection efficiency by using computer vision technology to collect and process the driver's image data in real time; and the fatigue driving detection system does not require special sensor equipment, and only needs to use ordinary cameras to realize fatigue driving detection, thereby reducing the detection cost; and the fatigue driving detection system can timely and accurately detect the driver's fatigue state, and remind the driver through sound or display, thereby improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The figure is a flow chart of the fatigue driving detection method of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The present invention provides a technical solution: a fatigue driving detection system based on a convolutional neural network, comprising a data acquisition module, a data preprocessing module, a feature extraction module, a fatigue detection module and a result output module;

[0033] Specifically, the data acquisition module is used to collect image data of the driver, and the image data collected by the data acquisition module is specifically a facial image and an eye image of the driver;

[0034] Specifically, the data preprocessing module is used to preprocess the collected image data, including operations such as image cropping, normalization, and grayscale conversion;

[0035] Specifically, the feature extraction module is used for convolutional neural network to extract features from the preprocessed image data to obtain a feature vector of the image. The convolutional neural network model is VGG-16, which has high accuracy and generalization ability. The classifier is a support vector machine (SVM);

[0036] Specifically, the fatigue detection module is used to detect the fatigue state of the driver using a classifier according to the feature vector of the image to determine whether the driver is in a fatigue state;

[0037] Specifically, the result output module is used to output the result of fatigue detection to the user. The detection result output by the result output module to the user can be output visually or auditorily, and the visual display is used to display the driver through a display, and the auditory prompts the driver through sound.

[0038] Combination Figure 1 , a fatigue driving detection method based on convolutional neural network, comprising the following steps:

[0039] For image data collection, a camera is used to collect the driver's image data, including facial images and eye images, and the image data is stored in a local hard disk after collection;

[0040] Preprocess the image data, including image cropping, normalization, grayscale conversion and other operations. In the image data preprocessing, the collected image data needs to be cropped to a fixed size, normalized to normalize the pixel values ​​of the image to the range of [0,1], and grayscaled to convert the image into a grayscale image;

[0041] For image feature extraction, a convolutional neural network is used to extract features from the preprocessed image data to obtain the feature vector of the image. When extracting image features, a VGG-16 convolutional neural network is used to extract features from the preprocessed image data. During the training process, the VGG-16 model is initialized using the pretrained weight file, and then the model is fine-tuned using the dataset. The dataset used is a public fatigue driving dataset, such as the NTHU-DDD dataset, the UTA-RLDD dataset, etc. The dataset contains a large amount of driver image data and the corresponding fatigue status labels, which are used to train and verify the convolutional neural network model to improve the performance of the model. The training method is the stochastic gradient descent (SGD) algorithm. The algorithm has a faster convergence speed and better generalization ability. During the training process, the cross entropy loss function is used as the loss function of the model to measure the difference between the prediction result of the model and the true label. In order to prevent overfitting, L2 regularization and Dropout technology are used to optimize the model. The test method adopted by the present invention is a single picture test and a video test. In the single picture test, the picture to be tested is input into the trained convolutional neural network model to obtain the fatigue state prediction result of the picture. In the video test, the video to be tested is input into the trained convolutional neural network model to obtain the fatigue state prediction result of each frame in the video, and then the fatigue state in the video is judged according to certain rules;

[0042] For fatigue state detection, a classifier is used to detect the fatigue state of the driver according to the feature vector of the image to determine whether the driver is in a fatigue state. When detecting the fatigue state, a support vector machine (SVM) classifier is used to detect the fatigue state of the driver according to the feature vector of the image. The SVM classifier is trained using the data set during the training process;

[0043] The test results are output to the user, such as by displaying the test results on a display or prompting the driver through sound. When the results are output, the fatigue test results are output to the user, including the driver's fatigue status, which is divided into fatigue or alertness, and an assessment of the degree of fatigue, such as mild fatigue, moderate fatigue, severe fatigue, etc., and the test results are displayed on a display or the driver is prompted through sound.

[0044] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A fatigue driving detection system based on convolutional neural network, characterized in that: It includes data acquisition module, data preprocessing module, feature extraction module, fatigue detection module and result output module; The data acquisition module is used to collect image data of the driver; The data preprocessing module is used to preprocess the collected image data, including operations such as image cropping, normalization, and grayscale conversion; The feature extraction module is used for performing feature extraction on the preprocessed image data by using a convolutional neural network to obtain a feature vector of the image; The fatigue detection module is used to detect the fatigue state of the driver using a classifier according to the feature vector of the image to determine whether the driver is in a fatigue state; The result output module is used to output the result of fatigue detection to the user.

2. The fatigue driving detection system based on convolutional neural network according to claim 1 is characterized in that: The convolutional neural network model is VGG-16, and the classifier is a support vector machine (SVM).

3. The fatigue driving detection system based on convolutional neural network according to claim 1 is characterized in that: The data acquisition module acquires the driver's image data, specifically facial images and eye images.

4. The fatigue driving detection system based on convolutional neural network according to claim 1 is characterized in that: The detection result outputted by the result output module to the user can be outputted visually or auditorily, and the visual output is displayed to the driver through a display, and the auditory output is used to prompt the driver through sound.

5. A detection method for a fatigue driving detection system based on a convolutional neural network, characterized in that: The following steps are involved: Step 1: Collect image data, using a camera to collect the driver's image data, including facial images and eye images; Step 2: Preprocess the image data, preprocess the collected image data, including image cropping, normalization, grayscale and other operations; Step 3: Extract image features, use convolutional neural network to extract features from preprocessed image data to obtain the feature vector of the image; Step 4: Fatigue state detection: Based on the feature vector of the image, a classifier is used to detect the driver's fatigue state to determine whether the driver is in a fatigue state; Step 5: The LL outputs the detection result and outputs the result of the fatigue detection to the user, for example, by displaying the detection result on a display or by prompting the driver through sound.

6. The detection method of a fatigue driving detection system based on a convolutional neural network according to claim 5 is characterized in that: In the step 1, the image data is collected and stored in the local hard disk.

7. The detection method of a fatigue driving detection system based on a convolutional neural network according to claim 5 is characterized in that: In the image data preprocessing in step 2, the collected image data needs to be cropped to a fixed size, normalized to normalize the pixel values ​​of the image to the range of [0, 1], and grayscaled to convert the image into a grayscale image.

8. The detection method of a fatigue driving detection system based on a convolutional neural network according to claim 5 is characterized in that: When extracting image features in step three, a VGG-16 convolutional neural network is used to extract features from the preprocessed image data. During the training process, the VGG-16 model is initialized using a pretrained weight file, and then the model is fine-tuned using the data set.

9. The detection method of a fatigue driving detection system based on a convolutional neural network according to claim 5, characterized in that: During the fatigue state detection in step 4, a support vector machine (SVM) classifier is used to detect the driver's fatigue state according to the feature vector of the image, and the SVM classifier is trained using the data set during the training process.

10. The detection method of a fatigue driving detection system based on a convolutional neural network according to claim 5, characterized in that: When the result in step five is output, the result of the fatigue detection will be output to the user, including the driver's fatigue state and the evaluation of the fatigue degree, and the detection result will be displayed on a display or the driver will be prompted by sound.

Images