Fatigue driving detection method, system, equipment and medium

Abstract

The invention provides a fatigue driving detection method, system and device and a medium, and belongs to the technical field of fatigue detection, and the method comprises the following steps: collecting facial features and sound features of a to-be-detected driver; the method comprises the following steps: constructing a sound fatigue detection model CNN-SE-MN4 and an eye fatigue detection model Yov8n-MN4; inputting the sound features converted into the two-dimensional image spectrogram into CNN-SE-MN4, extracting time sequence features and spatial features, and judging sound fatigue features according to the time sequence features and the spatial features; using Yov8n-MN4 to extract an eye key area in the facial features, and using MobileNet v4 to determine fatigue features of the eye key area; and performing weight distribution on the sound fatigue characteristics and the fatigue characteristics of the eye key areas by using a fuzzy analytic hierarchy process, and determining the fatigue state of the to-be-detected driver according to a weight distribution result. According to the invention, the accuracy of fatigue state detection can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fatigue detection, and in particular relates to a fatigue driving detection method, system, equipment and medium. Background Art

[0002] Fatigue driving is the leading cause of life-threatening situations during long-distance driving. When a driver is fatigued, their ability to observe road conditions, operate the vehicle, and respond quickly to emergencies is significantly impaired. Statistics show that 25%-30% of traffic accidents are caused by fatigue driving. To overcome this problem, driver fatigue detection is a key approach to reducing accidents and improving traffic safety. Deep learning-based fatigue detection primarily utilizes facial features, physiological features, and features based on vehicle behavior. Early approaches primarily used single-modality feature extraction. Vision-based feature extraction is susceptible to sensor clarity, lighting conditions, and whether the driver's face is obstructed. This can lead to system instability in limited visual conditions. The sole use of in-vehicle cameras also raises concerns about driver privacy.

[0003] A literature proposes an eye-based fatigue detection algorithm, which extracts features from the eyes, uses eye aspect ratio EAR, blink frequency (BF), PERCLOS, and pupil occlusion rate (POR), performs multi-feature fusion on their calculation results, uses comprehensive evaluation to allocate weights, and obtains performance.

[0004] A multi-stream facial feature fusion convolutional neural network (FFF-CNN) has been proposed to optimize fatigue detection even with low-quality input. This method utilizes a global face stream and two local eye streams and introduces feature fusion to improve eye state classification. To facilitate feature fusion, two preprocessing modules are used: the Feature Attention Module (FAM) and the Stream Interaction Module (SIM). The FAM emphasizes important features in the global face stream, while the SIM corrects information from the two local eye streams.

[0005] However, existing facial feature fatigue detection methods cannot accurately identify the face when it is occluded or in a dark environment, resulting in low accuracy of fatigue detection results. Summary of the Invention

[0006] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a method for detecting fatigue driving, comprising the following steps:

[0007] Collect facial features and voice features of the driver to be detected;

[0008] Taking the deep separable convolutional neural network MobileNetV4 as the base model, the LSTM neural network and SE attention mechanism are introduced into MobileNet v4 to obtain the voice fatigue detection model CNN-SE-MN4. Taking the object detection model Yolov8n as the base model, the backbone network of Yolov8n is replaced with MobileNet v4, and the parameter-free attention mechanism is introduced into the neck network of Yolov8n to obtain the eye fatigue detection model Yolov8n-MN4.

[0009] The sound features are converted into a two-dimensional image spectrogram, which is then fed into CNN-SE-MN4. LSTM and SE are used to extract the temporal and spatial features of the two-dimensional image spectrogram. The sound fatigue features are then identified based on the temporal and spatial features.

[0010] Input facial features into Yolov8n-MN4, use Yolov8n-MN4 to extract the key eye area in the facial features, and use MobileNet v4 to determine the fatigue characteristics of the key eye area;

[0011] The fuzzy hierarchical analysis method is used to assign weights to the sound fatigue features and fatigue features of key eye areas, and the fatigue state of the driver to be detected is determined based on the results of the weight assignment.

[0012] Preferably, the method of using the fuzzy hierarchical analysis method to weight the sound fatigue characteristics and the fatigue characteristics of the key eye areas, and determining the fatigue state of the driver to be detected according to the result of the weight allocation, comprises the following steps:

[0013] Construct a fuzzy judgment matrix based on the fatigue characteristics of voice and key eye areas;

[0014] The fuzzy hierarchical analysis method is used to assign optimal weights to different indicators in the fuzzy judgment matrix, and the different weights after the optimal weight assignment are weighted to obtain the compatibility index;

[0015] The fatigue state of the driver to be detected is determined based on the compatibility index.

[0016] Preferably, the fatigue characteristics of the key eye area include eye closure state and blinking frequency, and the eye closure state is determined by the following formula:

[0017]

[0018] Among them, F EC Refers to the number of frames in which the eye is closed during the period when the eyelid covers 80% of the pupil, F S Refers to the total number of frames during the period when the eyelid covers 20% of the pupil;

[0019] The blink frequency is determined by the following formula:

[0020]

[0021] Among them, N tb Yes, the number of blinks per unit time, N t is the unit time, P b is the blink frequency per unit time.

[0022] Preferably, the voice fatigue feature is the percentage of fatigue spectrogram to total number of spectrogram frames, and its expression is as follows:

[0023]

[0024] Among them, N T represents the number of fatigue spectrograms per unit time, N represents the total number of spectrogram frames per unit time, and VOICE represents the percentage of fatigue spectrograms in the total number of spectrogram frames per unit time.

[0025] Preferably, before converting the sound features into a two-dimensional image spectrogram, the method further includes pre-processing the sound features by pre-emphasis, framing and windowing, speech denoising and endpoint detection.

[0026] The present invention also provides a fatigue driving detection system, comprising:

[0027] A data acquisition module, used to collect facial features and voice features of the driver to be detected;

[0028] The model building module is used to build a voice fatigue detection model CNN-SE-MN4 based on the deep separable convolutional neural network MobileNet V4. The module also uses the object detection model Yolov8n as the base model, replaces the backbone network of Yolov8n with MobileNet v4, and introduces a parameter-free attention mechanism into the neck network of Yolov8n to build the eye fatigue detection model Yolov8n-MN4.

[0029] The sound feature detection module is used to convert sound features into a two-dimensional image spectrogram, input the two-dimensional image spectrogram into the CNN-SE-MN4, use LSTM and SE to extract the temporal and spatial features of the two-dimensional image spectrogram, and identify the sound fatigue characteristics based on the temporal and spatial features;

[0030] Eye feature detection module, which is used to input facial features into Yolov8n-MN4, use Yolov8n-MN4 to extract the key eye area in the facial features, and use MobileNet v4 to determine the fatigue characteristics of the key eye area;

[0031] The fatigue state acquisition module is used to use the fuzzy hierarchical analysis method to weight the sound fatigue characteristics and the fatigue characteristics of the key areas of the eyes, and determine the fatigue state of the driver to be detected based on the results of the weight allocation.

[0032] The present invention also provides a computer device, comprising a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the fatigue driving detection method.

[0033] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the fatigue driving detection method.

[0034] The fatigue driving detection method provided by the present invention has the following beneficial effects:

[0035] The present invention introduces the LSTM neural network and SE attention mechanism into MobileNet v4 to obtain a voice fatigue detection model CNN-SE-MN4, replaces the backbone network of Yolov8n with MobileNet v4, and introduces a parameter-free attention mechanism into the neck network of Yolov8n to obtain an eye fatigue detection model Yolov8n-MN4; by using the LSTM and SE of CNN-SE-MN4, the temporal features and spatial features of the two-dimensional image spectrogram can be extracted, and the voice fatigue features can be discriminated based on the temporal features and spatial features; by using Yolov8n-MN4 to extract the key eye areas in the facial features, and by using MobileNet v4 to determine the fatigue features of the key eye areas, this process can accurately propose the fatigue features of the eyes; by using the fuzzy hierarchical analysis method to weight the fatigue features of the voice fatigue features and the fatigue features of the key eye areas, the fatigue state of the driver to be detected can be determined according to the result of the weight allocation, and this process can flexibly adjust the weights of different fatigue features according to the detection environment, thereby reducing the influence of the environment on the detection of facial fatigue features and improving the accuracy of fatigue state detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] To more clearly illustrate the embodiments of the present invention and its design, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0037] Figure 1 This is a flow chart of a fatigue driving detection method according to an embodiment of the present invention; Figure 2 This is a diagram of the sound information preprocessing process; Figure 3 This is the network structure diagram of the CNN-SE-MN4 model; Figure 4 This is the network structure diagram of the Yolov8n-MN4 model; Figure 5 It is a structural diagram of the SimAM module; Figure 6 This is a diagram showing the standard human eye opening degree of P80; Figure 7 is the accuracy change of model comparison; Figure 8 is the change in loss rate for model comparison; Figure 9 The accuracy and loss rate changes of the eye classification model; Figure 10 Comparison results of fatigue driving identification experiment. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the technical solution of the present invention and to be able to implement it, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.

[0039] Example 1

[0040] The present invention provides a method for detecting fatigue driving, specifically Figure 1 As shown, the following steps are included:

[0041] Step 1: Collect the facial features and voice features of the driver to be detected.

[0042] Step 2: Taking the deep separable convolutional neural network MobileNet V4 as the base model, introduce the LSTM neural network and SE attention mechanism into MobileNet v4 to obtain the voice fatigue detection model CNN-SE-MN4; taking the object detection model Yolov8n as the base model, replace the backbone network of Yolov8n with MobileNet v4, and introduce the parameter-free attention mechanism into the neck network of Yolov8n to obtain the eye fatigue detection model Yolov8n-MN4.

[0043] Step 3: Convert the sound features into a two-dimensional image spectrogram, input the two-dimensional image spectrogram into CNN-SE-MN4, use LSTM and SE to extract the temporal and spatial features of the two-dimensional image spectrogram, and discriminate the sound fatigue characteristics based on the temporal and spatial features.

[0044] The acquisition of sound information is subject to noise. To eliminate this noise, preprocessing is required. After preprocessing, the sound information is converted into a two-dimensional image spectrogram representation according to the same time segment. The two-dimensional image spectrogram (referred to as spectrogram) is input into the attention mechanism and deep separable convolutional neural network MobileNet V4 for training.

[0045] In the process of sound denoising, the main methods include pre-emphasis, framing and windowing, speech denoising, and endpoint detection. Figure 2 shown.

[0046] Pre-emphasis is due to the fact that when a person makes a sound, the high-frequency information is affected by the organs around the vocal cords, causing it to decay rapidly. The spectrum value of the high-frequency information is relatively small, which is not conducive to better utilization of the sound information. It is necessary to improve the high-frequency information of the sound in a digital filter. The digital filter is shown in the following formula 1:

[0047] H(Z)=1-μZ -1 ;

[0048] Where H(Z) is the frequency domain signal obtained after high-pass filtering, Z is the input signal after conversion to the frequency domain, and μ is the pre-emphasis coefficient, which is usually set to 0.95.

[0049] After this processing, the spectrum is relatively flat, and the information contained in the generated spectrogram is more obvious.

[0050] Framing is the process of dividing a continuous sound signal into equal time segments, which can effectively address its time-varying characteristics and facilitate spectrogram conversion. Windowing is the process of weighting each time segment to make the sound signal smoother in both the time and frequency domains. The principle is to strengthen adjacent segmented sound segments and weaken the remaining segments. Windowing uses a Hamming window, and its formula is as follows:

[0051]

[0052] Where w(n) is the Hamming window expression, N is the window length, and n is the index of the discrete time series.

[0053] The length of the time segments must be precisely calculated based on the signal sampling rate and analysis objectives. Specifically, the frame length is typically set to 10-30 milliseconds. This setting maintains time-domain resolution while ensuring stability in frequency-domain analysis. Furthermore, overlapping frames are often designed with a sliding step size of 1 / 3 to 1 / 2 the frame length, for example, 10 milliseconds. This partial overlap effectively mitigates edge effects caused by signal truncation, ensuring more efficient data processing.

[0054] Speech denoising processes audio containing human voices to remove background noise and enhance the human voice. The data collected in this invention is in a real road environment, where relatively stable noises are present, such as tire noise and engine noise. This is because a Wiener filter is used for denoising. This invention uses Wiener filtering to process stable noises. This algorithm is a classic algorithm based on the minimum mean square error (MSE) criterion. Its core concept is to achieve optimal linear filtering by estimating the statistical characteristics of the signal and noise.

[0055] Endpoint detection is crucial for feature extraction. Because driving is not a continuous process, but rather intermittent, it's crucial to isolate the valid sound components from the continuous audio. This allows for efficient extraction of the required dataset during model training. Endpoint detection can be performed using both time-domain and frequency-domain methods. The present invention employs a dual-threshold method based on time-domain principles, primarily utilizing the speech signal's short-time energy (STE) and short-time zero-crossing rate (STZCR) to detect the start and end points of speech. Its core principle is to analyze the amplitude of the speech signal using two preset upper and lower thresholds. Signal components with amplitudes between these two thresholds are identified as valid speech content, while those outside this range are considered noise or silence. This method effectively eliminates background noise and silence, resulting in clearer, purer speech information. This method is simple, computationally efficient, and suitable for real-time speech processing systems.

[0056] When calculating short-time energy, the reference signal needs to be framed and windowed before this processing method. The average energy is calculated as shown in the following formula:

[0057]

[0058] Where x(n) is a pure signal, h(n) = w 2 (n), w(n) is the window function in the windowing process, m is the window length, and n is the frame length.

[0059] The short-time zero-crossing rate indicates the number of times a waveform crosses zero in a frame of speech signal. In continuous signals, which are analog and whose waveforms change continuously on the time axis, a zero crossing refers to when the waveform actually crosses zero from a positive value to a negative value or vice versa. Discrete signals, on the other hand, are digital and consist of sampling points. Zero crossings are determined based on the sign changes of adjacent sampling points. The sign function is defined as follows:

[0060]

[0061] Where x represents the speech signal.

[0062] Sound signal x n The zero-crossing rate Z of (m) n As shown in the following formula:

[0063]

[0064] Where N is the window length, x n (m-1) represents the previous sampling point of the n-frame speech signal.

[0065] The dual-threshold method is an effective speech endpoint detection method, particularly suitable for scenarios with high real-time requirements. By properly selecting the threshold value, the speech signal can be effectively separated from the noise.

[0066] By pre-processing the driver's voice signal to improve the quality of the voice signal, the processed voice signal is processed according to Figure 2 The process is converted into a spectrogram, and then the converted spectrogram is input into the CNN-SE-MN4 model for fatigue judgment. The CNN-SE-MN4 model is as follows Figure 3 shown.

[0067] The fatigue index of the discriminant structure is calculated as the percentage of the fatigue spectrogram with a unit of 3s to the total number of spectrogram frames. The expression is shown in the following formula:

[0068]

[0069] Among them, N T represents the number of fatigue spectrograms per unit time, N represents the total number of spectrogram frames per unit time, and VOICE represents the percentage of fatigue spectrograms in the total number of frames per unit time.

[0070] Step 4: Input the facial features into Yolov8n-MN4, use Yolov8n-MN4 to extract the key eye area in the facial features, and use MobileNet v4 to determine the fatigue features of the key eye area.

[0071] The facial feature detection method uses eye fatigue information. When people are tired, the blink rate will increase and the eyes will be squinting. Therefore, based on Yolov8n, the backbone network of the algorithm is improved to MobileNet v4 module and the non-parameter attention mechanism is introduced into the neck network to form the Yolov8n-MN4 model. The improved model is as follows Figure 4 As shown, the non-parameter attention mechanism is Figure 4 The SimAM module is in the middle. The specific structure of the SimAM module is as follows Figure 5After the key eye regions are detected, they are fed into the MobileNet v4 of the Yolov8n-MN4 model. Classification is then performed using the MobileNet v4 network improved by depthwise separable convolution, and the decision result is obtained. The model structure is shown in Table 1:

[0072] Table 1 MobileNet v4 network structure diagram

[0073]

[0074] Fatigue determination in the eye area is usually calculated using the duration of eye closure and blink frequency. PERCLOS (percentage of eyelid closure over the pupil overtime) has been verified by numerous experiments to be a very reliable calculation method. PERCLOS refers to the percentage of time a driver's eyes are closed above a certain threshold within a certain period of time and is used to determine driver fatigue. Common methods for determining eye closure include EM (Eye Movement), P70, and P80, representing the states when the eyelids are closed to 50%, 70%, and 80% of the pupil, respectively. Studies have shown that P80 is the most effective criterion through experiments, and therefore the present invention uses P80 as a basis for determining the driver's eye closure.

[0075] like Figure 6 The horizontal axis shown in the figure represents that the eyelid covers 20% of the pupil, and t3 to t4 represents that the eyelid covers 80% of the pupil, which means that the eyes are closed. According to the P80 definition, the calculation formula is as follows:

[0076]

[0077] Among them, F EC Refers to the number of frames of eye closure from t3 to t4, F S Refers to the total number of frames from t2 to t5.

[0078] Fatigue judgment method based on blinking frequency:

[0079] The driver's blinking frequency changes significantly when driving in a non-fatigued and fatigued state. When driving in a non-fatigued state, the average blink frequency is about 6 times per minute, while when driving in a fatigued state, the blink frequency can reach 16-30 times per minute. Therefore, the present invention also uses the blink frequency as a fatigue determination method for the eye area. The calculation formula is as follows:

[0080]

[0081] Among them, Ntb Yes, the number of blinks per unit time, N t is the unit time, P b is the blink frequency per unit time.

[0082] Step 5: Use the fuzzy hierarchical analysis method to assign weights to the sound fatigue features and the fatigue features of the key areas of the eyes, and determine the fatigue state of the driver to be detected based on the results of the weight assignment.

[0083] The present invention is based on multimodal data and classifies driver fatigue by improving different models based on their voice and eye features. Different feature discrimination methods are then used to calculate the classification results of the driver fatigue detection model. Different indicators have different impacts on fatigue determination. The present invention uses multiple indicators to jointly determine the multimodal data and assigns personalized weights to the calculation results of different indicators. First, the fuzzy analytic hierarchy process (FAHP) is used to calculate the optimal weight distribution of different indicators, and the fatigue index is then calculated using the different weights.

[0084] In the case of mouth area data, the fuzzy judgment matrix is ​​constructed by the sound signal and the eye area. Only VOICE, PERCLOS and P b Judgment index, therefore according to PERCLOS=VOICE>P b The importance of is sorted and the fuzzy judgment matrix D is constructed as shown in Table 2 below.

[0085] Table 2 Fuzzy judgment matrix D

[0086] VOICE PERCLOS <![CDATA[P b ]]> VOICE 0.5 0.5 0.7 PERCLOS 0.5 0.5 0.7 <![CDATA[P b ]]> 0.3 0.3 0.5

[0087] Similarly, the weight vectors of the fuzzy judgment matrix D are 0.367, 0.367, and 0.267. The calculated compatibility index is 0.054<α=0.1, and the conclusion is that the consistency conclusion is passed, so the constructed fuzzy judgment matrix is ​​reasonable.

[0088] The present invention also provides a fatigue driving detection system, comprising:

[0089] A data acquisition module, used to collect facial features and voice features of the driver to be detected;

[0090] The model building module is used to build a voice fatigue detection model CNN-SE-MN4 based on the deep separable convolutional neural network MobileNetV4. The module also uses the object detection model Yolov8n as the base model, replaces the backbone network of Yolov8n with MobileNet v4, and introduces a parameter-free attention mechanism into the neck network of Yolov8n to build the eye fatigue detection model Yolov8n-MN4.

[0091] The sound feature detection module is used to convert sound features into a two-dimensional image spectrogram, input the two-dimensional image spectrogram into the CNN-SE-MN4, use LSTM and SE to extract the temporal and spatial features of the two-dimensional image spectrogram, and identify the sound fatigue characteristics based on the temporal and spatial features;

[0092] Eye feature detection module, which is used to input facial features into Yolov8n-MN4, use Yolov8n-MN4 to extract the key eye area in the facial features, and use MobileNet v4 to determine the fatigue characteristics of the key eye area;

[0093] The fatigue state acquisition module is used to use the fuzzy hierarchical analysis method to weight the sound fatigue characteristics and the fatigue characteristics of the key areas of the eyes, and determine the fatigue state of the driver to be detected based on the results of the weight allocation.

[0094] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the fatigue driving detection method.

[0095] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor to execute the fatigue driving detection method.

[0096] Example 2

[0097] Most traditional fatigue driving detection methods are based on a single feature, and the data set required to use a single feature is also single. The advantage of the present invention is that it uses a multivariate data set collected by a multimodal sensor, and uses an RGB camera and a microphone to collect the driver's facial RGB image and sound features. Since the camera can continuously collect images while the sound is not continuous during driving, the sound data set extracted from the video is relatively small. In the simulated environment, the sound data set is collected to solve the above-mentioned problem of insufficient extraction.

[0098] The present invention uses sleep deprivation to create fatigue. A total of 18 university student volunteers with driving qualifications were recruited, of whom 10 were selected to form a sleep deprivation group and 8 to form a normal sleep group. When collecting data sets, the sleep deprivation group played online games the night before and drove a motor vehicle on roads around the school from 8:00 to 8:30 a.m. the next day to collect a half-hour driving data set. The volunteers were also asked to make sounds to control the voice interaction of new energy vehicles, such as playing music, turning on the air conditioner, navigating to the company, turning off the air conditioner, etc. The normal sleep group rested according to the school's normal schedule and drove a motor vehicle on roads around the school from 8:00 to 8:30 a.m. the next day to collect a 30-minute driving data set. Because the sound data set is discontinuous, after collecting the video data set, a 10-minute sound data set was collected for each of the sleep deprivation group and the normal sleep group. The data set was collected with the car stationary on the side of the road to simulate an environment with road noise in the real environment.

[0099] The present invention collected a total of 5 hours of video data sets of eye fatigue and 4 hours of non-fatigue video data sets (see Table 3), as well as 100 minutes of fatigue sound data sets and 80 minutes of non-fatigue data sets (see Table 4). Through data set preprocessing and volunteer screening, the available data sets were selected as follows:

[0100] Table 3 Eye region dataset

[0101] category Total number of samples training set Test set fatigue 3000 2400 600 Non-fatigue 2400 1920 480 total 5400 4320 1080

[0102] Table 4 Sound dataset

[0103] category Total number of samples training set Test set fatigue 2400 1920 480 Non-fatigue 1920 1536 384 total 4320 3456 864

[0104] The present invention uses python 3.10 and the deep learning framework TensorFlow 2.14.0 to train the model. The system of the experimental environment uses the Windows 10 operating system. For the operating hardware, the present invention uses the Intel Core i7-12700 processor and the GeForce GTX3060 graphics card.

[0105] In the experimental phase of the present invention, a number of key indicators are used to comprehensively evaluate the performance of the model. These indicators include: Precision, Miss Rate, Recall, F1 Score, Average Precision and FPS (Frames Per Second).

[0106] Precision refers to the percentage of correctly predicted positive classes out of all predicted positive classes, that is, the accuracy of the model prediction. Its calculation formula is as follows:

[0107]

[0108] The MissRate indicates the percentage of positive class errors, reflecting the proportion of targets that the model failed to detect. Its calculation formula is as follows:

[0109]

[0110] Recall indicates the percentage of correctly predicted positive classes to all actual positive classes, that is, the ability of the model to identify targets is evaluated. The calculation formula is as follows:

[0111]

[0112] The F1 score represents the harmonic mean of precision and recall, providing a comprehensive evaluation of the overall performance of the model. Its calculation formula is as follows:

[0113]

[0114] Among them, TP, FP, and FN represent the number of correct positive predictions, the number of negative predictions as positive, and the number of positive predictions as negative, respectively.

[0115] Mean Average Precision (mAP) refers to the average precision of target detection across all categories. mAP is calculated based on AP, which stands for the area under the Precision-Recall curve. Its calculation formula is as follows:

[0116]

[0117] Where k represents the number of categories and P(r) represents the Precision-Recall curve.

[0118] FPS (Frames Per Second) refers to the number of images processed by the model per second, which is used to measure the efficiency of the model's processing speed. Through the organic combination of these indicators, the present invention can conduct an in-depth multi-dimensional analysis of the model's performance, thereby drawing more comprehensive and accurate conclusions.

[0119] In order to verify the effectiveness of the method of the present invention, the following experiments were conducted for comparison:

[0120] (1) Comparative experiment of CNN-SE-MN4 models.

[0121] This paper aims to verify the effectiveness of the CNN-SE-MN4 model in the task of sound signal classification. By comparing it with existing advanced models, which cover different architectures and design concepts, including the classic Resnet 34, GoogLeNet, and VGG16 models, as well as the MobileNet v3 and MobileNet v4 models, the paper verifies the advantages and disadvantages of the CNN-SE-MN4 model in terms of precision, recall, F1 score, and mAP, providing solid data support and theoretical basis for further optimization and practical application of the model.

[0122] Table 5 compares the CNN-SE-MN4 model with other models in terms of precision, recall, F1 score, and mAP. The experimental results show that the proposed driver voice signal discrimination model CNN-SE-MN4 is effective, achieving excellent performance in precision, recall, and F1 score. The CNN-SE-MN4 model achieves a precision of 95.9%, while the comparison models MobileNet v4, MobileNet v3, ResNet 34, GoogLeNet, and VGG16 achieve accuracies of 94.4%, 91.4%, 84.6%, 90.8%, and 90.1%, respectively. CNN-SE-MN4 improves upon these comparison models by 1.5%, 4.5%, 11.3%, 5.1%, and 5.8%, respectively. The CNN-SE-MN4 achieved a recall rate of 96.1%, compared to 95.2%, 91.2%, 84.8%, 90.4%, and 89.0% for the comparison models MobileNet v4, MobileNet v3, Resnet 34, GoogLeNet, and VGG16, respectively. These improvements represent improvements of 0.9%, 4.9%, 11.3%, 5.7%, and 7.1%, respectively. The F1 score, the harmonic mean of the precision and recall of a sound signal discrimination model, was 96.0% for the CNN-SE-MN4, compared to 94.8%, 91.3%, 84.7%, 90.6%, and 89.5% for the comparison models MobileNet v4, MobileNet v3, Resnet34, GoogLeNet, and VGG16, respectively. Compared with the comparison model, the performance improved by 1.2%, 4.7%, 11.3%, 5.4%, and 6.5%, respectively. The improved CNN-SE-MN4 model of the present invention has better performance than other models, so the model is suitable for fatigue determination based on spectrograms, which are graphical representations of the driver's voice signals. Therefore, the model is suitable for fatigue determination based on voice signals in the multimodal fusion fatigue driving detection method based on voice and facial features.

[0123] Table 5 Comparison experimental results of CNN-SE-MN4 model and other classification algorithms

[0124]

[0125]

[0126] Figure 7 The accuracy of driver fatigue assessment models based on acoustic signals varies. Among existing classic models, the MobileNet v4 model achieves the highest accuracy, reaching 95.2%. This is followed by the MobileNet v3, GoogLeNet, VGG16, and ResNet34 models, with accuracy rates of 91.2%, 90.9%, 89.2%, and 84.8%, respectively. The ResNet 34 model performs the worst. Therefore, this paper improves upon the latest MobileNet v4 by incorporating a channel attention mechanism and LSTM into the model's input, taking advantage of the temporal nature of spectrograms. Its purpose is to extract spatial and temporal characteristics, and also to change the image dimension. The accuracy of the improved network CNN-SE-MN4 model based on MobileNet v4 in this invention is 96.2%, which is 1% higher than the existing model MobileNet v4. The accuracy is increased by 5%, 5.3%, 7% and 11.4% compared with MobileNet v3, GoogLeNet, VGG16 and Resnet34 respectively.

[0127] Figure 8 The loss rate of the driver's acoustic signal-based fatigue assessment model shows changes. Among classic models, MobileNet v4 has the lowest loss rate of 13.6%, followed by MobileNet v3, GoogLeNet, VGG16, and Resnet34, with loss rates of 15.5%, 17.5%, 19.8%, and 25.3%, respectively. Resnet 34 has the highest loss rate. The loss rate of the CNN-SE-MN4 model proposed in this paper is 8.8%, which is 4.8%, 6.7%, 8.7%, 11%, and 16.5% lower than that of MobileNet v4, MobileNet v3, GoogLeNet, VGG16, and Resnet 34, respectively. Experimental data comparison demonstrates the effectiveness and advancement of the improved CNN-SE-MN4 proposed in this paper in fatigue assessment based on acoustic signals.

[0128] (2) Ablation experiment of CNN-SE-MN4 model improvement strategy.

[0129] In the above model comparison, MobileNet v4 achieved better performance, reflecting the applicability of the model in this task. It is higher than other models in terms of precision and accuracy, showing that the MobileNetv4 model proposed by the Google development team is excellent. Therefore, the present invention decided to improve on this model and add CNN modules, SE modules and LSTM modules in a serial connection in front of the model to form a CNN-SE-MN4 network model. In order to verify the rationality of adding modules and the improvement effect on the model, CNN, SE and LSTM were added separately on the basis of MobileNet v4. Then two modules were added, namely CNN and SE, CNN and LSTM, and SE and LSTM. Finally, the method of the present invention added all three modules. CNN was added to improve the spatial feature extraction capability and to reduce the dimension of the input image.

[0130] Experiments were conducted on a self-built dataset and comprehensive analysis was performed. The experimental results are shown in Table 6.

[0131] Table 6 Ablation experiment results of CNN-SE-MN4 model improvement strategy

[0132] method Precision Recall F1 Accuracy Ours 95.9％ 96.1％ 96.0％ 96.2％ MobileNet v4 94.4％ 95.2％ 94.8％ 95.2％ +CNN 94.4％ 93.9％ 94.1％ 94.2％ +SE 94.8％ 95.2％ 95.0％ 95.2％ +LSTM 95.4％ 94.7％ 95.1％ 95.4％ +CNN, SE 94.9％ 94.5％ 94.7％ 94.8％ +CNN, LSTM 95.4％ 95.0％ 92.2％ 95.4％ +SE, LSTM 95.6％ 95.5％ 95.5％ 95.7％

[0133] The experimental results in Table 6 show that the CNN-SE-MN4 model, formed by adding the three components to the MobileNet v4 model, achieves optimal precision, recall, and accuracy. Its precision reaches 95.4%, a 1.5% improvement over MobileNet v4, and its recall reaches 96.1%, a 0.9% improvement over MobileNet v4. The improved model achieves 96.2% accuracy, a 1% improvement over the base model. Adding only the CNN module decreases recall and accuracy by 1.3% and 1%, respectively. Adding only the LSTM module significantly increases precision by 1% and accuracy by 0.2%. Both modules showed improvements, but the increase was more pronounced when the SE and LSTM modules were added. Compared to MobileNet v4, the precision increased by 1.2%, the recall increased by 0.3%, and the accuracy increased by 0.5%. Comprehensive analysis shows that CNN-SE-MN4 adds different modules to the MobileNet v4 model to pre-process the input data of MobileNet v4, which results in different improvements. However, the addition of modules affects the calculation of the model but does not affect the overall efficiency.

[0134] The above two experiments verified the effectiveness and advancement of the improved CNN-SE-MN4 model based on MobileNet v4. When designing additional modules, the present invention ensures the lightweightness of MobileNet v4 and effectively improves the accuracy of driver voice signal fatigue classification.

[0135] (3) Eye feature extraction verification experiment.

[0136] In order to verify the effectiveness and advancement of the Yolov8n-MN4 model in extracting the driver's eye and mouth areas, Yolov8n-MN4 is compared with Yolov8s, Yolov8n, Yolov5n, Dlib, MTCNN, cascade regression tree, CLNF, and ShuffleNetv2. The effectiveness and real-time performance of the Yolov8n-MN4 model are verified by the precision, recall, F1 score, mAP, and FPS of eye and mouth area extraction. The experimental results of the model comparison are shown in Table 7.

[0137] As shown in the model comparison results in Table 7, among classic detection algorithms, Yolov8n achieved the highest accuracy of 97.4%, 0.4% higher than MTCNN's 97.0%. Its recall rate reached 97.2%, 0.6% lower than MTCNN's 97.8%. Yolov8n achieved an FPS of 175, while MTCNN achieved an FPS of 98. Yolov8n's higher accuracy and speed made it the preferred model for eye and mouth region detection. By incorporating MobileNet v4 principles into Yolov8n, we proposed the Yolov8n-MN4 detection model. By using the present invention in combination with public datasets and self-built datasets for training, the accuracy rate reached 99.2%, which is 1.8 percentage points higher than Yolov8n's 97.4%. The recall rate was 1.2 percentage points higher than Yolov8n, and the FPS was 17% higher than Yolov8n. This proves the advancement and rationality of the Yolov8n-MN4 model, and its detection speed also meets the real-time detection requirements of the fatigue detection system.

[0138] Table 7 Comparative experimental results of Yolov8n-MN4 model

[0139] method Precision Recall F1 mAP FPS Ours 99.2％ 98.4％ 98.8％ 98.6％ 206 Yolov8n 97.4％ 97.2％ 97.3％ 97.2％ 175 Yolov8s 96.6％ 96.8％ 96.7％ 96.1％ 130 Yolov5n 88.1％ 87.8％ 88.0％ 84.0％ 157 Cascaded Regression Trees 94.3％ 93.8％ 94.0％ 94.0％ 20 Dlib 96.1％ 95.4％ 95.7％ 95.5％ 272 CLNF 95.8％ 95.3％ 95.5％ 95.4％ 24 ShuffleNet v2 95.5％ 94.8％ 95.1％ 96.3％ 12 MTCNN 97.0％ 97.8％ 97.4％ 95.8％ 98

[0140] (3) Ablation experiment of Yolov8n-MN4 model improvement strategy

[0141] This paper improves Yolov8n to the Yolov8n-MN4 model. The experiment is divided into two parts: the first part is to replace the backbone network of Yolov8n with the lightweight network MobileNet v4 method to form the Yolov8n-v4 model method. The experiment verifies the effectiveness of the Yolov8n-v4 method in scenarios with limited computing resources. The second part is based on the improved model Yolov8n-v4 and adds classic attention mechanisms to the neck network, such as SimAM, SE, GAM, and GCT. The changes in indicators of the added attention mechanism are compared through experiments, and the experimental results are recorded as shown in Table 8.

[0142] Table 8 Ablation experiment results of Yolov8n-MN4 model improvement strategy

[0143] method Precision Recall F1 mAP FPS Ours 99.2％ 98.4％ 98.8％ 98.6％ 206 Yolov8n 97.4％ 97.2％ 97.3％ 97.2％ 175 Yolov8n-v4 98.8％ 97.2％ 97.9％ 94.4％ 210 +SimAM 98.7％ 97.8％ 98.2％ 98.4％ 170 +GCT 92.6％ 92.0％ 92.3％ 92.6％ 170 +SE 96.4％ 95.7％ 96.0％ 96.6％ 168 +GAM 95.1％ 90.9％ 93.0％ 92.8％ 169

[0144] The experimental results in Table 8 show that the Yolov8n-v4 model in the first section significantly improves precision, recall, F1, and mAP compared to the original Yolov8n. Precision increased from 97.4% to 98.8%, while recall remained at 97.2%. F1 increased from 97.3% to 97.9%, and mAP increased from 97.2% to 94.4%. The FPS of the Yolov8n-v4 model increased from 175 to 210, indicating that the introduction of a lightweight backbone network significantly improved the model's inference speed while maintaining high detection accuracy. In the second section, SimAM improved detection accuracy while maintaining high inference speed, achieving excellent overall performance. The introduction of GCT resulted in a significant decrease in detection accuracy, which may be unsuitable for the current model structure and requires further optimization. The addition of SE may be effective in improving detection accuracy, but its precision performance is lower than that of SimAM and has a certain negative impact on inference speed. The introduction of GAM resulted in a decrease in both detection accuracy and inference speed, possibly due to excessive computational overhead. Comprehensive comparisons do not justify this approach. By replacing the Yolov8n backbone network with the lightweight MobileNet v4 network and introducing the SimAM attention mechanism into the neck network, the Yolov8n-MN4 model was successfully improved. Experimental results demonstrate that the improved model significantly improves both detection accuracy and inference speed, demonstrating its advanced performance in detecting driver eye and mouth feature regions.

[0145] (4) Eye discrimination verification experiment.

[0146] The MobileNetv4 model was used in the eye region discrimination model comparison experiment. The model was trained using a method combining a self-built dataset. This model, MobileNet v4, was compared with the MobileNet v3, Resnet34, GoogLeNet, and VGG16 classic models. The training results are saved as shown in Table 9.

[0147] Table 9 MobileNetv4 eye discrimination comparison experimental results

[0148] method Precision Recal F1 Accuracy Ours 95.9％ 95.6％ 95.7％ 95.9％ MobileNet v3 92.4％ 92.0％ 92.2％ 92.3％ Resnet 34 85.6％ 85.7％ 85.6％ 85.7％ GoogLeNet 90.9％ 90.4％ 90.6％ 90.8％ VGG16 92.3％ 91.8％ 90.2％ 92.2％

[0149] The experimental results in Table 9 show that MobileNet v4_Conv_Small performs best in precision, exceeding MobileNet v3 by 3.5 percentage points and surpassing the lowest precision of the Resnet-34 model by 10.3 percentage points. This indicates that MobileNet v4 has the highest proportion of samples predicted as positive that are actually positive. MobileNet v4_Conv_Small also achieves a 3.6 percentage point higher recall than its predecessor, MobileNet v3, and a 9.9% higher recall than the lowest Resnet-34.

[0150] Figure 9 The accuracy and loss rate of the MobileNet v4 model during training for 100 epochs show a maximum accuracy of 95% during training and 95.9% on the test dataset. The loss rate curve shows a minimum loss of 12.7% on the training set and 12.3% on the test set. This experimental comparison demonstrates that MobileNet v4 can be applied to eye fatigue detection.

[0151] (5) Multimodal fusion experiment.

[0152] The model discrimination results are calculated using multiple indicators. Different indicators have different importance in the overall evaluation, so different weights need to be assigned to different indicators. The present invention adopts the fuzzy analytic hierarchy process to assign weights. The rationality of the present invention is verified by self-built multimodal data.

[0153] The effectiveness of the method of the present invention was verified by comparing multiple indicators with the comparative method. Figure 10 As shown, the fatigue index used in the present invention is the voice signal VOICE, and the eye area calculation index is P bThe weights calculated for the judgment matrix constructed using the fuzzy analytic hierarchy process (FAHP) for the PERCLOS and PERCLOS methods were 0.367, 0.367, and 0.267, respectively. The comparison methods used the same weight of 1 / 3 and recorded the comprehensive experimental results for voice and eye features. Their F1 score was 97.1, while the comparison method achieved 0.914.

[0154] In order to improve the accuracy of fatigue detection, the present invention proposes a multimodal fatigue driving detection method based on the combination of eye and sound features. According to the results, the use of a multimodal method can solve the problem of model failure caused by a single modality in a restricted environment. Compared with a single modality, the use of a multimodal method can improve its accuracy. Looking to the future, in recent years, the development and popularization of new energy vehicles have been rapid. New energy vehicles are more intelligent and voice operation is more convenient. The use of a method combined with sound can eliminate the problem of privacy leakage caused by facial features. In the future, as the industry develops, the use of physiological signal detection equipment in the steering wheel can better detect the physiological state of fatigue and effectively avoid the problem of implicit leakage.

[0155] The above-described embodiments are only preferred specific implementation methods of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention fall within the protection scope of the present invention.

Claims

1. A method for detecting fatigue driving, characterized in that: include: Collect facial features and voice features of the driver to be detected; Taking the deep separable convolutional neural network MobileNetV4 as the base model, the LSTM neural network and SE attention mechanism are introduced into MobileNet v4 to obtain the voice fatigue detection model CNN-SE-MN4. Taking the object detection model Yolov8n as the base model, the backbone network of Yolov8n is replaced with MobileNet v4, and the parameter-free attention mechanism is introduced into the neck network of Yolov8n to obtain the eye fatigue detection model Yolov8n-MN4. The sound features are converted into a two-dimensional image spectrogram, which is then fed into CNN-SE-MN4. LSTM and SE are used to extract the temporal and spatial features of the two-dimensional image spectrogram. The sound fatigue features are then identified based on the temporal and spatial features. Input facial features into Yolov8n-MN4, use Yolov8n-MN4 to extract the key eye area in the facial features, and use MobileNet v4 to determine the fatigue characteristics of the key eye area; The fuzzy hierarchical analysis method is used to assign weights to the sound fatigue features and fatigue features of key eye areas, and the fatigue state of the driver to be detected is determined based on the results of the weight assignment.

2. The fatigue driving detection method according to claim 1, characterized in that: The method of using the fuzzy hierarchical analysis method to weight the sound fatigue characteristics and the fatigue characteristics of the key eye area, and determining the fatigue state of the driver to be detected based on the weight distribution results, includes the following steps: Construct a fuzzy judgment matrix based on the fatigue characteristics of voice and key eye areas; The fuzzy hierarchical analysis method is used to assign optimal weights to different indicators in the fuzzy judgment matrix, and the different weights after the optimal weight assignment are weighted to obtain the compatibility index; The fatigue state of the driver to be detected is determined based on the compatibility index.

3. The fatigue driving detection method according to claim 1, characterized in that: The fatigue characteristics of the key eye area include eye closure state and blinking frequency, and the eye closure state is determined by the following formula: Among them, F EC Refers to the number of frames in which the eye is closed during the period when the eyelid covers 80% of the pupil, F S Refers to the total number of frames during the period when the eyelid covers 20% of the pupil; The blink frequency is determined by the following formula: Among them, N tb Yes, the number of blinks per unit time, N t is the unit time, P b is the blink frequency per unit time.

4. The fatigue driving detection method according to claim 1, characterized in that: The voice fatigue feature is the percentage of fatigue spectrogram to the total number of spectrogram frames, and its expression is as follows: Among them, N T represents the number of fatigue spectrograms per unit time, N represents the total number of spectrogram frames per unit time, and VOICE represents the percentage of fatigue spectrograms in the total number of spectrogram frames per unit time.

5. The fatigue driving detection method according to claim 1, characterized in that: Before converting the sound features into a two-dimensional image spectrogram, the sound features are pre-processed by pre-emphasis, framing and windowing, speech denoising and endpoint detection.

6. A fatigue driving detection system, characterized in that: include: A data acquisition module, used to collect facial features and voice features of the driver to be detected; The model building module is used to build a voice fatigue detection model CNN-SE-MN4 based on the deep separable convolutional neural network MobileNet V4. The module also uses the object detection model Yolov8n as the base model, replaces the backbone network of Yolov8n with MobileNet v4, and introduces a parameter-free attention mechanism into the neck network of Yolov8n to build the eye fatigue detection model Yolov8n-MN4. The sound feature detection module is used to convert sound features into a two-dimensional image spectrogram, input the two-dimensional image spectrogram into the CNN-SE-MN4, use LSTM and SE to extract the temporal and spatial features of the two-dimensional image spectrogram, and identify the sound fatigue characteristics based on the temporal and spatial features; Eye feature detection module, which is used to input facial features into Yolov8n-MN4, use Yolov8n-MN4 to extract the key eye area in the facial features, and use MobileNet v4 to determine the fatigue characteristics of the key eye area; The fatigue state acquisition module is used to use the fuzzy hierarchical analysis method to weight the sound fatigue characteristics and the fatigue characteristics of the key areas of the eyes, and determine the fatigue state of the driver to be detected based on the results of the weight allocation.

7. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the fatigue driving detection method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the fatigue driving detection method according to any one of claims 1 to 5.

Images