A non-contact fatigue detection method and system based on multi-sensor
Through a multi-sensor method combined with IMU and vision camera, a discriminant model is constructed using support vector mechanism and lightweight neural network, solving the problem of insufficient comprehensive single indicators and interference in the existing technology, achieving interference-free and accurate Driver fatigue detection improves the reliability of the detection system and the efficiency of computing resource utilization.
Patent Information
- Application Number
- CN202410917706.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-10
AI Technical Summary
The existing driver fatigue detection technology has problems such as insufficient single indicators, interference to drivers, and high computing resources, which leads to inaccurate detection results and low reliability.
Using a multi-sensor non-contact method based on IMU and vision camera, first- and second-level discriminant models are constructed through support vector mechanisms and lightweight neural networks, combining vehicle motion data and driver facial expression image data to perform fatigue state detection and warn.
It realizes interference-free and accurate driver fatigue status detection, improves the reliability of the detection system and the utilization efficiency of computing resources, and ensures driving safety.
Smart Images

Figure CN118470694B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of driver fatigue detection, and in particular relates to a non-contact fatigue detection method and system based on multiple sensors. Background Art
[0002] The Chinese patent (a driver driving status monitoring system CN117958742A) provides an active monitoring method for the driver's driving status, that is, a fatigue detection mark is displayed through a head-up display device (such as: HUD, etc.), and a fatigue index is obtained according to the number of times the eyes focus on the fatigue detection mark per unit time. If the fatigue index is greater than the fatigue threshold, a warning mark is displayed through the head-up display device, thereby improving the driver's driving safety; and the fatigue threshold is also linked to the driver's wearable device, and the fatigue threshold is corrected according to the night sleep index collected by the wearable device, that is, the worse the sleep, the lower the fatigue threshold, thereby improving the monitoring sensitivity.
[0003] The invention evaluates the driver's fatigue status through the head-up display device and the number of eye focusing times, and corrects the fatigue threshold in combination with the nighttime sleep index. However, a single indicator may not be able to fully reflect the fatigue status, which may lead to inaccurate monitoring results. At the same time, relying on the driver's wearable device connection may be affected by the driver's acceptance and the accuracy of the device, which may limit the actual application effect of the method.
[0004] The Chinese patent (A fatigue detection signal acquisition device based on multimodal data fusion CN117958839A) proposes a fatigue detection signal acquisition device based on multimodal data fusion of ADS1299, MPU6050 and STM32F407ZGT6. First, the physiological signal is processed by a second-order low-pass filter and then enters the ADS1299 integrated chip. The EEG signal amplified by PGA is converted into analog to digital, and the converted EEG data is sent to the microcontroller MCU through the SPI interface for processing and analysis. At the same time, the motion data of the human head collected by the MPU6050 gyroscope is sent to the microcontroller through IIC. Finally, the collected data is sent to the Bluetooth module through the asynchronous communication serial port for wireless data transmission, and finally, it is transmitted to the host computer.
[0005] The invention proposes a lightweight and convenient head-mounted device for collecting EEG and EOG for further fatigue monitoring. However, the accuracy and stability of sensors such as ADS1299 and MPU6050 may be affected by environmental interference, affecting the accuracy of fatigue detection results. In addition, compared with non-contact sensors, head-mounted devices may have a certain impact on the driver's normal driving, such as increasing the driver's discomfort or interfering with their normal operation. Especially during long-term driving, wearing a head-mounted device may cause fatigue in the driver, thereby affecting driving behavior and safety.
[0006] The Chinese patent (Driving fatigue detection method and system based on lightweight neural network image enhancement CN117789181A) proposes a driving fatigue detection method based on lightweight neural network image enhancement, which is characterized by: real-time acquisition of dynamic images of the driver in the cockpit, and enhancement processing of the dynamic images; capturing the driver's facial image stream in the enhanced dynamic images through a lightweight fast face detector; inputting the facial image stream into a preset facial key feature point extraction network model to extract facial key feature points of each frame image of the facial image stream, wherein the facial key feature point extraction network model is obtained by constructing and training a heavily parameterized lightweight convolutional network; judging whether the driver has facial fatigue features based on the facial key feature points, and determining that the driver is driving fatigued when the facial fatigue features are detected.
[0007] This invention uses machine vision to identify fatigue driving. However, judging only by facial fatigue features may be subjective and limited. Because the manifestation of fatigue features may vary from individual to individual, and this method may not cover all fatigue manifestations, the accuracy and reliability of the detection results are affected. In addition, the system requires high-frequency image data acquisition for model inference, which consumes a lot of computing resources. Summary of the invention
[0008] In order to solve the above technical problems, the present invention proposes a multi-sensor based non-contact fatigue detection method and system to solve the above problems existing in the prior art.
[0009] To achieve the above object, the present invention provides a non-contact fatigue detection method based on multiple sensors, comprising:
[0010] Acquire vehicle motion data and driver facial expression image data based on IMU and visual camera;
[0011] Processing the vehicle motion data and the driver's facial expression image data to obtain a motion data set and an image data set;
[0012] Constructing a primary discrimination model by using a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result;
[0013] Building a secondary discrimination model based on a lightweight neural network structure, inputting the image data set into the secondary discrimination model, and obtaining a secondary discrimination result;
[0014] The driver's fatigue state is detected and warned based on the primary discrimination result and the secondary discrimination result.
[0015] Preferably, the IMU is used to collect steering wheel motion data when the vehicle is moving, and the sampling frequency is 100HZ;
[0016] The visual camera is used to collect the driver's facial expression data, and the sampling frequency is 10 frames per second.
[0017] Preferably, the process of processing the vehicle motion data and the driver's facial expression image data to obtain a motion data set and an image data set includes:
[0018] Processing the vehicle motion data based on a sliding smoothing filter algorithm to obtain filtered data, and then performing feature extraction on the filtered data to obtain the motion data set;
[0019] The driver's facial expression image data is annotated, and then data enhancement is performed on the annotated image data based on a Gaussian noise method to obtain the image data set.
[0020] Preferably, the process of extracting features from the filtered data includes:
[0021] Calculate the variance, mean, maximum value, and sum of squares SSI in the filtered data;
[0022] Construct a 3-channel feature vector matrix based on the variance, mean, maximum, square, SSI, acceleration of the X and Y axes and the angular velocity of the Z axis in the filtered data;
[0023] The 3-channel feature vector matrix is merged and expanded to obtain a feature vector with a dimension of 12 corresponding to each joint node in the IMU signal sample;
[0024] The feature vector corresponding to the dimension of 12 is normalized to obtain the motion data set.
[0025] Preferably, the process of constructing a primary discrimination model by using a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result comprises:
[0026] SVM is used as the basic model, with a window size of 500ms, and a first-level discriminant model is established based on the IMU data in each window;
[0027] Construct fatigue thresholds based on historical prediction results;
[0028] Inputting the motion data set into the primary discriminant model for calculation to obtain fatigue degree prediction results within multiple moving windows;
[0029] The fatigue degree prediction results in the multiple moving windows are compared with the fatigue degree threshold to obtain the primary discrimination result.
[0030] Preferably, the process of constructing a secondary discrimination model based on a lightweight neural network structure, inputting the image data set into the secondary discrimination model, and obtaining a secondary discrimination result includes:
[0031] Building a lightweight neural network structure based on a depth-separable convolution structure, an inverted residual block, a Squeeze-and-Excitation module, and a HardSwish activation function, and training the lightweight neural network structure to obtain the secondary discriminant model;
[0032] The image data set is input into the secondary discrimination model to obtain the secondary discrimination result.
[0033] Preferably, the loss function used by the lightweight neural network structure is a focal loss function, and the expression of the focal loss function is:
[0034] ;
[0035] in, is the focal loss function, y' is the predicted probability value; γ is the adjustment parameter of the focal loss, usually taking a positive value; log() is the natural logarithm function.
[0036] To achieve the above object, the present invention also provides a non-contact fatigue detection system based on multiple sensors, comprising:
[0037] Sensor data acquisition module, used to obtain vehicle motion data and driver facial expression image data based on IMU and visual camera;
[0038] A data preprocessing module, used for processing the vehicle motion data and the driver's facial expression image data to obtain a motion data set and an image data set;
[0039] A primary fatigue discrimination module, used for constructing a primary discrimination model through a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result;
[0040] The secondary fatigue discrimination module is used to construct a secondary discrimination model based on a lightweight neural network structure, input the image data set into the secondary discrimination model, and obtain a secondary discrimination result.
[0041] Compared with the prior art, the present invention has the following advantages and technical effects:
[0042] 1. Non-contact multi-sensor application: The present invention utilizes a variety of sensors, including MU and visual cameras, to complete a comprehensive detection of the driver's fatigue status in a non-contact manner. This innovation means that the system will not have any impact on the driver, avoiding the limitation of traditional fatigue detection methods that require the driver to wear sensors or devices. By comprehensively utilizing the data collected by different sensors, the system can more comprehensively and accurately evaluate the driver's status, improving the reliability and practicality of the fatigue detection system.
[0043] 2. Combination of primary and secondary discrimination: The present invention divides fatigue detection into primary and secondary discrimination. The primary discrimination is based on IMU data to quickly predict the driver's fatigue level, while the secondary discrimination is based on machine vision technology to further confirm the driver's fatigue state. Since the secondary discrimination is only performed when necessary, excessive consumption of computing resources can be avoided, making the system more efficient. This hierarchical discrimination strategy fully considers the effective use of computing resources while achieving accurate fatigue detection, thereby improving the overall performance of the system.
[0044] 3. Utilizing time features in primary discrimination: The present invention uses comprehensive analysis of data in multiple continuous time windows to more accurately determine the driver's fatigue state. This comprehensive utilization method based on time features improves the system's monitoring accuracy and reliability of the driver's state, providing more effective protection for driving safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0046] Figure 1 This is a flow chart of a non-contact fatigue detection method based on multiple sensors according to an embodiment of the present invention;
[0047] Figure 2 A schematic diagram of the installation position and installation direction of the IMU according to an embodiment of the present invention;
[0048] Figure 3 A diagram of a primary discrimination process according to an embodiment of the present invention;
[0049] Figure 4This is a diagram of the data transmission process of each functional module in an embodiment of the present invention. DETAILED DESCRIPTION
[0050] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0051] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0052] Embodiment 1
[0053] like Figure 1 As shown, this embodiment provides a non-contact fatigue detection method based on multiple sensors, including:
[0054] Acquire vehicle motion data and driver facial expression image data based on IMU and visual camera;
[0055] Processing the vehicle motion data and the driver's facial expression image data to obtain a motion data set and an image data set;
[0056] Constructing a primary discrimination model by using a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result;
[0057] Building a secondary discrimination model based on a lightweight neural network structure, inputting the image data set into the secondary discrimination model, and obtaining a secondary discrimination result;
[0058] The driver's fatigue state is detected and warned based on the primary discrimination result and the secondary discrimination result.
[0059] Further optimizing the scheme, the IMU is used to collect steering wheel motion data when the vehicle is moving, and the sampling frequency is 100HZ;
[0060] The visual camera is used to collect the driver's facial expression data, and the sampling frequency is 10 frames per second.
[0061] Further optimizing the scheme, the process of processing the vehicle motion data and the driver's facial expression image data to obtain the motion data set and the image data set includes:
[0062] Processing the vehicle motion data based on a sliding smoothing filter algorithm to obtain filtered data, and then performing feature extraction on the filtered data to obtain the motion data set;
[0063] The driver's facial expression image data is annotated, and then data enhancement is performed on the annotated image data based on a Gaussian noise method to obtain the image data set.
[0064] To further optimize the scheme, the process of extracting features from the filtered data includes:
[0065] Calculate the variance, mean, maximum value, and sum of squares SSI in the filtered data;
[0066] Construct a 3-channel feature vector matrix based on the variance, mean, maximum, square, SSI, acceleration of the X and Y axes and the angular velocity of the Z axis in the filtered data;
[0067] The 3-channel feature vector matrix is merged and expanded to obtain a feature vector with a dimension of 12 corresponding to each joint node in the IMU signal sample;
[0068] The feature vector corresponding to the dimension of 12 is normalized to obtain the motion data set.
[0069] Further optimizing the scheme, the process of constructing a primary discrimination model by using a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result includes:
[0070] SVM is used as the basic model, with a window size of 500ms, and a first-level discriminant model is established based on the IMU data in each window;
[0071] Construct fatigue thresholds based on historical prediction results;
[0072] Inputting the motion data set into the primary discriminant model for calculation to obtain fatigue degree prediction results within multiple moving windows;
[0073] The fatigue degree prediction results in the multiple moving windows are compared with the fatigue degree threshold to obtain the primary discrimination result.
[0074] Further optimizing the scheme, the process of constructing a secondary discrimination model based on a lightweight neural network structure, inputting the image data set into the secondary discrimination model, and obtaining the secondary discrimination result includes:
[0075] Building a lightweight neural network structure based on a depth-separable convolution structure, an inverted residual block, a Squeeze-and-Excitation module, and a HardSwish activation function, and training the lightweight neural network structure to obtain the secondary discriminant model;
[0076] The image data set is input into the secondary discrimination model to obtain the secondary discrimination result.
[0077] Further optimizing the scheme, the loss function used by the lightweight neural network structure is the focus loss function, and the expression of the focus loss function is:
[0078] ;
[0079] in, is the focal loss function, y' is the predicted probability value; γ is the adjustment parameter of the focal loss, usually taking a positive value; log() is the natural logarithm function.
[0080] Embodiment 2
[0081] This embodiment provides a multi-sensor based non-contact fatigue detection method, including:
[0082] S1. Sensor selection and data collection. Select sensors suitable for fatigue detection and place them inside the vehicle, including inertial measurement units (IMUs) and visual cameras. The IMU is used to collect steering wheel movement data when the vehicle is moving, with a sampling frequency of 100 Hz; the visual camera is used to collect driver's facial expression data, with a sampling frequency of 10 frames per second.
[0083] S1-1. The steering wheel motion data acquisition module when the vehicle is moving uses the N100 nine-axis attitude sensor produced by Lunqu Technology Co., Ltd. To ensure that it does not affect the driver's normal driving, the N100MINI model was selected, with sizes of 24.5mm, 24.5mm and 14.5mm. The sensor has a data output frequency of up to 400Hz, an accuracy of 0.05°, and high stability, and can collect X, Y, and Z three-axis angular acceleration, acceleration and Euler angle data in real time. At the same time, it supports TTL serial port, CAN and RS485 communication methods, providing a flexible data transmission method. The IMU is installed under the steering wheel, and the sensor is fixed with a soft bracket. When the steering wheel is straightened, the positive direction of the X-axis is parallel to the steering wheel plane and points directly upwards. The IMU installation position and installation direction are as follows: Figure 2 shown.
[0084] S1-2, the driver's facial expression data acquisition module uses the RealSense D435i camera produced by Intel. The camera module has an RGB image resolution of 1920x1080 and a size of approximately 90×25×25 mm. The camera is installed above the windshield to ensure that the driver's facial expression data can be clearly captured without affecting the driver's normal operation.
[0085] S1-3, according to the task requirements, the IMU sampling rate is set to 100Hz and the camera sampling frequency is set to 10 frames per second. The higher IMU sampling rate can provide more detailed steering wheel motion data, while the lower sampling frequency of the camera is sufficient to capture changes in facial expressions and avoid data redundancy.
[0086] S2. Primary discrimination based on IMU data. The collected IMU data is preprocessed and feature extracted to establish a primary discrimination model, and a support vector machine (SVM) is used to perform fatigue degree regression prediction. At the same time, a fatigue degree threshold is set, and the final primary discrimination result is determined based on the fatigue prediction results of three consecutive moving windows.
[0087] S2-1, IMU data acquisition and preprocessing. Considering that the IMU sensor may be affected by noise when collecting data, such as its own internal circuit, interference from the external environment and other factors. In order to ensure the quality and accuracy of the collected data, the sliding smoothing filter algorithm is used to filter the acceleration and other signals collected by the inertial sensor. The principle of sliding smoothing filtering is to treat the obtained n consecutive sampling values as a queue, with a fixed queue length. The new data obtained by each sampling will be placed at the end of the queue, and the data at the head of the previous queue will be discarded. Then, the average value of the n data in the queue is calculated to obtain new filtered data.
[0088] S2-2, IMU feature extraction. The feature selection of IMU extraction is variance VAR, mean MEAN, maximum MAX, and square sum SSI. The calculation process is shown in the following formula.
[0089] (1)
[0090] (2)
[0091] (3)
[0092] (4)
[0093] in, x l For the l measurements, L is the number of IMU sensor measurements, and max{} is the maximum value function.
[0094] Considering the steering wheel rotation characteristics during car driving, the IMU signal data set contains the corresponding IMU data of each sample, including the acceleration ACC of the X and Y axes, and the angular velocity GYO of the Z axis, a total of 3 channels, to construct the above 3-channel feature vector matrix :
[0095] (5)
[0096] in, , representing the acceleration of the X-axis, the acceleration of the Y-axis and the angular velocity of the Z-axis respectively. VAR p A vector representing the acceleration of the X-axis, the acceleration of the Y-axis, and the variance of the angular velocity of the Z-axis; MEAN p A vector representing the mean of the acceleration of the X-axis, the acceleration of the Y-axis, and the angular velocity of the Z-axis; MAX p A vector representing the maximum value of the acceleration of the X-axis, the acceleration of the Y-axis, and the angular velocity of the Z-axis; SSI p The vector represents the sum of the acceleration of the X axis, the acceleration of the Y axis, and the square of the angular velocity of the Z axis. The feature vectors of the three channels are combined and expanded to construct a feature vector with a dimension of 12 corresponding to each joint node in each IMU signal sample. :
[0097] (6)
[0098] Among them, T ACC_x The characteristic vector composed of four features representing the acceleration of the X-axis; T ACC_y The characteristic vector composed of four features representing the acceleration of the Y axis; T GYO_x A feature vector consisting of four features representing the angular velocity of the Z axis.
[0099] In the feature extraction stage, different types of feature quantities have different dimensions and data variation ranges. The difference in feature quantity dimensions will seriously affect the results of the classifier, so the dimensions need to be unified. This study uses the 0-1 normalization method to obtain the maximum value Max and the minimum value Min of the data, and then convert the data to between 0 and 1. The calculation formula is as follows.
[0100] (7)
[0101] Where, t i is the original data x i The value result after linear normalization.
[0102] S2-3, establish a fatigue regression model. The fatigue regression model is established based on the IMU data in each window with a window size of 500ms. SVM is used as the basic model, and the normalized dimension of the feature vector described in S2-2 is 12 as input, and fatigue level (0-100%) as model output. The specific model parameter settings are as follows: the number of iterations is 300, the Gaussian kernel function is used, the model parameter C is 20.7, and the model parameter γ is 0.083.
[0103] S2-4. Set the fatigue threshold. Set the fatigue threshold to 40%, which is determined based on the results of model training and previous research experience. When the fatigue level predicted by the model exceeds this threshold, the driver will be considered to be in a fatigued state.
[0104] S2-5, fatigue prediction based on multiple moving windows. The final fatigue judgment result can be determined by statistical analysis of the fatigue prediction results in the three moving windows. Among them, the window size is 500ms and the step size is 200ms. Assume that the three moving windows are represented as T1, T2 and T3 respectively, input the data of each moving window into the fatigue degree regression model to obtain the corresponding prediction result. After that, the AND gate in the logical operation is used for combination. If the prediction results in the three windows all indicate that the driver is in a fatigue state, the final first-level judgment result is a fatigue state; otherwise, the driver is deemed to be in a non-fatigue state. The process is as follows Figure 3 shown.
[0105] S3. Secondary discrimination based on machine vision. By collecting the driver's facial image data, combined with deep learning models and data enhancement technology, the driver's state detection is realized, which is specifically divided into "relaxed closed", "talking" and "yawning".
[0106] S3-1. Yawning image data collection. When the driver is in a state of fatigue, the brain usually releases characteristic behavioral signals such as yawning to remind the human body that it is tired and needs to rest. Therefore, during the driving process, the driver's mouth has three states: relaxed and closed, talking and yawning. Data collection is carried out in multiple vehicles of different models, such as BMW X1, Volkswagen Tiguan, etc. The camera is fixed in a unified position, that is, above the windshield, to ensure that the driver's complete face can be captured. Data collection is carried out during the day and at night to consider the mouth state performance under different lighting conditions. During the data collection process, it is necessary to ensure that the driver is in a normal driving posture and position to ensure that the collected images can accurately reflect the real driver's state. At the same time, it is necessary to guide the driver to present three states of relaxed and closed, talking and yawning fatigue. Data annotation is performed on the collected images, and corresponding category labels are added to the images according to the driver's situation: "relaxed and closed", "talking" and "yawning".
[0107] S3-2, image data enhancement. Vehicle shaking may occur during driving. To simulate this situation, Gaussian noise with a variance ranging from 0.1% to 30% of the original image is introduced to enhance the robustness of the model. Assume that the pixel value of the image is , the picture obtained after applying the Gaussian noise method is:
[0108] (8)
[0109] in, is the pixel value of the image obtained after applying the Gaussian noise method, To meet the requirement of mean 0, the variance is The random noise, is the pixel value of the image.
[0110] S3-3. This study uses MobileNet V3-s as the basic model for yawn discrimination. This is a lightweight neural network structure that includes a depth-separable convolution structure, an inverted residual block, a Squeeze-and-Excitation module, and a Hard Swish activation function.
[0111] S3-4. Model training hyperparameter settings. The training input size is 300×300, the batch size is set to 32, and the training lasts for 300 epochs. The Adam optimizer is used, the initial learning rate is 0.001, and the cosine annealing scheduler is used to update the learning rate, annealing every 10 epochs.
[0112] During the collection process, the driver's mouth is in a relaxed and closed state most of the time, which leads to class imbalance in the collected data. Therefore, the traditional cross entropy loss function is replaced by the focus loss function. The calculation formula of the focus loss function is:
[0113] (9)
[0114] in, is the focal loss function, y' is the predicted probability value; γ is the adjustment parameter of the focal loss, usually taking a positive value; log() is the natural logarithm function, y is the true sample value, taking a value of 0 or 1.
[0115] S3-5. After the yawn discrimination model predicts, the driver's mouth state is used to perform a secondary fatigue discrimination. If the driver's mouth state is predicted to be "yawning", it is judged as a fatigue state, triggering the corresponding fatigue warning mechanism; if the mouth state is "relaxed and closed" or "talking", it is judged as a non-fatigue state, and normal driving monitoring continues without additional processing.
[0116] Embodiment 3
[0117] like Figure 4 As shown, this embodiment provides a non-contact fatigue detection method based on multiple sensors, including:
[0118] System integration. Carry out modular design and standardized integration of data transmission to ensure the stability and compatibility between system modules.
[0119] The first step is to decompose the system into independent functional modules, each of which is responsible for a specific task or function. Specifically, it includes: sensor data acquisition module, data preprocessing module, primary fatigue discrimination module and secondary fatigue discrimination module. Among them, the sensor data acquisition module is responsible for collecting the movement inside the vehicle and the driver's facial expression data. The data preprocessing module is responsible for processing and feature extraction of the collected raw data. The primary fatigue discrimination module makes a preliminary assessment of the degree of fatigue based on the IMU data, and uses the SVM algorithm for prediction and threshold setting to achieve rapid identification of the driver's fatigue state. The secondary fatigue discrimination module analyzes and identifies the driver's facial image based on the deep learning model to further confirm the driver's fatigue state.
[0120] In the second step, the data between the sensor data acquisition module and the data preprocessing module are transmitted in binary format through the UART communication protocol; the processed data are transmitted to the primary fatigue discrimination module and the secondary fatigue discrimination module in the JSON data structure, and the transmission network protocol is TCP / IP; the primary fatigue degree prediction result is transmitted to the secondary fatigue discrimination module in the JSON data format, and the transmission network protocol is TCP / IP.
[0121] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A non-contact fatigue detection method based on multiple sensors, characterized in that: The following steps are involved: Acquire vehicle motion data and driver facial expression image data based on IMU and visual camera; Processing the vehicle motion data and the driver's facial expression image data to obtain a motion data set and an image data set; Constructing a primary discrimination model by using a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result; Building a secondary discrimination model based on a lightweight neural network structure, inputting the image data set into the secondary discrimination model, and obtaining a secondary discrimination result; Detecting and warning the driver's fatigue state based on the primary discrimination result and the secondary discrimination result; The process of constructing a primary discrimination model by using a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result includes: SVM is used as the basic model, with a window size of 500ms, and a first-level discriminant model is established based on the IMU data in each window; Construct fatigue thresholds based on historical prediction results; Inputting the motion data set into the primary discriminant model for calculation to obtain fatigue degree prediction results within multiple moving windows; Comparing the fatigue degree prediction result in the multiple moving windows with the fatigue degree threshold to obtain the primary discrimination result; The process of constructing a secondary discrimination model based on a lightweight neural network structure, inputting the image data set into the secondary discrimination model, and obtaining a secondary discrimination result includes: Building a lightweight neural network structure based on a depth-separable convolution structure, an inverted residual block, a Squeeze-and-Excitation module, and a HardSwish activation function, and training the lightweight neural network structure to obtain the secondary discriminant model; Inputting the image data set into the secondary discrimination model to obtain the secondary discrimination result; The IMU is used to collect steering wheel motion data when the vehicle is moving, with a sampling frequency of 100 Hz; The visual camera is used to collect the driver's facial expression data, with a sampling frequency of 10 frames per second; The steering wheel motion data when the vehicle is moving includes the acceleration of the X-axis, the acceleration of the Y-axis and the angular velocity of the Z-axis, and the acceleration of the X-axis, the acceleration of the Y-axis and the angular velocity of the Z-axis are used to construct a 3-channel feature vector matrix; The 3-channel feature vector matrix is merged and expanded to construct a feature vector with a dimension of 12 corresponding to each joint node in each IMU signal sample; The process of processing the vehicle motion data and the driver's facial expression image data to obtain a motion data set and an image data set includes: Processing the vehicle motion data based on a sliding smoothing filter algorithm to obtain filtered data, and then performing feature extraction on the filtered data to obtain the motion data set; Annotating the driver's facial expression image data, and then performing data enhancement on the annotated image data based on a Gaussian noise method to obtain the image data set; The process of extracting features from the filtered data includes: Calculate the variance, mean, maximum value, and sum of squares (SSI) in the filtered data; Construct a 3-channel feature vector matrix based on the variance, mean, maximum value, square sum SSI, acceleration of the X and Y axes and angular velocity of the Z axis in the filtered data; The 3-channel feature vector matrix is merged and expanded to obtain a feature vector with a dimension of 12 corresponding to each joint node in the IMU signal sample; Normalizing the feature vector corresponding to the dimension of 12 to obtain the motion data set; The primary discrimination model quickly predicts the driver's fatigue level based on the steering wheel movement data when the vehicle is moving; The secondary discrimination model is based on machine vision technology to further confirm the driver's fatigue state; The secondary discrimination model is performed when necessary to implement a hierarchical discrimination strategy.
2. The multi-sensor based non-contact fatigue detection method according to claim 1, characterized in that: The loss function used by the lightweight neural network structure is the focus loss function, and the expression of the focus loss function is: ; in, is the focal loss function, y' is the predicted probability value; γ is the adjustment parameter of the focal loss, and log() is the natural logarithm function.
3. The detection system of the multi-sensor based non-contact fatigue detection method according to any one of claims 1 to 2, characterized in that: include: Sensor data acquisition module, used to obtain vehicle motion data and driver facial expression image data based on IMU and visual camera; A data preprocessing module, used for processing the vehicle motion data and the driver's facial expression image data to obtain a motion data set and an image data set; A primary fatigue discrimination module, used for constructing a primary discrimination model through a support vector machine, inputting the motion data set into the primary discrimination model, and obtaining a primary discrimination result; The secondary fatigue discrimination module is used to construct a secondary discrimination model based on a lightweight neural network structure, input the image data set into the secondary discrimination model, and obtain a secondary discrimination result.
Citation Information
Patent Citations
Driving fatigue detection method and system based on lightweight neural network image enhancement
CN117789181A
Driver driving state monitoring system
CN117958742A
Fatigue detection signal acquisition device based on multi-modal data fusion
CN117958839A
Driver fatigue level recognition system based on bimodal feature fusion
CN110781872A
Fatigue driving stopping method based on visual features, steering wheel operation detection and active intervention
CN114973212A
