Real-time detection method and system for fatigue driving based on monitoring camera

Abstract

The application relates to the technical field of automobile safety detection, and discloses a fatigue driving real-time detection method and system based on a monitoring camera. The method collects multi-source sensing data streams of a driver's face region, generates a synchronized physiological signal set through time sequence alignment processing; multi-modal feature extraction is performed on the physiological signal set, so that an eye movement trajectory feature vector, a head posture change sequence and a facial muscle activity intensity atlas are obtained; an eye movement fatigue index and a head behavior abnormality degree score are respectively generated, a comprehensive fatigue grade index is obtained by combining a weighted decision algorithm; a grading early warning trigger strategy is generated according to the comprehensive fatigue grade index and real-time driving environment parameters, a corresponding alarm instruction is called to generate a multi-level warning signal output stream; the physiological state change trend of the driver is monitored in real time, the dynamic priority of the warning signal output stream is adjusted, the alarm instruction set is updated, and fatigue driving intervention closed-loop control is completed.

Description

Technical Field

[0001] This invention relates to the field of automotive safety detection technology, specifically to a method and system for real-time detection of driver fatigue based on a surveillance camera. Background Technology

[0002] With the rapid development of the road transportation industry, driving safety has gradually become a key issue of social concern, among which fatigue driving is one of the important factors causing traffic accidents. During long-term driving, drivers are prone to inattention and decreased reaction speed. If these conditions are not identified and addressed in time, they can easily lead to serious personal injury and property damage. Currently, various fatigue driving detection technologies have emerged in the industry. Some technologies rely on drivers wearing special devices to collect physiological signals, such as heart rate monitoring bracelets and electroencephalogram (EEG) sensors. However, these devices require drivers to actively wear them, and in actual driving scenarios, they suffer from poor wearing comfort and are easily affected by driving movements, resulting in insufficient data collection stability and difficulty in achieving continuous and effective real-time detection.

[0003] Other technologies rely on indirect judgments based on vehicle operating parameters, such as analyzing steering wheel rotation frequency and vehicle trajectory deviation to infer driver fatigue. However, these methods are greatly affected by external factors such as road conditions and driving habits. When the vehicle is driving on a winding road or encountering traffic congestion, misjudgments or omissions are likely to occur, making it impossible to accurately reflect the driver's true physiological state.

[0004] Camera-based visual detection technology has become a research hotspot in the field of fatigue driving detection due to its advantages such as no contact with the driver and convenient installation. However, existing visual detection technologies mostly use single-modal data for analysis. For example, they only use visible light cameras to collect facial images and extract eye or head features for fatigue judgment. In poor lighting conditions, such as at night, in tunnels, or in rainy weather, image quality deteriorates significantly, leading to reduced feature extraction accuracy and affecting the reliability of the detection results. Furthermore, existing technologies lack effective fusion and synchronous processing of multi-source sensor data during data processing. Discrepancies in the time series of different data types result in a lack of a unified data foundation for subsequent feature analysis and fatigue judgment, making it difficult to comprehensively and accurately assess the driver's fatigue level. In addition, existing warning mechanisms mostly use fixed alarm modes and cannot adjust warning strategies according to the driver's real-time physiological changes and driving environment, resulting in limited warning effectiveness and difficulty in effectively waking up fatigued drivers. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for real-time detection of driver fatigue based on a surveillance camera, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a real-time fatigue driving detection method based on a surveillance camera, the method comprising:

[0007] Multi-source sensor data streams of the driver's facial region are collected, and time-series alignment processing is performed on the multi-source sensor data streams to generate a synchronized set of physiological signals.

[0008] Multimodal feature extraction was performed on the synchronized physiological signal set to obtain eye movement trajectory feature vectors, head posture change sequences, and facial muscle activity intensity maps;

[0009] An eye movement fatigue index is generated by dynamically comparing and analyzing the feature vector of eye movement trajectory with preset fatigue threshold parameters.

[0010] By integrating head posture change sequences with facial muscle activity intensity maps, a head behavior abnormality score is calculated using a neural network model.

[0011] A weighted decision algorithm was used to generate a comprehensive fatigue level index by combining the eye movement fatigue index and the head behavior abnormality score.

[0012] Based on the comprehensive fatigue level index and real-time driving environment parameters, a graded early warning triggering strategy is generated.

[0013] Based on the tiered early warning triggering strategy, the corresponding alarm instruction set is invoked to generate a multi-level warning signal output stream;

[0014] Real-time monitoring of the driver's physiological state changes and dynamic priority adjustment of multi-level warning signal output streams;

[0015] The alarm instruction set is updated based on the dynamic priority adjustment results to complete the closed-loop control for fatigue driving intervention.

[0016] Preferably, the step of acquiring multi-source sensor data streams of the driver's facial region and performing time-series alignment processing on the multi-source sensor data streams to generate a synchronized physiological signal set includes:

[0017] The driver's facial thermal imaging data stream is acquired through an infrared camera, while a visible light video stream of the face is acquired through a visible light camera.

[0018] The temperature distribution matrix sequence in the thermal imaging data stream and the grayscale frame sequence in the visible light video stream were extracted respectively.

[0019] Timestamp matching is performed on the temperature distribution matrix sequence and the grayscale frame sequence to establish a cross-modal data synchronization index table;

[0020] Based on the cross-modal data synchronization index table, the temperature distribution matrix sequence and grayscale frame sequence are resampled to generate time-aligned infrared signal sequences and visible light signal sequences.

[0021] Time-aligned infrared signal sequences and visible light signal sequences are merged into a synchronized set of physiological signals.

[0022] Preferably, the step of performing multimodal feature extraction on the synchronized physiological signal set to obtain eye movement trajectory feature vectors, head posture change sequences, and facial muscle activity intensity maps includes:

[0023] Extract eye region image subsequences from visible light signal sequences and generate a continuous set of eye movement trajectory coordinates using a pupil tracking algorithm;

[0024] Velocity and acceleration are calculated on the coordinate set of eye movement trajectories to generate an eye movement trajectory feature vector containing blink frequency, eye closure duration, and pupil displacement.

[0025] Based on infrared signal sequences, temperature change data of key facial points are extracted, and facial muscle activity intensity maps are calculated using a thermodynamic model.

[0026] Simultaneously, the three-dimensional coordinates of head feature points are identified from the visible light signal sequence, and a head posture change sequence is generated through a posture calculation algorithm.

[0027] Preferably, the step of generating an eye movement fatigue index by dynamically comparing and analyzing the feature vector of eye movement trajectory with a preset fatigue threshold parameter includes:

[0028] Obtain a preset set of fatigue threshold parameters, including minimum blink frequency threshold, maximum eye closure duration threshold, and pupil displacement tolerance value;

[0029] The blink frequency in the eye movement trajectory feature vector is compared with the minimum blink frequency threshold to generate a frequency deviation coefficient;

[0030] The duration of eye closure in the eye movement trajectory feature vector is compared with the maximum duration threshold of eye closure to generate a duration exceedance coefficient;

[0031] The pupil displacement in the eye movement trajectory feature vector is compared with the pupil displacement tolerance value to generate a displacement anomaly coefficient.

[0032] An eye movement fatigue index is generated by fusing the frequency deviation coefficient, duration exceedance coefficient, and displacement anomaly coefficient using a multiple regression model.

[0033] Preferably, the fusion of head posture change sequence and facial muscle activity intensity map, and the calculation of head behavior abnormality score through a neural network model, includes:

[0034] The head posture change sequence is processed by Fourier transform to extract the frequency domain feature vector of head motion;

[0035] The facial muscle activity intensity map was processed by region segmentation to extract sub-maps of muscle activity intensity in the forehead, cheek, and jaw.

[0036] The head motion frequency domain feature vector and muscle activity intensity sub-map are input into a pre-trained convolutional neural network model;

[0037] The abnormality score of head behavior is output through a multi-layer feature fusion layer of a convolutional neural network model.

[0038] Preferably, the step of combining the eye movement fatigue index and the head behavior abnormality score, and using a weighted decision algorithm to generate a comprehensive fatigue level index, includes:

[0039] Obtain current driving environment parameters, including vehicle speed, road type identification, and time period code;

[0040] The weighting coefficient of the eye movement fatigue index is adjusted according to the vehicle speed value, and the weighting coefficient of the head behavior abnormality score is adjusted according to the road type sign.

[0041] The weighted average of the eye movement fatigue index and the head behavior abnormality score was calculated using a weighted summation algorithm.

[0042] Based on the time period coding, the weighted average value is compensated for time decay to generate a comprehensive fatigue level index.

[0043] Preferably, the step of generating a graded early warning triggering strategy based on the comprehensive fatigue level index and real-time driving environment parameters includes:

[0044] Set multiple fatigue threshold ranges, including mild fatigue threshold range, moderate fatigue threshold range, and severe fatigue threshold range;

[0045] The comprehensive fatigue level index is matched with the multi-level fatigue threshold range to determine the current fatigue level identifier.

[0046] The current fatigue level indicator is adjusted based on real-time driving environment parameters such as visibility, lane departure frequency, and steering wheel grip strength.

[0047] A graded early warning triggering strategy is generated based on the adjusted current fatigue level identifier.

[0048] Preferably, the step of invoking the corresponding alarm instruction set according to the graded early warning triggering strategy to generate a multi-level warning signal output stream includes:

[0049] Based on the warning level code in the graded warning triggering strategy, retrieve the corresponding acoustic alarm parameters, optical alarm parameters, and tactile alarm parameters from the alarm instruction library;

[0050] Generate acoustic alarm waveform data, optical flashing pattern data, and tactile vibration sequence data;

[0051] The acoustic alarm waveform data, optical flashing pattern data, and tactile vibration sequence data are combined into a multi-level warning signal output stream.

[0052] Preferably, the real-time monitoring of the driver's physiological state changes and the dynamic priority adjustment of the multi-level warning signal output streams include:

[0053] Continuously collect facial expression response data and body movement response data of drivers;

[0054] Analyze the trends in pupil constriction rate and facial muscle tension in facial expression response data;

[0055] Analyze the trends in steering wheel operation force and pedal depressing frequency in the limb movement response data;

[0056] Adjust the priority order of acoustic, optical, and tactile alarms in the multi-level warning signal output stream according to the changing trend.

[0057] Preferably, the method further includes a real-time fatigue driving detection system based on a surveillance camera, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the above-described real-time fatigue driving detection method based on a surveillance camera.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] This real-time fatigue driving detection method based on surveillance cameras collects multi-source sensor data streams from the driver's facial region and performs time-series alignment processing to generate a synchronized set of physiological signals. This solves the problems of asynchronous time and inconsistent data foundations in existing technologies, providing unified and reliable data support for subsequent multimodal feature extraction and analysis. The introduction of multi-source sensor data, combining infrared and visible light modalities, enables stable acquisition of driver facial information under various lighting conditions. This avoids the problem of insufficient acquisition accuracy caused by the influence of ambient light on single-modal data. Facial physiological feature data can be effectively acquired at night, in tunnels, or in rainy weather, ensuring the continuity and integrity of data acquisition.

[0060] In the feature extraction stage, multimodal feature extraction operations are used to obtain eye movement trajectory feature vectors, head posture change sequences, and facial muscle activity intensity maps. This captures the driver's physiological state features from multiple dimensions, including the eyes, head, and facial muscles. Compared with existing technologies that only extract features from a single part, this method can more comprehensively reflect the driver's fatigue-related information, reduce the detection bias caused by single features, and make subsequent fatigue judgment more comprehensive and accurate.

[0061] An eye movement fatigue index is generated by dynamically comparing and analyzing the feature vector of eye movement trajectory with preset fatigue threshold parameters. At the same time, a head behavior abnormality score is calculated by integrating head posture change sequence and facial muscle activity intensity map through a neural network model. The two assessment indicators reflect the fatigue state from different physiological feature dimensions. Then, a weighted decision algorithm is combined to generate a comprehensive fatigue level index, which can comprehensively consider the fatigue performance of different physiological parts of the driver, avoid the limitations of single indicator assessment, make the fatigue level assessment more in line with the driver's true fatigue level, and reduce the occurrence of misjudgment and omission.

[0062] By combining comprehensive fatigue level indicators with real-time driving environment parameters to generate a tiered warning trigger strategy, the warning standards can be adjusted according to actual driving scenarios. For example, in high-speed driving or complex road environments, the warning trigger conditions can be adjusted accordingly, making the warning strategy more in line with actual driving needs. Compared with existing fixed warning modes, it is more flexible and targeted. Based on the tiered warning trigger strategy, the corresponding alarm instruction set is invoked to generate a multi-level warning signal output stream. Through different levels and types of warning signals, warning measures of appropriate intensity can be taken according to differences in fatigue levels, avoiding the problem of warning signals being too strong and causing driver panic, or too weak and failing to wake the driver up.

[0063] This system monitors the driver's physiological state in real time and dynamically adjusts the priority of multi-level warning signal output streams. It can optimize the output method of warning signals based on the driver's response. For example, when the driver's response to acoustic alarms is weak, the priority of optical or tactile alarms can be increased to ensure that the warning signals are effectively transmitted to the driver, improving the warning effect. Based on the dynamic priority adjustment results, the system updates the alarm instruction set, completing a closed-loop control for fatigue driving intervention. This forms a complete process from data collection, feature analysis, fatigue assessment, graded warnings to dynamic adjustment, enabling continuous monitoring and effective intervention for fatigue driving, effectively reducing the risk of traffic accidents caused by fatigue driving, and ensuring road traffic safety. Attached Figure Description

[0064] Figure 1 This is a schematic diagram illustrating the working principle of the real-time fatigue driving detection method based on a surveillance camera as described in this invention.

[0065] Figure 2A schematic diagram illustrating the working principle of a synchronized set of physiological signals;

[0066] Figure 3 A diagram illustrating the working principle of generating the eye movement fatigue index. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Please see Figure 1 This invention provides a method and system for real-time detection of driver fatigue based on a surveillance camera. The method includes acquiring multi-source sensor data streams of the driver's facial area using a surveillance camera, and performing time-series alignment processing on these data streams to generate a synchronized physiological signal set. The multi-source sensor data streams include information from different sensors, such as infrared and visible light cameras, and data synchronization is ensured through timestamp matching and resampling. The synchronized physiological signal set provides a unified time reference for subsequent feature extraction. Multimodal feature extraction is performed on the synchronized physiological signal set to obtain eye movement trajectory feature vectors, head posture change sequences, and facial muscle activity intensity maps. The feature extraction process involves pupil tracking, posture calculation, and thermodynamic model calculation to capture fatigue-related physiological indicators. An eye movement fatigue index is generated by dynamically comparing the eye movement trajectory feature vectors with preset fatigue threshold parameters. This index is obtained by fusing multiple eye movement parameters through a multivariate regression model. The head posture change sequence and facial muscle activity intensity map are fused, and a head behavior anomaly score is calculated using a neural network model. The neural network uses convolutional and fusion layers to process frequency and spatial features. Combining eye movement fatigue index and head behavior abnormality score, a weighted decision algorithm is used to generate a comprehensive fatigue level index. The weighted decision takes into account real-time driving environment parameters such as vehicle speed and road type. Based on the comprehensive fatigue level index and real-time driving environment parameters, a tiered warning triggering strategy is generated. The strategy determines the warning level based on multi-level fatigue threshold ranges and environmental adjustments. According to the tiered warning triggering strategy, the corresponding alarm command set is invoked, generating a multi-level warning signal output stream, including acoustic, optical, and tactile alarm data. The driver's physiological state changes are monitored in real time, and the multi-level warning signal output stream is dynamically prioritized. The alarm sequence is optimized by analyzing facial expression and body movement response data. Based on the dynamic priority adjustment results, the alarm command set is updated to complete the closed-loop control for fatigue driving intervention, ensuring the system adapts to changes in driver state.

[0069] Example 1: See Figure 2The acquisition and synchronization of the driver's facial multi-source sensor data stream, as well as subsequent multimodal feature extraction, begins with the simultaneous acquisition of data by an infrared camera and a visible light camera installed in the cockpit. Both the infrared and visible light cameras are intelligent sensors with integrated environmental perception capabilities. These intelligent sensors can detect the light intensity, humidity, and dust concentration in the cockpit in real time. When a sudden drop in light intensity is detected (such as when entering a tunnel), the infrared camera automatically increases the signal-to-noise ratio of the thermal imaging signal, and the visible light camera simultaneously activates its night vision enhancement mode. If excessive dust concentration is detected, potentially affecting lens clarity, the sensor will also trigger a built-in lens cleaning reminder mechanism to ensure that the multi-source sensor data stream maintains stable acquisition quality, providing a reliable data foundation for subsequent time-series alignment processing. The infrared camera captures facial thermal radiation distribution at a rate of 30 frames per second, forming a thermal imaging data stream, while the visible light camera acquires a high-resolution facial video stream at the same frame rate, forming a visible light video stream. The two data streams are initially aligned through a hardware synchronization signal, with each data frame marked with a timestamp accurate to the millisecond level, providing a time reference for subsequent processing. Each frame in the thermal imaging data stream is parsed into a temperature distribution matrix, with each element in the matrix corresponding to a temperature measurement value of a specific area of ​​the face. The visible light video stream is then converted into a continuous grayscale frame sequence by a decoder. Grayscale processing reduces the data dimensionality while retaining the necessary texture information.

[0070] After extracting the temperature distribution matrix sequence and the grayscale frame sequence, the system performs timestamp matching processing. This processing establishes a cross-modal data synchronization index table by comparing the timestamp differences between the two data streams. The index table is stored using a hash table structure, with the key being the unified timestamp and the value being the position index of the corresponding frame in its respective stream. The matching algorithm uses a nearest neighbor interpolation strategy, automatically selecting the closest frame for association when there is a slight deviation in the timestamps. Based on the cross-modal data synchronization index table, the temperature distribution matrix sequence and the grayscale frame sequence are resampled. The resampling algorithm uses a linear interpolation method to convert the non-uniformly sampled data into a signal sequence with a fixed time interval. The resampling of the infrared signal sequence is based on the spatiotemporal continuity of the temperature matrix, and missing frames are filled using bilinear interpolation. For the visible light signal sequence, pixel-level temporal interpolation is performed on the grayscale frames, ultimately generating infrared and visible light signal sequences that are perfectly aligned on the time axis. The aligned bimodal sequences were merged into a synchronized physiological signal set, which was stored in a circular buffer structure. Each time point contained a paired temperature matrix and grayscale image. The buffer size was set to a sliding window lasting 5 seconds to ensure data continuity during real-time processing.

[0071] When performing multimodal feature extraction on synchronized physiological signal sets, the system extracts eye region image subsequences from visible light signal sequences. The eye region is located using a Haar cascade classifier trained on a large number of eye images from driving scenarios, enabling it to adapt to head rotation and lighting changes. The extracted eye image subsequences are input into a pupil tracking algorithm. This algorithm, based on an improved ellipse fitting model, locates the pupil center coordinates through edge detection and Hough transform, generating a continuous set of eye movement trajectory coordinates. The coordinate set is stored as a two-dimensional point sequence, with each point containing the pupil's horizontal and vertical coordinates in the image coordinate system. The sequence length is consistent with the time window. Velocity and acceleration are calculated from the coordinate set of eye movement trajectories. The velocity value is obtained by the ratio of the difference between the coordinates of consecutive frames to the time interval, while the acceleration value is calculated based on the quadratic difference of the velocity sequence. The final generated eye movement trajectory feature vector contains three types of parameters: blink frequency is calculated by detecting the periodic signal of the eyelid covering the pupil, and the peak detection algorithm is used to count the number of blinks per unit time; eye closure duration is determined by the pupil visibility status of consecutive frames, and when the pupil is covered for more than a threshold number of frames, it is recorded as an eye closure event; pupil displacement is calculated by the standard deviation of the trajectory coordinates, reflecting the average movement amplitude of the pupil per unit time.

[0072] The process of extracting facial muscle activity intensity maps based on infrared signal sequences requires locating key facial points, including anatomical landmarks such as the glabella, root of the nose, and corners of the mouth. The localization algorithm combines texture features from the grayscale signal sequence with spatial gradient information of temperature distribution. The extracted key point temperature change data is input into a thermodynamic model, which is built upon biological heat transfer equations. This model attributes changes in skin surface temperature to heat changes generated by subcutaneous muscle contraction and calculates muscle activity intensity by solving the inverse problem of partial differential equations. The calculated intensity values ​​are mapped into a grayscale image of facial muscle activity intensity, where the grayscale value of each pixel corresponds to the real-time activity level of the underlying muscle; higher grayscale values ​​indicate greater muscle contraction intensity. This paper describes a visual acquisition unit for extracting the 3D coordinates of head feature points from a visible light signal sequence. Equipped with a smart sensor featuring dynamic target tracking, this sensor can lock onto the driver's head area in real time. Even with slight head movements or forward leaning, the sensor uses a built-in motion prediction algorithm to continuously and accurately locate feature points such as the inner and outer corners of the eyes and the tip of the nose, preventing feature point loss due to head movement and ensuring continuous generation of head posture change sequences. This provides complete time-series data for subsequent calculation of head behavior anomaly scores. Stable features such as the inner and outer corners of the eyes, the tip of the nose, and the corners of the mouth are selected as feature points. The 3D coordinates are recovered using stereo vision principles. When using only a monocular camera, the PnP algorithm is used to estimate the 3D posture from a 2D projection. The posture calculation algorithm describes head movement based on rotation matrices and translation vectors, avoiding gimbal lock issues through quaternion representation. The generated head posture change sequence includes time-series data for pitch, yaw, and roll angles, with the sequence sampling rate consistent with the original video stream.

[0073] The data flow in the multimodal feature extraction process adopts a parallel processing architecture. Feature extraction of visible light signal sequences and infrared signal sequences is performed synchronously in two independent processing threads, with the threads exchanging synchronization timestamp information through shared memory. The calculation of eye movement trajectory feature vectors uses a sliding window mechanism with a window width of 3 seconds and a step size of 1 second to achieve continuous real-time updates. The solution of head posture change sequences is processed using a Kalman filter to reduce the impact of instantaneous jitter on posture estimation. The generation of facial muscle activity intensity maps uses a multi-scale analysis method to calculate intensity values ​​at different spatial resolutions and finally fuse them into a unified map. All feature data are normalized before output to eliminate scale differences caused by different units. The normalization method uses a min-max scaling technique to map feature values ​​to the [0,1] interval. The output of the feature extraction module is a time-aligned multimodal feature stream, which is passed to subsequent analysis modules through a message queue. The queue mechanism ensures that no data frames are lost under high load. The entire processing chain adopts a pipeline design, with a timeout protection mechanism set at each processing stage. When a certain stage times out, the current frame is automatically skipped and an exception log is recorded to ensure the real-time requirements of the system.

[0074] Example 2: See Figure 3 The generation of the eye movement fatigue index begins with retrieving a preset set of fatigue threshold parameters from the system configuration. This set includes a minimum blink frequency threshold, a maximum eye closure duration threshold, and a pupil displacement tolerance value. These threshold parameters are derived statistically from a large amount of driving behavior research data. Reasonable limits are set by analyzing the distribution range of eye movement patterns under normal driving conditions. The minimum blink frequency threshold is typically set as the lower limit of blinks per minute to detect abnormalities such as reduced blinking. The maximum eye closure duration threshold defines the maximum allowable duration of a single eye closure event. The pupil displacement tolerance value specifies the normal fluctuation range of pupil movement distance per unit time. These threshold parameters are stored numerically in the system database and support dynamic adjustments based on individual driver differences. For example, baseline eye movement data of the driver can be collected during the initial calibration phase to fine-tune the thresholds, enhancing personalized adaptability.

[0075] After obtaining the threshold parameters, the system dynamically compares and analyzes the real-time calculated eye movement trajectory feature vector with these thresholds. The eye movement trajectory feature vector contains three core parameters: blink frequency, eye closure duration, and pupil displacement. These parameters are derived from a continuous set of eye movement trajectory coordinates. The comparison analysis compares the blink frequency with the minimum blink frequency threshold to generate a frequency deviation coefficient. The comparison process uses a relative deviation calculation method, dividing the difference between the actual blink frequency and the threshold by the threshold to obtain a standardized deviation value. When the actual frequency is lower than the threshold, the deviation is positive; otherwise, it is negative. However, the absolute value is usually taken because fatigue is mainly manifested as reduced blinking. Similarly, the comparison of eye closure duration with the maximum eye closure duration threshold generates a duration exceedance coefficient. The coefficient is calculated based on the proportion of eye closure duration exceeding the threshold, using a piecewise linear function mapping. When the eye closure duration is within the threshold, the coefficient is zero; after exceeding the threshold, it increases linearly with the increase in the duration of exceedance. The displacement coefficient is generated by comparing the pupil displacement amount with the pupil displacement tolerance value. This coefficient is obtained by calculating the ratio of the fluctuation amplitude of the pupil movement trajectory to the tolerance value, and the pupil instability is assessed by combining the randomness of the movement direction.

[0076] After generating three coefficients, the system uses a multiple regression model to perform a fusion calculation to generate an eye movement fatigue index. The multiple regression model adopts a linear regression form, with independent variables being the frequency deviation coefficient, duration exceedance coefficient, and displacement anomaly coefficient, and the dependent variable being the eye movement fatigue index. Model parameters are obtained through offline training, using a driving experiment dataset labeled with fatigue states. The least squares method is used to fit the linear relationship between the coefficients and the degree of fatigue. In practical applications, the model directly outputs a continuous eye movement fatigue index after inputting the three coefficient values. The index value is normalized to between 0 and 1, with higher values ​​indicating more severe fatigue. To adapt to different driving scenarios, the model also incorporates an environmental correction factor, such as temporarily adjusting the coefficient weights at tunnel exits where lighting changes drastically, to reduce misjudgments.

[0077] The entire comparative analysis process runs continuously in a sliding window mode, with a window width of 5 seconds and a step size of 1 second, ensuring real-time performance while smoothing out instantaneous fluctuations. The calculation of the frequency deviation coefficient also incorporates historical trend analysis, enhancing the sensitivity to progressive fatigue detection by comparing the blink frequency change rate between the current and previous windows. The generation of the duration exceedance coefficient distinguishes between conscious eye closure (such as intentional blinking) and fatigue-related eye closure. The algorithm filters by analyzing eye movement patterns before and after eye closure, counting only events accompanied by upward eye movement or stagnation as valid events. The calculation of the displacement anomaly coefficient employs a multi-scale analysis method, assessing micro-tremors within a short time window and detecting macro-drift within a long time window, comprehensively reflecting pupil movement abnormalities. The implementation of the multiple regression model includes an online update mechanism. The system periodically collects new eye movement data and driver feedback, adjusting model parameters through incremental learning algorithms to adapt to population drift or individual habit changes. The eye movement fatigue index output by the model is also smoothed in the time domain, using an exponentially weighted moving average method to eliminate short-term noise and highlight long-term trends. The index value is ultimately output along with the confidence index. The confidence index is calculated based on the quality and completeness of the input coefficients. When data is missing or noise is excessive, the index weight is reduced to avoid unreliable decisions. The system also includes an outlier detection module. When a coefficient suddenly changes drastically, a review process is triggered, for example, by checking the consistency of adjacent frames to verify the validity of the data.

[0078] Example 3: The calculation process for the head behavior anomaly score begins with Fourier transform processing of the head posture change sequence to extract frequency domain features. The head posture change sequence originates from the three-dimensional coordinate data of head feature points in the visible light signal sequence, typically including time series of pitch, yaw, and roll angles, with the sampling rate consistent with the video stream frame rate. The Fourier transform is implemented using the Fast Fourier Transform algorithm, converting the time-domain posture sequence into a frequency domain representation to capture the periodic features of head movement, such as the frequency components of nodding or shaking. During the transform process, the Hanning window function is used to reduce spectral leakage and improve frequency resolution. The extraction of frequency domain feature vectors focuses on specific frequency bands, such as the 0.1Hz to 2Hz range, which is closely related to fatigue-related head micro-movements. The feature vectors include the amplitude and phase information of the main frequency components. The facial muscle activity intensity map is segmented into regions based on facial anatomy, dividing the map into forehead, cheek, and mandibular sub-regions. The forehead region corresponds to the activity of the corrugator supercilii and frontalis muscles, the cheek region involves the risorius and zygomaticus major muscles, and the mandibular region includes the masseter and mentalis muscles. The muscle activity intensity sub-maps for each sub-region are generated by statistical analysis of pixel-level grayscale values. The size of the sub-maps is normalized to a uniform resolution to facilitate subsequent processing. The segmentation algorithm uses an adaptive thresholding method, combined with the spatial gradient information of thermal imaging data, to accurately define the region boundaries.

[0079] The head motion frequency domain feature vector and muscle activity intensity sub-map are input into a pre-trained convolutional neural network model. The neural network model employs a multi-input architecture, with one branch processing the frequency domain feature vector and the other processing the image-based sub-map. The frequency domain feature vector is dimensionality-reduced through fully connected layers, transforming it into a feature map. The image-based sub-map extracts spatial features through convolutional and pooling layers. The features from the two branches are fused in an intermediate layer. The fusion layer uses connection operations to merge the frequency domain features and image features, and further abstracts them through multiple convolutional layers, outputting a head behavior anomaly score. The neural network model is trained using a large amount of labeled data, with annotations based on expert assessments of the degree of head behavior anomaly. The loss function is mean squared error loss, and the optimizer uses the Adam algorithm. Data augmentation techniques are introduced during training to improve generalization ability.

[0080] In Fourier transform processing, the head pose change sequence is represented as a discrete-time signal, let the sequence be... ,in For time indexing, the value range is from 0 to... , Given the sequence length, the frequency domain representation is obtained after Fourier transform. ,in For frequency indexing. The transformation formula is:

[0081]

[0082] in: Indicates time index The head pose value at that location can be either angle or coordinate data; Indicates frequency index The complex frequency domain coefficients at a given point represent the intensity of that frequency component. The imaginary unit; This is the total length of the sequence; It is a natural constant. The frequency domain eigenvectors are derived from... Extract the amplitude spectrum and take the first... The main frequency components form a vector.

[0083] The facial muscle activity intensity maps for region segmentation are derived from the thermodynamic model output of the infrared signal sequence. The map size is normalized to 256x256 pixels. During segmentation, key facial points, such as the glabella and corners of the mouth, are located, and regions of interest are defined based on the connections between these key points. The forehead sub-map covers the area above the eyebrows, the cheek sub-map includes the area around the cheekbone, and the mandibular sub-map focuses on the chin area. Each sub-map is cropped to 64x64 pixels and its grayscale value is normalized to the 0-1 range. The calculation of the muscle activity intensity sub-maps is based on the rate of temperature change, and the activity region is enhanced through differential calculation between adjacent frames. The sub-map generation is performed in real time, maintaining time synchronization with the head pose sequence. The convolutional neural network model consists of five convolutional layers and three fully connected layers. The frequency domain feature vectors are reconstructed into 1D feature maps, which are input to a one-dimensional convolutional layer with a kernel size of 3. The image-based sub-maps are input to a two-dimensional convolutional layer with a kernel size of 5x5. The feature fusion layer flattens and concatenates one-dimensional and two-dimensional feature maps, introducing non-linearity through the ReLU activation function, and a Dropout layer prevents overfitting. The model output is a scalar score, ranging from 0 to 1, normalized by the sigmoid function; a higher score indicates a more severe abnormality in head behavior. During inference, the model processes data using a sliding window approach with a window length of 5 seconds and a step size of 1 second to ensure real-time performance. Fourier transform is accelerated using the FFT library, region segmentation is optimized using integral images, and neural network inference is computed in parallel using GPUs. Model parameters are updated periodically, adapting to new drivers through online learning. The anomaly score, along with a confidence index, is passed to the decision module. The confidence score is calculated based on input data quality, such as the signal-to-noise ratio and the proportion of missing frames.

[0084] Example 4: The generation of comprehensive fatigue level indicators and the formulation of graded early warning triggering strategies begin with the system acquiring current driving environment parameters, including vehicle speed values ​​read in real time from onboard sensors, road type identifiers parsed from GPS map data, and time period codes determined according to system time. Vehicle speed values ​​are continuously updated in kilometers per hour, road type identifiers are classified into categories such as highways, urban roads, and rural roads, and time period codes distinguish between different time periods such as daytime, dusk, and nighttime to reflect the influence of light and biological clock. In a specific example, assuming a car is traveling at 90 kilometers per hour on a highway, and the current time is 2:00 AM, the system labels the time period as late night. These parameters are encapsulated into a data structure and passed to the weight adjustment module.

[0085] The weighting coefficient of the eye movement fatigue index is adjusted based on vehicle speed. The adjustment logic is based on a preset mapping relationship within the speed range. For example, when the speed is higher than 80 km / h, the weight of the eye movement index increases from the baseline value of 0.5 to 0.7 because visual fatigue has a more significant impact on safety during high-speed driving. The weighting coefficient is stored in a dynamic configuration table, supporting real-time querying and updating. Simultaneously, the weighting coefficient of the head behavior anomaly score is adjusted based on road type identification. For instance, in highway environments, the head behavior weight decreases from 0.5 to 0.3 because head posture changes are smaller during straight-line driving, while in urban roads, the weight is increased to 0.7 to capture fatigue signs caused by frequent turning. Weighting adjustments are achieved by looking up a predefined rule table, which is based on driving scenario risk assessment. The weighted average of the eye movement fatigue index and the head behavior anomaly score is calculated using a weighted summation algorithm. The algorithm formula is as follows: The eye movement fatigue index and head behavior abnormality score have been normalized to the range of 0-1, and the weighted average is output as a preliminary fatigue indicator. Subsequently, time decay compensation is applied to the weighted average according to the time period code. For example, a decay factor of 0.9 is introduced for late night time. The weighted average is multiplied by this factor to amplify the fatigue effect. The compensated value is mapped to a comprehensive fatigue level index with a value range of 0 to 100, divided into discrete levels such as normal, mild fatigue, moderate fatigue, and severe fatigue.

[0086] After generating the comprehensive fatigue level index, the system immediately enters the generation stage of the graded early warning trigger strategy. Referring to Table 1, multiple fatigue threshold ranges are set. These ranges are based on historical driving data statistics. For example, normal state corresponds to 0-30, mild fatigue corresponds to 31-60, moderate fatigue corresponds to 61-80, and severe fatigue corresponds to 81-100. The threshold ranges are stored in a configuration table for real-time querying.

[0087] Table 1: Multi-level Fatigue Threshold Intervals

[0088]

[0089] The comprehensive fatigue level index is matched with multi-level fatigue threshold intervals to determine the current fatigue level label. The matching algorithm compares which interval the index value falls into; for example, a comprehensive fatigue level index of 75 matches a moderate fatigue level. The current fatigue level label is adjusted based on real-time driving environment parameters such as visibility, lane departure frequency, and steering wheel grip strength. Visibility is obtained from a weather sensor, measured in meters; when visibility is below 100 meters, the fatigue level is increased by one level. Lane departure frequency is counted by the lane recognition system; if it exceeds 3 times per minute, the level is increased. Steering wheel grip strength is monitored by a torque sensor. The sensor collecting the steering wheel grip strength value is an intelligent sensor with data self-calibration capabilities. This intelligent sensor can periodically compare with preset standard grip strength benchmarks and automatically correct measurement deviations caused by prolonged use. It can also distinguish between "normal driving grip strength" and "instantaneous grip strength fluctuations caused by bumps and steering," retaining valid grip strength data through a built-in filtering algorithm to ensure the accuracy of the steering wheel grip strength value in real-time driving environment parameters, providing a reliable basis for adjusting the fatigue level label. A continuous decrease in grip strength also triggers a level adjustment.

[0090] Based on the adjusted current fatigue level identifier, a tiered warning trigger strategy is generated. The strategy defines different response measures for different levels; for example, mild fatigue triggers a voice prompt "Please rest," moderate fatigue triggers seat vibration and flashing lights, and severe fatigue triggers a strong alarm and suggests pulling over. The strategy generation module outputs a structure containing fields such as warning level, trigger conditions, and alarm type for subsequent modules to use. The strategy is recalculated when environmental parameters change to ensure real-time performance, and all operation logs are used for auditing and optimization. Assuming the system detects an eye movement fatigue index of 0.8, a head behavior anomaly score of 0.6, a vehicle speed of 90 km / h, a highway road type, and a time period of late night, with adjusted weights of 0.7 for eye movement and 0.3 for head movement, the weighted average is... After applying a time decay factor of 0.9, the comprehensive fatigue level index is: The value is rounded to 67, matching the moderate fatigue level. At this point, real-time environmental parameters show good visibility (200 meters), but lane departures occur 4 times per minute. Therefore, the fatigue level is adjusted to severe, generating a corresponding warning strategy and triggering an advanced alert. A de-jitter mechanism is incorporated into the system processing to avoid frequent level switching and ensure stability.

[0091] Example 5: The generation and dynamic adjustment process of multi-level warning signal output stream. This process begins with the system receiving the warning level code from the graded warning triggering strategy. For example, when the strategy determines severe fatigue, the corresponding code is L3. The system retrieves the corresponding parameter set from a preset alarm instruction library based on this code. The alarm instruction library adopts a hierarchical storage structure. The first layer indexes the warning level, and the second layer categorizes and stores acoustic alarm parameters, optical alarm parameters, and tactile alarm parameters. Acoustic parameters include the sound wave frequency range, loudness in decibels, and alarm rhythm mode. Optical parameters cover LED color codes, flashing frequency, and light intensity levels. Tactile parameters include the vibration motor amplitude level, vibration waveform mode, and duration configuration. The retrieval process uses a hash algorithm for rapid location, ensuring that parameter retrieval is completed within milliseconds. The retrieved parameter set is encapsulated into a temporary data structure and transmitted to the signal generation module.

[0092] When generating acoustic alarm waveform data, the system calls the waveform synthesis function in the digital signal processing library to generate a specific mode of audio signal based on the retrieved acoustic parameters. For example, for a severe fatigue alarm, a sine wave with alternating frequencies between 2000 Hz and 3000 Hz is generated, with the loudness controlled between 75 dB and 85 dB, using a rapid intermittent rhythm mode. The generation of optical flashing mode data is achieved by controlling an LED driver chip. The chip receives the color code and flashing frequency parameters and outputs the corresponding PWM signal. For example, in severe fatigue, a red LED is driven to flash at a frequency of 5 times per second, and the light intensity automatically adapts to the ambient light sensor reading. The generation of tactile vibration sequence data relies on an embedded vibration motor controller. The controller generates a drive current sequence based on the amplitude level and waveform mode, for example, using a gradually increasing-decreasing waveform to simulate a pulsating vibration effect. The three types of data are stored in different buffer queues: acoustic data is buffered in PCM format, optical data is stored as a timing instruction set, and tactile data is recorded as a motor control parameter sequence.

[0093] When combining acoustic alarm waveform data, optical flashing pattern data, and tactile vibration sequence data into a multi-level warning signal output stream, the system employs a time alignment mechanism to ensure synchronous output of multi-modal signals. Acoustic waveform data is converted into analog signals by an audio codec chip and output to the speaker. Optical data controls the LED group via I / O ports, and tactile data activates the vibration motor via a motor drive circuit. The output stream uses a timestamp synchronization strategy, assigning the same timestamp to all three types of data at the beginning of each processing cycle, strictly maintaining timing consistency during output. For example, the start time of the acoustic alarm and the start time of the LED flashing deviate by no more than 10 milliseconds. The output stream also includes a priority flag field, with the initial priority order preset according to the warning level. For example, in cases of severe fatigue, the tactile alarm has the highest priority, followed by the acoustic alarm, and then the optical alarm has the lowest priority. The real-time monitoring module continuously collects facial expression and body movement response data from the driver. Facial expression data is captured at 30 frames per second by a visible light camera, and image processing algorithms extract pupil diameter change curves and facial muscle tension indices. Body movement data is collected by a steering wheel pressure sensor and a pedal travel sensor. The pressure sensor records changes in hand grip force, and the travel sensor monitors pedal frequency. When analyzing the pupil constriction rate in the facial expression response data, the algorithm calculates the slope of pupil diameter change per unit time and assesses the trend of alertness changes by combining it with a baseline pupil size. Facial muscle tension is quantified by analyzing the displacement velocity of feature points such as frown lines and drooping corners of the mouth. The body movement response data is analyzed to determine the fluctuation period of steering wheel grip force and pedal depressor intervals, establishing a time-series model of operational force and frequency.

[0094] Based on the analyzed trends, the priority order of acoustic, optical, and tactile alarms in the multi-level warning signal output stream is dynamically adjusted. The adjustment algorithm is based on fuzzy logic rules. For example, if a 20% increase in pupil contraction rate and increased facial muscle tension are detected within 3 seconds, it is determined that the driver is more sensitive to tactile stimulation, and the tactile alarm priority is raised to the highest level. During the adjustment process, the system continuously evaluates the effectiveness of the alarms. If no expected response (such as pupil contraction or change in operating force) is detected after two consecutive triggers of an alarm of a certain modality, the priority of that alarm is automatically reduced. The priority adjustment information is updated in real time to the output stream data structure and simultaneously transmitted to the alarm instruction library through a feedback loop, forming an adaptive optimization mechanism. In a specific example, when the system triggers the severe fatigue alarm for the first time, it simultaneously outputs an acoustic alarm (3000 Hz buzzer), an optical alarm (rapid red flashing), and a tactile alarm (strong continuous vibration) according to the initial settings. Within the next two seconds, the monitoring module detected a 15% increase in the driver's pupil constriction rate, but a 5% decrease in steering wheel grip strength. Based on this, the system determined that the acoustic alarm was of limited effectiveness while the tactile alarm began to take effect. Therefore, it dynamically adjusted the priorities: the tactile alarm maintained the highest priority and its vibration amplitude was increased; the optical alarm was changed to slow flashing to reduce interference; and the acoustic alarm was changed to an intermittent beep. After the adjusted output stream continued running for three seconds, the system detected that the pedal frequency had returned to normal. The system then further reduced the priority of the optical alarm, creating a dynamic alarm intensity decay curve.

[0095] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0096] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time detection of driver fatigue based on a surveillance camera, characterized in that, include: Multi-source sensor data streams of the driver's facial region are collected, and time-series alignment processing is performed on the multi-source sensor data streams to generate a synchronized set of physiological signals. Multimodal feature extraction was performed on the synchronized physiological signal set to obtain eye movement trajectory feature vectors, head posture change sequences, and facial muscle activity intensity maps; An eye movement fatigue index is generated by dynamically comparing and analyzing the feature vector of eye movement trajectory with preset fatigue threshold parameters. By integrating head posture change sequences with facial muscle activity intensity maps, a head behavior abnormality score is calculated using a neural network model. A weighted decision algorithm was used to generate a comprehensive fatigue level index by combining the eye movement fatigue index and the head behavior abnormality score. Based on the comprehensive fatigue level index and real-time driving environment parameters, a graded early warning triggering strategy is generated. Based on the tiered early warning triggering strategy, the corresponding alarm instruction set is invoked to generate a multi-level warning signal output stream; Real-time monitoring of the driver's physiological state changes and dynamic priority adjustment of multi-level warning signal output streams; The alarm instruction set is updated based on the dynamic priority adjustment results to complete the closed-loop control for fatigue driving intervention. The process involves acquiring multi-source sensor data streams from the driver's facial region and performing time-series alignment processing on the multi-source sensor data streams to generate a synchronized set of physiological signals, including: The driver's facial thermal imaging data stream is acquired through an infrared camera, while a visible light video stream of the face is acquired through a visible light camera. The temperature distribution matrix sequence in the thermal imaging data stream and the grayscale frame sequence in the visible light video stream were extracted respectively. Timestamp matching is performed on the temperature distribution matrix sequence and the grayscale frame sequence to establish a cross-modal data synchronization index table; Based on the cross-modal data synchronization index table, the temperature distribution matrix sequence and grayscale frame sequence are resampled to generate time-aligned infrared signal sequences and visible light signal sequences. Time-aligned infrared signal sequences and visible light signal sequences are merged into a synchronized set of physiological signals; The step of dynamically comparing and analyzing the feature vector of eye movement trajectory with preset fatigue threshold parameters to generate an eye movement fatigue index includes: Obtain a preset set of fatigue threshold parameters, including minimum blink frequency threshold, maximum eye closure duration threshold, and pupil displacement tolerance value; The blink frequency in the eye movement trajectory feature vector is compared with the minimum blink frequency threshold to generate a frequency deviation coefficient; The duration of eye closure in the eye movement trajectory feature vector is compared with the maximum duration threshold of eye closure to generate a duration exceedance coefficient; The pupil displacement in the eye movement trajectory feature vector is compared with the pupil displacement tolerance value to generate a displacement anomaly coefficient. An eye movement fatigue index is generated by fusing the frequency deviation coefficient, duration exceedance coefficient, and displacement anomaly coefficient using a multiple regression model. The method combines the eye movement fatigue index and head behavior abnormality score, and uses a weighted decision algorithm to generate a comprehensive fatigue level index, including: Obtain current driving environment parameters, including vehicle speed, road type identification, and time period code; The weighting coefficient of the eye movement fatigue index is adjusted according to the vehicle speed value, and the weighting coefficient of the head behavior abnormality score is adjusted according to the road type sign. The weighted average of the eye movement fatigue index and the head behavior abnormality score was calculated using a weighted summation algorithm. Based on the time period coding, the weighted average value is compensated for time decay to generate a comprehensive fatigue level index.

2. The method for real-time detection of driver fatigue based on a surveillance camera according to claim 1, characterized in that, The process of performing multimodal feature extraction on the synchronized physiological signal set to obtain eye movement trajectory feature vectors, head posture change sequences, and facial muscle activity intensity maps includes: Extract eye region image subsequences from visible light signal sequences and generate a continuous set of eye movement trajectory coordinates using a pupil tracking algorithm; Velocity and acceleration are calculated on the coordinate set of eye movement trajectories to generate an eye movement trajectory feature vector containing blink frequency, eye closure duration, and pupil displacement. Based on infrared signal sequences, temperature change data of key facial points are extracted, and facial muscle activity intensity maps are calculated using a thermodynamic model. Simultaneously, the three-dimensional coordinates of head feature points are identified from the visible light signal sequence, and a head posture change sequence is generated through a posture calculation algorithm.

3. The method for real-time detection of driver fatigue based on a surveillance camera according to claim 2, characterized in that, The fusion of head posture change sequences and facial muscle activity intensity maps, along with the calculation of a head behavior anomaly score using a neural network model, includes: The head posture change sequence is processed by Fourier transform to extract the frequency domain feature vector of head motion; The facial muscle activity intensity map was processed by region segmentation to extract sub-maps of muscle activity intensity in the forehead, cheek, and jaw. The head motion frequency domain feature vector and muscle activity intensity sub-map are input into a pre-trained convolutional neural network model; The abnormality score of head behavior is output through a multi-layer feature fusion layer of a convolutional neural network model.

4. The method for real-time fatigue driving detection based on a surveillance camera according to claim 3, characterized in that, The step of generating a graded early warning triggering strategy based on the comprehensive fatigue level index and real-time driving environment parameters includes: Set multiple fatigue threshold ranges, including mild fatigue threshold range, moderate fatigue threshold range, and severe fatigue threshold range; The comprehensive fatigue level index is matched with the multi-level fatigue threshold range to determine the current fatigue level identifier. The current fatigue level indicator is adjusted based on real-time driving environment parameters such as visibility, lane departure frequency, and steering wheel grip strength. A graded early warning triggering strategy is generated based on the adjusted current fatigue level identifier.

5. The method for real-time fatigue driving detection based on a surveillance camera according to claim 4, characterized in that, The process of invoking the corresponding alarm instruction set according to the tiered early warning triggering strategy to generate a multi-level warning signal output stream includes: Based on the warning level code in the graded warning triggering strategy, retrieve the corresponding acoustic alarm parameters, optical alarm parameters, and tactile alarm parameters from the alarm instruction library; Generate acoustic alarm waveform data, optical flashing pattern data, and tactile vibration sequence data; The acoustic alarm waveform data, optical flashing pattern data, and tactile vibration sequence data are combined into a multi-level warning signal output stream.

6. The method for real-time detection of driver fatigue based on a surveillance camera according to claim 5, characterized in that, The real-time monitoring of the driver's physiological state changes and the dynamic priority adjustment of the multi-level warning signal output streams include: Continuously collect facial expression response data and body movement response data of drivers; Analyze the trends in pupil constriction rate and facial muscle tension in facial expression response data; Analyze the trends in steering wheel operation force and pedal depressing frequency in the limb movement response data; Adjust the priority order of acoustic, optical, and tactile alarms in the multi-level warning signal output stream according to the changing trend.

7. A real-time fatigue driving detection system based on a surveillance camera, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the real-time fatigue driving detection method based on a monitoring camera as described in any one of claims 1 to 6.

Images