Hyperspectral driver fatigue detection system and method based on dynamic exposure compensation and multi-modal feature fusion

Abstract

The invention discloses a hyperspectral driver fatigue detection system and method based on dynamic exposure compensation and multi-modal feature fusion, and relates to the technical field of intelligent driving and health monitoring, and the system comprises a hyperspectral camera hardware module, a fatigue feature extraction module and a multi-modal fusion decision module. The hyperspectral camera hardware module monitors illumination through a wide dynamic range sensor, triggers segmented dual-channel acquisition and dynamically adjusts parameters, outputs a standardized spectral cube through radiometric calibration, and the fatigue feature extraction module synchronously extracts physiological indexes such as blood oxygen saturation and heart rate variability and behavior features such as eyelid closure and pupil microtremor. The multi-modal fusion decision module calculates a fusion weight through a gating attention mechanism, and outputs a fatigue state discrimination result in combination with an SVM and LSTM classifier and a dynamic weighted voting mechanism, the adaptability in a complex illumination environment is improved, collaborative analysis of physiological and behavior characteristics is realized, and the accuracy and reliability of driver fatigue detection are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving and health monitoring technology, specifically to a hyperspectral driver fatigue detection system and method based on dynamic exposure compensation and multimodal feature fusion. Background Technology

[0002] With the rapid development of intelligent transportation systems, driver fatigue monitoring has become a key technological support for ensuring road safety. Fatigue driving leads to decreased neuromotor coordination and weakened situational awareness, and in severe cases, even loss of consciousness, making it one of the core causes of road traffic accidents. In real-world driving environments, lighting conditions are complex and variable, such as sudden changes in light when entering or exiting tunnels, and low-light or strong backlighting at night, all of which place stringent demands on the environmental adaptability of fatigue detection systems. Therefore, developing a detection system capable of stable operation under complex lighting conditions and providing early warning of fatigue is of significant practical importance for improving road traffic safety.

[0003] Currently, driver fatigue detection solutions primarily rely on RGB cameras to capture facial images and extract behavioral features such as eyelid closure and head posture for status assessment. However, these solutions have significant technical limitations: First, RGB imaging is susceptible to changes in ambient lighting, easily leading to overexposure, underexposure, or loss of detail in scenarios with sudden changes in lighting, resulting in misjudgment or missed detection. Second, limited by the imaging principle, it can only acquire exogenous behavioral representations and cannot effectively retrieve endogenous physiological indicators reflecting the essence of fatigue, such as blood oxygen saturation and heart rate variability, keeping fatigue assessment at a superficial level. Third, some distributed systems typically rely on high-latency data transmission architectures to upload video streams captured by cameras to the cloud for behavioral recognition and fatigue analysis, resulting in average data latency that is difficult to meet real-time warning requirements. Furthermore, existing technologies generally suffer from high data processing latency and weak anti-interference capabilities, making it difficult to meet the real-time and reliability requirements of actual driving environments.

[0004] In existing technologies, traditional visual fatigue detection schemes rely on RGB imaging technology, which is not only susceptible to interference from sudden changes in lighting, leading to unstable image quality, but also unable to simultaneously acquire deep physiological indicators and behavioral characteristics that reflect the essence of fatigue. This makes it difficult to achieve accurate and stable identification of fatigue state under complex lighting conditions, and this problem has become the core bottleneck restricting the practical application of fatigue detection technology.

[0005] In summary, existing fatigue detection technologies struggle to achieve stable extraction and collaborative analysis of physiological indicators and behavioral characteristics under complex driving environments such as sudden changes in lighting, resulting in insufficient robustness and accuracy. To address this critical issue, there is an urgent need to propose a fatigue detection scheme with strong environmental adaptability that can simultaneously integrate physiological and behavioral characteristics to meet the real-time and reliability requirements of actual driving scenarios. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies by providing a hyperspectral driver fatigue detection system and method based on dynamic exposure compensation and multimodal feature fusion. By integrating a dynamic exposure compensation unit into the hyperspectral camera hardware module, and utilizing a wide dynamic range sensor to monitor ambient light in real time, a segmented dual-channel acquisition mode is triggered when there are sudden changes in light intensity. Combined with a feedback control algorithm, image sensor parameters are adjusted, and standardized spectral data is output after radiometric calibration. This enables stable acquisition of high-quality spectral information under complex lighting conditions. Simultaneously, a fatigue feature extraction module extracts physiological indicators such as blood oxygen saturation and behavioral features such as eyelid closure. A multimodal fusion decision module performs feature calibration, weight calculation, and classifier fusion decision-making, enabling collaborative analysis of physiological and behavioral features. This improves the accuracy and environmental adaptability of fatigue detection, overcoming the shortcomings of existing technologies, such as susceptibility to light interference and limited feature diversity.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, a hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion, comprising:

[0008] A hyperspectral camera hardware module is used to acquire spectral data of the driver's facial region through a snapshot-type hyperspectral imaging unit. The spectral data covers the visible light band and the near-infrared band. The hyperspectral camera hardware module integrates a dynamic exposure compensation unit.

[0009] The dynamic exposure compensation unit includes a wide dynamic range sensor, an exposure control unit, and a data normalization unit.

[0010] The wide dynamic range sensor monitors ambient light intensity in real time;

[0011] The exposure control unit triggers a segmented dual-channel acquisition mode when there is a sudden change in illumination. The short exposure mode is used to suppress overexposure in the highlight area, and the long exposure mode is used to enhance the details in the shadow area. The integral time and gain of the image sensor are dynamically adjusted through a feedback control algorithm.

[0012] The data normalization unit performs nonlinear correction on the original spectral data using a radiometric calibration plate and outputs a normalized spectral cube.

[0013] The fatigue feature extraction module, connected to the hyperspectral camera hardware module, is used to simultaneously extract physiological indicators and behavioral features from the standardized spectral cube.

[0014] The physiological indicators include blood oxygen saturation, heart rate variability, and skin moisture loss.

[0015] The behavioral characteristics include eyelid closure degree, pupillary micro-tremor and micro-expression changes;

[0016] The multimodal fusion decision module, connected to the fatigue feature extraction module, is used to perform timestamp synchronization and spatial posture calibration of physiological indicators and behavioral features. It generates a fusion feature vector by feature concatenation, calculates cross-modal fusion weights based on a gating attention mechanism, and inputs them into parallel SVM and LSTM classifiers to generate fatigue probability vectors. Finally, it outputs the fatigue state discrimination result through a dynamic weighted voting mechanism, wherein the voting weight corresponding to the physiological indicators is not less than 60% of the total weight.

[0017] Furthermore, the calculation of blood oxygen saturation in the fatigue feature extraction module uses the following formula:

[0018]

[0019] Among them, R 660 R represents the reflectivity in the 660nm wavelength band. 940 This indicates the reflectivity in the 940nm band. This represents the calibration reference reflectance in the 660nm band. This represents the calibration reference reflectance in the 940nm band, where K is the linear scaling factor and C is the calibration constant offset.

[0020] The following formula is used to calculate skin moisture loss:

[0021]

[0022] Among them, R 970 R represents the reflectivity in the 970nm wavelength band. 900 This represents the reference reflectance in the 900nm band, α is the skin type correction factor, and β is the moisture content baseline offset.

[0023] Furthermore, the pupillary microtremor detection specifically includes:

[0024] Based on near-infrared band images, an edge detection algorithm and Hough transform are used to determine the initial pupil region;

[0025] Pupil contours are extracted using the Zernike subpixel edge detection model, and temporal filtering is performed on the continuous frame pupil center coordinate sequence.

[0026] An adaptive bandpass filter was used to separate the pupillary micro-tremor signal, and the phase synchronization entropy value was calculated as an indicator of rhythm stability.

[0027] The eyelid closure degree detection specifically includes:

[0028] A three-dimensional eyelid motion model was established, integrating the visible pupil ratio and the change in palpebral fissure height.

[0029] Non-fatigue eye-closing events are identified using a bidirectional long short-term memory network. The inputs of the bidirectional long short-term memory network include eyelid closure timing signals, head motion acceleration, and pupillary micro-tremor signals.

[0030] Furthermore, the spectral analysis of micro-expression change features includes the following steps:

[0031] Reflectance data of predefined sensitive bands are extracted from key areas of the face. The sensitive bands include the characteristic absorption band of oxyhemoglobin, the characteristic absorption band of deoxyhemoglobin, and the isoabsorption point reference band, wherein the isoabsorption point reference band is selected in the wavelength range of 570nm to 575nm.

[0032] The reflectance ratio of the sensitive band to the reference band is calculated to form a time-series spectral feature vector;

[0033] The time-series spectral feature vectors are subjected to bandpass filtering, with the passband range set to 0.5 Hz to 2.0 Hz;

[0034] Abrupt changes are detected in the filtered vector to identify events of instantaneous changes in subcutaneous blood volume.

[0035] Based on the spatial distribution and temporal patterns of mutation points, micro-expression categories are output through a pre-trained classification model.

[0036] Furthermore, the characteristic absorption band of the oxyhemoglobin is a narrow-band spectrum in the wavelength range of 540nm to 545nm, with a bandwidth of no more than 10nm;

[0037] The characteristic absorption band of the deoxyhemoglobin is a narrow band spectrum in the wavelength range of 555nm to 565nm, with a bandwidth of no more than 10nm;

[0038] The reflectance ratio calculation uses the standardized spectral cube output by the dynamic exposure compensation module as input, and suppresses ambient light fluctuations through the isoabsorption point reference band.

[0039] Furthermore, the timestamp synchronization of the multimodal fusion decision module specifically involves:

[0040] A unified timestamp was applied to blood oxygen saturation signals, heart rate variability signals, eyelid closure signals, and micro-expression feature signals;

[0041] The spatial attitude calibration specifically includes:

[0042] The coordinate transformation of the physiological indicator detection area is performed based on the head posture angle data to eliminate feature offset caused by head deflection.

[0043] Furthermore, the dynamic weighted voting mechanism specifically includes:

[0044] The first fatigue probability vector output by the SVM classifier and the second fatigue probability vector output by the LSTM classifier are linearly superimposed according to the weight coefficients, which are dynamically adjusted according to the light stability.

[0045] When the rate of change in ambient illuminance exceeds the threshold, the voting weight of the physiological indicators is increased to 70% of the total weight.

[0046] Furthermore, the feedback control algorithm employs a proportional-integral-derivative controller, with the input being the difference between the measured ambient illuminance and the target illuminance, and the output being the image sensor integral time adjustment and gain adjustment.

[0047] In the segmented dual-channel acquisition mode, the exposure time for the short exposure mode is shorter than that for the long exposure mode, and the exposure time for the long exposure mode is no less than 10 times that for the short exposure mode.

[0048] Furthermore, the data quality of the standardized spectral cube satisfies a signal-to-noise ratio not lower than a preset threshold;

[0049] The wide dynamic range sensor operates from 0.1 to 100,000 lux.

[0050] The near-infrared active illumination uses an LED array with a center wavelength of 850nm and a radiation intensity lower than the safety threshold for the human eye.

[0051] On the other hand, a hyperspectral driver fatigue detection method based on dynamic exposure compensation and multimodal feature fusion includes the following specific steps:

[0052] S100: Real-time monitoring of ambient light intensity through a wide dynamic range sensor; triggering segmented dual-channel acquisition when light intensity changes abruptly; and adjusting image sensor parameters using a feedback control algorithm.

[0053] S200: Corrects the original spectral data through radiometric calibration, outputs a standardized spectral cube, and simultaneously extracts blood oxygen saturation, heart rate variability, skin moisture loss, eyelid closure, pupillary micro-tremor and micro-expression features from the standardized spectral cube.

[0054] S300: The extracted features are time-stamped and spatially calibrated, and the standardized features are input into the gating attention mechanism to calculate the fusion weights;

[0055] S400: The probability vector is generated in parallel by SVM classifier and LSTM classifier, and the fatigue state is output by dynamic weighted voting, in which the weight of physiological features accounts for no less than 60%.

[0056] Compared with existing technologies, this hyperspectral driver fatigue detection system and method based on dynamic exposure compensation and multimodal feature fusion has the following advantages:

[0057] I. This invention integrates a dynamic exposure compensation unit into the hardware module of a hyperspectral camera. It utilizes a wide dynamic range sensor to monitor ambient light intensity in real time, triggering a segmented dual-channel acquisition mode when light intensity changes abruptly. A feedback control algorithm dynamically adjusts the image sensor parameters, and data normalization is performed using radiometric calibration. This enables stable acquisition of high-quality spectral data under complex lighting conditions, effectively avoiding overexposure, underexposure, or loss of detail. Simultaneously, the fatigue feature extraction module extracts endogenous physiological indicators such as blood oxygen saturation and behavioral features such as eyelid closure, achieving synergistic analysis of physiological and behavioral features. This solves the problems of traditional technologies relying on single behavioral features and being susceptible to light interference, leading to insufficient detection stability.

[0058] Second, this invention uses a multimodal fusion decision module to synchronize the extracted physiological and behavioral features with timestamps and calibrate their spatial pose. It calculates cross-modal fusion weights based on a gated attention mechanism and uses parallel decision-making with SVM and LSTM classifiers and a dynamic weighted voting mechanism to output the results. The voting weight of physiological indicators is kept at a high proportion, which can give full play to the stability advantage of physiological indicators in complex environments and the real-time advantage of behavioral features, improve the accuracy of feature fusion and decision robustness, and thus solve the problems of insufficient accuracy in fatigue state discrimination and weak anti-interference ability caused by insufficient feature fusion and single decision mechanism in traditional detection technology.

[0059] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0061] Figure 1 This is a flowchart of the process of the present invention;

[0062] Figure 2 This is a diagram of the system hardware and data flow architecture of the present invention;

[0063] Figure 3This is a flowchart of the multimodal feature fusion decision-making process of the present invention. Detailed Implementation

[0064] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structure, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0065] Example 1

[0066] like Figure 2 and Figure 3 As shown, this embodiment discloses the specific application process of a hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion in complex lighting scenarios. When a vehicle encounters drastic changes in ambient light during operation, the system uses a hyperspectral camera hardware module to sense the changes in light in real time and activates a dynamic exposure compensation mechanism to ensure the stability of spectral data acquisition. Subsequently, the fatigue feature extraction module simultaneously acquires the driver's endogenous physiological indicators and exogenous behavioral characteristics. Finally, the multimodal fusion decision module achieves accurate identification and early warning of fatigue status. This embodiment verifies the system's environmental adaptability, feature extraction accuracy, and decision reliability in scenarios with sudden changes in lighting, providing a practical and feasible technical solution for driver fatigue detection.

[0067] The system deployment and initial environment setup are implemented as follows;

[0068] In this embodiment, a hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion is deployed in the cockpit of a small car. The hyperspectral camera hardware module is installed inside the A-pillar of the cockpit, with the lens facing the driver's face to ensure complete acquisition of spectral data of key facial areas (including the forehead, eyes, and glabella). After the system is initially powered on, it automatically completes hardware self-test and parameter initialization, specifically including: the snapshot-type hyperspectral imaging unit of the hyperspectral camera hardware module is activated, and the spectral coverage is adjusted to the visible light and near-infrared bands; the wide dynamic range sensor enters real-time monitoring mode, and its working range covers low-light to strong light environments that may be encountered during vehicle operation; the exposure control parameters of the dynamic exposure compensation unit, the feature extraction threshold of the fatigue feature extraction module, and the classification model parameters of the multimodal fusion decision module are all loaded with preset initial values ​​to ensure that the system enters the ready-to-work state.

[0069] In the initial stage of vehicle operation, the ambient lighting is stable, meeting normal daytime lighting conditions. At this time, the wide dynamic range sensor monitors the ambient illuminance in real time and finds it to be within a stable range. The hyperspectral camera hardware module operates in a conventional single-channel exposure mode, and the image sensor's integration time and gain remain at the baseline value. After preliminary processing by the data normalization unit, the acquired raw spectral data outputs a standardized spectral cube signal-to-noise ratio that meets the preset threshold, providing a stable data foundation for subsequent feature extraction.

[0070] The dynamic exposure compensation process of the hyperspectral camera hardware module is implemented as follows;

[0071] When a vehicle travels through a mountainous area, it needs to pass through a long tunnel. At this time, the ambient light changes rapidly from strong daytime light (illuminance of about 100,000 lux) to low illumination inside the tunnel (illuminance of about 50 lux), creating a typical scene of sudden change in illumination. The wide dynamic range sensor captures this dramatic change in illumination in real time, triggering the segmented dual-channel acquisition mode of the dynamic exposure compensation unit.

[0072] Upon receiving a sudden change in illumination, the exposure control unit immediately activates a dual-channel parallel acquisition mechanism: a short exposure mode suppresses overexposure in residual highlight areas at the tunnel entrance by shortening the image sensor integration time to avoid image saturation caused by windshield reflections and strong light at the tunnel entrance; a long exposure mode is activated simultaneously, extending the integration time to enhance details in facial shadow areas, ensuring clear spectral information for key areas such as the driver's eyes and forehead. Simultaneously, the proportional-integral-derivative (PID) controller built into the exposure control unit begins operation. Its input is the difference between the current ambient illuminance measurement and the target illuminance. By calculating and dynamically outputting adjustments to the image sensor integration time and gain, real-time optimization of exposure parameters is achieved. The exposure time for the long exposure mode is set to be more than 10 times that of the short exposure mode to ensure effective acquisition of information from different illumination areas.

[0073] During dual-channel acquisition, the data normalization unit operates synchronously. Radiation distortion in the raw spectral data caused by changes in illumination is nonlinearly corrected using a radiometric calibration plate. The calibration plate provides standard reflectance benchmarks for each band. The data normalization unit compares the acquired raw reflectance with the benchmark reflectance, eliminating errors caused by illumination gradients through a nonlinear correction algorithm, ultimately outputting a standardized spectral cube. This standardized spectral cube maintains a signal-to-noise ratio above a preset threshold before and after abrupt changes in illumination, effectively avoiding overexposure, underexposure, or detail loss issues that occur with traditional imaging techniques during dramatic illumination changes, providing high-quality spectral data support for subsequent fatigue feature extraction. Furthermore, the near-infrared active illumination unit automatically activates during this process, employing an LED array with a center wavelength of 850nm. Its radiation intensity is strictly controlled below the human eye safety threshold, ensuring clear acquisition of facial near-infrared spectral information even in low-light environments without interfering with the driver's visual perception.

[0074] The feature extraction process of the fatigue feature extraction module is implemented as follows;

[0075] Based on the standardized spectral cube output by the hyperspectral camera hardware module, the fatigue feature extraction module simultaneously initiates the extraction process of physiological indicators and behavioral features.

[0076] For physiological indicator extraction, blood oxygen saturation was first measured. The system extracted reflectance data from the 660nm and 940nm bands within a standardized spectral cube, where 660nm represents the characteristic absorption peak of deoxyhemoglobin and 940nm represents the characteristic absorption peak of oxyhemoglobin. Based on the blood oxygen saturation calculation formula... The reflectivity (R) of the 660nm band 660 ) and the calibration reference reflectivity of this band The ratio, divided by the reflectivity (R) of the 940nm band. 940 ) and the calibration reference reflectivity of this band The ratio of the two values ​​is then corrected using a linear proportionality coefficient (K) and a calibration constant offset (C) to finally calculate the driver's blood oxygen saturation value. The test results showed that the driver's blood oxygen saturation was slightly lower than when the light intensity was stable, reflecting a tendency towards fatigue.

[0077] Skin moisture loss is detected based on the characteristic absorption properties of water molecules in the 970nm wavelength band. The system extracts the 970nm reflectance (R0) from the normalized spectral cube. 970 ) and 900nm reference band reflectivity (R 900 ), through formula The natural logarithm of the ratio was calculated, and then corrected for by a skin type correction factor (α) and a moisture content baseline offset (β) to obtain the percentage of skin moisture loss. The results showed a slight increase in skin moisture loss, further confirming the physiological changes under fatigue.

[0078] In terms of behavioral feature extraction, pupillary microtremor detection is based on near-infrared images. First, an edge detection algorithm and Hough transform are used to determine the initial pupil region. This process initially locates the pupil boundary by identifying the grayscale difference between the pupil and iris in the image. Then, a Zernike moment sub-pixel edge detection model is introduced to accurately extract the pupil contour, achieving sub-pixel-level edge localization and improving the accuracy of pupil center coordinate calculation. Temporal filtering is applied to the pupil center coordinate sequence of consecutive frames to remove high-frequency noise interference. Then, an adaptive bandpass filter is used to separate the pupillary microtremor signal, and the phase synchronization entropy value of this signal is calculated as an indicator of rhythmic stability. The results show that an increase in the phase synchronization entropy value indicates a decrease in the rhythmic stability of pupillary microtremors, suggesting a deepening of fatigue.

[0079] Eyelid closure detection is achieved by establishing a three-dimensional eyelid motion model. This model integrates the visible pupil ratio and the change in palpebral fissure height to comprehensively reflect the eyelid motion state. To avoid interference from non-fatigue-related eye-closing events (such as normal blinking) on ​​the detection results, the system introduces a bidirectional long short-term memory network. Its inputs include eyelid closure timing signals, head motion acceleration, and pupillary micro-tremor signals. By learning the differences in timing patterns between normal and fatigue-related eye-closing, the system automatically eliminates non-fatigue-related eye-closing events. An increase in the calculated eyelid closure percentage after correction indicates that the driver is engaging in frequent eye-closing behavior.

[0080] Micro-expression feature extraction focuses on key facial areas (such as the area between the eyebrows and the corners of the eyes). The system first extracts reflectance data from predefined sensitive bands, including the characteristic absorption bands of oxyhemoglobin (540nm to 545nm), deoxyhemoglobin (555nm to 565nm), and isoabsorption reference bands (570nm to 575nm), where the isoabsorption reference bands are used to suppress interference from ambient light fluctuations. The reflectance ratio of the sensitive bands to the reference bands is calculated to construct a temporal spectral feature vector. This vector is then bandpass filtered (passband range 0.5Hz to 2.0Hz) to retain effective signals related to micro-expressions. Abrupt changes are detected on the filtered vector to identify instantaneous changes in subcutaneous blood volume caused by micro-expressions. Based on the spatial distribution and temporal patterns of these abrupt changes, a pre-trained classification model outputs the micro-expression category. The detection results show an increased frequency of fatigue-related micro-expressions such as frowning.

[0081] The decision-making process of the multimodal fusion decision module is implemented as follows;

[0082] After receiving the physiological indicators and behavioral features output by the fatigue feature extraction module, the multimodal fusion decision module first performs timestamp synchronization and spatial posture calibration. Timestamp synchronization ensures consistency of each feature in the time dimension by assigning a unified timestamp to the blood oxygen saturation signal, heart rate variability signal, eyelid closure signal, and micro-expression feature signal. Spatial posture calibration transforms the coordinates of the physiological indicator detection area based on head posture angle data to eliminate the spatial position offset of features caused by the driver's head rotation, ensuring spatial consistency of features.

[0083] After feature calibration, the system concatenates physiological indicator feature groups (including blood oxygen saturation, heart rate variability, and skin moisture loss) with behavioral feature groups (including eyelid closure, pupillary micro-tremors, and micro-expression changes) to generate a fused feature vector. This vector is then input into a gated attention mechanism, which learns the contribution of each feature to fatigue detection and calculates cross-modal fusion weights, giving higher weights to features sensitive to fatigue states.

[0084] The feature vectors are simultaneously input into parallel SVM and LSTM classifiers. The SVM classifier transforms high-dimensional features into a linearly separable space through kernel function mapping, outputting a first fatigue probability vector. The LSTM classifier, leveraging its ability to capture temporal features, analyzes the trend of feature changes over time, outputting a second fatigue probability vector. The system employs a dynamic weighted voting mechanism to fuse the two probability vectors, with the voting weight for physiological indicators set at 60% of the total weight. Due to the current scenario of sudden changes in illumination (ambient illuminance change rate exceeding a threshold), the system automatically increases the voting weight of physiological indicators to 70% to utilize the stability advantage of physiological features under complex lighting conditions.

[0085] Through dynamic weighted voting calculation, the system finally outputs the fatigue state judgment result, determining that the driver is currently in a state of moderate fatigue.

[0086] The system's early warning and feedback process is implemented as follows;

[0087] Once the multimodal fusion decision module outputs a moderate fatigue state assessment result, the system immediately activates a tiered early warning mechanism. The in-vehicle human-machine interface first displays a fatigue warning icon visually, and simultaneously issues a prompt message via the voice module, reminding the driver to take a break. To further enhance the warning effect, the system combines vehicle driving status (such as current speed and road segment) to display information about nearby service areas or parking areas on the in-vehicle screen, suggesting that the driver stop and rest in a timely manner.

[0088] During the warning process, the system continuously monitors changes in the driver's condition and ambient lighting. The hyperspectral camera hardware module continues to collect spectral data and perform dynamic exposure compensation, while the fatigue feature extraction module and multimodal fusion decision module update the feature extraction and decision results in real time. If the driver does not take any rest measures and their fatigue continues to worsen, the system will gradually escalate the warning level, including increasing the frequency of voice prompts and seat vibration alerts, until the driver responds to the warning or the system determines that there is a serious safety risk. In this case, the system will send a warning message to the relevant safety management platform via the vehicle network.

[0089] In summary, this embodiment fully demonstrates the working process of a hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion by simulating a sudden change in lighting conditions when a vehicle passes through a tunnel. The hyperspectral camera hardware module, through a dynamic exposure compensation mechanism, can still output high-quality standardized spectral cubes under drastic lighting changes, solving the data distortion problem of traditional imaging technologies under complex lighting conditions. The fatigue feature extraction module achieves simultaneous and accurate extraction of endogenous physiological indicators and exogenous behavioral features, providing multi-dimensional evidence for fatigue state identification. The multimodal fusion decision module, through a dynamic weighted voting mechanism, fully leverages the stability advantages of physiological features in complex environments, ensuring the accuracy of fatigue state identification.

[0090] This embodiment verifies the system's environmental adaptability, feature extraction effectiveness, and decision reliability under complex lighting conditions. It can achieve early warning of driver fatigue, providing strong technical support for improving road driving safety, which is in line with the design purpose and technical effect of this invention.

[0091] Example 2

[0092] like Figure 1 As shown in Example 1, this example elaborates on the specific steps of the hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion during operation. The specific steps are as follows:

[0093] S100. After system startup, the wide dynamic range sensor continuously monitors the ambient light intensity. When a sudden change in light intensity is detected, a segmented dual-channel acquisition mode is triggered. The short exposure channel is used to avoid overexposure in highlight areas, while the long exposure channel is used to enhance details in dark areas. The exposure control unit adjusts the integration time and gain of the image sensor in real time through a feedback control algorithm to adapt to the current lighting conditions.

[0094] The S200 system collects raw spectral data, which undergoes radiometric calibration and nonlinear correction via a data normalization unit, outputting a standardized spectral cube with a signal-to-noise ratio that meets requirements. Based on this cube, the system simultaneously extracts multiple physiological indicators and behavioral characteristics, including blood oxygen saturation, heart rate variability, skin moisture loss, eyelid closure, pupillary micro-tremors, and micro-expression changes.

[0095] S300: The extracted features are unified with timestamps and calibrated with spatial pose to eliminate biases caused by time differences in acquisition or head movements. The calibrated features are then weighted and fused using a gated attention mechanism to form a fused feature vector with cross-modal correlation.

[0096] S400: The fused feature vectors are input in parallel to the SVM and LSTM classifiers, which output fatigue probability vectors based on static and temporal features, respectively. The system dynamically adjusts the weights of the two classification results based on the stability of the current ambient light, and outputs the final fatigue state determination result using a weighted voting method. The weight of physiological features in the voting is always no less than 60%, increasing to 70% when there are drastic changes in light intensity, to enhance the system's robustness in complex environments.

[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion, characterized in that, include: A hyperspectral camera hardware module is used to acquire spectral data of the driver's facial region through a snapshot-type hyperspectral imaging unit. The spectral data covers the visible light band and the near-infrared band. The hyperspectral camera hardware module integrates a dynamic exposure compensation unit. The dynamic exposure compensation unit includes a wide dynamic range sensor, an exposure control unit, and a data normalization unit. The wide dynamic range sensor monitors ambient light intensity in real time; The exposure control unit triggers a segmented dual-channel acquisition mode when there is a sudden change in illumination. The short exposure mode is used to suppress overexposure in the highlight area, and the long exposure mode is used to enhance the details in the shadow area. The integral time and gain of the image sensor are dynamically adjusted through a feedback control algorithm. The data normalization unit performs nonlinear correction on the original spectral data using a radiometric calibration plate and outputs a normalized spectral cube. The fatigue feature extraction module, connected to the hyperspectral camera hardware module, is used to simultaneously extract physiological indicators and behavioral features from the standardized spectral cube. The physiological indicators include blood oxygen saturation, heart rate variability, and skin moisture loss. The behavioral characteristics include eyelid closure degree, pupillary micro-tremor and micro-expression changes; The multimodal fusion decision module, connected to the fatigue feature extraction module, is used to perform timestamp synchronization and spatial posture calibration of physiological indicators and behavioral features. It generates a fusion feature vector by feature concatenation, calculates cross-modal fusion weights based on a gating attention mechanism, and inputs them into parallel SVM and LSTM classifiers to generate fatigue probability vectors. Finally, it outputs the fatigue state discrimination result through a dynamic weighted voting mechanism, wherein the voting weight corresponding to the physiological indicators is not less than 60% of the total weight.

2. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 1, characterized in that, The calculation of blood oxygen saturation in the fatigue feature extraction module uses the following formula: Among them, R 660 R represents the reflectivity in the 660nm wavelength band. 940 This indicates the reflectivity in the 940nm band. This represents the calibration reference reflectance in the 660nm band. This represents the calibration reference reflectance in the 940nm band, where K is the linear scaling factor and C is the calibration constant offset. The following formula is used to calculate skin moisture loss: Among them, R 970 R represents the reflectivity in the 970nm wavelength band. 900 This represents the reference reflectance in the 900nm band, α is the skin type correction factor, and β is the moisture content baseline offset.

3. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 1, characterized in that, The pupillary microtremor detection specifically includes: Based on near-infrared band images, an edge detection algorithm and Hough transform are used to determine the initial pupil region; Pupil contours are extracted using the Zernike subpixel edge detection model, and temporal filtering is performed on the continuous frame pupil center coordinate sequence. An adaptive bandpass filter was used to separate the pupillary micro-tremor signal, and the phase synchronization entropy value was calculated as an indicator of rhythm stability. The eyelid closure degree detection specifically includes: A three-dimensional eyelid motion model was established, integrating the visible pupil ratio and the change in palpebral fissure height. Non-fatigue eye-closing events are identified using a bidirectional long short-term memory network. The inputs of the bidirectional long short-term memory network include eyelid closure timing signals, head motion acceleration, and pupillary micro-tremor signals.

4. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 1, characterized in that, The spectral analysis of micro-expression change features includes the following steps: Reflectance data of predefined sensitive bands are extracted from key areas of the face. The sensitive bands include the characteristic absorption band of oxyhemoglobin, the characteristic absorption band of deoxyhemoglobin, and the isoabsorption point reference band, wherein the isoabsorption point reference band is selected in the wavelength range of 570nm to 575nm. The reflectance ratio of the sensitive band to the reference band is calculated to form a time-series spectral feature vector; The time-series spectral feature vectors are subjected to bandpass filtering, with the passband range set to 0.5 Hz to 2.0 Hz; Abrupt changes are detected in the filtered vector to identify events of instantaneous changes in subcutaneous blood volume. Based on the spatial distribution and temporal patterns of mutation points, micro-expression categories are output through a pre-trained classification model.

5. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 4, characterized in that, The characteristic absorption band of the oxyhemoglobin is a narrow band spectrum in the wavelength range of 540nm to 545nm, with a bandwidth of no more than 10nm. The characteristic absorption band of the deoxyhemoglobin is a narrow band spectrum in the wavelength range of 555nm to 565nm, with a bandwidth of no more than 10nm; The reflectance ratio calculation uses the standardized spectral cube output by the dynamic exposure compensation module as input, and suppresses ambient light fluctuations through the isoabsorption point reference band.

6. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 1, characterized in that, The timestamp synchronization of the multimodal fusion decision module is specifically as follows: A unified timestamp was applied to blood oxygen saturation signals, heart rate variability signals, eyelid closure signals, and micro-expression feature signals; The spatial attitude calibration specifically includes: The coordinate transformation of the physiological indicator detection area is performed based on the head posture angle data to eliminate feature offset caused by head deflection.

7. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 1, characterized in that, The dynamic weighted voting mechanism specifically includes: The first fatigue probability vector output by the SVM classifier and the second fatigue probability vector output by the LSTM classifier are linearly superimposed according to the weight coefficients, which are dynamically adjusted according to the light stability. When the rate of change in ambient illuminance exceeds the threshold, the voting weight of the physiological indicators is increased to 70% of the total weight.

8. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 1, characterized in that, The feedback control algorithm uses a proportional-integral-derivative controller. The input is the difference between the measured ambient illuminance and the target illuminance, and the output is the image sensor integral time adjustment and gain adjustment. In the segmented dual-channel acquisition mode, the exposure time for the short exposure mode is shorter than that for the long exposure mode, and the exposure time for the long exposure mode is no less than 10 times that for the short exposure mode.

9. The hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion according to claim 1, characterized in that, The data quality of the standardized spectral cube meets the requirement that the signal-to-noise ratio is not lower than a preset threshold. The wide dynamic range sensor operates from 0.1 to 100,000 lux. The near-infrared active illumination uses an LED array with a center wavelength of 850nm and a radiation intensity lower than the safety threshold for the human eye.

10. A hyperspectral driver fatigue detection method based on dynamic exposure compensation and multimodal feature fusion, applicable to the hyperspectral driver fatigue detection system based on dynamic exposure compensation and multimodal feature fusion as described in any one of claims 1-9, characterized in that, The specific steps of this method are as follows: S100: Real-time monitoring of ambient light intensity through a wide dynamic range sensor; triggering segmented dual-channel acquisition when light intensity changes abruptly; and adjusting image sensor parameters using a feedback control algorithm. S200: Corrects the original spectral data through radiometric calibration, outputs a standardized spectral cube, and simultaneously extracts blood oxygen saturation, heart rate variability, skin moisture loss, eyelid closure, pupillary micro-tremor and micro-expression features from the standardized spectral cube. S300: The extracted features are time-stamped and spatially calibrated, and the standardized features are input into the gating attention mechanism to calculate the fusion weights; S400: The probability vector is generated in parallel by SVM classifier and LSTM classifier, and the fatigue state is output by dynamic weighted voting, in which the weight of physiological features accounts for no less than 60%.

Images