Driver behavior monitoring devices and electronic equipment based on multimodal vision recognition

Abstract

This invention relates to the field of vehicle driving safety monitoring technology, and more particularly to a driver behavior monitoring device and electronic equipment based on multimodal visual recognition. The device includes a physical device and a control device. The control device is integrated into an embedded computing platform and includes a data acquisition module that acquires vehicle dynamic parameters through a vehicle bus interface and controls a multispectral imaging unit to acquire video streams; a feature extraction module that fuses spatial features and temporal motion features from the video stream to generate a spatiotemporal feature tensor; a behavior analysis module that generates standard behavioral features through a generative adversarial network and calculates statistical deviations; a risk decision module that uses a dynamic weighted algorithm to output a comprehensive risk level and confidence assessment; and a control output module that executes early warning operations and sends feedback data to the behavior analysis module for model optimization. This invention reduces the false positive rate and improves the timeliness of early warnings through multimodal data fusion and closed-loop optimization mechanisms.

Description

Technical Field

[0001] This invention relates to the field of vehicle driving safety monitoring technology, and in particular to a driver behavior monitoring device and electronic equipment based on multimodal visual recognition. Background Technology

[0002] Existing driver condition monitoring technologies suffer from the following technical challenges: In the cab of hazardous materials transport vehicles, current driver condition monitoring methods often rely on threshold judgments of single behavioral characteristics, such as monitoring only eyelid closure or steering wheel grip strength, without simultaneously collecting and integrating multi-dimensional behavioral data such as facial expressions, gaze direction, and head posture. This one-sided judgment method struggles to distinguish between genuine fatigue and distraction and normal driving operations. For example, a brief visual shift caused by the driver checking the rearview mirror may be misjudged as distraction, or a momentary squinting due to strong light may be falsely reported as fatigue, leading to frequent false alarms. This not only interferes with the driver's normal operations but also reduces the sensitivity to warnings of real risks due to alarm redundancy, failing to provide timely and effective intervention in the early stages of dangerous driving behavior, thus creating safety hazards in the high-risk transportation of hazardous materials. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a driver behavior monitoring device and electronic equipment based on multimodal visual recognition, which solves the technical problems of high misjudgment rate and delayed early warning for dangerous states such as fatigue and distraction caused by the single nature and insufficient judgment dimensions of existing driver state monitoring methods.

[0004] To solve the above-mentioned technical problems, the specific contents of the present invention are as follows: In a first aspect, the driver behavior monitoring device based on multimodal visual recognition provided by the present invention includes a physical device and a control device, wherein the control device establishes a communication connection with the physical device. The physical device includes a multispectral imaging unit, an embedded computing platform, a multimodal alarm, and a vehicle bus interface module; The control unit is integrated into the embedded computing platform and includes: The data acquisition module is configured to acquire vehicle dynamic parameters through the vehicle bus interface module, generate imaging control commands based on the vehicle dynamic parameters and send them to the multispectral imaging unit, receive the raw video stream returned by the multispectral imaging unit, preprocess the raw video stream and add scene tags, and output a video stream with scene tags. The feature extraction module is configured to receive the video stream with scene labels output by the data acquisition module, extract the spatial features and temporal motion features of facial key points from the video stream, parse physiological indicators from the video stream, and fuse the spatial features and temporal motion features into a spatiotemporal feature tensor. The behavior analysis module is configured to receive the spatiotemporal feature tensor output by the feature extraction module and the vehicle dynamic parameters provided by the data acquisition module, generate standard behavior features based on the vehicle dynamic parameters, calculate the statistical deviation between the spatiotemporal feature tensor and the standard behavior features, predict behavior trends based on historical behavior data, and output the abnormal deviation and behavior trend prediction vector. The risk decision-making module is configured to receive the abnormal deviation degree and behavior trend prediction vector output by the behavior analysis module, as well as the physiological indicators parsed by the feature extraction module. It uses a dynamic weighting algorithm to fuse and calculate the abnormal deviation degree, behavior trend prediction vector and physiological indicators, and outputs a comprehensive risk level and confidence assessment. The control output module is configured to control the multimodal alarm to perform early warning operations based on the comprehensive risk level output by the risk decision module, collect data after the early warning from the spatiotemporal feature tensor output by the feature extraction module as response data, and send the response data to the behavior analysis module for model optimization.

[0005] Furthermore, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a data acquisition module comprising: The parameter acquisition unit is configured to periodically read vehicle speed, lateral acceleration and turn signal status data from the vehicle CAN bus, and output vehicle dynamic parameters to the fuzzy inference unit. The fuzzy inference unit is configured to receive vehicle dynamic parameters output by the parameter acquisition unit, input the vehicle dynamic parameters into a preset fuzzy rule base for calculation, and output the scene complexity evaluation result to the imaging control unit. The imaging control unit is configured to receive the scene complexity assessment result output by the fuzzy inference unit, generate imaging control commands based on the scene complexity assessment result, and send the imaging control commands to the multispectral imaging unit. The preprocessing unit is configured to receive the raw video stream acquired by the multispectral imaging unit, perform non-uniformity correction and scene label addition processing on the raw video stream, and output the video stream with scene labels to the feature extraction module.

[0006] Furthermore, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a feature extraction module comprising: The spatial feature extraction unit is configured to receive the video stream with scene labels output by the data acquisition module, extract the spatial coordinates of facial key points in a single frame image through a convolutional neural network, and output the spatial coordinates to the feature fusion unit. The temporal feature extraction unit is configured to receive the video stream with scene labels output by the data acquisition module, analyze the motion trajectory of key points in the continuous frame sequence through the Transformer architecture, and output the motion trajectory to the feature fusion unit. The feature fusion unit is configured to receive the spatial coordinates output by the spatial feature extraction unit and the motion trajectory output by the temporal feature extraction unit, and fuse the spatial coordinates and motion trajectory through a cross-attention mechanism to output a spatiotemporal feature tensor to the behavior analysis module.

[0007] Furthermore, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a behavior analysis module comprising: The baseline generation unit is configured to receive vehicle dynamic parameters provided by the data acquisition module, generate a standard behavioral feature sequence through a generative adversarial network, and output the standard behavioral feature sequence to the deviation calculation unit. The deviation calculation unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module and the standard behavioral feature sequence output by the benchmark generation unit, calculate the statistical difference between the spatiotemporal feature tensor and the standard behavioral feature sequence using the Mahalanobis distance algorithm, and output the abnormal deviation to the risk decision module. The trend prediction unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module, analyze the spatiotemporal feature sequence in historical behavior data through a long short-term memory network, and output the behavior trend prediction vector to the risk decision module.

[0008] Furthermore, the risk decision module of the driver behavior monitoring device based on multimodal visual recognition described in this invention is configured as follows: The data organization unit is configured to receive the abnormal deviation degree and behavior trend prediction vector output by the behavior analysis module, and the physiological indicators parsed by the feature extraction module, and organize the abnormal deviation degree, behavior trend prediction vector and physiological indicators into a parallel data stream. The entropy calculation unit is configured to receive the parallel data stream output by the data organization unit and calculate the information entropy values ​​of the abnormal deviation, behavioral trend prediction vector and physiological indicators within a sliding time window. The weight allocation unit is configured to receive the information entropy value output by the entropy value calculation unit and dynamically allocate weight coefficients based on the information entropy value. The fusion assessment unit is configured to receive the weight coefficients output by the weight allocation unit and the parallel data stream output by the data organization unit, perform fusion calculations on the weighted indicators, and generate a comprehensive risk level and confidence assessment based on the dispersion of the weight distribution.

[0009] Furthermore, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a control output module comprising: The graded early warning unit is configured to receive the comprehensive risk level and confidence assessment output by the risk decision module, determine the alarm intensity level based on the combination relationship between the comprehensive risk level and the confidence assessment, and output the alarm control signal to the multimodal alarm. The response monitoring unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module, monitor the time delay of the behavior feature returning to normal state, and output the time delay data to the data feedback unit. The data feedback unit is configured to receive time delay data output by the response monitoring unit and comprehensive risk level and confidence assessment output by the risk decision module, encapsulate the time delay data and comprehensive risk level and confidence assessment into a structured data packet, and output the data packet to the behavior analysis module.

[0010] Furthermore, the driver behavior monitoring device based on multimodal visual recognition described in this invention further includes a behavior analysis module that includes: The model optimization unit is configured to receive structured data packets sent by the control output module, extract feature patterns corresponding to early warning events from the structured data packets, adjust the generator parameters in the generative adversarial network through an incremental learning algorithm, and output the generative adversarial network parameters to the baseline generation unit.

[0011] Furthermore, the driver behavior monitoring device based on multimodal visual recognition described in this invention also includes: The multispectral imaging unit is configured to receive imaging control commands sent by the data acquisition module, acquire driver video data according to the imaging control commands, and send the raw video stream to the preprocessing unit in the data acquisition module. The multimodal alarm is configured to receive the early warning control signal sent by the control output module and execute the audible and visual alarm operation according to the early warning control signal; The vehicle bus interface module is configured to read vehicle dynamic parameters from the vehicle CAN bus and send the vehicle dynamic parameters to the parameter acquisition unit in the data acquisition module.

[0012] Furthermore, the driver behavior monitoring device based on multimodal visual recognition described in this invention also includes: The control output module is configured to send the collected driver response data to the model optimization unit in the behavior analysis module; The behavior analysis module is configured to receive driver response data, adjust the parameters of the generated adversarial network through the model optimization unit, and update the standard behavior feature sequence. The risk decision-making module is configured to receive historical early warning effect data and adjust the weight coefficients in the weight allocation unit according to the early warning effect.

[0013] Secondly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the functions of each module in the driver behavior monitoring device based on multimodal visual recognition as described above.

[0014] Beneficial effects of this invention; This invention effectively solves the problems of high false alarm rate and delayed early warning caused by the single method and insufficient judgment dimensions in existing driver status monitoring through the systematic design of multimodal data fusion and closed-loop optimization mechanism. The data acquisition module adaptively adjusts the acquisition strategy of the multispectral imaging unit based on vehicle dynamic parameters, and overcomes the interference of light change through infrared and visible light multi-band imaging, improving data quality in complex environments. The feature extraction module uses spatial and temporal dual-path fusion to generate spatiotemporal feature tensors, avoiding the limitations of single feature analysis, and can simultaneously capture the static position and dynamic movement pattern of key facial points. The behavior analysis module constructs a virtual driver model through generative adversarial network and combines it with long short-term memory network to predict behavior trends, achieving accurate identification of abnormal states and risk prediction. The risk decision module uses the entropy weight method dynamic weighting algorithm to focus on key risk signals, and allocates weight coefficients according to the entropy value of each indicator information to improve the accuracy of assessment. The hierarchical early warning and feedback mechanism of the control output module drives the model optimization unit to continuously adjust the generative adversarial network parameters, forming a closed-loop self-optimizing system. Ultimately, in complex scenarios such as hazardous chemical transportation, it significantly reduces the false alarm rate caused by environmental interference and improves the timeliness of early warning. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on the accompanying drawings without creative effort.

[0016] Figure 1 The system architecture diagram of the driver behavior monitoring device based on multimodal visual recognition provided by the present invention is shown. Detailed Implementation

[0017] To make the technical solution of the present invention clearer, the present invention will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention provided by various embodiments will be described in detail below with reference to the accompanying drawings. To better understand the purpose of the present invention, the present invention will be described in further detail below.

[0018] Firstly, please refer to Figure 1 The driver behavior monitoring device based on multimodal visual recognition provided by the present invention includes a physical device and a control device, wherein the control device establishes a communication connection with the physical device. The physical device includes a multispectral imaging unit, an embedded computing platform, a multimodal alarm, and a vehicle bus interface module; In the physical device, the vehicle bus interface module connects to the vehicle system via a CAN bus to acquire dynamic parameters such as vehicle speed and lateral acceleration in real time. The multispectral imaging unit receives imaging control commands from the data acquisition module, acquires driver video streams, and transmits them to the preprocessing unit in the embedded computing platform. The embedded computing platform integrates control devices, processes the video streams and vehicle data, and generates warning commands. The multimodal alarm receives warning control signals from the control output module and executes audible and visual alarm operations. All components establish a closed-loop data flow through standardized communication interfaces, enabling collaborative operation from parameter acquisition and image processing to risk warning.

[0019] The control unit is integrated into the embedded computing platform and includes: The data acquisition module is configured to acquire vehicle dynamic parameters through the vehicle bus interface module, generate imaging control commands based on the vehicle dynamic parameters and send them to the multispectral imaging unit, receive the raw video stream returned by the multispectral imaging unit, preprocess the raw video stream and add scene tags, and output a video stream with scene tags. The feature extraction module is configured to receive the video stream with scene labels output by the data acquisition module, extract the spatial features and temporal motion features of facial key points from the video stream, parse physiological indicators from the video stream, and fuse the spatial features and temporal motion features into a spatiotemporal feature tensor. The behavior analysis module is configured to receive the spatiotemporal feature tensor output by the feature extraction module and the vehicle dynamic parameters provided by the data acquisition module, generate standard behavior features based on the vehicle dynamic parameters, calculate the statistical deviation between the spatiotemporal feature tensor and the standard behavior features, predict behavior trends based on historical behavior data, and output the abnormal deviation and behavior trend prediction vector. The risk decision-making module is configured to receive the abnormal deviation degree and behavior trend prediction vector output by the behavior analysis module, as well as the physiological indicators parsed by the feature extraction module. It uses a dynamic weighting algorithm to fuse and calculate the abnormal deviation degree, behavior trend prediction vector and physiological indicators, and outputs a comprehensive risk level and confidence assessment. The control output module is configured to control the multimodal alarm to perform early warning operations based on the comprehensive risk level output by the risk decision module, collect data after the early warning from the spatiotemporal feature tensor output by the feature extraction module as response data, and send the response data to the behavior analysis module for model optimization.

[0020] The driver behavior monitoring device based on multimodal visual recognition monitors the driver's state through the coordinated operation of physical and control devices. The physical devices include a multispectral imaging unit, an embedded computing platform, a multimodal alarm, and a vehicle bus interface module. The vehicle bus interface module connects to the vehicle control system via a CAN bus, periodically reading vehicle speed, lateral acceleration, and turn signal status data to form vehicle dynamic parameters. The multispectral imaging unit acquires a video stream of the driver's face according to imaging control commands, overcoming interference from lighting variations through multi-band infrared and visible light imaging. The embedded computing platform integrates the control device, and the multimodal alarm executes audible and visual alarm operations upon receiving a warning signal.

[0021] After the control unit's data acquisition module obtains vehicle dynamic parameters through the vehicle bus interface module, the parameter acquisition unit extracts real-time data such as vehicle speed and lateral acceleration from the CAN bus. The fuzzy inference unit calculates the vehicle dynamic parameters based on a preset fuzzy rule base, for example, mapping high-speed curve scenes to a high complexity level and outputting scene complexity assessment results. The imaging control unit dynamically adjusts the acquisition parameters of the multispectral imaging unit according to the scene complexity, such as increasing the sampling weight of the infrared band for backlight scenes. The preprocessing unit performs non-uniformity correction on the raw video stream to eliminate sensor noise and adds scene labels such as highway night driving, outputting a scene-labeled video stream to the feature extraction module.

[0022] After receiving the video stream with scene labels, the feature extraction module uses a spatial feature extraction unit to extract the 3D spatial coordinates of facial key points such as the corners of the eyes and mouth from a single frame image using a convolutional neural network. The temporal feature extraction unit uses a Transformer architecture to analyze motion features such as head rotation trajectories in a continuous frame sequence. The feature fusion unit uses a cross-attention mechanism to align and fuse spatial coordinates with motion trajectories, generating a spatiotemporal feature tensor that includes both static position and dynamic changes. This module also analyzes physiological indicators such as pupil diameter changes from the video stream, providing multidimensional data support for subsequent decision-making.

[0023] The behavior analysis module combines spatiotemporal feature tensors with vehicle dynamic parameters for analysis. The benchmark generation unit constructs a virtual driver model using a generative adversarial network (GAN) and generates standard behavior feature sequences based on scenario parameters such as straight-line driving. The deviation calculation unit uses the Mahalanobis distance algorithm to calculate the statistical difference between real-time spatiotemporal features and the standard sequences, quantifying the degree of abnormal deviation. The trend prediction unit analyzes historical behavior data through a long short-term memory (LSTM) network to predict the evolution trend of fatigue levels and outputs a behavior trend prediction vector. This multi-dimensional analysis can effectively distinguish between normal operations such as observing rearview mirrors and real risk states.

[0024] After receiving abnormal deviation, behavioral trend prediction vectors, and physiological indicators, the risk decision-making module integrates multi-source data into a parallel data stream. The entropy calculation unit analyzes the information entropy value of each indicator within a sliding time window; for example, it increases the entropy weight when gaze deviation data fluctuates drastically. The weight allocation unit dynamically allocates weight coefficients based on the entropy values, ensuring the system prioritizes head posture indicators in lane-changing scenarios. The fusion evaluation unit performs fusion calculations on the weighted indicators and, combined with the weight distribution dispersion, outputs a comprehensive risk level and confidence assessment, forming a quantitative decision-making basis.

[0025] The hierarchical early warning unit of the control output module determines the alarm intensity based on the combination of risk level and confidence level, such as triggering a high-frequency audible and visual alarm in high-risk, high-confidence scenarios. The response monitoring unit collects data after the early warning from the spatiotemporal feature tensor and monitors the time delay for behavioral characteristics to return to normal. The data feedback unit encapsulates the time delay and risk information into a structured data packet and sends it to the behavior analysis module. The model optimization unit adjusts the parameters of the generative adversarial network through an incremental learning algorithm, adapting the standard behavioral feature sequence to individual driver differences, forming a closed-loop optimization mechanism.

[0026] In complex scenarios such as hazardous chemical transportation, this system significantly reduces the misjudgment rate caused by environmental interference through multimodal data fusion and dynamic weighting strategies. For example, when driving on a curve, the system can combine vehicle lateral acceleration data with driver line-of-sight deviation characteristics to distinguish between active steering operations and distracted driving behavior, thereby improving the timeliness of warnings.

[0027] Specifically, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a data acquisition module comprising: The parameter acquisition unit is configured to periodically read vehicle speed, lateral acceleration and turn signal status data from the vehicle CAN bus, and output vehicle dynamic parameters to the fuzzy inference unit. The fuzzy inference unit is configured to receive vehicle dynamic parameters output by the parameter acquisition unit, input the vehicle dynamic parameters into a preset fuzzy rule base for calculation, and output the scene complexity evaluation result to the imaging control unit. The imaging control unit is configured to receive the scene complexity assessment result output by the fuzzy inference unit, generate imaging control commands based on the scene complexity assessment result, and send the imaging control commands to the multispectral imaging unit. The preprocessing unit is configured to receive the raw video stream acquired by the multispectral imaging unit, perform non-uniformity correction and scene label addition processing on the raw video stream, and output the video stream with scene labels to the feature extraction module.

[0028] The data acquisition module achieves environmentally adaptive data acquisition through a multi-level processing flow. The parameter acquisition unit reads real-time data such as vehicle speed, lateral acceleration, and turn signal status from the vehicle's CAN bus at a fixed sampling frequency to form standardized vehicle dynamic parameters. In hazardous chemical transportation scenarios, when the vehicle enters a curve at a high speed, the lateral acceleration data will increase significantly. At this time, the parameter acquisition unit will transmit the complete parameter set, including the turn signal activation status, to the fuzzy inference unit.

[0029] The fuzzy rule base built into the fuzzy inference unit defines the mapping relationships for various typical driving scenarios. The rule base converts continuous variables such as vehicle speed and lateral acceleration into fuzzy sets, and defuzzifies them using the centroid method to output a quantified scenario complexity assessment result. For example, when the system simultaneously detects high vehicle speed, large lateral acceleration, and the turn signal is not activated, the fuzzy inference unit will classify it as a high-risk curve scenario and output a high complexity score to the imaging control unit.

[0030] The imaging control unit dynamically configures the operating mode of the multispectral imaging unit based on scene complexity. In low-complexity scenes, such as driving at a constant speed in a straight line, a standard sampling frequency and visible light-dominated imaging strategy are adopted. When receiving high-complexity evaluation results, it switches to a high-frequency sampling mode and increases the weight of the near-infrared band. This adaptive mechanism is particularly crucial when driving on curves at night, effectively overcoming the impact of insufficient visible light on facial feature acquisition by enhancing infrared imaging.

[0031] The preprocessing unit uses a two-point correction method to correct the non-uniformity of the original video stream, eliminating fixed-pattern noise caused by sensor dark current. For different driving scenarios, the system adds structured tags to the video stream; for example, highways and strong lighting conditions are labeled "highway daytime scene," and urban roads and rainy weather are labeled "urban rainy daytime scene." The corrected and tagged video stream forms a standard data format with scene tags and is then sent to the feature extraction module for further processing.

[0032] Each processing unit achieves efficient data flow through a pipelined architecture. Vehicle dynamic parameters output by the parameter acquisition unit directly drive fuzzy inference decisions, while the scene complexity assessment results generated by the fuzzy inference unit determine the instruction generation strategy of the imaging control unit. This chain-like processing mechanism is particularly effective in tunnel entry and exit scenarios: when a vehicle suddenly enters a tunnel from a bright environment, the abrupt change in light intensity triggers a reassessment of scene complexity, prompting the imaging control unit to quickly adjust exposure parameters to ensure stable facial image acquisition quality. The preprocessing unit's final output, a scene-labeled video stream, provides a standardized data source with controllable quality and clear context for subsequent feature extraction modules.

[0033] Specifically, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a feature extraction module comprising: The spatial feature extraction unit is configured to receive the video stream with scene labels output by the data acquisition module, extract the spatial coordinates of facial key points in a single frame image through a convolutional neural network, and output the spatial coordinates to the feature fusion unit. The temporal feature extraction unit is configured to receive the video stream with scene labels output by the data acquisition module, analyze the motion trajectory of key points in the continuous frame sequence through the Transformer architecture, and output the motion trajectory to the feature fusion unit. The feature fusion unit is configured to receive the spatial coordinates output by the spatial feature extraction unit and the motion trajectory output by the temporal feature extraction unit, and fuse the spatial coordinates and motion trajectory through a cross-attention mechanism to output a spatiotemporal feature tensor to the behavior analysis module.

[0034] The feature extraction module receives the video stream with scene labels output by the data acquisition module and extracts spatial and temporal features through parallel processing. The spatial feature extraction unit uses a convolutional neural network architecture to process single-frame images, performs key point localization for the facial region, and outputs the three-dimensional spatial coordinates of feature points such as the corners of the eyes and mouth. In the scenario of hazardous chemical transport vehicles driving at night, when the multispectral imaging unit acquires the driver's facial image under strong light interference, the convolutional neural network extracts illumination-invariant features through multiple convolutional kernels, ensuring that the spatial coordinates can still be stably output to the feature fusion unit under alternating light and dark environments.

[0035] The temporal feature extraction unit analyzes continuous frame sequences based on the Transformer architecture to capture the motion trajectories of facial key points. The Transformer's self-attention mechanism can model long-distance dependencies; for example, during a driver's head rotation, the trajectory of gaze changes is calculated by analyzing the displacement of the pupil position in consecutive frames. This temporal analysis is particularly suitable for identifying progressive fatigue states. When the system processes long-distance driving video streams on highways, it can effectively capture subtle changes such as a gradual increase in blink frequency and output the motion trajectory to the feature fusion unit.

[0036] The feature fusion unit achieves deep fusion of spatial coordinates and motion trajectories through a cross-attention mechanism. This mechanism uses spatial coordinates as the query vector and the motion trajectory as the key-value pair for feature alignment, generating a spatiotemporal feature tensor that simultaneously includes static spatial information and dynamic temporal changes. Taking a driver's yawning action as an example, the spatial feature extraction unit locates the coordinates of key points on the mouth, while the temporal feature extraction unit captures the motion patterns of the mouth opening and closing. The resulting spatiotemporal feature tensor can represent both the mouth opening amplitude and the duration of the yawn.

[0037] The three processing units form a collaborative workflow. The spatial feature extraction unit and the temporal feature extraction unit process the same video stream in parallel, extracting complementary features from the spatial and temporal dimensions, respectively. The feature fusion unit eliminates feature ambiguity through a weighted fusion strategy. For example, when a driver naturally turns their head to check the rearview mirror while driving on a curve, the system can distinguish between active observation behavior and abnormal head movements through the spatiotemporal feature tensor. The final output spatiotemporal feature tensor is sent to the behavior analysis module to provide multi-dimensional feature support for subsequent risk decision-making.

[0038] The internal data flow of the module follows a logical hierarchy of feature extraction, from the raw video stream to primary feature extraction of spatial coordinates and motion trajectories, and then to advanced feature fusion of spatiotemporal feature tensors. This hierarchical processing mechanism exhibits strong robustness in complex driving scenarios. For example, when a vehicle passes through a tunnel and lighting conditions change abruptly, the scene-labeled video stream can prompt the feature extraction unit to adaptively adjust its feature extraction strategy, ensuring that the quality of the generated spatiotemporal feature tensors is not affected by environmental interference.

[0039] Specifically, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a behavior analysis module comprising: The baseline generation unit is configured to receive vehicle dynamic parameters provided by the data acquisition module, generate a standard behavioral feature sequence through a generative adversarial network, and output the standard behavioral feature sequence to the deviation calculation unit. The deviation calculation unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module and the standard behavioral feature sequence output by the benchmark generation unit, calculate the statistical difference between the spatiotemporal feature tensor and the standard behavioral feature sequence using the Mahalanobis distance algorithm, and output the abnormal deviation to the risk decision module. The trend prediction unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module, analyze the spatiotemporal feature sequence in historical behavior data through a long short-term memory network, and output the behavior trend prediction vector to the risk decision module.

[0040] The behavior analysis module achieves dynamic evaluation and prediction of driver behavior through multi-level processing. The baseline generation unit receives vehicle dynamic parameters from the data acquisition module and constructs a virtual driver model using a generative adversarial network (GAN). The generator in the GAN synthesizes standard behavioral feature sequences based on current driving scenario parameters, such as outputting ideal head posture and gaze direction sequences during smooth straight-line driving. The discriminator continuously optimizes the generator's output quality through adversarial training, ensuring that the standard sequences accurately reflect normal driving behavior patterns in specific scenarios. The generated sequences are then fed to the deviation calculation unit, providing a benchmark reference for subsequent anomaly detection.

[0041] The deviation calculation unit synchronously receives the spatiotemporal feature tensor output by the feature extraction module and the standard behavioral feature sequence provided by the benchmark generation unit. This unit uses the Mahalanobis distance algorithm to calculate the multidimensional statistical difference between the real-time features and the standard sequence. Mahalanobis distance effectively quantifies the degree of abnormal deviation by introducing a feature covariance matrix to eliminate correlation interference between indicators. Taking the nighttime driving scenario of hazardous chemical transport vehicles as an example, when the system detects that the driver's line of sight is continuously deviating from the road centerline, the Mahalanobis distance algorithm, combined with historical data distribution, calculates a deviation index significantly higher than the threshold. The calculated abnormal deviation is transmitted to the risk decision module in real time as a key input for risk level assessment.

[0042] The trend prediction unit analyzes spatiotemporal feature sequences in historical behavioral data based on a Long Short-Term Memory (LSTM) network architecture. The LTM network captures long-term dependencies in time series through gating mechanisms, such as identifying gradual changes in fatigue levels from several hours of driving data. The historical information carried in the network's hidden states is combined with the current spatiotemporal feature tensor to output a behavioral trend prediction vector. The vector dimension can represent indicators such as the probability of fatigue escalation and the trend of distraction. When a vehicle is en route on a long journey, the system can identify potential risks of decreased driver reaction speed through trend prediction, enabling early warning.

[0043] The three processing units form a progressive analysis chain. The behavioral benchmark established by the benchmark generation unit provides a normalized reference for deviation calculation, while the real-time anomaly indicators output by the deviation calculation unit complement the forward-looking analysis of the trend prediction unit. In complex scenarios such as meeting oncoming traffic on curves, this multi-dimensional analysis mechanism can effectively distinguish between proactive avoidance behavior and abnormal distraction states. For example, when the driver's gaze briefly shifts while the vehicle is turning normally, the system compares the Mahalanobis distance between the standard sequence and real-time features, combined with historical trend data, to determine a safe operation. The abnormal deviation and behavioral trend prediction vector ultimately output by the behavior analysis module provide quantitative basis for the risk decision-making module, constituting the core judgment logic of driver state monitoring.

[0044] Specifically, the risk decision module of the driver behavior monitoring device based on multimodal visual recognition described in this invention is configured as follows: The data organization unit is configured to receive the abnormal deviation degree and behavior trend prediction vector output by the behavior analysis module, and the physiological indicators parsed by the feature extraction module, and organize the abnormal deviation degree, behavior trend prediction vector and physiological indicators into a parallel data stream. The entropy calculation unit is configured to receive the parallel data stream output by the data organization unit and calculate the information entropy values ​​of the abnormal deviation, behavioral trend prediction vector and physiological indicators within a sliding time window. The weight allocation unit is configured to receive the information entropy value output by the entropy value calculation unit and dynamically allocate weight coefficients based on the information entropy value. The fusion assessment unit is configured to receive the weight coefficients output by the weight allocation unit and the parallel data stream output by the data organization unit, perform fusion calculations on the weighted indicators, and generate a comprehensive risk level and confidence assessment based on the dispersion of the weight distribution.

[0045] The risk decision-making module quantifies the driver's risk status through multi-level processing. The data organization unit receives the abnormal deviation and behavioral trend prediction vectors output by the behavior analysis module, and integrates physiological indicators such as pupil diameter changes and heart rate variability analyzed by the feature extraction module. This unit integrates multi-source heterogeneous data into a time-aligned parallel data stream. In the scenario of high-speed driving of hazardous chemical transport vehicles, when the system simultaneously detects slight steering wheel vibration and abnormal gaze drift, the data organization unit uses buffer management technology to ensure that the sampling time of each indicator is synchronized, forming a standardized data format that is then sent to the entropy calculation unit.

[0046] The entropy calculation unit employs a sliding time window mechanism to analyze the information entropy of parallel data streams. The window length is dynamically adjusted based on the driving scenario; for example, a longer time window is used to capture the cumulative effect of fatigue during long-distance highway driving. The unit calculates the information entropy of abnormal deviation, behavioral trend prediction vector, and physiological indicators within the window. When a driver experiences frequent yawning accompanied by arrhythmia, the entropy value of the yawning frequency data increases significantly, reflecting uncertain changes in physiological state. The entropy calculation results characterize the information content of each indicator in the current time period, providing a quantitative basis for weight allocation.

[0047] The weight allocation unit dynamically adjusts weights based on entropy calculations. Indicators with higher information entropy values ​​receive greater weight coefficients, allowing the system to prioritize risk signals with high volatility. For example, in a scenario where a hazardous materials transport vehicle encounters another vehicle at night, when the oncoming vehicle's glare causes the driver's pupils to constrict sharply, the entropy value of physiological indicators rises. The system automatically increases the weight of pupil change indicators while decreasing the weight of relatively stable head posture indicators. This adaptive mechanism ensures that risk assessment focuses on the most sensitive risk dimensions.

[0048] The fusion assessment unit receives the weighted indicator data stream and calculates the comprehensive risk value using a linear weighted fusion algorithm. The unit simultaneously analyzes the dispersion of the weight coefficients; higher dispersion results in a lower confidence assessment, indicating uncertainty in the decision-making process. In a cornering braking scenario, when the system detects abnormal driver foot movements but normal hand grip, the weight dispersion increases, and the fusion assessment unit outputs a medium risk level with a medium confidence level, providing a basis for early warning decisions. The final comprehensive risk level and confidence assessment are then sent to the control output module, completing the conversion from multi-source information to decision output.

[0049] Each processing unit forms a progressive analysis chain. Data integration by the data organization unit provides input for entropy calculation, and the entropy output drives the weight allocation strategy. The weighted data is finally fused and evaluated to generate a decision result. In complex scenarios such as continuous driving through tunnels, this pipeline architecture can adapt to environmental changes in real time. For example, when vehicles alternately enter and exit tunnels, causing frequent changes in lighting, the system maintains the stability of risk assessment through a dynamic reweighting mechanism, effectively reducing the probability of misjudgment.

[0050] Specifically, the driver behavior monitoring device based on multimodal visual recognition described in this invention includes a control output module comprising: The graded early warning unit is configured to receive the comprehensive risk level and confidence assessment output by the risk decision module, determine the alarm intensity level based on the combination relationship between the comprehensive risk level and the confidence assessment, and output the alarm control signal to the multimodal alarm. The response monitoring unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module, monitor the time delay of the behavior feature returning to normal state, and output the time delay data to the data feedback unit. The data feedback unit is configured to receive time delay data output by the response monitoring unit and comprehensive risk level and confidence assessment output by the risk decision module, encapsulate the time delay data and comprehensive risk level and confidence assessment into a structured data packet, and output the data packet to the behavior analysis module.

[0051] The control output module achieves closed-loop management of driver status intervention through a tiered early warning and feedback mechanism. The tiered early warning unit receives the comprehensive risk level and confidence assessment output from the risk decision module and determines the alarm intensity level through predefined mapping rules. For example, when the system detects a high risk level coupled with a high confidence assessment, a high-frequency audible and visual alarm mode is triggered; a medium risk level coupled with a low confidence assessment activates a mild visual alert. The alarm control signal is transmitted to the multimodal alarm. In scenarios involving fatigued driving in hazardous materials transport vehicles, the system dynamically adjusts the alarm frequency based on a quantitative score of fatigue level to avoid excessive interference with driver operation.

[0052] The response monitoring unit extracts post-warning data from the spatiotemporal feature tensor output by the feature extraction module, continuously tracking the recovery process of driver behavior characteristics. The unit calculates the time delay for behavioral characteristics to return to a normal state by comparing timestamps; for example, after a driver receives a distraction warning, the system records the time taken from a shift in gaze to refocusing on the road. This time delay data reflects the driver's response efficiency to the warning. In long-haul transportation tasks, the response monitoring unit identifies periodic changes in the driver's attention state by analyzing the time delay trends of multiple warning events.

[0053] The data feedback unit integrates the time delay data output by the response monitoring unit and the comprehensive risk level and confidence assessment provided by the risk decision-making module. The unit uses a standardized data format to encapsulate multi-source information into structured data packets, which include fields such as warning timestamps, risk indicators, and response delays. In continuous driving scenarios within tunnel complexes, when the system records multiple short delays accompanied by low-risk levels, the data feedback unit automatically marks them as valid warning cases and sends the data packets to the model optimization unit of the behavior analysis module.

[0054] The control output module forms a complete chain from warning execution to data feedback. The output of the hierarchical warning unit directly drives the multimodal alarm, the response monitoring unit collects changes in driver behavior in real time, and the data feedback unit completes information encapsulation and transmission. In complex scenarios such as passing oncoming traffic on curves, this design enables the system to adjust the alarm strategy based on real-time risks and optimize model parameters through feedback data. For example, when vehicles frequently pass through curves, the system adjusts the warning threshold by analyzing historical response data, reducing false alarms caused by normal steering operations.

[0055] The modules are tightly interconnected through data flow. The alarm decisions of the tiered early warning unit are based on risk assessment results, the data collection of the response monitoring unit depends on the timing of the early warning trigger, and the data feedback unit ensures complete information feedback. This structure is particularly effective during nighttime driving of hazardous materials. When the driver shows signs of drowsiness, the system wakes the driver with a progressive alarm and records the response data for subsequent model training, ultimately improving the adaptability and accuracy of the early warning system.

[0056] Specifically, the driver behavior monitoring device based on multimodal visual recognition described in this invention further includes a behavior analysis module that includes: The model optimization unit is configured to receive structured data packets sent by the control output module, extract feature patterns corresponding to early warning events from the structured data packets, adjust the generator parameters in the generative adversarial network through an incremental learning algorithm, and output the generative adversarial network parameters to the baseline generation unit.

[0057] The model optimization unit receives structured data packets from the control output module. These packets include multi-dimensional information such as the timestamp of the warning event, risk level, and driver response delay. In hazardous materials transportation scenarios, when the system repeatedly records drivers quickly recovering from warnings of visual deviation while driving on curves, the structured data packets mark these events as valid samples, providing a data foundation for model optimization. The data packets are transmitted in a standardized format to ensure compatibility with the data interface of the behavior analysis module.

[0058] When extracting feature patterns corresponding to warning events from structured data packets, the model optimization unit uses a pattern matching algorithm to identify key feature sequences, such as head posture change trajectories or gaze focus patterns in consecutive frames. Taking nighttime highway driving as an example, the system establishes a typical feature library of fatigue behavior by analyzing the feature pattern of drivers yawning accompanied by increased blinking frequency in multiple warning events. The feature extraction process is based on a time window sliding mechanism to capture the dynamic evolution of behavioral patterns.

[0059] The generator parameters in the generative adversarial network (GAN) are adjusted using an incremental learning algorithm. The model optimization unit employs an online learning mechanism, fine-tuning the generator weights only based on newly added warning event data, avoiding retraining with the entire dataset. For example, when new data indicates that a driver exhibits abnormal pupil constriction behavior under specific lighting conditions, the incremental learning algorithm dynamically adjusts the generator parameters of the GAN, making the virtual driver model more closely resemble the actual driving environment. The adjustment process uses a gradient descent optimization strategy, gradually converging to the optimal parameter space.

[0060] After the generative adversarial network (GAN) parameters are output to the baseline generation unit, the baseline generation unit uses the updated parameters to generate a more accurate sequence of standard behavioral features. In long-distance transportation tasks, this continuous optimization mechanism allows the system to gradually adapt to the driver's individual behavioral habits, such as head turning amplitude or gaze scanning frequency. Parameter updates trigger the baseline generation unit to resynthesize standard behavioral features, forming a closed-loop self-optimizing system, thereby reducing the false positive rate and improving the timeliness of early warnings.

[0061] Specifically, the driver behavior monitoring device based on multimodal visual recognition described in this invention further includes: The multispectral imaging unit is configured to receive imaging control commands sent by the data acquisition module, acquire driver video data according to the imaging control commands, and send the raw video stream to the preprocessing unit in the data acquisition module. The multimodal alarm is configured to receive the early warning control signal sent by the control output module and execute the audible and visual alarm operation according to the early warning control signal; The vehicle bus interface module is configured to read vehicle dynamic parameters from the vehicle CAN bus and send the vehicle dynamic parameters to the parameter acquisition unit in the data acquisition module.

[0062] After receiving imaging control commands from the data acquisition module, the multispectral imaging unit dynamically adjusts the video acquisition parameters according to the commands. The imaging control commands include configuration information such as frame rate, resolution, and spectral bands. The unit simultaneously acquires visible light and infrared video streams through multi-band sensors. In nighttime driving scenarios for hazardous materials transport vehicles, when the system detects a sudden change in lighting at a tunnel entrance, the imaging control commands instruct the unit to increase the weight of the infrared band to compensate for the inability to capture facial details in low-light conditions. The acquired raw video stream is transmitted to the preprocessing unit in the data acquisition module via a high-speed interface for subsequent correction processing. This adaptive acquisition mechanism effectively overcomes the impact of environmental changes on image quality.

[0063] The multimodal alarm receives warning control signals from the control output module and analyzes the alarm level and mode parameters within the signals. The unit executes differentiated alarm operations based on the risk level; for example, high risk triggers a high-frequency buzzer and flashing red warning lights, while medium risk activates intermittent voice prompts. In driver fatigue warning scenarios, the alarm uses a sound pressure level gradient algorithm to avoid sudden blaring that could cause secondary interference. The audible and visual alarm elements employ a combination of directional speakers and matrix LEDs to ensure sufficient visibility of the alarm signal in a noisy cockpit environment. Real-time status data is fed back to the control output module during execution, forming a closed-loop control system.

[0064] The vehicle bus interface module periodically scans the vehicle's CAN bus protocol frames to extract dynamic parameters such as vehicle speed and lateral acceleration. The module's built-in protocol parser identifies signal mapping relationships in the standard CAN database file, for example, converting byte streams in data frames with specific identifiers into physical quantities. In cornering scenarios, the interface module captures steering angle change data with millisecond-level latency and eliminates bus interference through checksum verification mechanisms. The parsed vehicle dynamic parameters are encapsulated in a unified format and transmitted to the parameter acquisition unit in the data acquisition module, providing a real-time data source for scenario evaluation.

[0065] Each physical unit collaborates via a hard real-time communication link. The video acquisition by the multispectral imaging unit and the parameter reading by the vehicle bus interface module are synchronized in time; for example, in high-speed lane-changing scenarios, facial image frames and steering wheel angle data are timestamped. The execution delay of the multimodal alarm is incorporated into the response time evaluation system. When the system detects an abnormality in the driver's pupil focusing time after an alarm, the feedback data is used to optimize the warning threshold. This integrated design improves the accuracy of status judgment through multi-source data synchronization in complex scenarios such as long downhill sections of hazardous materials transportation.

[0066] Specifically, the driver behavior monitoring device based on multimodal visual recognition described in this invention further includes: The control output module is configured to send the collected driver response data to the model optimization unit in the behavior analysis module; The behavior analysis module is configured to receive driver response data, adjust the parameters of the generated adversarial network through the model optimization unit, and update the standard behavior feature sequence. The risk decision-making module is configured to receive historical early warning effect data and adjust the weight coefficients in the weight allocation unit according to the early warning effect.

[0067] After completing the warning operation, the control output module collects the driver's response data from the spatiotemporal feature tensor. This data includes dynamic changes in behavioral characteristics after the warning is triggered, such as the time sequence of the driver's gaze refocusing on the road or the trajectory points where the head posture returns to normal. The collected response data is transmitted to the model optimization unit in the behavior analysis module via the embedded system's internal data bus. A timestamp alignment mechanism ensures data timing consistency during transmission. In long-distance driving scenarios for hazardous materials transport vehicles, when the system issues a distraction warning, the control output module records the complete behavioral sequence of the driver from gaze deviation to refocusing on the road, forming a time-stamped response data packet.

[0068] After receiving driver response data, the model optimization unit of the behavior analysis module first parses the feature patterns in the data packets, such as extracting the blink frequency recovery curve or the steering wheel grip stability index. The unit uses an incremental learning algorithm to fine-tune the generator parameters in the generative adversarial network, optimizing only some network layer weights through gradient descent to avoid retraining the entire model. After parameter adjustment, the baseline generation unit resynthesizes standard behavioral feature sequences using the new parameters, making the virtual driver model more closely resemble actual driving habits. Taking nighttime highway driving as an example, when the system repeatedly records effective responses from drivers quickly adjusting their seating posture after fatigue warnings, the model optimization unit gradually lowers the anomaly detection threshold for such scenarios, improving model adaptability.

[0069] The risk decision-making module periodically extracts historical warning effectiveness data from the system log library, including indicators such as warning trigger accuracy, false alarm count, and response delay statistics. The weight allocation unit dynamically adjusts the weight coefficients of each input indicator based on the warning effectiveness. For example, when historical data indicates a high false alarm rate for physiological indicators in curve scenarios, the unit automatically reduces the weight of the heart rate variability indicator. The adjustment strategy is based on the contribution evaluation of indicators within a sliding window, ensuring that the weight coefficients reflect the latest driving status. In continuous curve driving scenarios, the system analyzes recent warning effectiveness and prioritizes increasing the fusion weight of vehicle dynamic parameters and facial features to reduce decision-making bias caused by environmental interference.

[0070] The control output module, behavior analysis module, and risk decision-making module form a collaborative optimization closed loop through data flow. Feedback data from the control output module drives model updates in the behavior analysis module. The updated standard feature sequence influences the deviation calculation in the risk decision-making module, while the weight adjustments in the decision-making module further optimize the accuracy of subsequent warnings. In scenarios where hazardous chemical transport vehicles travel across regions, this closed-loop mechanism enables the system to adapt to different road conditions and individual driver differences. For example, it automatically switches the dominant evaluation indicators between mountain curves and flat highway sections, continuously improving the robustness and warning accuracy of the monitoring system.

[0071] Secondly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the functions of each module in the driver behavior monitoring device based on multimodal visual recognition as described above.

[0072] The driver behavior monitoring device based on multimodal visual recognition solves the problems of high false alarm rates and delayed warnings caused by the single monitoring method in existing technologies by working collaboratively with physical and control devices. The physical device includes a multispectral imaging unit, an embedded computing platform, a multimodal alarm, and a vehicle bus interface module. The multispectral imaging unit acquires driver video streams according to imaging control commands, adapting to changes in lighting conditions through multi-band infrared and visible light imaging; the vehicle bus interface module reads dynamic parameters such as vehicle speed and lateral acceleration in real time from the vehicle's CAN bus; the embedded computing platform integrates the control device; and the multimodal alarm executes warning operations. All components establish a closed-loop data flow through standardized interfaces.

[0073] After the data acquisition module of the control device obtains the vehicle's dynamic parameters through the vehicle bus interface module, the parameter acquisition unit periodically reads CAN bus data. The fuzzy inference unit calculates the scene complexity based on the fuzzy rule base. The imaging control unit generates imaging control commands to adjust the acquisition strategy of the multispectral imaging unit. The preprocessing unit performs non-uniformity correction on the original video stream and adds scene labels. In the scenario of hazardous chemical transport vehicles driving at night, when the system detects a sudden change in illumination at the tunnel entrance, the imaging control command increases the weight of the infrared band to compensate for the facial feature capture capability in low-light environments and outputs a video stream with scene labels to the feature extraction module.

[0074] The feature extraction module receives video streams with scene labels. The spatial feature extraction unit extracts the spatial coordinates of facial key points such as the corners of the eyes and mouth in a single frame image using a convolutional neural network. The temporal feature extraction unit uses a Transformer architecture to analyze the head motion trajectory in a continuous frame sequence. The feature fusion unit generates a spatiotemporal feature tensor through a cross-attention mechanism. This module simultaneously analyzes physiological indicators such as changes in pupil diameter. For example, during driving on a curve, the system integrates the head rotation trajectory with changes in gaze direction to form a feature representation that includes both static position and dynamic changes, and outputs the spatiotemporal feature tensor to the behavior analysis module.

[0075] The behavior analysis module's baseline generation unit constructs a virtual driver model using a generative adversarial network (GAN) and generates standard behavioral feature sequences based on vehicle dynamic parameters. The deviation calculation unit uses the Mahalanobis distance algorithm to calculate the statistical difference between the real-time spatiotemporal feature tensor and the standard sequence. The trend prediction unit analyzes historical data through a long short-term memory network (LSTM) to predict behavioral trends. In long-distance driving scenarios, when the system detects a gradual increase in blink frequency accompanied by gaze drift, the trend prediction unit outputs a fatigue aggravation vector, which, along with the abnormal indicators from the deviation calculation unit, is sent to the risk decision module.

[0076] The risk decision-making module's data organization unit integrates anomaly deviation, behavioral trend prediction vectors, and physiological indicators into a parallel data stream; the entropy calculation unit calculates the information entropy of each indicator within a sliding time window; the weight allocation unit dynamically adjusts the weight coefficients, prioritizing fluctuation signals with high entropy values; and the fusion assessment unit outputs a comprehensive risk level and confidence assessment. Taking a scenario of a hazardous chemical transport vehicle encountering strong light as an example, when the entropy value of pupil constriction data increases significantly, the system increases the weight of physiological indicators and decreases the weight of relatively stable head posture, generating a weighted risk assessment result.

[0077] The hierarchical early warning unit of the control output module triggers differentiated alarms based on a combination of risk level and confidence level. High-risk, high-confidence scenarios activate high-frequency audible and visual alarms. The response monitoring unit collects the behavioral recovery time delay after an early warning from the spatiotemporal feature tensor. The data feedback unit encapsulates the delay data and risk assessment results and sends them to the behavior analysis module. After the driver receives a distraction warning, the system records the time taken from gaze deviation to refocusing on the road. The model optimization unit incrementally adjusts the generated adversarial network parameters to adapt the standard behavioral feature sequence to individual driving habits, forming a closed-loop optimization.

[0078] This invention's device, through multimodal data fusion and real-time feedback mechanisms, effectively distinguishes between active steering operations and abnormal distraction states in complex environments such as continuous curved road sections used in hazardous chemical transportation. This significantly reduces false alarms caused by environmental interference and improves the timeliness of early warnings. The data flow between modules is tightly integrated, forming a self-optimizing system from parameter acquisition and feature extraction to decision output, meeting the patent law's requirements for a clear and complete specification.

[0079] This invention effectively addresses the high misjudgment rate and delayed early warning issues in existing driver status monitoring systems, which suffer from limited data sources and analytical dimensions, through a systematic design of multimodal data fusion and closed-loop optimization mechanisms. Specifically, the device integrates multispectral imaging units, vehicle bus interfaces, and other sensing devices at the physical level, and constructs a complete processing chain from data acquisition to risk decision-making at the control level, forming a monitoring closed loop with adaptive capabilities.

[0080] During the data acquisition phase, the system acquires vehicle dynamic parameters such as vehicle speed and lateral acceleration in real time through the vehicle bus interface module, while simultaneously using a multispectral imaging unit to acquire a video stream of the driver's face. The multispectral imaging unit can adaptively adjust its imaging mode according to lighting conditions, for example, enhancing infrared imaging capabilities in nighttime or tunnel scenarios, overcoming the poor recognition performance of a single visible light sensor in low-light environments. The acquired raw video stream undergoes non-uniformity correction and scene labeling to form standardized data with scene labels, providing a quality-controlled input for subsequent feature extraction.

[0081] The feature extraction module employs a parallel processing architecture with both spatial and temporal pathways. The spatial feature extraction unit locates the coordinates of key facial points using a convolutional neural network, while the temporal feature extraction unit analyzes motion trajectories in consecutive frames using a Transformer architecture. These are then fused into a spatiotemporal feature tensor through a cross-attention mechanism. This multi-dimensional feature extraction approach can simultaneously capture both the static spatial information and dynamic changes in the driver's facial posture, avoiding the limitations of single-dimensional feature analysis. For example, during cornering, the system can integrate head rotation angle and gaze deviation trajectory to distinguish between normal driver behavior of actively checking the rearview mirror and abnormal states of distraction.

[0082] The behavior analysis module incorporates a generative adversarial network to construct a virtual driver model, generating a standard behavioral feature sequence based on the current driving scenario as a comparison benchmark. The statistical deviation between real-time features and the standard sequence is calculated using the Mahalanobis distance algorithm, and behavioral trends are predicted using a long short-term memory network. In long-distance driving scenarios, the system can identify the accumulation process of fatigue by analyzing the gradual change pattern of blink frequency and the trend of steering wheel grip force, thus enabling early warning.

[0083] The risk decision-making module employs a dynamic weighted algorithm to fuse multi-source indicators. This module first calculates the information entropy value of each indicator to assess its volatility, and then dynamically assigns weight coefficients, allowing the system to prioritize risk signals with significant changes. For example, when the system detects abnormal fluctuations in pupil diameter accompanied by an increase in the coefficient of variation of heart rate, it automatically increases the weight of physiological indicators, forming a weighted comprehensive risk level and confidence assessment. This strategy effectively reduces false alarms caused by environmental interference; for instance, in glare-filled scenarios, the system can reduce false alarms of momentary squinting by adjusting the weights.

[0084] The control output module achieves coordinated hierarchical early warning and closed-loop optimization. Differential alarms are triggered based on a combination of risk level and confidence level: high-risk scenarios activate audible and visual alarms, while medium- and low-risk scenarios use visual cues. After an alert, the system continuously monitors driver response data, such as recording the recovery time from a shift in gaze to refocusing on the road. This data is encapsulated and fed back to the behavior analysis module. The model optimization unit adjusts the generative adversarial network parameters through incremental learning algorithms, gradually adapting standard behavioral feature sequences to individual driver habits, forming a continuously optimized closed-loop system.

[0085] In practical applications such as hazardous materials transportation, this device demonstrates significant advantages. For example, when vehicles continuously pass through tunnel complexes, the system can quickly adapt to changes in lighting through a multispectral imaging unit, combining vehicle dynamic parameters to determine the complexity of the driving scenario and effectively distinguish between physiological reactions caused by sudden changes in light within the tunnel and actual fatigue characteristics. Through multimodal data fusion and a closed-loop learning mechanism, the system maintains high robustness in complex environments, significantly reducing false alarm rates and improving the timeliness of early warnings. The closed-loop data flow formed between modules enables the system to continuously evolve, ultimately realizing a shift in driver status monitoring from passive response to proactive prevention.

[0086] This invention demonstrates the actual workflow of a driver behavior monitoring device based on multimodal visual recognition through specific application scenarios. In a scenario of a hazardous chemical transport vehicle traveling on a highway at night, as the vehicle is about to enter a tunnel complex, the system detects in real time through the vehicle bus interface module that the vehicle speed remains high and the lateral acceleration is stable. The parameter acquisition unit reads dynamic parameters from the CAN bus. A sudden change in light intensity at the tunnel entrance triggers the re-evaluation mechanism of the fuzzy inference unit. Based on a preset fuzzy rule base, the scene complexity is marked as high. The imaging control unit then sends an adjustment command to the multispectral imaging unit to increase the sampling weight of the infrared band to compensate for the low-light environment inside the tunnel. After the driver's facial video stream acquired by the multispectral imaging unit is corrected for non-uniformity by the preprocessing unit, a "nighttime tunnel scene" label is added, and the video stream with the scene label is output to the feature extraction module. The spatial feature extraction unit extracts the spatial coordinates of key points such as the corners of the eyes and mouth under low-light conditions through a convolutional neural network. The temporal feature extraction unit uses a Transformer architecture to analyze the continuous change trajectory of the head posture during tunnel travel. The feature fusion unit generates a spatiotemporal feature tensor that fuses static position and dynamic motion. The behavior analysis module's baseline generation unit generates a standard line-of-sight direction sequence based on straight tunnel driving parameters. The deviation calculation unit uses the Mahalanobis distance algorithm to detect that the driver's momentary squinting due to strong light adaptation deviates little from the standard sequence. The trend prediction unit, combined with historical data, determines this to be a normal physiological reaction, avoiding misreporting sudden changes in light intensity as fatigue. The risk decision module assigns low weights to physiological indicators, dynamically weights them, and outputs a low-risk level, preventing the control output module from triggering an alarm. This process demonstrates that the device effectively overcomes the misjudgment defects of a single visible light sensor under sudden changes in light intensity through multispectral adaptive imaging and multi-dimensional feature fusion.

[0087] In a fatigue monitoring scenario during long-distance hazardous chemical transport, the vehicle enters a straight highway section after driving continuously for several hours. The data acquisition module detects stable vehicle speed and near-zero lateral acceleration via the vehicle bus interface. The fuzzy inference unit assesses the scenario as low complexity, and the imaging control unit maintains the standard imaging mode. The feature extraction module parses features from the continuous video stream, revealing a gradual increase in the driver's blinking frequency and an increase in head nodding micro-movements. The spatiotemporal feature tensor displays the fatigue accumulation trend. The behavior analysis module compares the standard behavioral feature sequence generated by the generative adversarial network with real-time data. The Mahalanobis distance algorithm calculates a continuously increasing abnormal deviation, and the long short-term memory network predicts that fatigue is entering an acceleration phase based on data from the previous few hours. The risk decision module's data organization unit integrates deviation indicators, trend prediction vectors, and physiological data on pupil diameter changes. The entropy calculation unit detects a significant increase in the blinking frequency entropy, and the weight allocation unit dynamically increases its temporal weight. The fusion assessment unit outputs a high-risk level with high confidence, and the control output module's graded warning unit triggers a high-frequency audible and visual alarm. After the driver is awakened and adjusts their posture, the response monitoring unit records that the behavioral characteristics return to normal within a short period of time. The data feedback unit encapsulates the response delay data and sends it to the behavior analysis module. The model optimization unit adjusts the parameters of the generative adversarial network through incremental learning, making the standard sequence more closely match the driver's fatigue resistance characteristics, thus forming a closed-loop optimization. This embodiment demonstrates that the device achieves accurate early warning in the early stages of fatigue accumulation through long-term trend prediction and dynamic weighting strategies, avoiding the reaction lag problem of existing threshold methods in critical states.

[0088] In a scenario involving oncoming traffic on a series of bends in a mountainous area, a hazardous materials transport vehicle navigates an S-shaped bend at medium speed. The vehicle bus interface module reads the periodically changing lateral acceleration and turn signal activation signals. The parameter acquisition unit transmits dynamic parameters to the fuzzy inference unit, and the scene is marked as a series of high-risk bends. The multispectral imaging unit automatically enhances infrared imaging under backlight conditions, and the preprocessing unit labels it as a "mountainous bend backlight scene." The spatial feature extraction unit of the feature extraction module captures the spatial coordinates of the driver's natural head steering movements, the temporal feature extraction unit analyzes the regular scanning motion trajectory of the driver's gaze in the rearview mirror and ahead of the road, and the feature fusion unit generates a spatiotemporal feature tensor that fuses spatial steering and gaze scanning. The benchmark generation unit of the behavior analysis module generates a standard sequence including normal observation actions based on the bend parameters, the deviation calculation unit distinguishes between active steering operations and abnormal head movements using Mahalanobis distance, and the trend prediction unit confirms that the behavior conforms to a safe mode by combining historical bend data. The weight allocation unit of the risk decision module reduces the weight of head movement indicators, prioritizes abnormal gaze drift indicators, and the fusion evaluation unit outputs a low-risk conclusion. The device uses multimodal data cross-validation to eliminate false alarms from normal head movements performed by drivers to observe road conditions, significantly reducing the false alarm rate in complex road conditions. This processing flow demonstrates the advantages of spatiotemporal feature fusion and scene-adaptive weighting in distinguishing operational intentions.

[0089] In a mixed road condition scenario in a suburban area during dusk, a hazardous materials transport vehicle experiences frequent starts and stops and gradual changes in light. The vehicle bus interface module reads data showing drastic speed fluctuations and frequent turn signal switching, and the fuzzy inference unit assesses the scene complexity as a dynamic change level. The multispectral imaging unit simultaneously activates visible light and infrared imaging under twilight conditions, and the preprocessing unit adds a "dusk mixed road" scene label. The feature extraction module analyzes the physiological indicators of the driver briefly closing their eyes to rest while waiting at a red light at an intersection. The behavior analysis module generates a standard behavior sequence including reasonable rest intervals through a generative adversarial network, and the deviation calculation unit identifies the statistical difference between short-term eye closure and fatigue-induced ptosis. When the vehicle restarts, the trend prediction unit determines that the speed of attention recovery meets the safety threshold based on the behavior sequence. The risk decision module assigns time decay weights to short-term physiological indicators to avoid misjudging normal rest as sudden fatigue. The control output module uses gentle visual cues instead of acoustic alarms based on confidence assessment to reduce interference with urban driving. This embodiment demonstrates that the device achieves risk assessment in complex traffic environments through temporal context analysis and a weighted decay strategy, improving the accuracy of warnings and user experience.

[0090] This invention relates to a convolutional neural network (CNN), a deep learning model specifically designed for processing image data. It automatically extracts spatial features, such as edges and textures, through convolutional layers and is widely used in computer vision. In this invention, the CNN is integrated into the spatial feature extraction unit of the feature extraction module. It processes the scene-labeled video stream output from the data acquisition module, locating the three-dimensional spatial coordinates of key facial points of the driver, such as the corners of the eyes and mouth, from a single frame image. These coordinates are then used as basic spatial features and fed to the feature fusion unit, providing a static information foundation for subsequent temporal analysis.

[0091] The Transformer architecture employs a self-attention mechanism to process sequential data, enabling it to capture long-range dependencies and making it particularly suitable for analyzing global contextual information in time series. In the temporal feature extraction unit of the feature extraction module, the Transformer architecture parses continuous video frame sequences, tracks the motion trajectories of facial key points (such as head rotation or gaze changes), and outputs dynamic motion features to the feature fusion unit, forming complementary temporal information with the spatial coordinates extracted by the convolutional neural network.

[0092] Cross-attention is a feature fusion technique that aligns and integrates data from different modalities through query key-value interactions, improving the fusion effect of multi-source information. In the feature fusion unit, cross-attention uses the spatial coordinates generated by the convolutional neural network as the query vector and the motion trajectory output by the Transformer architecture as the key-value pair, performing feature cross-alignment to generate a spatiotemporal feature tensor that includes both static position and dynamic changes. This tensor is then fed into the behavior analysis module to provide a unified feature representation for calculating behavior deviation.

[0093] Generative Adversarial Networks (GANs) consist of a generator and a discriminator. They generate realistic data samples through adversarial training and are often used to simulate complex distributions. In the baseline generation unit of the behavior analysis module, the GAN receives vehicle dynamic parameters and generates a sequence of standard behavioral features of the virtual driver, such as an ideal gaze direction or head posture sequence, as a reference benchmark for normal driving behavior. This sequence is then output to the deviation calculation unit for subsequent real-time data comparison.

[0094] Mahalanobis distance is a statistical distance metric that considers the covariance structure of data, effectively eliminating interference from correlations between features, and is suitable for anomaly detection in multidimensional spaces. In the deviation calculation unit of the behavior analysis module, Mahalanobis distance calculates the statistical difference between the spatiotemporal feature tensor and the standard sequence provided by the generative adversarial network, quantifying the degree of deviation between real-time driving behavior and the ideal state, and outputting the anomaly deviation to the risk decision module as a key input for risk assessment.

[0095] Long Short-Term Memory (LSTM) networks are a variant of recurrent neural networks that memorize long-term historical information through gating mechanisms, making them adept at handling time series prediction tasks. In the trend prediction unit of the behavior analysis module, LTM networks analyze historical spatiotemporal feature tensor sequences to identify gradual patterns in fatigue or distraction behaviors, such as an increasing trend in blink frequency. They then output behavior trend prediction vectors to the risk decision-making module, providing forward-looking risk indicators.

[0096] The fuzzy inference algorithm, based on fuzzy logic theory, handles uncertainty and fuzzy concepts, mapping continuous inputs to quantized outputs through a rule base. In the fuzzy inference unit of the data acquisition module, the algorithm calculates scene complexity assessment results based on vehicle dynamic parameters such as speed and lateral acceleration using a preset fuzzy rule base, and outputs the results to the imaging control unit. This drives the multispectral imaging unit to adaptively adjust its acquisition strategy, for example, by increasing the weight of infrared imaging in low-light environments.

[0097] Information entropy calculation originates from information theory and measures the uncertainty of random variables; a higher entropy value indicates greater data volatility. In the entropy calculation unit of the risk decision-making module, the information entropy value is calculated within a sliding time window for abnormal deviation, behavioral trend prediction vector, and physiological indicators. The information content of each indicator is evaluated, and the entropy value result is output to the weight allocation unit to provide a quantitative basis for dynamic weighting.

[0098] The dynamic weighting algorithm adjusts weight coefficients based on the importance of real-time data, enabling the system to prioritize indicators that show significant changes. In the weight allocation unit of the risk decision module, the algorithm dynamically allocates weights based on the information entropy calculation results. For example, when the entropy value of a physiological indicator increases, its weight is automatically increased, forming a weighted data stream that is sent to the fusion evaluation unit to ensure that the risk assessment focuses on key risk signals.

[0099] Incremental learning algorithms allow models to gradually update parameters on new data, avoiding full retraining and improving system adaptability. In the model optimization unit of the behavior analysis module, the incremental learning algorithm processes the structured data packets fed back from the control output module, adjusts the generator parameters in the generative adversarial network, continuously optimizes the standard behavioral feature sequences, adapts to individual driver differences, and forms a closed-loop self-optimization mechanism.

Claims

1. A driver behavior monitoring device based on multi-modal visual recognition, characterized in that, The physical device and the control device are connected in communication; The physical device comprises a multi-spectral imaging unit, an embedded computing platform, a multi-modal alarm and a vehicle bus interface module; The control device is integrated in the embedded computing platform and comprises: A data acquisition module configured to acquire vehicle dynamic parameters through the vehicle bus interface module, generate imaging control instructions according to the vehicle dynamic parameters and send the imaging control instructions to the multi-spectral imaging unit, receive original video streams returned by the multi-spectral imaging unit, pre-process the original video streams and attach scene labels to the original video streams, and output video streams with scene labels; A feature extraction module configured to receive the video streams with scene labels output by the data acquisition module, extract spatial features and time sequence motion features of facial key points from the video streams, and analyze physiological indicators from the video streams, and fuse the spatial features and the time sequence motion features into a spatio-temporal feature tensor; A behavior analysis module configured to receive the spatio-temporal feature tensor output by the feature extraction module and the vehicle dynamic parameters provided by the data acquisition module, generate standard behavior features according to the vehicle dynamic parameters, calculate statistical deviation degrees of the spatio-temporal feature tensor and the standard behavior features, predict behavior trends based on historical behavior data, and output abnormal deviation degrees and behavior trend prediction vectors; A risk decision module configured to receive the abnormal deviation degrees and the behavior trend prediction vectors output by the behavior analysis module, and the physiological indicators analyzed by the feature extraction module, fuse and calculate the abnormal deviation degrees, the behavior trend prediction vectors and the physiological indicators by using a dynamic weighting algorithm, and output a comprehensive risk level and a confidence evaluation; A control output module configured to control the multi-modal alarm to perform a warning operation according to the comprehensive risk level output by the risk decision module, collect data after the warning from the spatio-temporal feature tensor output by the feature extraction module as response data, and send the response data to the behavior analysis module for model optimization.

2. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 1, wherein, The data acquisition module comprises: A parameter acquisition unit configured to periodically read vehicle speed, lateral acceleration and steering light state data from a vehicle CAN bus, and output vehicle dynamic parameters to a fuzzy reasoning unit; The fuzzy reasoning unit is configured to receive the vehicle dynamic parameters output by the parameter acquisition unit, input the vehicle dynamic parameters into a preset fuzzy rule library for calculation, and output a scene complexity evaluation result to an imaging control unit; The imaging control unit is configured to receive the scene complexity evaluation result output by the fuzzy reasoning unit, generate imaging control instructions according to the scene complexity evaluation result, and send the imaging control instructions to the multi-spectral imaging unit; The preprocessing unit is configured to receive the original video streams collected by the multi-spectral imaging unit, perform non-uniformity correction and scene label attachment processing on the original video streams, and output video streams with scene labels to the feature extraction module.

3. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 2, wherein, The feature extraction module comprises: The spatial feature extraction unit is configured to receive the video streams with scene labels output by the data acquisition module, extract spatial coordinates of facial key points in a single frame image through a convolutional neural network, and output the spatial coordinates to the feature fusion unit; The time feature extraction unit is configured to receive the video stream with scene labels output by the data acquisition module, analyze key point motion trajectories in a continuous frame sequence through a Transformer architecture, and output the motion trajectories to the feature fusion unit. The feature fusion unit is configured to receive the spatial coordinates output by the spatial feature extraction unit and the motion trajectories output by the time feature extraction unit, perform feature fusion on the spatial coordinates and the motion trajectories through a cross-attention mechanism, and output a spatiotemporal feature tensor to the behavior analysis module.

4. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 3, wherein, The behavior analysis module includes: The reference generation unit is configured to receive vehicle dynamic parameters provided by the data acquisition module, generate a standard behavior feature sequence through a generative adversarial network, and output the standard behavior feature sequence to the deviation calculation unit. The deviation calculation unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module and the standard behavior feature sequence output by the reference generation unit, calculate the statistical difference between the spatiotemporal feature tensor and the standard behavior feature sequence using a Mahalanobis distance algorithm, and output an abnormal deviation degree to the risk decision module. The trend prediction unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module, analyze the spatiotemporal feature sequence in historical behavior data through a long short-term memory network, and output a behavior trend prediction vector to the risk decision module.

5. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 4, wherein, The risk decision module is configured to: The data organization unit is configured to receive the abnormal deviation degree and the behavior trend prediction vector output by the behavior analysis module, receive physiological indicators parsed by the feature extraction module, and organize the abnormal deviation degree, the behavior trend prediction vector, and the physiological indicators into a parallel data stream. The entropy value calculation unit is configured to receive the parallel data stream output by the data organization unit, calculate the information entropy values of the abnormal deviation degree, the behavior trend prediction vector, and the physiological indicators within a sliding time window. The weight allocation unit is configured to receive the information entropy values output by the entropy value calculation unit and dynamically allocate weight coefficients according to the information entropy values. The fusion evaluation unit is configured to receive the weight coefficients output by the weight allocation unit and the parallel data stream output by the data organization unit, perform fusion calculation on the weighted indicators, and generate a comprehensive risk level and a confidence evaluation according to the weight distribution dispersion.

6. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 5, wherein, The control output module includes: The hierarchical warning unit is configured to receive the comprehensive risk level and the confidence evaluation output by the risk decision module, determine an alarm intensity level according to the combination relationship of the comprehensive risk level and the confidence evaluation, and output an alarm control signal to the multi-modal alarm. The response monitoring unit is configured to receive the spatiotemporal feature tensor output by the feature extraction module, monitor the time delay for the behavior features to return to a normal state, and output time delay data to the data feedback unit. The data feedback unit is configured to receive the time delay data output by the response monitoring unit and the comprehensive risk level and the confidence evaluation output by the risk decision module, encapsulate the time delay data and the comprehensive risk level and the confidence evaluation into a structured data packet, and output the data packet to the behavior analysis module.

7. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 6, wherein, The behavior analysis module further includes: The model optimization unit is configured to receive structured data packets sent by the control output module, extract feature patterns corresponding to early warning events from the structured data packets, adjust the generator parameters in the generative adversarial network through an incremental learning algorithm, and output the generative adversarial network parameters to the baseline generation unit.

8. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 7, wherein, Also includes: The multispectral imaging unit is configured to receive imaging control commands sent by the data acquisition module, acquire driver video data according to the imaging control commands, and send the raw video stream to the preprocessing unit in the data acquisition module. The multimodal alarm is configured to receive the early warning control signal sent by the control output module and execute the audible and visual alarm operation according to the early warning control signal; The vehicle bus interface module is configured to read vehicle dynamic parameters from the vehicle CAN bus and send the vehicle dynamic parameters to the parameter acquisition unit in the data acquisition module.

9. The driver behavior monitoring device based on multi-modal visual recognition as claimed in claim 8, wherein, Also includes: The control output module is configured to send the collected driver response data to the model optimization unit in the behavior analysis module; The behavior analysis module is configured to receive driver response data, adjust the parameters of the generated adversarial network through the model optimization unit, and update the standard behavior feature sequence. The risk decision-making module is configured to receive historical early warning effect data and adjust the weight coefficients in the weight allocation unit according to the early warning effect.

10. An electronic device, comprising: It includes a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the functions of each module in the driver behavior monitoring device based on multimodal visual recognition as described in any one of claims 1 to 9.

Images