Driver attention evaluation method based on multi-modal feature fusion and physiological monitoring

Abstract

This invention discloses a driver attention assessment method based on multimodal feature fusion and physiological monitoring, belonging to the field of intelligent driving assistance technology. The method includes: acquiring facial images of the driver, real-time road condition images, and chassis dynamics signals; generating a full-face image of the driver, an rPPG signal extraction area, and a local eye image based on the facial image; extracting multimodal features based on the acquired and extracted data; constructing a multimodal spatiotemporal fusion assessment model, inputting multimodal features, and outputting a distraction level; and implementing graded intervention based on the distraction level. The method seamlessly integrates physiological and behavioral indicators, effectively solving the problem of unseen detection. By introducing rPPG technology, even when the driver is wearing sunglasses or their face is partially obscured, physiological signals can still assist in judging their state. Combining environmental context, it distinguishes between valid and invalid observations, significantly reducing false alarms.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving assistance technology, and in particular to a method for assessing driver attention based on multimodal feature fusion and physiological monitoring. Background Technology

[0002] With the development of intelligent driving technology, driver monitoring systems (DMS) have become a standard feature for vehicle safety.

[0003] Existing distraction detection technologies can be mainly categorized as follows: 1) Vehicle behavior-based methods: These judge based on data such as steering wheel angle, lane departure, and speed changes. The drawback is strong lag; detection only occurs when the vehicle is in an abnormal state, making early warning impossible. 2) Traditional computer vision-based methods: These capture the driver's face using a camera and calculate PERCLOS (percentage of eye closure), blink frequency, or head posture. The drawback is susceptibility to lighting conditions (e.g., nighttime, tunnels) and obstructions (e.g., sunglasses, masks), and difficulty in detecting the driver's eye characteristics (cognitive distraction). 3) Contact-based physiological detection: This requires the driver to wear a wristband or headband to detect electroencephalography (EEG) or electrical conductance analysis (EDA). The drawback is strong invasiveness, poor user experience, and unsuitability for daily driving.

[0004] Problems existing in the prior art include: 1. Inability to effectively detect cognitive distraction: Existing vision solutions can only detect whether a driver is looking at the road, but cannot determine whether the driver is distracted. For example, a driver's eyes are fixed on the road, but their mind is thinking about other things; this dangerous state (cognitive tunneling effect) is currently difficult to identify.

[0005] 2. Lack of contextual information leads to a high false alarm rate: Traditional eye-tracking does not consider road conditions. For example, checking the rearview mirror while changing lanes is a safe behavior, but traditional systems may misinterpret it as distraction; while looking down at a mobile phone while driving at high speed is a high-risk behavior. There is a lack of correlation analysis between driving scenarios and eye-tracking behavior.

[0006] 3. Insufficient robustness: Under complex lighting conditions (backlighting, uneven lighting) and facial occlusion, single feature extraction is prone to failure. Summary of the Invention

[0007] The purpose of this invention is to provide a driver attention assessment method based on multimodal feature fusion and physiological monitoring, aiming to solve or improve at least one of the above-mentioned technical problems.

[0008] To achieve the above objectives, the present invention provides the following solution: Driver attention assessment methods based on multimodal feature fusion and physiological monitoring include: Collect facial images of the driver, real-time road condition images, and chassis dynamics signals; Based on facial images, generate a full-face image of the driver, an rPPG signal extraction area, and a local eye image; Multimodal feature extraction is performed based on the collected and extracted data; Construct a multimodal spatiotemporal fusion evaluation model, inputting multimodal features and outputting the distraction level; Interventions are tiered based on the level of distraction.

[0009] Furthermore, facial images are captured by a near-infrared narrow-band DMS camera deployed inside the A-pillar or in the steering column cover of the vehicle.

[0010] Furthermore, based on the facial image, a full-face image of the driver, an rPPG signal extraction region, and a local eye map are generated, including: Based on facial images, a deep neural network is used for face detection, generating facial bounding boxes, extracting facial features, and determining facial geometric topological relationships. Based on the facial bounding box, the pixel margins are extended outward for adaptive cropping to obtain a full-face image that includes the complete head pose and hand interaction area; Based on facial geometric topology, specific areas of the forehead center and cheeks are dynamically identified as rPPG signal extraction areas. Based on facial geometric topology, the eye contour is extracted and the pixel margin is expanded outward to crop and generate a local image of the eye.

[0011] Furthermore, based on facial images, a deep neural network is used for face detection, generating facial bounding boxes, extracting facial features, and determining facial geometric topological relationships, including: A contrast-limited adaptive histogram equalization algorithm is used for facial images, and Gaussian filtering is introduced for noise reduction. Based on the denoised facial image, a face detection model is used to detect faces and generate facial bounding boxes; Based on the face detection model, a face detection model is loaded, multiple facial features are regressed, and the geometric topological relationships of the face are determined.

[0012] Furthermore, multimodal features include: explicit visual behavioral features, implicit physiological features, environmental context features, and vehicle operation features.

[0013] Furthermore, the extraction process for explicit visual behavioral features, including the driver's head Euler angles, body movements, and gaze placement, is as follows: Based on the full-face image and local eye image, a Backbone network with shared weights is used to extract full-face feature maps and eye feature maps; The Euler angles of the driver's head are calculated based on the full-face feature map; The full-face feature map is input into the YOLOv8-nano model and the skeletal keypoint detection network respectively to detect target objects and recognize limb movements; A 3D geometric model of the vehicle interior is constructed. Based on the eye feature map, a 3D gaze vector is output through L2CS-Net. The 3D gaze vector is then mapped to the vehicle interior coordinates to determine the gaze point.

[0014] Furthermore, the extraction process for hidden physiological characteristics is as follows: Based on the image information extracted from the rPPG signal region, the average pixel value is calculated to generate the original spatiotemporal signal sequence; A planar orthogonal projection algorithm is used to denoise the spatiotemporal signal sequence and generate a clean signal. A bandpass filter is applied to the clean signal to generate rPPG waveform data; Perform a fast Fourier transform on the rPPG waveform data, obtain the frequency corresponding to the energy peak through spectrum analysis, and calculate the real-time heart rate; The peak interval RR of rPPG waveform data is extracted. Based on the RR interval sequence, the standard deviation SDNN and the root mean square of the difference between adjacent intervals RMSSD are calculated.

[0015] Furthermore, the environmental context features extraction process is as follows: A saliency detection network is used to generate a saliency map based on real-time traffic images; The driver's line of sight is mapped to the two-dimensional coordinate system of the real-time traffic image through coordinate transformation, and the spatial overlap between the line of sight and the highly saliency area in the saliency map is calculated.

[0016] Furthermore, the vehicle operation characteristics extraction process is as follows: Time-domain characteristics: Using a sliding window to statistically analyze time-domain data, the standard deviation and entropy of the vehicle steering wheel angle are calculated; the variances of lateral acceleration and yaw rate are also calculated. Frequency domain characteristics: Perform a fast Fourier transform on the steering wheel angle signal to generate frequency domain data, extract the energy proportion of specific frequency bands, and identify the control lag or high-frequency jitter characteristics caused by increased driver cognitive load.

[0017] Furthermore, the multimodal spatiotemporal fusion evaluation model includes: a spatial feature extraction module, a temporal dynamic modeling module, and an adaptive fusion module; The spatial feature extraction module performs spatial mapping on the input features of the four modalities to generate spatial features, expressed as follows: ; In the formula, Spatial features; It is the ReLU activation function; The weight matrix is ​​a learnable weight matrix; For input features; For the corresponding mode The bias term; For time step; The temporal dynamic modeling module, based on spatial characteristics, adopts a multi-scale temporal downsampling strategy and generates temporal features through cascaded max pooling operations; The adaptive fusion module fuses the temporal features of each modality and hides the features; The output layer generates a prediction of cognitive load level based on the hidden features.

[0018] Furthermore, the temporal features of each modality are fused and the features are hidden, including: Calculate the correlation weights between the two modal features to generate weighted features, expressed as follows: ; ; ; In the formula, For target mode The query matrix; Source mode The key matrix; Source mode The value matrix; , , The linear projection weight matrix for the query, key, and value; Let be the temporal characteristics of the j-th mode; Score for attention; Weighted features; The weighted features are concatenated and then fused and mapped through a fully connected layer to generate fused features, expressed as: ; ; ; In the formula, Features of splicing; Let i be the temporal feature of the i-th mode; For feature splicing operations; This is a joint representation of the splicing features; Features of fusion; For activation functions; This is the weight matrix for the fusion mapping; The bias vector for the fusion mapping; The dimension of the fusion feature; The fused features are non-linearly transformed through two fully connected layers, with batch normalization and Dropout regularization introduced in between, to generate hidden features, expressed as follows: ; ; In the formula, These are primary hidden features; For the final hidden features; and This is the weight matrix of the fully connected layer; and This is the bias vector of the fully connected layer; This represents the probability of dropping items in Dropout.

[0019] Furthermore, generating predictions of cognitive load levels based on hidden features includes: For binary classification, the expression is: ; In the formula, This is a binary classification output; The weight vector of the output layer; For the bias term of the output layer; To hide features; The three-class classification is expressed as follows: , ; In the formula, Output in three categories; This is the weight vector of the output layer.

[0020] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention discloses a driver attention assessment method based on multimodal feature fusion and physiological monitoring. The method seamlessly integrates physiological and behavioral indicators, effectively solving the problem of detection that is invisible to the naked eye.

[0021] By introducing rPPG technology, physiological signals can still be used to help determine the driver's condition even when the driver is wearing sunglasses or has part of their face obscured.

[0022] By combining environmental context, valid and invalid observations can be distinguished, significantly reducing false alarms. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the rPPG signal processing flow in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the mapping between the line of sight and the salience of the road surface in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the multimodal spatiotemporal fusion evaluation model in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The purpose of this invention is to provide a driver attention assessment method based on multimodal feature fusion and physiological monitoring, aiming to solve or improve at least one of the above-mentioned technical problems.

[0027] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 As shown, this invention provides a driver attention assessment method based on multimodal feature fusion and physiological monitoring, including: Step 1: Acquire facial images of the driver, real-time road condition images, and chassis dynamic signals, including: Near-infrared (NIR) narrow-band DMS cameras are deployed inside the A-pillar or in the steering column cover of the vehicle to capture facial images of the driver.

[0029] In this embodiment, the hardware parameters are set as follows: resolution no less than 1280×720 (720p), frame rate stable between 30fps and 60fps, and lens field of view (FOV) set to 60°±5° horizontally and 40°±5° vertically to cover the driver's free head movement range. Active infrared illuminators with wavelengths of 850nm or 940nm are activated to ensure effective penetration of dark sunglasses and elimination of facial shadows in strong backlight, tunnel dim lighting, and complete darkness at night, thereby acquiring high signal-to-noise ratio images of the eyes and facial textures.

[0030] Simultaneously, the forward-looking ADAS wide-angle camera is invoked to collect real-time road condition images of the road ahead of the vehicle.

[0031] Vehicle chassis dynamics signals are read in real time via the vehicle CAN bus or FlexRay interface at a frequency of no less than 100Hz.

[0032] Chassis dynamics signals include: vehicle speed, steering wheel angle, yaw rate, brake pedal position, and turn signal activation status. All acquired timing data is timestamped and time-aligned with facial images and real-time road condition images to eliminate transmission delays between multiple sensors.

[0033] Step 2: Based on the facial image, generate the driver's full-face image, rPPG signal extraction area, and local eye image, including: Image enhancement is performed on facial images, face detection is performed using a deep neural network, facial bounding boxes are generated, facial features are extracted, and facial geometric topological relationships are determined, including: The Contrast-Limited Adaptive Histogram Equalization (CLAHE) algorithm was applied to the facial images, with a cropping limit of 2.0 and a grid size of 8×8, to eliminate local overexposure or underexposure caused by complex lighting inside the vehicle, significantly enhancing the contrast of facial contours and skin texture. Gaussian filtering was also introduced for noise reduction, smoothing the image while preserving edge details.

[0034] Based on the denoised facial images, RetinaFace or MTCNN deep neural networks are used for face detection to generate facial bounding boxes.

[0035] In this embodiment, the confidence threshold is set to 0.85 and the non-maximum suppression (NMS) threshold is set to 0.4 to filter out false background detections.

[0036] Based on the face detection model, a facial detection model (such as HRNet or PFLD) is loaded to regress multiple facial features and determine the geometric topological relationships of the face. These facial features include eyebrows, eye contours, nose tip, corners of the mouth, and jawline.

[0037] Based on the facial bounding box, the pixel margin is extended outward by 15%-20% for adaptive cropping to obtain a full-face image containing the complete head posture and hand interaction area, which is used to identify macroscopic distraction behaviors such as smoking, making phone calls, and gaze deviation.

[0038] Based on facial geometric topology, specific areas of the forehead center and cheeks are dynamically identified as rPPG (remote photoplethysmography) signal extraction areas.

[0039] In this embodiment, the extraction process should avoid interference from facial muscle movements, such as eyebrows and high-reflectivity areas on the face, so that physiological characteristic signals reflecting the driver's heart rate and heart rate variability (HRV) can be separated from the subtle color changes in the area.

[0040] Based on facial geometric topology, the eye contour is extracted and extended outward by 15%-20% of the pixel margin to generate a local eye image.

[0041] Step 3: Based on the full-face image, rPPG signal extraction area, eye local image, real-time road condition image, and chassis dynamics signal, perform multimodal feature extraction, including: Extract explicit visual behavioral features, including: Based on full-face images and local eye images, a Backbone network with shared weights (such as MobileNetV3 or EfficientNet) is used to extract full-face feature maps and eye feature maps; Based on the full-face feature map, the Euler angles (Yaw, Pitch, Roll) of the driver's head are calculated using the PnP (Perspective-n-Point) algorithm or a direct regression network. In this embodiment, if Yaw > a threshold (e.g., 30 degrees) for 2 seconds, it is determined that the driver is not looking at the area in front.

[0042] The full-face feature map is input into the YOLOv8-nano model and the SkeletonDetection network respectively to detect target objects, such as mobile phones, cigarettes, and horizontal objects. At the same time, the coordinates of human joints are located to recognize limb movements, such as taking the hand off the steering wheel or holding a phone.

[0043] Construct a 3D geometric model of the vehicle interior. Based on the eye feature map, output a 3D gaze vector through L2CS-Net. Map the 3D gaze vector to the vehicle interior coordinates to determine the gaze landing point, such as the instrument panel, central control screen, rearview mirror, left and right windows or windshield area.

[0044] Recessive physiological feature extraction (rPPG Physiological Features) includes: The absorption rate of specific wavelengths of light by hemoglobin in human blood varies periodically with heartbeats, causing subtle photometric fluctuations in the intensity of light reflected from facial skin. These fluctuations, captured by a camera, can be converted into photoplethysmography (rPPG) signals. When a driver's cognitive load increases (such as when thinking about complex problems), the autonomic nervous system regulates, leading to a decrease in heart rate variability (HRV). Therefore, by monitoring mutations in HRV, it is possible to effectively determine whether a driver is experiencing cognitive distraction.

[0045] Based on the image information extracted from the rPPG signal region, the average pixel value is calculated to generate the original spatiotemporal signal sequence; To address noise interference introduced by driver head movements, an algorithm based on Plane Orthogonal Projection (POS) is used to denoise the spatiotemporal signal sequence, generating a clean signal, including: Construct a projection plane to project the RGB / IR channel signals onto an orthogonal plane, eliminating motion artifacts. The expression is: In the formula, This is the pure signal after projection; and These are the chroma signals for different channels; The optimal projection coefficients are calculated based on the projection plane. For the pure signal, a bandpass filter is applied with a passband range of 0.7Hz-4Hz, corresponding to a physiological heart rate of 42-240 bpm, to generate rPPG waveform data; Perform a Fast Fourier Transform (FFT) on the rPPG waveform data, obtain the frequency corresponding to the energy peak through spectrum analysis, and calculate the real-time heart rate (HR).

[0046] Peak intervals (RR) are extracted from rPPG waveform data. Based on the RR interval sequence, the standard deviation (SDNN) and the root mean square (RMSSD) of the difference between adjacent intervals are calculated. This is used to quantify heart rate variability and thus assess the driver's cognitive load and distractibility.

[0047] Extracting environmental context features, including: Using real-time road condition images and employing a saliency detection network (such as U2-Net), a saliency map is generated, marking visually attractive areas in the current driving environment. These areas typically correspond to high-risk targets, such as pedestrians suddenly appearing, non-motorized vehicles, brake lights of vehicles ahead illuminating, or road obstacles.

[0048] The driver's gaze point is mapped to a two-dimensional coordinate system of the real-time traffic image through coordinate transformation. The spatial overlap between the gaze point and the highly salient region in the saliency map is calculated as a quantitative indicator of attention concentration.

[0049] If a highly salient target is detected on the road surface, but the point of view is deviated from the salient area, it is judged as a high-risk distraction state.

[0050] Based on chassis dynamics signals, vehicle handling characteristics are extracted, including: Temporal characteristics: Using sliding window statistical time-domain data, the standard deviation and entropy of the vehicle steering wheel angle are calculated to quantify the frequency and disorder of the driver's small corrections during lane keeping; the variance of lateral acceleration and yaw rate are calculated to evaluate the stability of the vehicle's lateral motion.

[0051] Frequency domain characteristics: Perform a fast Fourier transform (FFT) on the steering wheel angle signal to generate frequency domain data, extract the energy proportion of a specific frequency band (such as 0.5Hz-2Hz), and identify the control lag or high-frequency jitter characteristics caused by increased driver cognitive load.

[0052] By combining time-domain and frequency-domain features, vehicle operation features are generated.

[0053] Step 4: Construct a multimodal spatiotemporal fusion evaluation model, inputting multimodal features and outputting distraction levels, including: Set a sliding window and construct a sequence of feature vectors based on features from multiple models. The expression is as follows: In the formula, It is a dominant visual feature; Head posture features; Characteristics of body movements; These are latent physiological characteristics; Vehicle operating characteristics; This refers to environmental context features.

[0054] The multimodal spatiotemporal fusion evaluation model adopts a modular architecture, including: a spatial feature extraction module, a temporal dynamic modeling module, and an adaptive fusion module; The spatial feature extraction module mines spatial distribution patterns closely related to cognitive states from multi-channel physiological signals. Just as... Figure 4 As shown, the input features for four modalities (explicit visual behavioral features, implicit physiological features, environmental context features, and vehicle operation features) are presented. ,in Let m be the number of channels and T be the number of time steps. Spatial mapping is performed to generate spatial features, expressed as: ; In the formula, Spatial features; It is the ReLU activation function; For mapping input features to A learnable weight matrix in 3D space; For input features; For the corresponding mode The bias term; For time step.

[0055] The temporal dynamic modeling module adopts a multi-scale time downsampling strategy, which gradually compresses the time dimension through cascaded max pooling operations (pooling size = 4) while retaining key time patterns to generate temporal features.

[0056] At the feature representation level, a multi-head self-attention mechanism (2 heads, key dimension = 64) is introduced, enabling the model to autonomously learn the dependencies between different time points.

[0057] The expression for the time-series dynamic modeling module is: ; In the formula, It is a time-series feature; It is a time-series encoder that includes cascaded max pooling and multi-head self-attention mechanisms; In this embodiment, to output the time step, .

[0058] This design abandons the sequential computation mode of traditional RNNs / LSTMs, and significantly improves computational efficiency while maintaining the ability to model long sequences, making it more suitable for the real-time needs of actual vehicle environments.

[0059] The adaptive fusion module undertakes the key tasks of multimodal feature fusion and decision generation. After temporal modeling, this module extracts the received feature sequence from the spatial feature module. The temporal features of the four modalities are denoted as... The process is as follows: First, the cross-modal attention layer calculates the correlation weights between the two modal features for any target modality. Source mode Calculate cross-modal attention weights to achieve adaptive alignment and weighting, generating weighted features, expressed as: ; ; ; In the formula, For target mode The query matrix; Source mode The key matrix; Source mode The value matrix; , , The linear projection weight matrix for the query, key, and value; Let be the temporal characteristics of the j-th mode; The attention score represents the source modality. For target mode Attention weight distribution; For weighted features, source mode After being weighted according to attention weights, it is used to assist the target modality. Its characteristics.

[0060] The weighted features are concatenated and then fused and mapped through a fully connected layer to generate a fused feature, which serves as a unified joint representation of the four modalities. The expression is as follows: ; ; ; In the formula, Features of splicing; Let i be the temporal feature of the i-th mode; For feature splicing operations; This is a joint representation of the splicing features; Features of fusion; For activation functions; This is the weight matrix for the fusion mapping; The bias vector for the fusion mapping; The dimension of the fusion feature; The fused features are non-linearly transformed through two fully connected layers, with batch normalization and Dropout regularization introduced in between, to generate hidden features, expressed as follows: ; ; In the formula, These are primary hidden features; For the final hidden features; and This is the weight matrix of the fully connected layer; and This is the bias vector of the fully connected layer; This represents the probability of dropping data in Dropout.

[0061] Based on hidden features, either the Sigmoid activation function (for binary classification) or the Softmax activation function (for tri-class classification) is used to generate predictions of cognitive load levels, including: The expression for binary classification (high / low cognitive load) is: ; In the formula, This is a binary classification output; The weight vector of the output layer; For the bias term of the output layer; The expression for the three categories (low / medium / high load) is: , ; In the formula, Output in three categories; This is the weight vector of the output layer.

[0062] Step 5: Implement graded interventions based on the level of distraction, including: The dashboard icons flash as a visual distraction. When cognitive distraction occurs, the seat vibrates or makes an audible alert, "Please be aware of what's ahead," and the sensitivity of AEB (Automatic Emergency Braking) is increased. In extremely dangerous situations, a high-frequency alarm will sound and trigger the vehicle's ADAS system (such as pre-charged brakes and seatbelt tightening).

[0063] To verify the beneficial effects of the present invention, the method of the present invention was compared with the prior art, and the results are shown in Table 1.

[0064] According to the data in Table 1, the multimodal spatiotemporal fusion evaluation algorithm of this invention exhibits a significant performance advantage in driver distraction detection. Its accuracy reaches 97.19%, precision reaches 97.33%, recall reaches 97.19%, and AUC is 0.9738. All four core indicators are at a high level, while the accuracy of all comparative methods (including traditional machine learning and deep learning models such as Random Forest, LightGBM, LSTM, and CNN) does not exceed 86%. This result suggests that the algorithm achieves a relatively balanced performance between precision and recall: higher precision means a lower risk of false positives, potentially reducing unnecessary intervention in normal driving; higher recall indicates a smaller proportion of missed detections of real distraction events, helping to alleviate the common "blind spotting" problem in existing technologies. In contrast, better-performing comparative models such as LSTM have a recall of approximately 74%, and CNN-LSTM has approximately 62%, which may result in a higher false negative rate in practical applications. In summary, by fusing rPPG physiological signals, facial behavior, road condition information, and vehicle dynamics in a multimodal spatiotemporal manner, the model may be able to more effectively distinguish between effective and ineffective observations in complex scenarios such as sunglasses occlusion and changes in lighting, thereby possessing a certain degree of accuracy and robustness in assessing driving attention.

[0065] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0066] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A driver attention evaluation method based on multi-modal feature fusion and physiological monitoring, characterized in that, include: Facial images of drivers are captured using a near-infrared narrow-band DMS camera; real-time road condition images are captured using a forward-looking ADAS wide-angle camera. Chassis dynamic signals are acquired via the vehicle's CAN bus; Based on facial images, generate a full-face image of the driver, an rPPG signal extraction area, and a local eye image; Based on full-face images, rPPG signal extraction areas, eye local images, real-time road condition images, and chassis dynamics signals, explicit visual behavioral features, implicit physiological features, environmental context features, and vehicle operation features are extracted respectively. Construct a multimodal spatiotemporal fusion evaluation model, inputting multimodal features and outputting the distraction level; Interventions are tiered based on the level of distraction.

2. The driver attention evaluation method based on multi-modal feature fusion and physiological monitoring according to claim 1, characterized in that, The process of generating a full-face image of the driver, an rPPG signal extraction area, and a local eye image based on the facial image includes: Based on facial images, a deep neural network is used for face detection, generating facial bounding boxes, extracting facial features, and determining facial geometric topological relationships. Based on the facial bounding box, the pixel margins are extended outward for adaptive cropping to obtain a full-face image that includes the complete head pose and hand interaction area; Based on facial geometric topology, specific areas of the forehead center and cheeks are dynamically identified as rPPG signal extraction areas. Based on facial geometric topology, the eye contour is extracted and the pixel margin is expanded outward to crop and generate a local image of the eye.

3. The driver attention evaluation method based on multi-modal feature fusion and physiological monitoring according to claim 2, characterized in that, The process of face detection based on facial images, using a deep neural network to generate facial bounding boxes, extracting facial features, and determining facial geometric topological relationships includes: A contrast-limited adaptive histogram equalization algorithm is used for facial images, and Gaussian filtering is introduced for noise reduction. Based on the denoised facial image, a face detection model is used to detect faces and generate facial bounding boxes; Based on the face detection model, a face detection model is loaded, multiple facial features are regressed, and the geometric topological relationships of the face are determined.

4. The method for driver attention evaluation based on multi-modal feature fusion and physiological monitoring according to claim 1, characterized in that, The explicit visual behavioral features, including the driver's head Euler angles, body movements, and gaze placement, are extracted as follows: Based on the full-face image and local eye image, a Backbone network with shared weights is used to extract full-face feature maps and eye feature maps; The Euler angles of the driver's head are calculated based on the full-face feature map; The full-face feature map is input into the YOLOv8-nano model and the skeletal keypoint detection network respectively to detect target objects and recognize limb movements; A 3D geometric model of the vehicle interior is constructed. Based on the eye feature map, a 3D gaze vector is output through L2CS-Net. The 3D gaze vector is then mapped to the vehicle interior coordinates to determine the gaze point.

5. The method for driver attention evaluation based on multi-modal feature fusion and physiological monitoring according to claim 1, characterized in that, The extraction process for the aforementioned latent physiological features is as follows: Based on the image information extracted from the rPPG signal region, the average pixel value is calculated to generate the original spatiotemporal signal sequence; A planar orthogonal projection algorithm is used to denoise the spatiotemporal signal sequence and generate a clean signal. A bandpass filter is applied to the clean signal to generate rPPG waveform data; Perform a fast Fourier transform on the rPPG waveform data, obtain the frequency corresponding to the energy peak through spectrum analysis, and calculate the real-time heart rate; The peak interval RR of rPPG waveform data is extracted. Based on the RR interval sequence, the standard deviation SDNN and the root mean square of the difference between adjacent intervals RMSSD are calculated.

6. The method for driver attention evaluation based on multi-modal feature fusion and physiological monitoring according to claim 1, characterized in that, The extraction process for the environmental context features is as follows: A saliency detection network is used to generate a saliency map based on real-time traffic images; The driver's line of sight is mapped to the two-dimensional coordinate system of the real-time traffic image through coordinate transformation, and the spatial overlap between the line of sight and the highly saliency area in the saliency map is calculated.

7. The method for driver attention assessment based on multi-modal feature fusion and physiological monitoring according to claim 1, characterized in that, The extraction process for the vehicle operation features is as follows: Time-domain characteristics: Using a sliding window to statistically analyze time-domain data, the standard deviation and entropy of the vehicle steering wheel angle are calculated; the variances of lateral acceleration and yaw rate are also calculated. Frequency domain characteristics: Perform a fast Fourier transform on the steering wheel angle signal to generate frequency domain data, extract the energy proportion of specific frequency bands, and identify the control lag or high-frequency jitter characteristics caused by increased driver cognitive load. 8.The driver attention evaluation method based on multi-modal feature fusion and physiological monitoring according to claim 1, characterized in that, The multimodal spatiotemporal fusion evaluation model includes: a spatial feature extraction module, a temporal dynamic modeling module, and an adaptive fusion module; The spatial feature extraction module performs spatial mapping on the input features of the four modalities to generate spatial features, expressed as follows: ； In the formula, Spatial features; It is the ReLU activation function; The weight matrix is ​​a learnable weight matrix; For input features; For the corresponding mode The bias term; For time step; The temporal dynamic modeling module, based on spatial characteristics, adopts a multi-scale temporal downsampling strategy and generates temporal features through cascaded max pooling operations; The adaptive fusion module fuses the temporal features of each modality and hides the features; The output layer generates a prediction of cognitive load level based on the hidden features.

9. The driver attention evaluation method based on multi-modal feature fusion and physiological monitoring according to claim 8, characterized in that, The process of fusing and hiding the temporal features of each modality includes: Calculate the correlation weights between the two modal features to generate weighted features, expressed as follows: ； ； ； In the formula, For target mode The query matrix; Source mode The key matrix; Source mode The value matrix; , , The linear projection weight matrix for the query, key, and value; Let be the temporal characteristics of the j-th mode; Score for attention; Weighted features; The weighted features are concatenated and then fused and mapped through a fully connected layer to generate fused features, expressed as: ； ； ； In the formula, Features of splicing; Let i be the temporal feature of the i-th mode; For feature splicing operations; This is a joint representation of the splicing features; Features of fusion; For activation functions; This is the weight matrix for the fusion mapping; The bias vector for the fusion mapping; The dimension of the fusion feature; The fused features are non-linearly transformed through two fully connected layers, with batch normalization and Dropout regularization introduced in between, to generate hidden features, expressed as follows: ； ； where, is the primary hidden feature; is the final hidden feature; and is the weight matrix of the fully connected layer; and is the bias vector of the fully connected layer; is the dropout probability of Dropout.

10. The driver attention evaluation method based on multi-modal feature fusion and physiological monitoring according to claim 8, characterized in that, The generation of a cognitive load level prediction based on hidden features includes: For binary classification, the expression is: ； wherein is a binary classification output; is a weight vector for the output layer; is a bias term for the output layer; is a hidden feature; The three-class classification is expressed as follows: ， ； In the formula, Output in three categories; This is the weight vector of the output layer.

Images